TL;DR: Polars is a DataFrame library written in Rust that is significantly faster than pandas for most operations; it uses columnar memory layout (Apache Arrow), lazy evaluation, and multi-core parallelism by default. It has both a Rust native API and Python bindings (
pip install polars). Use it for ETL, data analysis, and processing large datasets. The lazy API (LazyFrame) builds a query plan and optimizes it before execution; similar to Spark's DataFrame API.
What Is Polars?
Polars is a columnar DataFrame library for Rust (and Python): built on Apache Arrow, optimized for multi-core CPUs, and designed to replace pandas for performance-critical data work.
Key properties:
- Columnar: data stored column-by-column → cache-efficient operations
- Lazy evaluation: builds a query plan, optimizes, then executes
- Parallel by default: uses all CPU cores automatically
- Apache Arrow: interoperates with other Arrow-based tools
- No GIL: Python bindings release the GIL during computation
How Do You Use Polars in Rust?
[dependencies]
polars = { version = "0.46", features = ["lazy", "csv", "parquet"] }use polars::prelude::*;
fn main() -> Result<(), PolarsError> {
// Create a DataFrame
let df = df! {
"name" => ["Alice", "Bob", "Carol", "Dave"],
"age" => [30u32, 25, 35, 28],
"salary" => [95000f64, 72000.0, 110000.0, 85000.0],
"active" => [true, true, false, true],
}?;
println!("{df}");
// Filter: active employees over 27
let result = df.lazy()
.filter(
col("active").eq(lit(true))
.and(col("age").gt(lit(27u32)))
)
.select([col("name"), col("salary")])
.sort(["salary"], SortMultipleOptions::default().with_order_descending(true))
.collect()?;
println!("{result}");
Ok(())
}How Does the Lazy API Work?
LazyFrame builds an optimized execution plan before running; predicate pushdown, projection pruning, and parallel execution happen automatically.
use polars::prelude::*;
fn analyze_sales(path: &str) -> Result<DataFrame, PolarsError> {
// LazyFrame; no data is read yet
let result = LazyCsvReader::new(path)
.with_has_header(true)
.finish()?
// All of this is a query plan; not executed yet
.filter(col("revenue").gt(lit(1000.0)))
.group_by([col("region")])
.agg([
col("revenue").sum().alias("total_revenue"),
col("revenue").mean().alias("avg_revenue"),
col("orders").count().alias("num_orders"),
])
.sort(["total_revenue"], SortMultipleOptions::default().with_order_descending(true))
.limit(10)
// Only now: Polars optimizes and executes the plan
.collect()?;
Ok(result)
}How Do You Read and Write Files?
use polars::prelude::*;
// Read CSV
let df = CsvReadOptions::default()
.with_has_header(true)
.try_into_reader_with_file_path(Some("data.csv".into()))?
.finish()?;
// Read Parquet
let mut file = std::fs::File::open("data.parquet")?;
let df = ParquetReader::new(&mut file).finish()?;
// Write CSV
let mut file = std::fs::File::create("output.csv")?;
CsvWriter::new(&mut file).finish(&mut df)?;
// Write Parquet
let mut file = std::fs::File::create("output.parquet")?;
ParquetWriter::new(&mut file).finish(&mut df)?;How Does Polars Compare to pandas?
| Polars | pandas | |
|---|---|---|
| Language | Rust (Python bindings) | Python (C/Fortran core) |
| Memory layout | Columnar (Arrow) | Row + columnar (NumPy) |
| Parallelism | Multi-core by default | Single-threaded (GIL) |
| Lazy evaluation | ✅ LazyFrame | ❌ Eager only |
| Speed (typical) | 5–50x faster | Baseline |
| Memory usage | Lower (Arrow zero-copy) | Higher |
| API maturity | Growing | Very mature |
| Ecosystem | Younger | pandas/sklearn/etc. |
Frequently Asked Questions
For performance: yes, significantly. For ecosystem maturity and breadth of integrations (scikit-learn, matplotlib, Jupyter notebooks): pandas still has more. Polars is the better choice when data size or processing speed is a concern.
Yes; pip install polars. The Python API is the most widely used interface. The Rust API gives you more control and avoids Python overhead entirely.
DataFrame is eager; operations execute immediately. LazyFrame is a query builder; operations are collected into a plan and executed all at once when you call .collect(). Use LazyFrame for complex queries; Polars can optimize the whole plan.
Yes; Polars uses Apache Arrow2 (now arrow-rs) as its in-memory format. You can zero-copy convert between Polars DataFrames and Arrow arrays, and interoperate with other Arrow-based tools like DataFusion.
Sources
Related Glossary Terms
- Iterator: Polars operations are lazily composed like iterators
- Rayon: Polars uses similar data-parallel patterns internally
- Serde: Polars supports serde for JSON I/O
- PyO3: Polars' Python bindings are part of the broader Rust-to-Python tooling story
Keep Reading
- Rust for Python Developers: Polars as the Rust replacement for pandas
- Rust vs Python Performance: why Polars outperforms pandas
- Building AI Agents in Rust: data processing pipelines for AI
