Migrating from pandas to polars: Step-by-Step Guide

Introduction: What is Polars?

Polars is a fast, modern DataFrame library designed for high-performance data analysis. Built entirely in Rust and exposed to Python via bindings, it provides a multi-threaded, vectorized query engine that can handle large datasets with ease. Unlike pandas, which relies on NumPy under the hood and often struggles with memory overhead and single-threaded execution, Polars is built on Apache Arrow—a columnar memory format—enabling zero-copy data interchange and near-instant serialization.

Polars offers both an eager API (immediate execution, similar to pandas) and a lazy API (query optimization and deferred computation), making it suitable for interactive exploration and production pipelines alike. It has no row index by default, which simplifies mental models and reduces accidental complexity. With its expression-based syntax, Polars encourages composability and clarity, often leading to more maintainable code.

Why Migrate from Pandas to Polars?

The migration is motivated by several critical advantages:

• Speed and Memory Efficiency: Polars outperforms pandas by significant margins—often 5–30x faster on common operations—while using far less RAM due to its Rust core and Arrow-backed columnar layout.
• Lazy Evaluation: Build a query plan, let Polars optimize it (predicate pushdown, projection pruning), and execute only when needed. This is ideal for ETL workloads.
• No Implicit Index: Without a default row index, operations are less error-prone. Grouping and joining rely purely on column values.
• Parallel Execution: All cores are automatically utilized; no need for manual partitioning.
• Better Null Handling: Polars distinguishes between missing values (null) and NaN cleanly, avoiding the ambiguous NaN pitfalls of pandas.
• Stable API: Polars’ API is designed to be consistent and predictable, reducing breaking changes over time.

Step-by-Step Migration Guide

1. Setting Up Polars

Installation is straightforward via pip:

pip install polars

Import in your scripts:

import polars as pl

You can optionally import pandas for comparison during migration:

import pandas as pd

2. Creating DataFrames

In pandas, you create a DataFrame from a dictionary of lists or columns:

# pandas
df_pd = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
    "salary": [70000, 80000, 95000]
})

In Polars, the equivalent is straightforward:

# polars
df_pl = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
    "salary": [70000, 80000, 95000]
})

You can also construct from a list of tuples (row-wise) with the orient="row" parameter or by specifying column names:

# polars from rows
df_pl = pl.DataFrame(
    [
        ("Alice", 25, 70000),
        ("Bob", 30, 80000),
        ("Charlie", 35, 95000)
    ],
    schema=["name", "age", "salary"],
    orient="row"
)

Reading CSV and Parquet files:

# pandas
df_pd = pd.read_csv("data.csv")
# polars eager
df_pl = pl.read_csv("data.csv")
# polars lazy (recommended for large files)
df_pl_lazy = pl.scan_csv("data.csv")  # returns LazyFrame

3. Selecting and Filtering Data

Pandas uses bracket indexing df['column'] or df[['col1','col2']] for column selection, and boolean masks for filtering:

# pandas select columns
cols = df_pd[['name', 'age']]
# pandas filter rows
filtered = df_pd[df_pd['age'] > 30]

Polars employs the select and filter methods with an expression API. Expressions are built from pl.col():

# polars select columns
cols = df_pl.select(pl.col("name", "age"))
# polars filter rows
filtered = df_pl.filter(pl.col("age") > 30)

You can combine multiple conditions using logical operators (&, |) or the .and_(), .or_() expression methods:

# polars multiple conditions
filtered = df_pl.filter(
    (pl.col("age") > 30) & (pl.col("salary") > 80000)
)

4. Column Operations and Transformations

Pandas offers assign or direct assignment for new columns:

# pandas
df_pd['age_squared'] = df_pd['age'] ** 2
# or with assign
df_pd = df_pd.assign(salary_k = df_pd['salary'] / 1000)

In Polars, you use with_columns to add or modify columns, or select to create a new DataFrame with only the desired columns:

# polars add columns
df_pl = df_pl.with_columns([
    (pl.col("age") ** 2).alias("age_squared"),
    (pl.col("salary") / 1000).alias("salary_k")
])
# polars select with transformations
transformed = df_pl.select(
    pl.col("name"),
    pl.col("age").mean().alias("avg_age")
)

String operations are similarly expressive. For example, uppercase a string column:

# pandas
df_pd['name_upper'] = df_pd['name'].str.upper()
# polars
df_pl = df_pl.with_columns(
    pl.col("name").str.to_uppercase().alias("name_upper")
)

5. Aggregations and Group-by

Pandas uses groupby (or groupby with agg):

# pandas
grouped = df_pd.groupby('department').agg(
    mean_salary=('salary', 'mean'),
    max_age=('age', 'max')
)

Polars uses group_by and agg. The column references are expressions:

# polars
grouped = df_pl.group_by("department").agg([
    pl.col("salary").mean().alias("mean_salary"),
    pl.col("age").max().alias("max_age")
])

Polars also supports group_by_dynamic for time-series rolling windows, and pivot/unpivot with a cleaner syntax than pandas’ pivot_table and melt.

6. Joins and Concatenations

Pandas provides pd.merge and pd.concat:

# pandas inner join
merged = pd.merge(left_df, right_df, on='key', how='inner')
# pandas vertical concat
combined = pd.concat([df1, df2], ignore_index=True)

Polars uses join and concat:

# polars join
merged = left_df.join(right_df, on='key', how='inner')
# polars vertical concat
combined = pl.concat([df1, df2], how='vertical')

Polars also supports asof joins, semi/anti joins, and cross joins. The lazy engine will reorder and optimize join operations automatically.

7. Handling Missing Data

Pandas uses NaN (a float value) to represent missing data, leading to issues with integer columns being upcasted to float. Polars uses explicit null (Arrow’s native missing value) which preserves types.

# pandas drop rows with any NaN
df_pd.dropna(inplace=True)
# pandas fill NaN
df_pd['salary'].fillna(0, inplace=True)
# polars drop rows with any null
df_pl = df_pl.drop_nulls()
# polars fill null
df_pl = df_pl.with_columns(
    pl.col("salary").fill_null(0)
)

Polars also provides fill_null with strategies like forward fill, backward fill, mean, etc., as expressions.

8. Working with LazyFrames

One of the biggest shifts is embracing lazy evaluation. Instead of eagerly loading and transforming data, you build a query graph and let Polars optimize and execute in one shot.

Start with scan_csv, scan_parquet, or create a lazy frame from an existing DataFrame with .lazy():

# polars lazy
lazy_df = pl.scan_csv("large_data.csv")
query = (
    lazy_df
    .filter(pl.col("age") > 30)
    .group_by("department")
    .agg(pl.col("salary").mean())
)
# Nothing executed yet
result = query.collect()   # triggers execution with optimizations

You can inspect the optimized plan with query.describe_optimized_plan(). Lazy evaluation reduces memory usage and speeds up complex pipelines by combining multiple steps into a single pass.

9. Converting between Pandas and Polars

During migration, you often need to interoperate. Polars can directly consume pandas DataFrames and export back:

# pandas to polars
df_pl = pl.from_pandas(df_pd)
# polars to pandas
df_pd = df_pl.to_pandas()

Under the hood, this uses Apache Arrow to convert with minimal overhead. For large data, you can go through Arrow tables explicitly to control chunk sizes. If you have nullable integer columns, the conversion will correctly map null to pd.NA (if using pandas nullable dtypes) or to NaN for float columns. It’s advisable to use pandas’ extension arrays with IntegerDtype to preserve nulls accurately.

Best Practices for Migration

Gradual Migration

Don’t rewrite your entire codebase at once. Start by introducing Polars in new data pipelines or isolated functions. Convert DataFrames at the boundaries: read data with Polars, perform heavy transformations, and then convert back to pandas if downstream code still expects pandas. Over time, expand the Polars territory.

Embrace Lazy Execution

Whenever loading external data (CSV, Parquet, databases), use the scan_* functions. Build your transformations as a lazy chain and call .collect() only when you need the final result. This lets Polars optimize the entire query graph, often reducing peak memory and runtime dramatically.

Rethink Indexes

Polars doesn’t have a row index, so operations like .iloc and index-based joins are not directly available. Instead, use explicit columns as keys. If you need row numbering, add an integer column with pl.int_range(). This forces cleaner, more portable logic.

Leverage the Expression API

Polars expressions are composable and can be reused. Instead of loops and apply (which are slow and single-threaded), express operations through pl.col(), .map_elements() (if necessary), and built-in methods. Almost every operation has an expression equivalent, enabling the query optimizer to fuse them.

Test and Profile

Before fully switching, write unit tests to verify that Polars produces equivalent results to your pandas code. Use tools like pandas.testing or custom assertions. Profile both memory and execution time to quantify the gains—this helps justify the migration to your team.

Conclusion

Migrating from pandas to Polars is a strategic investment in performance, scalability, and code clarity. By understanding the core differences—no implicit index, expression-based operations, lazy evaluation, and native Arrow integration—you can systematically replace pandas patterns with Polars equivalents. Start small, leverage lazy frames for large data, and gradually expand your Polars footprint. The result is faster, more memory-efficient data processing that scales effortlessly across modern hardware. With a vibrant open-source community and continuous improvements, Polars is poised to become the default DataFrame library for Python data professionals.