Polars vs Pandas for LMS Bulk Exports

When a bulk export is a few thousand submissions, the choice of dataframe library is aesthetic — pandas, polars, even a list comprehension all finish before you notice. When it is a multi-term engagement pull spanning millions of page-view events across every course in an institution, the choice decides whether a worker completes or dies with a MemoryError at 3 a.m. This guide is a decision framework, not an advocacy piece: it compares pandas and polars on the axes that actually matter for LMS bulk exports — memory model, lazy versus eager evaluation, streaming behaviour on large multi-term pulls, API ergonomics, and ecosystem — then gives a concrete switch-over rule and one script that runs the same flatten-and-tokenize flow both ways so the migration path is unambiguous.

The short version is that both tools produce the same canonical rows, and the difference is entirely about how much data must be resident at once. Pandas materializes every intermediate; polars can plan the whole transformation and stream it in chunks. For the small-to-medium exports that dominate day-to-day pipelines, pandas’ maturity and ubiquity win. For the tail of genuinely large exports, polars’ streaming engine is the difference between fitting in a worker’s heap and not. Knowing where that crossover sits — and writing the flatten step so you can swap libraries without rewriting the pipeline — is the whole skill.

The comparison that matters

Five axes decide the choice for an export workload. The API ingestion toolchain guidance frames pandas and polars as complementary rather than rival — each earns its place on a different part of the size curve — and this table makes the trade-offs concrete.

Memory model is the headline. Pandas evaluates eagerly: json_normalize, then astype, then a map each build a new full-size object, so peak memory is a multiple of the export size and the steep curve in the chart above is the direct consequence. Polars stores data in columnar Apache Arrow buffers and, under its streaming engine, never needs the entire frame resident — it pulls a chunk, transforms it, emits it, and moves on, which is why its curve stays nearly flat.

Lazy versus eager is the mechanism behind that. A polars LazyFrame does not execute anything when you write .with_columns(...); it records the operation into a query plan. When you finally .collect(), the optimizer reorders, fuses, and prunes — dropping columns you never reference, pushing filters earlier, and choosing a streaming execution when the plan allows. Pandas has no such planning layer; what you write is what runs, in order, now.

API ergonomics cut the other way. Pandas is forgiving and universally documented — the Canvas gradebook JSON flattening patterns most teams already have are pandas patterns, and every Stack Overflow answer assumes it. Polars’ expression API is stricter and less improvisational, which is a virtue at scale (fewer silent surprises) but a real cost when a one-off reconciliation just needs to be written quickly.

The decision rule

Reduce the trade-off to a single, testable threshold. Let $S$ be the size of the raw export the flatten step must hold, $M$ the worker’s memory ceiling, and $k$ pandas’ materialization multiplier — empirically 3 to 5 for a json_normalize plus astype chain on nested LMS JSON. Pandas is safe only while

$k \cdot S \;\lesssim\; \alpha \, M$

where $\alpha \approx 0.7$ leaves headroom for the interpreter and other allocations. The moment the projected working set $k \cdot S$ approaches that fraction of RAM, the eager model will thrash or die, and polars’ streaming engine — whose peak is a small multiple of a chunk, not the whole export — becomes the correct tool.

In practice this yields a simple operational rule:

• Use pandas for exports comfortably under the threshold — single-course gradebooks, term rosters, anything that fits in memory with room to spare. Its maturity, ergonomics, and ecosystem outweigh polars’ speed when speed is not the constraint.
• Reach for polars once an export is large enough that $k \cdot S$ crosses roughly 70% of the worker’s RAM — multi-term engagement pulls, institution-wide submission histories, any streaming flatten that must run in bounded memory. Combine it with cursor-based pagination for large course rosters so the export streams page by page and the whole path — fetch, flatten, write — stays flat.

Because both libraries emit the identical canonical schema, the switch is a local swap of the flatten function, not a pipeline rewrite — provided the flatten step is written to be library-agnostic, as below.

Both ways, side by side

The script flattens and normalizes the same nested LMS submission export with pandas and with a polars LazyFrame, tokenizing identifiers at the FERPA boundary in both, and asserts the two outputs match. It is the concrete mapping you follow when migrating a flatten step across the threshold.

import hashlib
import os

import pandas as pd
import polars as pl

ID_SALT = os.environ.get("EXPORT_SALT", "rotate-me-from-secrets-manager").encode()

# One page of a nested LMS export; placeholder ids only, never real students.
RAW: list[dict] = [
    {"id": 1, "assignment": {"id": 90, "points_possible": 100.0},
     "user": {"id": "u-1001", "login_id": "learner01"}, "score": 88.0},
    {"id": 2, "assignment": {"id": 90, "points_possible": 100.0},
     "user": {"id": "u-1002", "login_id": "learner02"}, "score": None},   # ungraded
]


def tokenize(value: object) -> str | None:
    """Irreversible FERPA-safe surrogate; null passes through as null."""
    if value is None:
        return None
    return "stu_" + hashlib.sha256(ID_SALT + str(value).encode()).hexdigest()[:24]


def flatten_pandas(records: list[dict]) -> pd.DataFrame:
    """Eager path: json_normalize, coerce to nullable dtypes, tokenize."""
    df = pd.json_normalize(records, sep="_")
    df = df.rename(columns={"assignment_id": "assignment_id",
                            "user_id": "student_id"})
    df = df.astype({"id": "Int64", "assignment_id": "Int64",
                    "assignment_points_possible": "Float64", "score": "Float64"})
    df["student_token"] = df["user_login_id"].map(tokenize)   # hash the login id
    df = df.drop(columns=["user_login_id"])                   # drop raw PII
    return df[["id", "assignment_id", "assignment_points_possible",
               "score", "student_token"]].sort_values("id").reset_index(drop=True)


def flatten_polars(records: list[dict]) -> pl.DataFrame:
    """Lazy/streaming path: same result, planned and chunked."""
    lf = pl.json_normalize(records, separator="_").lazy()     # build a query plan
    lf = lf.with_columns(
        pl.col("id").cast(pl.Int64),
        pl.col("assignment_id").cast(pl.Int64),
        pl.col("assignment_points_possible").cast(pl.Float64),
        pl.col("score").cast(pl.Float64),
        # map_elements applies the same tokenizer inside the plan
        pl.col("user_login_id").map_elements(tokenize, return_dtype=pl.String)
          .alias("student_token"),
    ).drop("user_login_id").select(
        "id", "assignment_id", "assignment_points_possible", "score", "student_token"
    ).sort("id")
    # streaming=True runs the plan in bounded-memory chunks for large exports
    return lf.collect(streaming=True)


if __name__ == "__main__":
    pdf = flatten_pandas(RAW)
    plf = flatten_polars(RAW)
    # Same canonical rows regardless of engine.
    assert pdf["student_token"].tolist() == plf["student_token"].to_list()
    assert pdf["score"].isna().tolist() == [v is None for v in plf["score"].to_list()]
    print("pandas:\n", pdf)
    print("polars:\n", plf)
    print("identical canonical output:", pdf.shape == (plf.height, plf.width))

The two flatten functions are interchangeable behind the pipeline: each takes a page of raw records and returns the same five-column canonical frame with a tokenized student_token and a preserved-null score. Swapping which one the export loop calls is the entire migration when a workload crosses the memory threshold.

Verification and output validation

Confirm the two engines truly agree before trusting either downstream:

• Row-for-row token match. assert pdf["student_token"].tolist() == plf["student_token"].to_list() — the salted hash is deterministic, so identical inputs must yield identical tokens across libraries.
• Null preserved, not zeroed. The ungraded row’s score must be <NA> in pandas and null in polars, never 0.0; conflating them deflates a student’s grade. The assertion in the script checks this explicitly.
• No raw identifiers survive. Assert neither frame contains user_login_id or user_id and that every student_token starts with stu_.
• Shape parity. pdf.shape == (plf.height, plf.width) — same rows, same columns, same order after the sort.
• Streaming actually engaged. For a large input, run the polars path under POLARS_VERBOSE=1 and confirm the log reports the streaming engine; a plan that falls back to in-memory execution defeats the purpose.

Troubleshooting

• AttributeError: module 'polars' has no attribute 'json_normalize'. You are on an older polars. Upgrade (pip install -U polars), or build the frame with pl.DataFrame(records) and .unnest("assignment", "user") to flatten struct columns manually.
• Polars map_elements is slow on a large export. Row-wise Python UDFs bypass the vectorized engine. For millions of rows, hash inside a vectorized expression or precompute tokens in a batch, reserving map_elements for small frames.
• Tokens differ between the two engines. One path stringified the id differently (an Int64 versus a Python str). Tokenize the same source column with the same str(...) coercion in both; the login id, not the numeric id, is the join key here.
• Pandas MemoryError on a multi-term pull. You crossed the threshold — k · S exceeded RAM. Switch that export to flatten_polars with collect(streaming=True), and stream the fetch page by page rather than buffering the whole export first.
• collect(streaming=True) raises or warns as deprecated. Newer polars uses collect(engine="streaming"). Match the call to your installed version; both request the same bounded-memory execution.
• Downstream code expects a pandas frame. Call plf.to_pandas() at the boundary. Arrow-backed conversion is cheap and lets polars own the heavy flatten while the rest of the pipeline stays on pandas.

Related

• Pagination Strategies for Bulk Exports — the parent export loop these flatten steps plug into, and the checkpoint model that keeps memory flat.
• Cursor-Based Pagination for Large Course Rosters — streaming the fetch side page by page so a polars streaming flatten never buffers the whole export.
• API Ingestion & Sync Workflows — the toolchain guidance that positions pandas and polars as complementary across the size curve.

Part of: Pagination Strategies for Bulk Exports