Validate pandas and Polars DataFrames

Use the ledger from Validate recorded rows to validate a pandas frame or stream a Polars lazy query. Select validator.fields in order and pass the rows to iter_results, which maintains the rolling history.

>>> from examples.accounting_breaks import EVENTS, Ledger, accounting_rows
>>> from modello.contrib.validation import WindowValidator
>>> validator = WindowValidator(Ledger, "rows")

Validate pandas rows

loc selects field names; itertuples supplies their stored values:

>>> import pandas as pd
>>> frame = pd.DataFrame(accounting_rows(), index=EVENTS, dtype=object)
>>> rows = frame.loc[:, list(validator.fields)].itertuples(index=False, name=None)
>>> frame["valid"] = pd.array(
...     [result.valid for result in validator.iter_results(rows)], dtype="boolean")
>>> frame["valid"].tolist()
[<NA>, True, False, True]

True means the row passes, False identifies a proven violation, and pd.NA means validity is assumed or undecidable. The output keeps the frame’s index, including duplicate labels and gaps; validation follows row order.

For nullable input columns, first use frame = frame.astype(object).where(frame.notna(), None) to replace pandas’ missing-value sentinels with None. dtype=object preserves exact integer and Decimal inputs; it cannot recover precision already lost to floats.

MultiIndex columns

For an existing accounts frame with columns such as ("holding", "quantity") and ("cash", ""), select validator.field_paths, padding shorter paths with empty strings:

>>> keys = [path + ("",) * (accounts.columns.nlevels - len(path)) for path in validator.field_paths]
>>> rows = accounts.loc[:, keys].itertuples(index=False, name=None)

Pass rows to iter_results as above. Adapt the keys when your column names or padding differ, preserving field order.

Stream a Polars lazy query

Scan a Parquet ledger at ledger_path. collect_batches streams ordered batches without collecting the whole result. Flatten their rows into one iterator so validation retains history across batches:

>>> import polars as pl
>>> from itertools import chain
>>> query = pl.scan_parquet(ledger_path).select(list(validator.fields))
>>> batches = query.collect_batches(engine="streaming", maintain_order=True)
>>> rows = chain.from_iterable(batch.iter_rows() for batch in batches)
>>> results = validator.iter_results(rows)
>>> [result.valid for result in results]
[None, True, False, True]

Polars supplies nulls as None. Consume results incrementally for large inputs; the list above only displays this small example. Keep one scan in event order for each independent history. Batch size is independent of validator.window_size. Polars currently marks collect_batches as unstable.

Both recipes check rows in Python/SymPy. See Validate recorded rows for inspecting failures and summarising coverage.