Validate pandas and Polars DataFrames
Use the ledger from Validate recorded rows to validate a pandas frame or
stream a Polars lazy query. Select validator.fields in order and pass the
rows to iter_results, which maintains the rolling history.
>>> from examples.accounting_breaks import EVENTS, Ledger, accounting_rows
>>> from modello.contrib.validation import WindowValidator
>>> validator = WindowValidator(Ledger, "rows")
Validate pandas rows
loc selects field names; itertuples supplies their stored values:
>>> import pandas as pd
>>> frame = pd.DataFrame(accounting_rows(), index=EVENTS, dtype=object)
>>> rows = frame.loc[:, list(validator.fields)].itertuples(index=False, name=None)
>>> frame["valid"] = pd.array(
... [result.valid for result in validator.iter_results(rows)], dtype="boolean")
>>> frame["valid"].tolist()
[<NA>, True, False, True]
True means the row passes, False identifies a proven violation, and
pd.NA means validity is assumed or undecidable. The output keeps the frame’s
index, including duplicate labels and gaps; validation follows row order.
For nullable input columns, first use
frame = frame.astype(object).where(frame.notna(), None) to replace pandas’
missing-value sentinels with None. dtype=object preserves exact integer
and Decimal inputs; it cannot recover precision already lost to floats.
MultiIndex columns
For an existing accounts frame with columns such as
("holding", "quantity") and ("cash", ""), select
validator.field_paths, padding shorter paths with empty strings:
>>> keys = [path + ("",) * (accounts.columns.nlevels - len(path)) for path in validator.field_paths]
>>> rows = accounts.loc[:, keys].itertuples(index=False, name=None)
Pass rows to iter_results as above. Adapt the keys when your column names
or padding differ, preserving field order.
Stream a Polars lazy query
Scan a Parquet ledger at ledger_path. collect_batches
streams ordered batches without collecting the whole result. Flatten their
rows into one iterator so validation retains history across batches:
>>> import polars as pl
>>> from itertools import chain
>>> query = pl.scan_parquet(ledger_path).select(list(validator.fields))
>>> batches = query.collect_batches(engine="streaming", maintain_order=True)
>>> rows = chain.from_iterable(batch.iter_rows() for batch in batches)
>>> results = validator.iter_results(rows)
>>> [result.valid for result in results]
[None, True, False, True]
Polars supplies nulls as None. Consume results incrementally for large
inputs; the list above only displays this small example. Keep one scan in
event order for each independent history.
Batch size is independent of validator.window_size. Polars currently marks
collect_batches as unstable.
Both recipes check rows in Python/SymPy. See Validate recorded rows for inspecting failures and summarising coverage.