Skip to content
PySuricata Logo

Build Status PyPI version Python versions License: MIT codecov Documentation Downloads

PySuricata

Exploratory data analysis for Python, built on streaming algorithms.

PySuricata generates self-contained HTML reports for pandas and polars DataFrames. It processes data in chunks using streaming algorithms, so memory usage stays bounded regardless of dataset size.

Try it in your browser with no install: drop a CSV, Parquet file or Excel workbook and get the real report back. The profiler is compiled to WebAssembly and runs in the page, so nothing is uploaded.

  • Live Demo

    Profile your own file in a browser tab. No install, no upload.

    Open the Demo

  • Quick Start

    Install PySuricata and generate your first report.

    Get Started

  • Why PySuricata?

    Understand the streaming architecture and design decisions.

    Learn More

  • User Guide

    Detailed guides for configuration, advanced features, and more.

    Read the Guide

  • API Reference

    profile(), summarize(), compare() and every option they take.

    API Docs

Features

  • Streaming processing — Data is processed in configurable chunks, keeping memory bounded in rows regardless of dataset size. Useful for datasets that don't fit in RAM.
  • Reads a source, not just a frame — A path, an Arrow table or reader, anything exporting __arrow_c_stream__, or a DuckDB relation, all a batch at a time and never materialised. 307 MB against 581 MB on a 180 MB Parquet file.
  • Numbers without the HTML — summarize() returns the same statistics as a versioned JSON payload, so a consumer can read it without parsing a report.
  • A gate, not just a report — pysuricata check compares a dataset against a stored baseline and exits non-zero when a threshold is crossed, so the same single pass runs in a notebook and in CI.
  • A diff between two datasets — compare(a, b) reports every delta: schema, dataset and per column, with the approximate ones marked.
  • Mathematically grounded — Welford's algorithm for numerically stable moments, Pébay's formulas for mergeable statistics, KMV sketches for distinct counts, Misra-Gries for heavy hitters — each with its error bound published rather than hidden.
  • Pandas and Polars support — Works natively with both pandas.DataFrame and polars.DataFrame / polars.LazyFrame.
  • Self-contained reports — A single HTML file with inline CSS, JS and SVG charts. No external assets or dependencies needed to view.
  • Reproducible by default — The seed is 0, not None, so re-running over unchanged data is a no-op rather than a set of sampling wobbles.

Installation

uv add pysuricata

pip install pysuricata

This installs PySuricata along with its dependencies: pandas, numpy (on Python ≥3.13), and markdown.

To also install polars support:

pip install pysuricata[polars]

Quick Example

import pandas as pd
from pysuricata import profile

# Two years of hourly bike rentals — a timestamp, two booleans, two
# categoricals, and numeric columns that genuinely correlate.
url = "https://raw.githubusercontent.com/alvarodiez20/pysuricata/main/docs/assets/bike_sharing.csv"
df = pd.read_csv(url, parse_dates=["rented_at"])

# Generate report
report = profile(df)
report.save_html("example_report.html")

Or from a shell, without writing a script:

pysuricata profile bike_sharing.csv --output example_report.html

This is the actual report generated from the code above (17,379 rows × 12 columns):

Can't see the report? Open in new tab →

How It Works

PySuricata reads data in chunks and updates lightweight accumulators for each column. This means:

Aspect Approach
Memory Bounded by chunk size + accumulator state, not dataset size
Speed Single pass over the data — each row is read once
Accuracy Exact for moments (mean, variance, skewness, kurtosis); approximate with known error bounds for distinct counts and top-k
Mergeability Accumulators can be merged across chunks or machines

Memory is bounded in rows. It is not bounded in columns: state is per column at roughly 529 KB each, and a 20,000 x 600 frame peaks at 631 MB against 344 MB for a 1,000,000 x 14 one on more cells. That is a known limit, tracked in #207.

Reports include per-column statistics, histograms, correlation chips, missing value analysis, outlier detection, and more — all computed during the single streaming pass.

Beyond the Report

  • Gate a build on drift

    pysuricata check compares against a stored baseline and exits non-zero when a threshold is crossed.

    Gating CI on drift

  • Read a source, not a frame

    Parquet, Arrow IPC and DuckDB relations, a batch at a time.

    Arrow, Parquet and DuckDB

  • Diff two datasets

    compare(a, b) reports every delta, with the approximate ones marked.

    Comparing two datasets

  • Read the numbers directly

    A versioned JSON payload with no HTML in the way.

    The summarize() schema

Next Steps

Community & Support

License

MIT License. See LICENSE for details.