Advanced Features

Advanced techniques for power users.

Examples on this page assume a DataFrame named df

Every snippet below that does not build its own frame expects one already in scope. Paste this first to follow along:

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
df = pd.DataFrame(
    {
        "id": range(5_000),
        "amount": rng.lognormal(3, 1, 5_000),
        "country": rng.choice(["ES", "FR", "DE"], 5_000),
        "signed_up": pd.date_range("2024-01-01", periods=5_000, freq="17min"),
        "active": rng.random(5_000) > 0.3,
    }
)

Custom Markdown Descriptions

Add rich descriptions to reports:

from pysuricata import profile, ProfileConfig

config = ProfileConfig()
config.render.description = """
# Q4 2024 Analysis

**Dataset**: Customer transactions  
**Period**: Oct-Dec 2024  
**Source**: production.transactions

## Key Findings

- Revenue up 15% YoY
- Average transaction: $87.50
- Peak hour: 2pm EST
"""

report = profile(df, config=config)

Streaming a Source Directly

Before hand-rolling a generator, check whether the input is one PySuricata already streams. A path, an Arrow table or reader, and a DuckDB relation are all read a batch at a time without being materialised:

import duckdb

from pysuricata import profile, summarize

profile("events.parquet")

con = duckdb.connect()
relation = con.sql('''
    SELECT o.*, c.segment
    FROM 'orders/*.parquet' o
    JOIN 'customers.parquet' c USING (customer_id)
    WHERE o.created_at > '2026-01-01'
''')
summarize(relation)

That last one is a query that has not run yet — a filtered join across several Parquet files, profiled without any of it existing as a frame. See Arrow, Parquet and DuckDB.

Streaming from Multiple Sources

For anything the built-in readers do not cover, combine sources yourself:

from pysuricata import profile
def multi_source_generator():
    # Source 1: CSV files
    for i in range(10):
        yield pd.read_csv(f"batch_{i}.csv")

    # Source 2: Parquet files
    for i in range(5):
        yield pd.read_parquet(f"archive_{i}.parquet")

    # Source 3: Database
    for chunk in pd.read_sql("SELECT * FROM logs", conn, chunksize=100_000):
        yield chunk

report = profile(multi_source_generator())

Parallel Processing with Dask

import dask.dataframe as dd
from pysuricata import profile

# Load with Dask
ddf = dd.read_csv("large_*.csv")

# Convert to generator
def dask_generator():
    for partition in ddf.partitions:
        yield partition.compute()

report = profile(dask_generator())

Custom Sampling Strategy

from pysuricata import profile
# Sample every Nth row for very large datasets
def sampled_generator(n=10):
    for chunk in pd.read_csv("huge.csv", chunksize=100_000):
        yield chunk[::n]  # Every 10th row

report = profile(sampled_generator())

Merging Accumulator States (Distributed)

import numpy as np

from pysuricata.accumulators import NumericAccumulator

rng = np.random.default_rng(0)
data_partition_1 = rng.lognormal(3, 1, 50_000)
data_partition_2 = rng.lognormal(3, 1, 50_000)

# Worker 1
acc1 = NumericAccumulator("amount")
acc1.update(data_partition_1)

# Worker 2
acc2 = NumericAccumulator("amount")
acc2.update(data_partition_2)

# Merge on coordinator
acc1.merge(acc2)
final_stats = acc1.finalize()

This is exact, not an approximation of a single-machine run. The moments merge by Pébay's formulas and the sketches by construction, so the merged result equals the result of one pass over the concatenation — benchmarks/accuracy.py asserts it. It is also why the order the partitions arrive in does not matter.

Conditional Profiling

Profile only rows meeting criteria:

from pysuricata import profile
def filtered_generator():
    for chunk in pd.read_csv("data.csv", chunksize=100_000):
        # Only active users
        yield chunk[chunk["status"] == "active"]

report = profile(filtered_generator())

Checkpointing Long-Running Profiles

For multi-hour pipelines analyzing massive datasets (i.e. over 100M rows), you can enable disk checkpoints. This saves the profiling state every N chunks, allowing you to resume or investigate partial state if the job is interrupted.

import numpy as np
import pandas as pd

from pysuricata import ProfileConfig, profile

rng = np.random.default_rng(0)

config = ProfileConfig()
# Checkpoint roughly every million rows at the default 50,000-row chunk. Leave
# chunk_size alone -- raising it costs memory and time both, and checkpoint
# frequency is what you are actually setting here.
config.compute.checkpoint.every_n_chunks = 20
config.compute.checkpoint.dir = "/tmp/pysuricata/nightly"
# Optionally dump a preview HTML page alongside the serialized pickle state
config.compute.checkpoint.write_html = True



def multi_source_generator():
    """Yield one chunk per source file, never holding more than one."""
    for i in range(4):
        yield pd.DataFrame({"amount": rng.lognormal(3, 1, 200_000)})


report = profile(multi_source_generator(), config=config)