Generated Reference

Everything below is generated from the source at build time, so it cannot drift from the code the way a hand-written page can. For the narrative version — what to reach for and why — see the High-Level API and the Configuration Guide.

Entry points

pysuricata.profile(data, config=None, *, preset=None, **options)

Compute statistics and render a self‑contained HTML report.

The function accepts in‑memory data (pandas or polars) or an iterable of pandas or polars chunks. Both pandas and polars DataFrames are processed consistently - chunking is handled by the engine based on the chunk_size configuration.

Parameters:

Name	Type	Description	Default
data	DataLike	Dataset to analyze. Supported: - pandas.DataFrame - polars.DataFrame or polars.LazyFrame - an Arrow table or reader, or anything exporting __arrow_c_stream__ - a DuckDB relation - a path to a .csv, .parquet, .json, .arrow, .feather, .ipc, .xlsx, .xlsm, .xlsb, .xls or .ods file - Iterable yielding pandas.DataFrame or polars.DataFrame chunks The last three stream a batch at a time and are never materialised as a whole frame. See the Arrow, Parquet and DuckDB guide.	required
config	ProfileConfig | None	Optional configuration overriding compute/render defaults. Set chunk_size=None to disable chunking for both pandas and polars. This is the full escape hatch and cannot be combined with preset or the keyword options below.	None
preset	str | None	"fast" or "thorough". One word for an intent, rather than working out which of twenty-one knobs to turn.	None
**options	Any	The most-reached-for settings, without the nesting: chunk_size, columns, sample, correlations, seed, title, progress.	{}

Returns:

Name	Type	Description
A	Report	class:Report object containing the HTML and the computed stats
	Report	mapping.

Raises:

Type	Description
TypeError	If data is not of a supported type.
ValueError	If data is None.
ConfigurationError	If an option or preset is not recognised, or config is combined with either.

Examples:

>>> profile(df)
>>> profile(df, chunk_size=50_000, correlations=False)
>>> profile(df, preset="fast")

Source code in pysuricata/api.py

def profile(
    data: DataLike,
    config: ProfileConfig | None = None,
    *,
    preset: str | None = None,
    **options: Any,
) -> Report:
    """Compute statistics and render a self‑contained HTML report.

    The function accepts in‑memory data (pandas or polars) or an iterable of
    pandas or polars chunks. Both pandas and polars DataFrames are processed
    consistently - chunking is handled by the engine based on the chunk_size
    configuration.

    Args:
        data: Dataset to analyze. Supported:
            - ``pandas.DataFrame``
            - ``polars.DataFrame`` or ``polars.LazyFrame``
            - an Arrow table or reader, or anything exporting
              ``__arrow_c_stream__``
            - a DuckDB relation
            - a path to a ``.csv``, ``.parquet``, ``.json``, ``.arrow``,
              ``.feather``, ``.ipc``, ``.xlsx``, ``.xlsm``, ``.xlsb``,
              ``.xls`` or ``.ods`` file
            - Iterable yielding ``pandas.DataFrame`` or ``polars.DataFrame`` chunks

            The last three stream a batch at a time and are never materialised
            as a whole frame. See the Arrow, Parquet and DuckDB guide.
        config: Optional configuration overriding compute/render defaults.
            Set chunk_size=None to disable chunking for both pandas and polars.
            This is the full escape hatch and cannot be combined with ``preset``
            or the keyword options below.
        preset: ``"fast"`` or ``"thorough"``. One word for an intent, rather
            than working out which of twenty-one knobs to turn.
        **options: The most-reached-for settings, without the nesting:
            ``chunk_size``, ``columns``, ``sample``, ``correlations``, ``seed``,
            ``title``, ``progress``.

    Returns:
        A :class:`Report` object containing the HTML and the computed stats
        mapping.

    Raises:
        TypeError: If ``data`` is not of a supported type.
        ValueError: If ``data`` is None.
        ConfigurationError: If an option or preset is not recognised, or
            ``config`` is combined with either.

    Examples:
        >>> profile(df)                                        # doctest: +SKIP
        >>> profile(df, chunk_size=50_000, correlations=False)  # doctest: +SKIP
        >>> profile(df, preset="fast")                          # doctest: +SKIP
    """
    if data is None:
        raise ValueError("Input data cannot be None")

    cfg = _resolve_config(config, preset, options)
    inp = _coerce_input(data)  # No more polars-specific wrapping!
    cfg = _to_engine_config(cfg)
    # The header names what was profiled. Only a path carries a name; an
    # in-memory frame has none, and the header renders without one. Set after
    # the conversion so an explicitly configured name is never overwritten.
    if not cfg.dataset_name:
        cfg.dataset_name = _source_name(data)

    # Always compute stats to return machine-readable mapping
    html, summary = report.build_report(inp, config=cfg, return_summary=True)  # type: ignore[misc]

    try:
        stats = dict(summary or {})
    except Exception:
        stats = {"dataset": {}, "columns": {}}
    return Report(html=html, stats=stats)

pysuricata.summarize(data, config=None, *, preset=None, **options)

Compute statistics only and return a JSON‑safe mapping.

This is the programmatic counterpart to :func:profile for code paths that do not need the HTML report (e.g., CI checks and data quality gates). Both pandas and polars DataFrames are processed consistently.

Parameters:

Name	Type	Description	Default
data	DataLike	Dataset to analyze. Same accepted types as :func:profile.	required
config	ProfileConfig | None	Optional configuration overriding compute/render defaults. Cannot be combined with preset or the keyword options.	None
preset	str | None	"fast" or "thorough".	None
**options	Any	Same keyword options as :func:profile.	{}

Returns:

Type	Description
Mapping[str, Any]	A nested mapping with dataset‑level and per‑column statistics. The
Mapping[str, Any]	result is safe to serialize to JSON.

Raises:

Type	Description
TypeError	If data is not of a supported type.
ValueError	If data is None.
ConfigurationError	If an option or preset is not recognised.

Source code in pysuricata/api.py

def summarize(
    data: DataLike,
    config: ProfileConfig | None = None,
    *,
    preset: str | None = None,
    **options: Any,
) -> Mapping[str, Any]:
    """Compute statistics only and return a JSON‑safe mapping.

    This is the programmatic counterpart to :func:`profile` for code paths that
    do not need the HTML report (e.g., CI checks and data quality gates).
    Both pandas and polars DataFrames are processed consistently.

    Args:
        data: Dataset to analyze. Same accepted types as :func:`profile`.
        config: Optional configuration overriding compute/render defaults.
            Cannot be combined with ``preset`` or the keyword options.
        preset: ``"fast"`` or ``"thorough"``.
        **options: Same keyword options as :func:`profile`.

    Returns:
        A nested mapping with dataset‑level and per‑column statistics. The
        result is safe to serialize to JSON.

    Raises:
        TypeError: If ``data`` is not of a supported type.
        ValueError: If ``data`` is None.
        ConfigurationError: If an option or preset is not recognised.
    """
    if data is None:
        raise ValueError("Input data cannot be None")

    cfg = _resolve_config(config, preset, options)
    inp = _coerce_input(data)  # No more polars-specific wrapping!
    cfg = _to_engine_config(cfg)
    # compute-only to skip HTML render
    _html, summary = report.build_report(
        inp, config=cfg, return_summary=True, compute_only=True
    )  # type: ignore[misc]
    stats = dict(summary or {})
    return stats

pysuricata.compare(before, after, *, seed=0, **options)

Compare two datasets and report what changed.

Parameters:

Name	Type	Description	Default
before	Any	The reference. A DataFrame, a path, an Arrow or DuckDB source, or a summarize() payload.	required
after	Any	The dataset to compare against it, in any of the same forms.	required
seed	int	Passed to summarize() for both sides. Defaulted rather than left to chance so that comparing a dataset against itself is a no-op rather than a set of sampling wobbles.	0
**options	Any	Any other keyword summarize() accepts, applied to both sides — comparing two profiles taken with different settings would report the settings as drift.	{}

Returns:

Type	Description
Comparison	The comparison.

Example

from pysuricata import compare

diff = compare(january, february)
diff.schema.added
diff.columns["amount"].median_shift_sigma

Source code in pysuricata/comparison.py

def compare(
    before: Any,
    after: Any,
    *,
    seed: int = 0,
    **options: Any,
) -> Comparison:
    """Compare two datasets and report what changed.

    Args:
        before: The reference. A DataFrame, a path, an Arrow or DuckDB source,
            or a `summarize()` payload.
        after: The dataset to compare against it, in any of the same forms.
        seed: Passed to `summarize()` for both sides. Defaulted rather than left
            to chance so that comparing a dataset against itself is a no-op
            rather than a set of sampling wobbles.
        **options: Any other keyword `summarize()` accepts, applied to both
            sides — comparing two profiles taken with different settings would
            report the settings as drift.

    Returns:
        The comparison.

    Example:
        ```python
        from pysuricata import compare

        diff = compare(january, february)
        diff.schema.added
        diff.columns["amount"].median_shift_sigma
        ```
    """
    # Profiled once each, not once per section: a source may be a generator or a
    # Parquet file, and reading it three times is between wasteful and wrong.
    first = _payload(before, seed, options)
    second = _payload(after, seed, options)
    return Comparison(
        dataset=_dataset_delta(first, second),
        schema=_schema_delta(first, second),
        columns=_column_deltas(first, second),
    )

Configuration

pysuricata.ProfileConfig dataclass

High-level configuration for data profiling.

This is the main configuration class used by the public API functions profile() and summarize(). It contains compute and render options that control how data is processed and how the output is generated.

Examples:

Basic usage with defaults

config = ProfileConfig()

Custom compute settings

config = ProfileConfig( compute=ComputeOptions( chunk_size=100_000, numeric_sample_size=10_000, random_seed=42 ) )

Attributes:

Name	Type	Description
compute	ComputeOptions	Compute-related options; see :class:ComputeOptions.
render	RenderOptions	Render-related options; see :class:RenderOptions.

Source code in pysuricata/api.py

@dataclass
class ProfileConfig:
    """High-level configuration for data profiling.

    This is the main configuration class used by the public API functions
    `profile()` and `summarize()`. It contains compute and render options
    that control how data is processed and how the output is generated.

    Examples:
        # Basic usage with defaults
        config = ProfileConfig()

        # Custom compute settings
        config = ProfileConfig(
            compute=ComputeOptions(
                chunk_size=100_000,
                numeric_sample_size=10_000,
                random_seed=42
            )
        )

    Attributes:
        compute: Compute-related options; see :class:`ComputeOptions`.
        render: Render-related options; see :class:`RenderOptions`.
    """

    compute: ComputeOptions = field(default_factory=ComputeOptions)
    render: RenderOptions = field(default_factory=RenderOptions)

pysuricata.ComputeOptions dataclass

Configuration for data processing and analysis.

These options control how data is streamed and how approximations are performed during computation. They are intentionally conservative by default to provide stable results for small to medium datasets, while still scaling to larger ones.

Examples:

For large datasets (memory constrained)

ComputeOptions(chunk_size=50_000, numeric_sample_size=5_000)

For high-quality analysis

ComputeOptions(chunk_size=500_000, numeric_sample_size=50_000)

For reproducible results

ComputeOptions(random_seed=42)

For specific columns only

ComputeOptions(columns=["age", "income", "education"])

With checkpointing for large datasets

ComputeOptions( chunk_size=100_000, checkpoint_every_n_chunks=10, checkpoint_dir="./checkpoints", checkpoint_write_html=True )

Attributes:

Name	Type	Description
chunk_size	int | None	Number of rows to process in each chunk. Default: 50,000. Bigger is not faster: the sketch merges are superlinear in batch size, so one 200,000-row batch costs more than four 50,000-row ones. Raise it only to trade memory for fewer chunk boundaries.
columns	Sequence[str] | None	Optional subset of columns to analyze. If None, all columns are analyzed. Default: None (all columns)
numeric_sample_size	int	Reservoir sample size for numeric statistics like quantiles and histograms. Larger samples give better accuracy but use more memory. Default: 20,000
max_uniques	int	Sketch size for approximate unique value counting. Larger sketches give better accuracy but use more memory. Default: 2,048
top_k	int	Maximum number of top categories to track for categorical columns. Default: 50
random_seed	int | None	Seed for reproducible sampling. Set to None for non-deterministic results. Default: 0 -- reproducible unless you ask for otherwise, so re-running the same data is a no-op.
progress	Any	Whether to report progress while streaming. True, False, "auto" (report only when stderr is a terminal) or a callable taking chunks/rows/elapsed. Always stderr, never stdout, so piped output stays parseable. Default: False
log_every_n_chunks	int	Log progress every N chunks. Set to 1 to log every chunk, higher values for less frequent logging. Default: 1
checkpoint_every_n_chunks	int	Create checkpoint every N chunks. Set to 0 to disable checkpointing. Default: 0 (disabled)
checkpoint_dir	str | None	Directory for checkpoint files. If None, uses current working directory. Default: None
checkpoint_prefix	str	Prefix for checkpoint filenames. Default: "pysuricata_ckpt"
checkpoint_write_html	bool	Whether to include HTML in checkpoints. Default: False
checkpoint_max_to_keep	int	Maximum number of checkpoints to retain. Default: 3
enable_auto_boolean_detection	bool	Whether to automatically detect 0/1 numeric columns as boolean. Default: True
boolean_detection_min_samples	int	Minimum number of samples required for boolean detection. Default: 100
boolean_detection_max_zero_ratio	float	Maximum ratio of zeros allowed for boolean detection (to avoid classifying mostly-zero columns as boolean). Default: 0.80
boolean_detection_require_name_pattern	bool	Whether to require boolean-like column names (e.g., 'is_', 'has_', 'can_') for detection. Default: False, so a 0/1 column is detected on its values whatever it is called.
force_column_types	dict[str, str] | None	Optional dictionary mapping column names to their forced types. Overrides automatic type inference. Default: None
compute_correlations	bool	Whether to compute pairwise correlations between numeric columns. Default: True
corr_threshold	float	Minimum absolute correlation value to report. Only correlations with |r| >= threshold are shown. Default: 0.5
corr_max_cols	int	Maximum number of numeric columns for correlation analysis. If exceeded, correlations are skipped. Default: 50
corr_max_per_col	int	Maximum number of top correlations to show per column. Default: 10

Source code in pysuricata/api.py

@dataclass
class ComputeOptions:
    """Configuration for data processing and analysis.

    These options control how data is streamed and how approximations are
    performed during computation. They are intentionally conservative by
    default to provide stable results for small to medium datasets, while still
    scaling to larger ones.

    Examples:
        # For large datasets (memory constrained)
        ComputeOptions(chunk_size=50_000, numeric_sample_size=5_000)

        # For high-quality analysis
        ComputeOptions(chunk_size=500_000, numeric_sample_size=50_000)

        # For reproducible results
        ComputeOptions(random_seed=42)

        # For specific columns only
        ComputeOptions(columns=["age", "income", "education"])

        # With checkpointing for large datasets
        ComputeOptions(
            chunk_size=100_000,
            checkpoint_every_n_chunks=10,
            checkpoint_dir="./checkpoints",
            checkpoint_write_html=True
        )

    Attributes:
        chunk_size: Number of rows to process in each chunk. Default: 50,000.
            Bigger is not faster: the sketch merges are superlinear in batch
            size, so one 200,000-row batch costs more than four 50,000-row
            ones. Raise it only to trade memory for fewer chunk boundaries.
        columns: Optional subset of columns to analyze. If None, all columns
            are analyzed. Default: None (all columns)
        numeric_sample_size: Reservoir sample size for numeric statistics like
            quantiles and histograms. Larger samples give better accuracy but
            use more memory. Default: 20,000
        max_uniques: Sketch size for approximate unique value counting.
            Larger sketches give better accuracy but use more memory.
            Default: 2,048
        top_k: Maximum number of top categories to track for categorical
            columns. Default: 50
        random_seed: Seed for reproducible sampling. Set to None for
            non-deterministic results. Default: 0 -- reproducible unless you
            ask for otherwise, so re-running the same data is a no-op.
        progress: Whether to report progress while streaming. True, False,
            "auto" (report only when stderr is a terminal) or a callable taking
            chunks/rows/elapsed. Always stderr, never stdout, so piped output
            stays parseable. Default: False
        log_every_n_chunks: Log progress every N chunks. Set to 1 to log every
            chunk, higher values for less frequent logging. Default: 1
        checkpoint_every_n_chunks: Create checkpoint every N chunks. Set to 0
            to disable checkpointing. Default: 0 (disabled)
        checkpoint_dir: Directory for checkpoint files. If None, uses current
            working directory. Default: None
        checkpoint_prefix: Prefix for checkpoint filenames. Default: "pysuricata_ckpt"
        checkpoint_write_html: Whether to include HTML in checkpoints.
            Default: False
        checkpoint_max_to_keep: Maximum number of checkpoints to retain.
            Default: 3
        enable_auto_boolean_detection: Whether to automatically detect 0/1 numeric
            columns as boolean. Default: True
        boolean_detection_min_samples: Minimum number of samples required for
            boolean detection. Default: 100
        boolean_detection_max_zero_ratio: Maximum ratio of zeros allowed for
            boolean detection (to avoid classifying mostly-zero columns as boolean).
            Default: 0.80
        boolean_detection_require_name_pattern: Whether to require boolean-like
            column names (e.g., 'is_', 'has_', 'can_') for detection.
            Default: False, so a 0/1 column is detected on its values whatever
            it is called.
        force_column_types: Optional dictionary mapping column names to their
            forced types. Overrides automatic type inference. Default: None
        compute_correlations: Whether to compute pairwise correlations between
            numeric columns. Default: True
        corr_threshold: Minimum absolute correlation value to report. Only
            correlations with |r| >= threshold are shown. Default: 0.5
        corr_max_cols: Maximum number of numeric columns for correlation analysis.
            If exceeded, correlations are skipped. Default: 50
        corr_max_per_col: Maximum number of top correlations to show per column.
            Default: 10
    """

    chunk_size: int | None = 50_000
    columns: Sequence[str] | None = None
    numeric_sample_size: int = 20_000
    max_uniques: int = 2_048
    top_k: int = 50
    random_seed: int | None = 0
    # True, False, "auto", or a callable taking chunks/rows/elapsed. Always
    # stderr, never stdout, so piped output stays parseable.
    progress: Any = False

    # Logging and checkpointing
    log_every_n_chunks: int = 1
    # Five of the twenty-two fields serve one concern most users never touch.
    # They stay here as fields for compatibility -- removing them would break
    # existing code for no gain -- and are also reachable as a named group via
    # the `checkpoint` view below, which is what makes the shape legible.
    checkpoint_every_n_chunks: int = 0  # 0 disables checkpointing
    checkpoint_dir: str | None = None
    checkpoint_prefix: str = "pysuricata_ckpt"
    checkpoint_write_html: bool = False
    checkpoint_max_to_keep: int = 3

    # Boolean detection options
    enable_auto_boolean_detection: bool = True
    boolean_detection_min_samples: int = 100
    boolean_detection_max_zero_ratio: float = 0.80
    boolean_detection_require_name_pattern: bool = False
    force_column_types: dict[str, str] | None = None

    # Correlation analysis options
    compute_correlations: bool = True
    corr_threshold: float = 0.5
    corr_max_cols: int = 50
    corr_max_per_col: int = 10

    def __post_init__(self) -> None:
        """Validate at construction. See :meth:`validate`."""
        self.validate()

    def validate(self) -> None:
        """Check every invariant, from wherever the options are about to be used.

        This is called at construction *and* again when the options are handed
        to the engine, because the dataclass is mutable and the second path is
        the one people take:

        ```python
        ComputeOptions(chunk_size=0)     # ValueError: chunk_size must be positive
        c = ComputeOptions()
        c.chunk_size = 0                 # accepted, and profiled happily
        ```

        The constructor guarded a door nobody walks through. Two rules for one
        field is also the class of inconsistency that produces a bug report you
        cannot reproduce, so there is now one rule, called from both places.

        Raises:
            ValueError: If a value is outside its documented range.
            ConfigurationError: If `progress` is not one of its four shapes.
        """
        if self.numeric_sample_size <= 0:
            raise ValueError("numeric_sample_size must be positive")
        if self.max_uniques <= 0:
            raise ValueError("max_uniques must be positive")
        if self.top_k <= 0:
            raise ValueError("top_k must be positive")
        if self.chunk_size is not None and self.chunk_size <= 0:
            raise ValueError("chunk_size must be positive")
        if self.log_every_n_chunks <= 0:
            raise ValueError("log_every_n_chunks must be positive")
        # Validated here as well as on EngineConfig: _to_engine_config falls
        # back to a direct mapping inside a bare `except Exception`, so an error
        # raised further in becomes a silently different configuration rather
        # than a message. Failing at the public boundary is the only place the
        # caller reliably sees it.
        if not (
            self.progress is None
            or isinstance(self.progress, bool)
            or callable(self.progress)
            or self.progress == "auto"
        ):
            raise ConfigurationError(
                "progress must be True, False, 'auto' or a callable, got "
                f"{self.progress!r}"
            )
        if self.checkpoint_every_n_chunks < 0:
            raise ValueError("checkpoint_every_n_chunks must be non-negative")
        if self.checkpoint_max_to_keep <= 0:
            raise ValueError("checkpoint_max_to_keep must be positive")
        if self.boolean_detection_min_samples <= 0:
            raise ValueError("boolean_detection_min_samples must be positive")
        if not 0 <= self.boolean_detection_max_zero_ratio <= 1:
            raise ValueError("boolean_detection_max_zero_ratio must be between 0 and 1")
        if self.force_column_types is not None:
            valid_types = {"numeric", "categorical", "datetime", "boolean"}
            for col_name, col_type in self.force_column_types.items():
                if col_type not in valid_types:
                    raise ValueError(
                        f"Invalid column type '{col_type}' for column '{col_name}'. Must be one of: {valid_types}"
                    )
        if not 0 <= self.corr_threshold <= 1:
            raise ValueError("corr_threshold must be between 0 and 1")
        if self.corr_max_cols <= 0:
            raise ValueError("corr_max_cols must be positive")
        if self.corr_max_per_col <= 0:
            raise ValueError("corr_max_per_col must be positive")

    @property
    def checkpoint(self) -> CheckpointView:
        """The five checkpointing settings as one named group.

        A view rather than a nested dataclass: the fields stay where they are,
        so nothing existing breaks, while `options.checkpoint.every_n_chunks`
        gives the concern a name and keeps it out of the way of the settings
        people actually reach for.

        ```python
        options = ComputeOptions()
        options.checkpoint.every_n_chunks = 10
        options.checkpoint.dir = "./checkpoints"
        ```
        """
        return CheckpointView(self)

    # --- Engine-aligned accessors (for backward compatibility) ---
    @property
    def numeric_sample_k(self) -> int:
        """Alias used by the engine; backed by ``numeric_sample_size``."""
        return int(self.numeric_sample_size)

    @property
    def uniques_k(self) -> int:
        """Alias used by the engine; backed by ``max_uniques``."""
        return int(self.max_uniques)

    @property
    def topk_k(self) -> int:
        """Alias used by the engine; backed by ``top_k``."""
        return int(self.top_k)

checkpoint property

The five checkpointing settings as one named group.

A view rather than a nested dataclass: the fields stay where they are, so nothing existing breaks, while options.checkpoint.every_n_chunks gives the concern a name and keeps it out of the way of the settings people actually reach for.

options = ComputeOptions()
options.checkpoint.every_n_chunks = 10
options.checkpoint.dir = "./checkpoints"

validate()

Check every invariant, from wherever the options are about to be used.

This is called at construction and again when the options are handed to the engine, because the dataclass is mutable and the second path is the one people take:

ComputeOptions(chunk_size=0)     # ValueError: chunk_size must be positive
c = ComputeOptions()
c.chunk_size = 0                 # accepted, and profiled happily

The constructor guarded a door nobody walks through. Two rules for one field is also the class of inconsistency that produces a bug report you cannot reproduce, so there is now one rule, called from both places.

Raises:

Type	Description
ValueError	If a value is outside its documented range.
ConfigurationError	If progress is not one of its four shapes.

Source code in pysuricata/api.py

def validate(self) -> None:
    """Check every invariant, from wherever the options are about to be used.

    This is called at construction *and* again when the options are handed
    to the engine, because the dataclass is mutable and the second path is
    the one people take:

    ```python
    ComputeOptions(chunk_size=0)     # ValueError: chunk_size must be positive
    c = ComputeOptions()
    c.chunk_size = 0                 # accepted, and profiled happily
    ```

    The constructor guarded a door nobody walks through. Two rules for one
    field is also the class of inconsistency that produces a bug report you
    cannot reproduce, so there is now one rule, called from both places.

    Raises:
        ValueError: If a value is outside its documented range.
        ConfigurationError: If `progress` is not one of its four shapes.
    """
    if self.numeric_sample_size <= 0:
        raise ValueError("numeric_sample_size must be positive")
    if self.max_uniques <= 0:
        raise ValueError("max_uniques must be positive")
    if self.top_k <= 0:
        raise ValueError("top_k must be positive")
    if self.chunk_size is not None and self.chunk_size <= 0:
        raise ValueError("chunk_size must be positive")
    if self.log_every_n_chunks <= 0:
        raise ValueError("log_every_n_chunks must be positive")
    # Validated here as well as on EngineConfig: _to_engine_config falls
    # back to a direct mapping inside a bare `except Exception`, so an error
    # raised further in becomes a silently different configuration rather
    # than a message. Failing at the public boundary is the only place the
    # caller reliably sees it.
    if not (
        self.progress is None
        or isinstance(self.progress, bool)
        or callable(self.progress)
        or self.progress == "auto"
    ):
        raise ConfigurationError(
            "progress must be True, False, 'auto' or a callable, got "
            f"{self.progress!r}"
        )
    if self.checkpoint_every_n_chunks < 0:
        raise ValueError("checkpoint_every_n_chunks must be non-negative")
    if self.checkpoint_max_to_keep <= 0:
        raise ValueError("checkpoint_max_to_keep must be positive")
    if self.boolean_detection_min_samples <= 0:
        raise ValueError("boolean_detection_min_samples must be positive")
    if not 0 <= self.boolean_detection_max_zero_ratio <= 1:
        raise ValueError("boolean_detection_max_zero_ratio must be between 0 and 1")
    if self.force_column_types is not None:
        valid_types = {"numeric", "categorical", "datetime", "boolean"}
        for col_name, col_type in self.force_column_types.items():
            if col_type not in valid_types:
                raise ValueError(
                    f"Invalid column type '{col_type}' for column '{col_name}'. Must be one of: {valid_types}"
                )
    if not 0 <= self.corr_threshold <= 1:
        raise ValueError("corr_threshold must be between 0 and 1")
    if self.corr_max_cols <= 0:
        raise ValueError("corr_max_cols must be positive")
    if self.corr_max_per_col <= 0:
        raise ValueError("corr_max_per_col must be positive")

pysuricata.RenderOptions dataclass

Render options for the HTML output.

The current HTML report is self-contained and styled with built-in assets. This class controls various rendering aspects of the report.

Attributes:

Name	Type	Description
title	str | None	Optional custom title for the HTML report. If not provided, defaults to "PySuricata EDA Report". This title appears in both the browser tab and the main heading of the report.
description	str | None	Optional user description to display in the summary section. If provided, this will be shown below the "Summary" heading with consistent styling. Can be used to provide context about the dataset or analysis.
include_sample	bool	Whether the report shows a handful of sample rows. Default: True. It removes the sample table, and only the sample table. Raw values still reach the page from four other places: a categorical card's top-value labels and its shortest/longest exemplars, and the numeric and datetime extremes. This is not a redaction switch, and the docs no longer describe it as one (#285).
sample_rows	int	How many rows the sample shows, when it is shown. Default: 10.

Source code in pysuricata/api.py

@dataclass
class RenderOptions:
    """Render options for the HTML output.

    The current HTML report is self-contained and styled with built-in assets.
    This class controls various rendering aspects of the report.

    Attributes:
        title: Optional custom title for the HTML report. If not provided,
            defaults to "PySuricata EDA Report". This title appears in both
            the browser tab and the main heading of the report.
        description: Optional user description to display in the summary section.
            If provided, this will be shown below the "Summary" heading with
            consistent styling. Can be used to provide context about the dataset
            or analysis.
        include_sample: Whether the report shows a handful of sample rows.
            Default: True.

            It removes the sample table, and **only** the sample table. Raw
            values still reach the page from four other places: a categorical
            card's top-value labels and its shortest/longest exemplars, and the
            numeric and datetime extremes. This is not a redaction switch, and
            the docs no longer describe it as one (#285).
        sample_rows: How many rows the sample shows, when it is shown.
            Default: 10.
    """

    title: str | None = None
    description: str | None = None
    # Both were documented against `config.render` for several releases while
    # living only on `EngineConfig`, so assigning them did nothing and said
    # nothing -- including in a recipe that offered `include_sample = False` as
    # a way to keep raw values out of a report (#266). A silent no-op is bad; a
    # silent no-op sold as a privacy control is worse.
    #
    # It is still not a privacy control, which is the correction #285 makes:
    # it governs the sample table and nothing else.
    include_sample: bool = True
    sample_rows: int = 10

Results

pysuricata.Report dataclass

Source code in pysuricata/api.py

@dataclass
class Report:
    html: str
    stats: Mapping[str, Any]

    """Container for a rendered report and its computed statistics.

    Attributes:
        html: The full HTML document for the report (self‑contained).
        stats: JSON‑serializable mapping with dataset‑level and per‑column
            statistics, suitable for programmatic consumption (e.g., CI checks).
    """

    def save_html(self, path: str) -> None:
        """Write the HTML report to disk.

        Args:
            path: Destination file path. Parent directories must exist.
        """
        with open(path, "w", encoding="utf-8") as f:
            f.write(self.html)

    def save_json(self, path: str) -> None:
        """Write the statistics mapping to a JSON file.

        Args:
            path: Destination file path. Parent directories must exist.
        """
        converted_stats = _convert_numpy_types(self.stats)

        with open(path, "w", encoding="utf-8") as f:
            json.dump(converted_stats, f, ensure_ascii=False, indent=2)

    def save(self, path: str) -> None:
        """Save the report based on the file extension.

        If the extension is ``.html``, the HTML is written. If it is ``.json``,
        the stats mapping is written as JSON.

        Args:
            path: Destination file path.

        Raises:
            ValueError: If the extension is not one of ``.html`` or ``.json``.
        """
        ext = os.path.splitext(path)[1].lower()
        if ext == ".html":
            self.save_html(path)
        elif ext == ".json":
            self.save_json(path)
        else:
            raise ValueError(f"Unknown extension for Report.save(): {ext}")

    def __repr__(self) -> str:
        """One line, not the whole document.

        The dataclass default rendered every byte of ``html``, so a bare
        ``report`` in a REPL printed over a megabyte and an exception
        traceback carried the entire report inline.
        """
        dataset = self.stats.get("dataset", {}) if self.stats else {}
        rows = dataset.get("rows_est")
        cols = len(self.stats.get("columns", {})) if self.stats else 0
        shape = f"{rows:,} rows" if isinstance(rows, int) else "unknown rows"
        return (
            f"<Report {shape} x {cols} columns, {len(self.html) / 1024:.0f} KB of HTML>"
        )

    # Jupyter-friendly inline display
    def _repr_html_(self) -> str:  # pragma: no cover - visual
        return self.html

    def display_in_notebook(self, width: str = "100%", height: str = "600px") -> None:
        """Display the report in a Jupyter notebook using an iframe.

        This method provides better display for large reports in Jupyter notebooks
        by using an iframe instead of inline HTML.

        Args:
            width: Width of the iframe (default: "100%")
            height: Height of the iframe (default: "600px")
        """
        try:
            import os
            import tempfile
            import threading
            import time

            from IPython.display import IFrame, display

            # Create a temporary file
            with tempfile.NamedTemporaryFile(
                mode="w", suffix=".html", delete=False
            ) as f:
                f.write(self.html)
                temp_path = f.name

            # Get the file URL for the iframe
            file_url = f"file://{temp_path}"

            # Display using iframe
            display(IFrame(file_url, width=width, height=height))

            # Clean up the temporary file after a delay
            def cleanup():
                time.sleep(5)  # Wait 5 seconds before cleanup
                try:
                    os.unlink(temp_path)
                except OSError:
                    pass

            cleanup_thread = threading.Thread(target=cleanup)
            cleanup_thread.daemon = True
            cleanup_thread.start()

        except ImportError:
            # Fallback to regular HTML display if IPython is not available
            return self._repr_html_()

    def show(self, width: str = "100%", height: str = "600px") -> None:
        """Alias for display_in_notebook for convenience."""
        return self.display_in_notebook(width, height)

save(path)

Save the report based on the file extension.

If the extension is .html, the HTML is written. If it is .json, the stats mapping is written as JSON.

Parameters:

Name	Type	Description	Default
path	str	Destination file path.	required

Raises:

Type	Description
ValueError	If the extension is not one of .html or .json.

Source code in pysuricata/api.py

def save(self, path: str) -> None:
    """Save the report based on the file extension.

    If the extension is ``.html``, the HTML is written. If it is ``.json``,
    the stats mapping is written as JSON.

    Args:
        path: Destination file path.

    Raises:
        ValueError: If the extension is not one of ``.html`` or ``.json``.
    """
    ext = os.path.splitext(path)[1].lower()
    if ext == ".html":
        self.save_html(path)
    elif ext == ".json":
        self.save_json(path)
    else:
        raise ValueError(f"Unknown extension for Report.save(): {ext}")

save_html(path)

Write the HTML report to disk.

Parameters:

Name	Type	Description	Default
path	str	Destination file path. Parent directories must exist.	required

Source code in pysuricata/api.py

def save_html(self, path: str) -> None:
    """Write the HTML report to disk.

    Args:
        path: Destination file path. Parent directories must exist.
    """
    with open(path, "w", encoding="utf-8") as f:
        f.write(self.html)

save_json(path)

Write the statistics mapping to a JSON file.

Parameters:

Name	Type	Description	Default
path	str	Destination file path. Parent directories must exist.	required

Source code in pysuricata/api.py

def save_json(self, path: str) -> None:
    """Write the statistics mapping to a JSON file.

    Args:
        path: Destination file path. Parent directories must exist.
    """
    converted_stats = _convert_numpy_types(self.stats)

    with open(path, "w", encoding="utf-8") as f:
        json.dump(converted_stats, f, ensure_ascii=False, indent=2)

show(width='100%', height='600px')

Alias for display_in_notebook for convenience.

Source code in pysuricata/api.py

def show(self, width: str = "100%", height: str = "600px") -> None:
    """Alias for display_in_notebook for convenience."""
    return self.display_in_notebook(width, height)

display_in_notebook(width='100%', height='600px')

Display the report in a Jupyter notebook using an iframe.

This method provides better display for large reports in Jupyter notebooks by using an iframe instead of inline HTML.

Parameters:

Name	Type	Description	Default
width	str	Width of the iframe (default: "100%")	'100%'
height	str	Height of the iframe (default: "600px")	'600px'

Source code in pysuricata/api.py

def display_in_notebook(self, width: str = "100%", height: str = "600px") -> None:
    """Display the report in a Jupyter notebook using an iframe.

    This method provides better display for large reports in Jupyter notebooks
    by using an iframe instead of inline HTML.

    Args:
        width: Width of the iframe (default: "100%")
        height: Height of the iframe (default: "600px")
    """
    try:
        import os
        import tempfile
        import threading
        import time

        from IPython.display import IFrame, display

        # Create a temporary file
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".html", delete=False
        ) as f:
            f.write(self.html)
            temp_path = f.name

        # Get the file URL for the iframe
        file_url = f"file://{temp_path}"

        # Display using iframe
        display(IFrame(file_url, width=width, height=height))

        # Clean up the temporary file after a delay
        def cleanup():
            time.sleep(5)  # Wait 5 seconds before cleanup
            try:
                os.unlink(temp_path)
            except OSError:
                pass

        cleanup_thread = threading.Thread(target=cleanup)
        cleanup_thread.daemon = True
        cleanup_thread.start()

    except ImportError:
        # Fallback to regular HTML display if IPython is not available
        return self._repr_html_()

pysuricata.Comparison dataclass

The full diff between two datasets.

Attributes:

Name	Type	Description
dataset	DatasetDelta	Whole-frame movement.
schema	SchemaDelta	Columns added, removed and retyped.
columns	dict[str, ColumnDelta]	Per-column deltas, for columns present in both.

Source code in pysuricata/comparison.py

@dataclass(frozen=True)
class Comparison:
    """The full diff between two datasets.

    Attributes:
        dataset: Whole-frame movement.
        schema: Columns added, removed and retyped.
        columns: Per-column deltas, for columns present in both.
    """

    dataset: DatasetDelta
    schema: SchemaDelta
    columns: dict[str, ColumnDelta]

    def to_dict(self) -> dict[str, Any]:
        """A JSON-serialisable view."""
        return {
            "dataset": asdict(self.dataset),
            "schema": {
                "added": dict(self.schema.added),
                "removed": dict(self.schema.removed),
                "retyped": {k: list(v) for k, v in self.schema.retyped.items()},
                "unchanged": list(self.schema.unchanged),
            },
            "columns": {name: asdict(delta) for name, delta in self.columns.items()},
        }

    def __repr__(self) -> str:
        moved = sum(1 for delta in self.columns.values() if _moved(delta))
        return (
            f"<Comparison {len(self.columns)} columns compared, {moved} moved, "
            f"{len(self.schema.added)} added, {len(self.schema.removed)} removed>"
        )

to_dict()

A JSON-serialisable view.

Source code in pysuricata/comparison.py

def to_dict(self) -> dict[str, Any]:
    """A JSON-serialisable view."""
    return {
        "dataset": asdict(self.dataset),
        "schema": {
            "added": dict(self.schema.added),
            "removed": dict(self.schema.removed),
            "retyped": {k: list(v) for k, v in self.schema.retyped.items()},
            "unchanged": list(self.schema.unchanged),
        },
        "columns": {name: asdict(delta) for name, delta in self.columns.items()},
    }

Errors

pysuricata.PySuricataError

Bases: Exception

Base class for every error this library raises deliberately.

The public surface used to raise TypeError for an unsupported input, ValueError for an unsupported file format and RuntimeError from the engine for the same class of mistake, so a caller had to catch three types to handle one situation. Everything raised on purpose now derives from this; the specific types remain as subclasses so existing except TypeError handlers keep working.

pysuricata.UnsupportedDataError

Bases: PySuricataError, TypeError

The input is not something this library can profile.

pysuricata.ConfigurationError

Bases: PySuricataError, ValueError

A configuration value is outside its documented range.

Readers

The batch readers behind the path, Arrow and DuckDB inputs, for when you want the batches rather than a profile.

pysuricata.sources

Stream Arrow, Parquet and DuckDB without materialising them first.

Every source used to arrive as a pandas or polars frame, which for a Parquet file or a DuckDB query meant reading the whole thing into memory before profiling it. That contradicts the one claim the library is positioned on -- bounded memory regardless of dataset size -- for exactly the inputs where the claim matters most.

The engine already consumes an iterable of chunks, so this is a reader per source rather than a change to the core. All three arrive as Arrow record batches, so there is really one reader and two entry points.

What this is not. The accumulators take numpy arrays, so each batch is converted on its way through; this is not a zero-copy Arrow path. What changes is that one batch is materialised at a time instead of the entire file.

One behavioural consequence, stated up front. Type inference reclassifies a numeric column as categorical from the distinct values it can see, which is sound evidence only when the whole column is in hand. A stream cannot offer that: a leading run of one value looks low-cardinality while the column is not. So a file that arrives in a single batch is handed to the engine as a frame -- identical results to pd.read_parquet -- and a file that does not is treated as what it is, a stream.

stream_parquet(path, *, batch_size=None, columns=None)

Yield a Parquet file one batch at a time.

Parameters:

Name	Type	Description	Default
path	str | PathLike	Path to a .parquet file.	required
batch_size	int | None	Rows per batch. Defaults to DEFAULT_BATCH_ROWS.	None
columns	list[str] | None	Optional subset to read. Columns not read are never decoded, which is where most of the saving is on a wide file.	None

Yields:

Type	Description
DataFrame	One pandas DataFrame per batch.

Raises:

Type	Description
ImportError	If pyarrow is not installed.
FileNotFoundError	If the file does not exist.

Source code in pysuricata/sources.py

def stream_parquet(
    path: str | os.PathLike,
    *,
    batch_size: int | None = None,
    columns: list[str] | None = None,
) -> Iterator[pd.DataFrame]:
    """Yield a Parquet file one batch at a time.

    Args:
        path: Path to a `.parquet` file.
        batch_size: Rows per batch. Defaults to `DEFAULT_BATCH_ROWS`.
        columns: Optional subset to read. Columns not read are never decoded,
            which is where most of the saving is on a wide file.

    Yields:
        One pandas DataFrame per batch.

    Raises:
        ImportError: If pyarrow is not installed.
        FileNotFoundError: If the file does not exist.
    """
    pq = _require("pyarrow.parquet", "reading Parquet")
    resolved = Path(path)
    if not resolved.exists():
        raise FileNotFoundError(f"File not found: {resolved}")

    # `pre_buffer=True` is pyarrow's default and it is aimed at remote storage:
    # it schedules ahead-of-time reads for row groups the caller has not asked
    # for yet, trading memory for hiding read latency. `stream_parquet` only
    # ever sees a local path (`resolved.exists()` above), so there is no
    # latency to hide and the prefetch is pure retained memory -- measured at
    # roughly 2x this reader's own working set on a text-heavy file under a
    # 512 MB ceiling (#92).
    handle = pq.ParquetFile(resolved, pre_buffer=False)
    batches = handle.iter_batches(
        batch_size=batch_size or DEFAULT_BATCH_ROWS, columns=columns
    )
    for batch in batches:
        yield batch.to_pandas()

stream_ipc(path, *, batch_size=None)

Yield an Arrow IPC file one batch at a time.

This is the format another runtime writes -- arrow::write_ipc_file() in R, Arrow.write() in Julia, the arrow crate in Rust -- which is what makes it worth reading directly rather than through a conversion (#247).

Three different framings can wear these extensions, and the extension does not say which. Dispatching on the suffix would load an R write_ipc_file() and fail on an Arrow.write(), since Julia defaults to the stream framing. So the first bytes decide:

magic	framing	read by
ARROW1	IPC file, with a footer	pa.ipc.open_file
ÿÿÿÿ	IPC stream, no footer	pa.ipc.open_stream
FEA1	Feather V1, not IPC at all	feather.read_table

Only the last materialises, and that format is legacy -- pyarrow itself deprecated writing it in 25.0.0 -- so it has no streaming reader to use.

Parameters:

Name	Type	Description	Default
path	str | PathLike	Path to a .arrow, .feather or .ipc file.	required
batch_size	int | None	Rows per batch, where the framing lets us choose. The two IPC framings carry the batching their writer chose and are read as they were written.	None

Yields:

Type	Description
DataFrame	One pandas DataFrame per batch.

Raises:

Type	Description
ImportError	If pyarrow is not installed.
FileNotFoundError	If the file does not exist.

Source code in pysuricata/sources.py

def stream_ipc(
    path: str | os.PathLike, *, batch_size: int | None = None
) -> Iterator[pd.DataFrame]:
    """Yield an Arrow IPC file one batch at a time.

    This is the format another runtime writes -- `arrow::write_ipc_file()` in
    R, `Arrow.write()` in Julia, the `arrow` crate in Rust -- which is what
    makes it worth reading directly rather than through a conversion (#247).

    **Three different framings can wear these extensions, and the extension
    does not say which.** Dispatching on the suffix would load an R
    `write_ipc_file()` and fail on an `Arrow.write()`, since Julia defaults to
    the stream framing. So the first bytes decide:

    | magic | framing | read by |
    |---|---|---|
    | `ARROW1` | IPC file, with a footer | `pa.ipc.open_file` |
    | `\xff\xff\xff\xff` | IPC stream, no footer | `pa.ipc.open_stream` |
    | `FEA1` | Feather V1, not IPC at all | `feather.read_table` |

    Only the last materialises, and that format is legacy -- pyarrow itself
    deprecated writing it in 25.0.0 -- so it has no streaming reader to use.

    Args:
        path: Path to a `.arrow`, `.feather` or `.ipc` file.
        batch_size: Rows per batch, where the framing lets us choose. The two
            IPC framings carry the batching their writer chose and are read as
            they were written.

    Yields:
        One pandas DataFrame per batch.

    Raises:
        ImportError: If pyarrow is not installed.
        FileNotFoundError: If the file does not exist.
    """
    pa = _require("pyarrow", "reading Arrow IPC")
    resolved = Path(path)
    if not resolved.exists():
        raise FileNotFoundError(f"File not found: {resolved}")

    with open(resolved, "rb") as handle:
        magic = handle.read(6)

    if magic.startswith(_FEATHER_V1_MAGIC):
        feather = _require("pyarrow.feather", "reading Feather V1")
        yield from stream_arrow(feather.read_table(resolved), batch_size=batch_size)
        return

    opener = (
        pa.ipc.open_file if magic.startswith(_IPC_FILE_MAGIC) else pa.ipc.open_stream
    )
    yield from stream_arrow(opener(resolved), batch_size=batch_size)

stream_arrow(source, *, batch_size=None)

Yield an Arrow source one batch at a time.

Parameters:

Name	Type	Description	Default
source	Any	A RecordBatchReader, Table, RecordBatch, or a pyarrow.dataset.Dataset.	required
batch_size	int | None	Rows per batch, where the source lets us choose. A RecordBatchReader already carries its own batching and is passed through as it is.	None

Yields:

Type	Description
DataFrame	One pandas DataFrame per batch.

Raises:

Type	Description
ImportError	If pyarrow is not installed.
TypeError	If the object is not an Arrow source.

Source code in pysuricata/sources.py

def stream_arrow(
    source: Any, *, batch_size: int | None = None
) -> Iterator[pd.DataFrame]:
    """Yield an Arrow source one batch at a time.

    Args:
        source: A `RecordBatchReader`, `Table`, `RecordBatch`, or a
            `pyarrow.dataset.Dataset`.
        batch_size: Rows per batch, where the source lets us choose. A
            `RecordBatchReader` already carries its own batching and is passed
            through as it is.

    Yields:
        One pandas DataFrame per batch.

    Raises:
        ImportError: If pyarrow is not installed.
        TypeError: If the object is not an Arrow source.
    """
    pa = _require("pyarrow", "reading Arrow data")
    rows = batch_size or DEFAULT_BATCH_ROWS

    if isinstance(source, pa.RecordBatchReader):
        # Its batching is a property of whatever produced it; re-chunking here
        # would mean buffering, which is the thing being avoided.
        for batch in source:
            yield batch.to_pandas()
        return

    if isinstance(source, pa.Table):
        for batch in source.to_batches(max_chunksize=rows):
            yield batch.to_pandas()
        return

    if isinstance(source, pa.RecordBatch):
        yield source.to_pandas()
        return

    # `RecordBatchFileReader` -- what `pa.ipc.open_file()` returns (#247). It
    # is the one Arrow reader that is *not* a `RecordBatchReader`: the IPC file
    # format has a footer listing every batch, so it offers random access by
    # index instead of a forward-only iterator. Reading by index keeps the
    # bounded-memory promise; `read_all()` would materialise the file.
    #
    # Duck-typed on the pair of members rather than the class name, for the
    # reason `_duckdb_reader` is: a check that names a class breaks when the
    # class moves, and these two members are specific enough together.
    count = getattr(source, "num_record_batches", None)
    get_batch = getattr(source, "get_batch", None)
    if isinstance(count, int) and callable(get_batch):
        for index in range(count):
            yield get_batch(index).to_pandas()
        return

    to_batches = getattr(source, "to_batches", None)
    if callable(to_batches):  # pyarrow.dataset.Dataset and friends
        for batch in to_batches(batch_size=rows):
            yield batch.to_pandas()
        return

    if hasattr(source, "__arrow_c_stream__"):
        # The PyCapsule interface: whatever produced this, pyarrow can read it.
        for batch in pa.RecordBatchReader.from_stream(source):
            yield batch.to_pandas()
        return

    raise TypeError(
        f"{type(source).__name__} is not an Arrow source. Pass a Table, a "
        "RecordBatch, a RecordBatchReader or a Dataset."
    )

stream_duckdb(relation, *, batch_size=None)

Yield a DuckDB relation or query result one batch at a time.

A relation is a query that has not run yet, so this profiles a result set without ever landing it in memory:

con.sql("SELECT * FROM 'events/*.parquet' WHERE ts > '2026-01-01'")

Parameters:

Name	Type	Description	Default
relation	Any	A DuckDB relation or result — anything with to_arrow_reader, or fetch_record_batch on older DuckDB.	required
batch_size	int | None	Rows per batch.	None

Yields:

Type	Description
DataFrame	One pandas DataFrame per batch.

Raises:

Type	Description
TypeError	If the object cannot produce Arrow batches.

Source code in pysuricata/sources.py

def stream_duckdb(
    relation: Any, *, batch_size: int | None = None
) -> Iterator[pd.DataFrame]:
    """Yield a DuckDB relation or query result one batch at a time.

    A relation is a query that has not run yet, so this profiles a result set
    without ever landing it in memory:

    ```python
    con.sql("SELECT * FROM 'events/*.parquet' WHERE ts > '2026-01-01'")
    ```

    Args:
        relation: A DuckDB relation or result — anything with
            `to_arrow_reader`, or `fetch_record_batch` on older DuckDB.
        batch_size: Rows per batch.

    Yields:
        One pandas DataFrame per batch.

    Raises:
        TypeError: If the object cannot produce Arrow batches.
    """
    fetch = _duckdb_reader(relation)
    if fetch is None:
        raise TypeError(
            f"{type(relation).__name__} is not a DuckDB relation. Pass the "
            "result of con.sql(...) or con.execute(...)."
        )
    for batch in fetch(batch_size or DEFAULT_BATCH_ROWS):
        yield batch.to_pandas()

is_arrow_source(obj)

Whether stream_arrow can read this object.

Two ways to qualify. A pyarrow object is recognised by module name, without importing pyarrow — an import inside a type check would make every profile() call pay for a dependency the caller may not have. Anything else qualifies by exporting __arrow_c_stream__, the Arrow PyCapsule interface, which is how the rest of the ecosystem hands data over without anyone agreeing on a type.

pandas and polars are excluded explicitly rather than by being checked first somewhere else: both export the capsule (pandas since 2.2), both have their own adapter, and a predicate that is only correct because of the order its caller happens to use it in is a trap for the next caller.

Source code in pysuricata/sources.py

def is_arrow_source(obj: Any) -> bool:
    """Whether `stream_arrow` can read this object.

    Two ways to qualify. A pyarrow object is recognised by module name, without
    importing pyarrow — an import inside a type check would make every
    `profile()` call pay for a dependency the caller may not have. Anything
    else qualifies by exporting `__arrow_c_stream__`, the Arrow PyCapsule
    interface, which is how the rest of the ecosystem hands data over without
    anyone agreeing on a type.

    pandas and polars are excluded explicitly rather than by being checked
    first somewhere else: both export the capsule (pandas since 2.2), both have
    their own adapter, and a predicate that is only correct because of the order
    its caller happens to use it in is a trap for the next caller.
    """
    module = type(obj).__module__ or ""
    if module.startswith(("pandas", "polars")):
        return False
    if hasattr(obj, "__arrow_c_stream__"):
        return True
    if not module.startswith("pyarrow"):
        return False
    return type(obj).__name__ in {
        "Table",
        "RecordBatch",
        "RecordBatchReader",
        # `pa.ipc.open_file()` and `open_stream()` (#247). The stream reader is
        # a `RecordBatchReader` subclass and would qualify anyway; the file
        # reader is not, and named here is the only way it qualifies.
        "RecordBatchFileReader",
        "RecordBatchStreamReader",
        "Dataset",
        "FileSystemDataset",
        "InMemoryDataset",
        "UnionDataset",
    }

is_duckdb_relation(obj)

Whether stream_duckdb can read this object.

Duck-typed on the batch-reader method, which is specific enough to be safe and survives DuckDB moving its classes between modules -- they live in _duckdb rather than duckdb, which a module-name check gets wrong.

Source code in pysuricata/sources.py

def is_duckdb_relation(obj: Any) -> bool:
    """Whether `stream_duckdb` can read this object.

    Duck-typed on the batch-reader method, which is specific enough to be safe
    and survives DuckDB moving its classes between modules -- they live in
    `_duckdb` rather than `duckdb`, which a module-name check gets wrong.
    """
    return _duckdb_reader(obj) is not None