Missing Values Analysis

Comprehensive guide to PySuricata's intelligent missing values analysis with adaptive display and chunk-level distribution tracking.

Examples on this page assume a DataFrame named df

Every snippet below that does not build its own frame expects one already in scope. Paste this first to follow along:

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
df = pd.DataFrame(
    {
        "id": range(5_000),
        "amount": rng.lognormal(3, 1, 5_000),
        "country": rng.choice(["ES", "FR", "DE"], 5_000),
        "signed_up": pd.date_range("2024-01-01", periods=5_000, freq="17min"),
        "active": rng.random(5_000) > 0.3,
    }
)

Overview

Missing data is ubiquitous in real-world datasets. PySuricata provides:

Per-column counts and percentages, exact, from the single pass
Chunk-level tracking: where in the stream the missing values fell, which is what turns "8% missing" into "the third file was empty"
Intelligent display: adaptive limits based on dataset size
Smart filtering: show only columns above a threshold
Expandable UI: progressive disclosure for many columns

What it does not do is cross-column pattern analysis — which columns go missing together. That needs the joint missingness matrix, which is quadratic in columns and not available from a bounded single pass. The two sections below say why, and what to do instead.

Missing Data Mechanisms

MAR, MCAR, MNAR

Missing Completely At Random (MCAR): - Missingness independent of observed/unobserved data - \(P(\text{missing} | X, Y) = P(\text{missing})\) - Example: Sensor randomly fails

Missing At Random (MAR): - Missingness depends on observed data only - \(P(\text{missing} | X, Y_{\text{obs}}) = P(\text{missing} | X)\) - Example: Older patients skip optional questions

Missing Not At Random (MNAR): - Missingness depends on unobserved values - Example: High earners don't report income

Detection not automated

Determining mechanism requires domain knowledge. PySuricata shows patterns to help investigation.

Mathematical Definitions

Missing Rate

For column with \(n_{\text{total}}\) observations:

\[ MR = \frac{n_{\text{missing}}}{n_{\text{total}}} \]

Missing Pattern Entropy

For \(k\) different missing patterns (combinations of missing columns):

\[ H_{\text{pattern}} = -\sum_{i=1}^{k} p_i \log_2 p_i \]

where \(p_i\) is the proportion of rows with pattern \(i\).

High entropy: Many different patterns (complex missingness)
Low entropy: Few patterns (systematic missingness)

Not computed

Pattern entropy is over joint patterns — the \(2^p\) possible combinations of which columns are missing in a row. Counting them needs either that many counters or a distinct-value sketch over row-level bitmasks, and either way the answer describes co-missingness, which is a cross-column question this single pass does not ask.

What is published is per-column and per-chunk: missing and missing_cells_pct, plus the chunk strip. If two columns go missing in the same chunks, the strip shows it.

Intelligent Display System

Dynamic Limits

Limits adapt to dataset size:

Dataset Size	Initial Display	Expanded Display
≤10 columns	All	All
11-50 columns	10	25
51-200 columns	12	25
>200 columns	15	25

Smart Filtering

Threshold: Only show columns with >\(t\)% missing (default \(t=0.5\)%)

Rationale: Columns with <0.5% missing are usually not concerning.

Expandable UI

For datasets with many missing columns: 1. Initial view: Show top \(n\) columns 2. Expand button: Reveal up to 25 total 3. Smooth animation: JavaScript-powered transition

Chunk-Level Distribution

Track missing data per chunk to identify: - Temporal patterns (early vs. late data) - Batch patterns (certain files have more missing) - System issues (outages, collection failures)

Visualization

Horizontal bar showing missing percentage per chunk:

Chunk 1  ████░░░░░░  40%
Chunk 2  ██░░░░░░░░  20%
Chunk 3  ░░░░░░░░░░   0%
Chunk 4  ███████░░░  70%

Reveals chunk 4 has data quality issue.

Configuration

from pysuricata import profile, ProfileConfig

config = ProfileConfig()

# Missing columns display threshold (default 0.5%)
# (Not yet configurable in current version)

# Maximum initial display (default: dynamic based on dataset size)
# (Not yet configurable in current version)

report = profile(df, config=config)

Implementation

MissingColumnsAnalyzer

class MissingColumnsAnalyzer:
    MIN_THRESHOLD_PCT = 0.5
    MAX_INITIAL_DISPLAY = 8
    MAX_EXPANDED_DISPLAY = 25

    def analyze_missing_columns(self, miss_list, n_cols, n_rows):
        """Analyze and filter missing columns"""
        # Filter significant missing
        significant = [
            item for item in miss_list
            if item[1] >= self.MIN_THRESHOLD_PCT
        ]

        # Determine limits
        initial_limit = self._get_initial_display_limit(n_cols, n_rows)
        expanded_limit = self._get_expanded_display_limit(n_cols, n_rows)

        # Build result
        return MissingColumnsResult(
            initial_columns=significant[:initial_limit],
            expanded_columns=significant[:expanded_limit],
            needs_expandable=len(significant) > initial_limit,
            total_significant=len(significant),
            total_insignificant=len(miss_list) - len(significant)
        )

Interpreting Results

High Missing Percentage (>50%)

Possible causes: - Optional field (by design) - Data collection issue - Recent column (added midway) - Rare event (e.g., "error_message" only on errors)

Actions: - Verify if intentional - Consider imputation or exclusion - Check data pipeline

Systematic Patterns

Multiple columns missing together:

Possible causes: - Related optional section (e.g., address fields) - Batch import failure - Survey skip logic

Actions: - Analyze co-occurrence - Check data source - Document business logic

Increasing Over Time

More missing in later chunks:

Possible causes: - Degrading data quality - System malfunction - Intentional change

Actions: - Investigate recent changes - Alert data engineering team

Why There Is No MCAR Test

Little's test (1988) compares the means of subgroups defined by missing pattern under \(H_0\): the data is missing completely at random. PySuricata does not run it, and this one is not a "not yet".

It needs the joint pattern structure — subgroups of rows sharing a missingness signature, and each subgroup's mean vector across every column. That is a second pass over data grouped by something you only know after the first one, which is the shape of computation this profiler is built not to do.

It is also the wrong altitude. Whether missingness is MCAR, MAR or MNAR is a modelling judgement about your domain, made once, deliberately, with the column semantics in hand. A profiler's job is to hand you the evidence — which columns, how much, and where in the stream — not to return a verdict on it.

Reference: Little, R.J.A. (1988), "A Test of Missing Completely at Random for Multivariate Data with Missing Values", JASA, 83(404): 1198–1202.

Imputation Considerations

Mean/Median Imputation

\[ x_{\text{imputed}} = \begin{cases} x & \text{if observed} \\ \bar{x} & \text{if missing} \end{cases} \]

Pros: Simple, fast
Cons: Reduces variance, distorts correlations

Multiple Imputation

Generate \(m\) complete datasets with different imputations, analyze separately, combine results.

Pros: Preserves uncertainty
Cons: Complex, computationally expensive

Model-Based

Use ML model to predict missing values from other columns.

Pros: Can capture complex relationships
Cons: Requires training, may introduce bias

PySuricata does not impute

PySuricata is a profiling tool, not a preprocessing tool. Imputation should be done separately based on domain knowledge.

Best Practices

Document missingness: Record why data is missing
Distinguish NULL types: NULL vs. empty string vs. "N/A"
Set thresholds: Define acceptable missing percentages
Monitor trends: Track missing rates over time
Investigate patterns: Look for systematic missingness

Examples

Basic Usage

import pandas as pd

from pysuricata import profile

# Dataset with missing values
df = pd.DataFrame({
    "age": [25, 30, None, 45, 50],
    "income": [50000, None, None, 80000, 90000],
    "city": ["NYC", "LA", None, "Chicago", None]
})

report = profile(df)
# Report shows missing percentages and patterns

Access Missing Statistics

from pysuricata import summarize

stats = summarize(df)
print(f"Missing cells: {stats['dataset']['missing_cells_pct']:.1f}%")

for col, col_stats in stats["columns"].items():
    missing_pct = col_stats.get("missing_pct", 0)
    if missing_pct > 10:
        print(f"{col}: {missing_pct:.1f}% missing")

References

Little, R.J.A., Rubin, D.B. (2019), Statistical Analysis with Missing Data, 3rd ed., Wiley.
Rubin, D.B. (1976), "Inference and Missing Data", Biometrika, 63(3): 581–592.
Schafer, J.L., Graham, J.W. (2002), "Missing Data: Our View of the State of the Art", Psychological Methods, 7(2): 147–177.
Wikipedia: Missing data - Link