Sentinel-Expanse — Multi-Dictionary Streaming Compression Engine

Compression, Measured Properly

A custom dictionary-based streaming compressor built from scratch. Not a wrapper — a complete pipeline with its own binary format, dictionary training system, and the Weisman Score metric that rewards both ratio and speed.

Sentinel v5.1 ZLIB Baseline LZMA Reference

View Source See Benchmarks ↓

Weisman Score = Compression Ratio × Throughput (MB/s) Higher is better — rewards both ratio AND speed

Live Benchmark Results · 10 MB Structured Logs · 5 Repetitions

Sentinel v5

1311

Weisman Score (±23.4)

Ratio6.98×

Speed187.98 MB/s

Integrity✓ BIT-PERFECT

ZLIB (Level 6)

853

Weisman Score (±13.2)

Ratio7.36×

Speed115.81 MB/s

Integrity✓ BIT-PERFECT

LZMA (Preset 1)

239

Weisman Score (±13.9)

Ratio7.98×

Speed29.89 MB/s

Integrity✓ BIT-PERFECT

SENTINEL

1311

ZLIB

853

LZMA

239

Why Sentinel Wins

Sentinel v5 wins the Weisman Score by 54% over ZLIB despite a slightly lower compression ratio (6.98× vs 7.36×). The adaptive dictionary delivers 62% higher throughput (188 vs 116 MB/s), which more than compensates for the ratio delta.

LZMA achieves the best ratio (7.98×) but its 30 MB/s throughput produces the lowest Weisman Score — 3.6× worse than ZLIB and 5.5× worse than Sentinel. This confirms the Weisman metric's design: it correctly penalizes slow-but-compressive algorithms for real-world use.

Roundtrip Integrity Verification

✓ test_god_tier_roundtrip — GodTierEngine (v5.1) PASSED

✓ test_sentinel_v6_python_roundtrip — SentinelV6 Multi-Dict PASSED

✓ benchmark_harness — SHA-256 Bit-Perfect Verification PASSED

How It Works

📖

Multi-Dictionary Training

Input is split into N segments. A separate dictionary is trained per segment using token frequency analysis. At compression time, each chunk is probed against all dictionaries and routed to the best one.

🌊

Streaming Architecture

Binary container format (SNTL) with per-chunk Adler-32 integrity checksums. Supports streaming compress and decompress without loading the full file into memory.

📊

Weisman Score Metric

Named after the fictional Pied Piper engineer. Measures Ratio × Speed (MB/s) — a single number that rewards both compression quality and throughput simultaneously.

⚡

Dual Implementation

Research engine in Python (v5.1 GodTier + v6 Multi-Dict) with production C implementation (v6 + v7). Both verified through automated bit-perfect roundtrip testing.

🖥️

Web Dashboard

Flask-based evaluation dashboard on port 5555. Upload two files, compress with both Sentinel and ZLIB, get side-by-side Weisman Scores with SHA-256 integrity proof.

🔒

VERITAS Certified

Engine integrity verified under the VERITAS Ω-1.3.1 10-gate pipeline. All test artifacts are cryptographically sealed and audit-ready.

Compression Pipeline

Input File │ ├─► Segment Splitter (N segments, configurable chunk_size) │ │ │ ├─► Dictionary Trainer (per-segment token frequency) │ │ │ │ │ └─► Dictionary Pool [D₁, D₂, ..., Dₙ] │ │ │ └─► Chunk Router │ │ │ ├─► Probe each chunk against all dictionaries │ ├─► Select best dictionary (highest ratio) │ └─► Compress with selected dictionary │ ├─► Binary Container Writer │ │ │ ├─► Header: magic bytes (SNTL), version, dict count │ ├─► Dictionary table (inline) │ └─► Chunk stream: [dict_id | compressed_data | adler32] │ └─► Output: .sntl compressed file

Quick Start

# Clone
git clone https://github.com/VrtxOmega/sentinel-expanse.git
cd sentinel-expanse

# Run benchmark
python python/benchmark_harness.py

# Run tests
pytest tests/test_roundtrip.py -v

# Launch dashboard
pip install flask
python dashboard/app.py     # → http://127.0.0.1:5555