Benchmarks & fidelity

Performance benchmarks and the round-trip fidelity dashboard, with methodology.

Rivoli PDF measures itself two ways. Performance benchmarks track how fast and how cheaply it generates PDFs as documents scale. Round-trip fidelity asks a harder question: when a PDF is parsed, rebuilt from the document model, and re-exported, does the result still look like the original? Both are deterministic and require no network access.

Performance benchmarks

The performance suite lives in benchmarks/Rivoli.Pdf.Benchmarks/ and is built on BenchmarkDotNet. It targets .NET 8 and .NET 10 and focuses on the generation path: building a document tree and rendering it to PDF bytes.

The suite is organized into four benchmark classes, each parameterized so you can see how cost grows with document size:

Benchmark class	Varies	Measures
`ParagraphHeavyBenchmarks`	10 / 100 / 1000 paragraphs	Text-heavy generation throughput
`TableHeavyBenchmarks`	10 / 50 / 100 tables	Table layout and rendering cost
`DocumentSizeBenchmarks`	1 / 10 / 100 / 1000 sections	Scaling across document size
`MixedContentBenchmarks`	50 / 100 / 200 blocks	Realistic mixed-content documents

Each one builds a Document, renders it through PdfDocumentRenderer to a MemoryStream, and returns the resulting byte[]. The [MemoryDiagnoser] attribute means every run reports allocations alongside timing, so you can watch both wall-clock and GC pressure as the parameters grow.

Running the benchmarks

cd benchmarks/Rivoli.Pdf.Benchmarks
dotnet run -c Release

BenchmarkDotNet presents a menu so you can pick a single class or run them all. Always use -c Release: benchmarking a Debug build measures the wrong thing. Results (Markdown tables, raw data) are written under the repo’s consolidated .outputs/ tree. For a quick iteration loop, pass a shorter job:

dotnet run -c Release -- --job short

Round-trip pixel-fidelity

Performance tells you the generator is fast; fidelity tells you it is correct in the way that matters most for documents: visually. The fidelity harness lives in fidelity/ and pulls PDFs ranging from trivial to pathological, runs each through a parse → rebuild → re-export loop, and compares the result to the original page by page, pixel by pixel.

Running the fidelity harness

The harness ships as a CLI, rivoli-rtf (see fidelity/README.md for the full option reference):

# from the repository root
dotnet build fidelity/Rivoli.Pdf.Fidelity.Cli -c Release
RTF=fidelity/Rivoli.Pdf.Fidelity.Cli/bin/Release/net8.0/rivoli-rtf

# quick smoke: 32 curated real-world PDFs
$RTF run --corpus public --dpi 150 --structural

# the full benchmark: the entire mozilla/pdf.js test corpus, 955 documents
$RTF run --corpus public-full --dpi 150 --structural \
    --out .outputs/results.json --html .outputs/dashboard.html

Downloaded corpus PDFs are cached under .outputs/corpus-cache/ and reused offline, so only the first run needs the network. --html writes a self-contained dashboard; --filter, --shard-start/--shard-count, --artifacts (diff images for failures), and --baseline + --fail-on-regression (CI gate) narrow or gate a run.

The loop

original.pdf ──parse+rebuild──▶ Document DOM ──re-export──▶ rebuilt.pdf
     │                                                            │
     └──── rasterize (oracle) ──▶  page PNGs  ◀── rasterize ──────┘
                                      │
                                pixel/perceptual compare → score

The trusted rasterizer (the oracle) is PDFium, used through a managed NuGet binding behind an IRasterizerOracle abstraction. Both the original and the rebuilt PDF are rasterized by the same oracle, so the comparison isolates parse + rebuild + re-export quality from any quirk of a particular renderer. The oracle lives only in the test and harness projects; it never enters the shipped library.

Why not “pixel-perfect”?

Byte-for-byte or 100%-identical-pixel regeneration of arbitrary PDFs is not achievable, and no engine does it: anti-aliasing, font hinting, rasterizer rounding, and JPEG re-encoding all introduce legitimate sub-pixel differences. So fidelity is a tiered model, not a single boolean:

Level	Metric	Passes when	Typical use
L0 Exact	per-pixel equality	100% identical pixels	simplest synthetic docs, AA disabled
L1 Near-exact	per-channel Δ, AA-aware	≥ 99.9% pixels within tolerance	synthetic + simple real docs
L2 Perceptual	SSIM / MS-SSIM	structural-similarity score above threshold	real-world documents
L3 Structural	text + region positions	text and region overlap above threshold	lossy / scanned / pathological
L4 Semantic	extracted-text equality	normalized text matches	known-limitation fallbacks

Each document declares the highest level it is expected to reach plus a per-feature budget, and runs are measured against a stored baseline scoreboard rather than against absolute perfection. “Pixel-perfect” is the aspiration for the simplest tier through a single deterministic oracle; real-world documents are graded perceptually.

The corpus ladder

Documents are organized on a complexity ladder, from one line of text to messy real-world files, so a regression can be pinned to the simplest tier that surfaces it:

Tier	Theme
0–1	One line / multi-paragraph with core-14 fonts and basic styling
2	Embedded TrueType with subsetting and ToUnicode
3–4	Vector graphics; raster images (JPEG/PNG/CMYK, transparency)
5–6	Tables, multi-column layout, lists and nested structure
7	CID/Type0 fonts, CJK, RTL, complex scripts
8–9	Forms and annotations; layers (OCG) and attachments
10–11	Encryption and signatures; standards (PDF/A, tagged/PDF-UA)
12	Pathological / real-world: scanned, huge, mixed, linearized

The corpus is drawn from three sources (a deterministic synthetic generator, public corpora, and a curated real-world set), each document tagged with its tier, the features it exercises, and its license.