Benchmarks

Head-to-Head: DOCX Parsing (29 Files)

The fairest comparison: 29 DOCX files that all 8 tools can parse. Same files, same format, same metrics. No format advantage — just parsing quality on the most important Office format.

29 DOCX files, 7 scoring dimensions. AILANG Parse leads by 8.5+ points even on pure DOCX. Scores include aspirational dimensions (bookmarks, footnote text, section breaks) that intentionally lower every tool's score — including ours. Parser versions as of 2026-04-06.

Full Suite: OfficeDocBench (69 Files, 11 Formats, 7 Metrics)

The full suite spans 11 formats with 54 core files plus 15 challenge files. Scores use 7 weighted metrics including ECMA-376 spec-driven dimensions. Coverage-Adjusted = Composite × format coverage — penalizes tools that skip formats.

Coverage-Adjusted = Composite × (files parsed / total files). Raw OOXML scores 84.4% on the 43 files it can parse, but only covers 62% of the benchmark. AILANG Parse and LlamaParse are the only tools with 100% format coverage; AILANG Parse leads on every dimension. Run date: 2026-04-08.

Per-Format Breakdown (AILANG Parse)

DOCX score drops to 90.9% due to aspirational metrics (bookmarks, footnotes, section breaks, hyperlinks). Scores below 100% represent real roadmap targets. Formats with blue borders are not supported by most competitors.

Exclusive Features AILANG Parse Only

These structural features are extracted by AILANG Parse but missed by most or all other parsers tested. They represent real capabilities that matter for document understanding — and they're why the composite gap exists.

Raw OOXML and Pandoc detect some features through XML/AST access, but miss others. No tool detects footnotes, bookmarks, section breaks, hyperlinks, or styles yet — these are aspirational targets for all.

Feature Detection Heatmap (17 Features)

Feature-by-feature detection across all 69 test files. Fractions show files where the feature was correctly detected out of files where it should be present. Features marked ASPIRATIONAL are ECMA-376 spec targets that no tool handles yet.

Aspirational features (purple rows) are ECMA-376 spec-referenced targets that intentionally lower every tool's score. They represent the next frontier — features that should be extracted but currently aren't by any parser. AILANG Parse uniquely detects equations and leads on track changes, comments, headers/footers, text boxes, and lists.

What We Measure: 7 Scoring Dimensions

OfficeDocBench uses 7 weighted scoring dimensions, many driven by ECMA-376 spec references. Each dimension scores [0, 1] and the composite is a weighted average. Aspirational sub-dimensions intentionally score 0 when ground truth data exists but no parser handles the feature yet.

Weights emphasize structural extraction (Structural Recall 20% + Element Count 15% = 35%) since that is the benchmark's primary purpose. Aspirational sub-dimensions within Structural Quality and Content Fidelity intentionally lower scores to create roadmap targets.

Eval-Driven Development

AILANG Parse is not hand-coded and then benchmarked. The benchmarks come first, and AI writes the parsing code against them. This is why the scores are high — and why we want to find documents that lower them.

Help Us Find Our Gaps

We score 93.9% today — intentionally including aspirational metrics that lower our own score. ECMA-376 alone is over 5,000 pages, and we're expanding into email, calendar, and more formats. There are documents in the wild that will break our parser. We want to find them.

If you have a document that AILANG Parse doesn't handle well, send it to us. If it reveals a gap that gets added to our eval corpus, you get 1 month of Business tier free (worth €99).

Your document is used only for eval purposes. We can anonymize content before adding it to the benchmark corpus. Documents that don't reveal new gaps are deleted. Any format welcome — Office, email, calendar, or anything else you throw at us.

PDF Benchmark (OmniDocBench)

For PDFs, AILANG Parse delegates to whatever AI model you plug in via AILANG's AI effect. The benchmark measures the model's accuracy, not our parsing code — and we're transparent about that.

Run the Benchmarks

Every number on this page comes from benchmarks/officedocbench/results/summary.json, regenerated whenever the eval runs. Three ways to reproduce, depending on what you have installed:

# 1. AILANG Parse only — instant, no extra dependencies
uv run benchmarks/officedocbench/eval_officedocbench.py

# 2. Single competitor — requires that adapter installed
uv run benchmarks/officedocbench/eval_officedocbench.py --adapter kreuzberg

# 3. Full leaderboard — all 8 adapters
uv pip install -e '.[competitors]'
uv run benchmarks/officedocbench/eval_officedocbench.py --all

Useful flags: --live re-parses files instead of using cached golden outputs, --format docx filters to one format, --json / --latex change report output.

The --all run writes summary.json (canonical) plus a mirror at docs/data/officedocbench-summary.json. The website reads the mirror at page load and rewrites every score in place — no rebuild step. Adding your own parser? Implement OfficeDocBenchAdapter in adapters/ and register it in eval_officedocbench.py:load_adapter().