How to Measure OCR Quality

The speaker, Vic, founder of Data Lab, discusses the challenges of evaluating AI model quality, particularly in the context of Optical Character Recognition (OCR) and document parsing. He highlights that when many models perform well on benchmarks, it becomes difficult to discern true quality. Data Lab has developed several state-of-the-art projects in OCR and document processing. **Key Issues with Current Benchmarking:** * **Benchmark Gaming:** Models can be optimized to perform well on specific benchmark formats and expectations, rather than improving underlying quality. This is exemplified by ParsBench, where a model improved its ranking through reinforcement learning without actual OCR gains. * **Flawed Metrics:** * **Edit Distance:** Over-penalizes minor formatting differences (e.g., HTML vs. Markdown) while under-penalizing actual content errors (e.g., incorrect digits in a number). * **Layout Metrics:** Highly sensitive to the specific definition of blocks (e.g., paragraphs) used by the benchmark author, leading to unfair penalization if a model's segmentation differs. * **Table Metrics:** Often poorly implemented, leading to inaccurate scoring of table data. * **LLM Judges:** Prone to errors, inconsistencies, and non-determinism. For instance, GPT-4.1 has been observed to incorrectly mark "not found" as correct when explicitly instructed otherwise. * **Ambiguous Ground Truth:** Benchmarks sometimes contain errors or ambiguities in their ground truth data, making it impossible for even good models to pass. Examples include incorrect values in charts or case-sensitive tests that don't match the actual text. * **Bias:** Benchmarks can be biased towards the creators, especially if their own models were used to generate ground truth data. This can lead to models performing significantly better on specific benchmark datasets without a genuine improvement in core capabilities. * **Saturation:** Benchmarks can become saturated, meaning that new models achieve higher scores through increasingly complex post-processing rather than fundamental improvements. Olmo CR is cited as an example where the benchmark started measuring how well models could "game" the system rather than actual quality. * **Lack of Transparency:** Many benchmarks are rushed and may contain scoring bugs, making it difficult to trust the results. **The Problem with Specific Benchmarks:** * **ParsBench (Llama Index):** Criticized for testing chart interpretation where values are not explicitly present for OCR, and for having incorrect ground truth values. It also heavily penalizes models not trained for chart analysis. * **ExtractBench (Llama Index):** Showed significant ranking improvements after fixing harness and scoring errors, highlighting potential inaccuracies in initial results. * **Olmo CR Bench (Allen AI):** While pioneering, it has become saturated. Models achieve higher scores through extensive post-processing rather than improving raw OCR quality. Some tests have ambiguous or incorrect ground truth, leading good models to fail. * **OmniDocBench:** Uses edit distance extensively, which is problematic for evaluating document parsing quality. **Criteria for a Good Metric:** 1. **Valid:** Correct outputs must pass, and incorrect outputs must fail. 2. **Specific:** Failures should clearly indicate why an output is wrong and how to fix it. Simpler metrics are preferred. 3. **Sourced:** The ground truth must be verifiable, ideally through human review, direct extraction from the source document, or authoring by reliable means, not by using LLMs to generate it. 4. **Discriminating:** The metric should differentiate between models, with some passing and some failing, rather than being universally passed or failed. **The "Almost OCR Style" Unit Test Approach:** The speaker advocates for a unit test methodology, pioneered by Allen AI's Almost OCR benchmark, as the most effective way to evaluate. This involves creating discrete, verifiable tests such as: * "Inside this page, I want to find this string." * "Inside this page, I want this string to come before the other string." * For tables: "Is this cell under this header?" or "Is this cell near this other cell?" These tests provide a clear pass/fail outcome. However, even this approach requires careful handling of ambiguities in ground truth. **How to Build Better Benchmarks and Evals:** 1. **Define Your Use Case:** Understand the specific requirements of your pipeline (e.g., RAG, extraction, accessibility). Different pipelines need different metrics. 2. **Segment Benchmarks:** Analyze performance on subsets of tests relevant to your use case. For example, if you need to extract numbers, focus on tests that measure numerical accuracy. 3. **Check Metric Validity:** * **Format Invariance:** Ensure the metric is not sensitive to format changes (e.g., Markdown vs. HTML). * **Corruption Tests:** Test if the metric correctly fails when a known correct output is corrupted (e.g., all lowercase, digits changed). * **Shared Failures:** Identify tests where all strong models fail, as these are likely problematic tests. 4. **Look for Bias:** Analyze if certain models show unusual uplifts on specific categories, which might indicate bias in the benchmark design or ground truth annotation. 5. **Source Ground Truth Carefully:** Use human verification, direct extraction from PDFs, publisher-provided ground truth, or well-vetted synthetic data. Avoid relying solely on LLM-generated ground truth. 6. **Be Format-Invariant:** Normalize outputs to ensure fair comparison across different formats (e.g., HTML vs. Markdown, different math notations). Compare equations by rendering, not string matching. Normalize numbers to handle variations in formatting (e.g., commas). 7. **Standardize Evaluation Conditions:** Use consistent prompts, timeouts, and evaluation rules for all models. 8. **Allow for Slicing:** Enable users to filter results by language, document type, or other relevant categories to focus on their specific needs. 9. **Inspect Failure Cases:** Thoroughly analyze why tests fail to identify issues with the benchmark itself or the models being tested. This is a crucial step for learning and improving. 10. **Human Verification is Critical:** Do not skip manual review of evaluation results, especially when dealing with complex documents or critical applications. **Practical Example: Choosing a Parser for RAG over Annual Filings:** * **Objective:** Select a parser that ensures high numerical accuracy for a RAG system processing annual filings. * **Process:** * Identify and sample documents representative of production use, covering edge cases. * Write verifiable questions with known, correct answers (document QA). * Identify potential failure modes (e.g., wrong numbers, table inaccuracies, ordering issues) that are critical for the use case. * Build unit tests based on these failure modes, ensuring they pass for correct outputs and fail for incorrect ones. * Normalize outputs to remove irrelevant formatting differences. * **Outcome:** Initial tests showed similar performance across parsers. To differentiate, harder edge cases (scanned, handwritten documents) were sampled, leading to more distinct results. A specific unit test example involved verifying the sum of revenue freight carried by checking if the sum of two constituent values matched the total. * **Analysis of Failures:** Examining failures revealed issues like headers split from values, characters in the wrong script, and incorrect number readings, which informed further benchmark refinement and model selection. **OmniParseBench:** Data Lab's benchmark is designed to be flexible, with open scorers and test cases, allowing users to apply the scoring engine to their own documents. **The Incentive Problem in Benchmarking:** There's a strong commercial incentive for companies to achieve top rankings on benchmarks, leading to a focus on "marketing wins" and "gaming" the system rather than genuine innovation. This erodes trust in the benchmarking process. The speaker suggests that the industry needs to rebuild trust through more robust, independent, and transparent evaluation methods. Publicly released benchmarks are often of lower quality because releasing comprehensive data could allow competitors to easily "hill climb" on them. Labs typically maintain extensive private evaluation sets. **Beyond OCR:** The speaker emphasizes that the issues with benchmarking extend beyond OCR to other AI model domains, including coding. He personally relies more on internal, private benchmarks for non-OCR tasks to assess model effectiveness for specific applications. These internal evals follow a similar process of defining ground truth, metrics, and running tests. **Conclusion:** The current state of AI benchmarking is problematic due to gaming, flawed metrics, ambiguous ground truth, and inherent biases. While challenging, building custom, well-sourced, and rigorously defined evaluation sets using a unit-test methodology is the most reliable way to assess model quality for specific use cases. Transparency and a focus on driving genuine innovation are crucial for rebuilding trust in the benchmarking ecosystem.