A Deep Dive on LLM Evaluation

This talk provides an opinionated summary of challenges and best practices in evaluating Large Language Models (LLMs), drawing from research experience at Al Luther AI, a nonprofit research lab known for open-source models like GPT-J and GPT-NeoX. The speaker, a maintainer of the LM Evaluation Harness library, highlights two primary difficulties in LLM evaluation: scoring and reproducibility. Scoring is challenging because reliably assessing the correctness of natural language responses is complex. A simple keyword match is insufficient, as demonstrated by the example of answering "Who is the current US president?". A response stating "It's definitely not Joe Biden" contains the correct name but conveys the opposite meaning. This fundamental problem underscores the need for robust evaluation methods beyond human eyeballing, which is costly and inconsistent. The talk then delves into common evaluation methods. The first is **log likelihoods**. LLMs output logits, which can be converted into probability distributions over a vocabulary of possible next tokens. This distribution is the basis for generating text, where a token is sampled and then fed back into the model to predict the subsequent token. Crucially, autoregressive LLMs can compute these probabilities for all tokens in a sequence in parallel during training and inference due to the causal masking in their attention mechanisms. This parallel processing allows for efficient calculation of the probability of an entire output string *y* given an input *x* by concatenating *x* and *y* and passing them through the model once. This log likelihood calculation is particularly useful for **multiple-choice question answering**. For a question *x* with potential answers A, B, C, and D, the model's probability of generating each answer can be calculated. The answer with the highest probability is considered the model's predicted answer. This method is cheaper than free-form generation, avoids parsing failures, and is suitable for smaller models. However, it deviates from real-world use cases that require open-ended generation and can be misleading if the model is evaluated on tasks it wasn't designed for. Another common method is **perplexity**, which measures how well a model fits a given data distribution. It's calculated by averaging the log probability the model assigns to the true next token across all tokens in a dataset. This is a straightforward self-supervised task, often using a validation set during training. While useful for assessing how well a model fits a specific data distribution (e.g., Wikipedia), perplexity can be misleading for instruction-tuned models or chatbots whose outputs might not align with the style or format of the perplexity dataset. A pitfall for both log likelihoods and perplexity is the dependence on the **tokenizer**. Different tokenizers can lead to non-comparable scores, necessitating normalization strategies. The third evaluation method is **generating text** from the model. This is essential for applications like chatbots. However, it reintroduces the challenge of scoring free-form natural language. Simple heuristic-based scoring, like looking for specific phrases, can penalize models that don't adhere to expected formats. Using another LLM as a judge is an option, but these judges are also fallible. A significant practical issue highlighted is the sensitivity of generation-based evaluation to subtle prompt details, such as trailing whitespace. The example of a code generation model showing a 5% performance drop due to a single newline and tab at the end of a prompt illustrates how minuscule implementation details can drastically alter evaluation results and conclusions. The difficulty in scoring and the sensitivity to implementation details make **reproducibility** a major challenge. Reproducibility, defined as the ability to obtain the same results by following published details, is crucial for research and development. Without shared evaluation code, it's hard to ensure fair comparisons, as advantages might stem from extensive prompt engineering rather than inherent model capability. The speaker argues that sharing evaluation code is paramount for reproducibility. Libraries like the LM Evaluation Harness, HELM, and OpenCompass aim to provide standardized, well-vetted code bases for evaluation, abstracting away intricate details like tokenization and normalization, allowing researchers to focus on the evaluation design itself. The LM Evaluation Harness, in particular, offers flexibility for practitioners to define their own tasks or modify prompts. For **production environments**, a distinction is made between "model evals" (general capability benchmarks like MMLU) and "downstream evals" (task-specific evaluations for concrete use cases, like a chatbot for company documentation). Downstream evaluations are generally preferred when a specific use case exists. Ideally, one should run the evaluation code themselves rather than relying solely on reported results. Furthermore, for subjective qualities, hiring human evaluators might be necessary. In downstream evaluations, overfitting to the evaluation metric can be beneficial if it directly translates to better performance on the desired task, unlike in general model evaluations where it signifies a flawed benchmark. High-level takeaways emphasize that **implementation details matter significantly**. Finicky prompt details, formatting, and measurement normalization can skew numerical scores and lead to incorrect conclusions about model superiority. This impacts both research (fair comparisons) and production (unexpected performance drops due to bugs). Beyond these issues, other challenges include data contamination, overfitting, evaluation saturation, measurement validity, and the suitability of the evaluation for the intended use case. The talk concludes by referencing a paper, "Lessons From The Trenches on Reproducible Evaluation of Language Models," and promoting the LM Evaluation Harness library, which now supports chat templating and is open to contributions. The speaker encourages the audience to check out the library's examples, the Open LLM Leaderboard, and to continue the discussion. Regarding specific questions from the audience: * **Benchmark errors:** Errors in benchmark datasets (like Hellaswag) are significant, especially when striving for high accuracy, as they can lead to overfitting and mask true model capability improvements. * **Parallel probability calculation:** The parallel processing capability of transformer architectures allows LLMs to compute probabilities for all tokens in a sequence in a single pass, enabling efficient calculation of string probabilities. * **LLM as a judge:** While LLMs can be valuable judges, their limitations and failure modes need careful consideration. They are generally better at simpler tasks like extracting multiple-choice answers than at complex, subjective reasoning. * **ARC benchmark difficulty:** ARC is harder because it focuses more on generalization and multi-hop reasoning, requiring more scaffolding than current benchmarks, which often test memorization. * **Reliable multiple-choice answers:** Structured generation tools or, for highly capable models, strict prompting can help produce reliable multiple-choice answers without extra fluff. However, without these, it can be tricky, as many models are trained to produce verbose outputs.