Model Selection & Token Efficiency

This session focuses on token optimization and model selection for efficient AI development, particularly within the Cursor platform. Santi Garza, a senior field engineer at SpaceX AI, explains the fundamental concepts and practical strategies for managing AI costs and improving performance. **Understanding Tokens and Their Cost:** A token is the basic unit of information that AI models process, both as input and output. While often correlating to words, a single word can sometimes be broken down into multiple tokens. Images and large files are also tokenized. The primary concern for developers is that tokens are the basis for billing. Models have different token costs for input (what you send to the model) and output (what the model generates). Output tokens are generally more expensive as they involve generation rather than just processing. For example, Claude Fable costs $10 per million input tokens and $50 per million output tokens. Opus is $5 per million input tokens. Grok 4.6, a model developed in conjunction with SpaceX AI, is significantly more cost-effective at $2 per million input tokens and $2.50 per million output tokens. Composer 2.5 is even cheaper at $0.50 per million input tokens and $2.50 per million output tokens. To mitigate costs associated with rereading context, models employ caching. Input tokens are written to a cache, and the model checks there first before processing them anew. This significantly reduces costs for repeated information. However, changes to rules or switching model providers can invalidate the cache. **The Cursor Harness and Agent Anatomy:** The "harness" in Cursor acts as the car around the AI model's engine. It manages prompts, pieces together necessary context, orchestrates tool calls, and communicates back and forth with the model. This harness is tuned for each specific model, optimizing its performance. Agents, while powerful, are non-deterministic, meaning they don't always produce the same result for the same input. Cursor's harness addresses many of the challenges associated with this non-determinism. Crucially, models have no memory between turns. The harness manages this by feeding context back into the model on each cycle. As the conversation grows, so does the context, potentially hitting the model's context window limit. When the context reaches approximately 90% of its limit, Cursor's harness begins to compact the conversation by summarizing the middle section to free up space. This summarization can lead to a loss of detail. The context served to the model typically consists of three parts: 1. **Prefix:** Essential information that every model turn needs, such as system prompts and rules. This part is never compacted. 2. **Conversation:** The growing history of the dialogue. This is the part that gets summarized during compaction. 3. **Most Recent Ask:** The user's current prompt, which cannot be compacted. **Token Consumption and Model Selection:** A common myth is that every agent action consumes tokens. In reality, agent actions themselves are free. The cost is primarily associated with the tokens that go into and out of the model. Actions like searching codebases or tab completions do not incur token costs, only the model's thinking and communication do. The AI landscape is rapidly evolving, with frontier models changing frequently. Cursor's key benefit is providing access to a wide range of models across the frontier, allowing users to avoid locking into a single provider. Comparing costs, Composer 2.5 can be up to 40x cheaper than Fable for certain tasks. Grok 4.6 is about 6x more expensive than Composer 2.5 but offers high performance. Internal benchmarks show Grok 4.6 performing comparably to Fable but at a significantly lower cost per task. Composer 2.5 is exceptionally cost-effective for discrete software engineering tasks. No single model excels in all categories. Different models have distinct strengths and limitations: * **Grok 4.6:** Sits on the "parao frontier," offering high performance at a competitive price. It's a general reasoning model capable of various tasks, including coding. * **Claude Fable:** Very expensive but excellent for highly complex problems requiring broad context, such as complex debugging or visual work. * **Opus:** A strong performer, particularly good at writing and execution, making it suitable for generating copy, notes, or plans. * **GPT-4.6 Soul:** Found to be effective for planning and reading codebases. * **Composer 2.5:** A highly specialized and efficient model for coding and software engineering tasks. It's fast and cost-effective, often paying for more relevant parameters than general reasoning models. **Controlling AI Spend with Cursor Features:** Cursor offers several knobs to control AI usage and cost: * **Effort Level:** This setting determines how much computational effort the model expends on a task, ranging from low to extra high. Higher effort levels involve more thinking loops, leading to more token spend but potentially more thoughtful results. * **Fast vs. Standard Mode:** "Fast" mode prioritizes getting to the front of the processing queue, reducing latency but significantly increasing cost. It's recommended only for demos or when time is critical. * **Cursor Router:** This feature automatically routes requests to the best model for the job, optimizing for cost, balance, or intelligence. * **Cost:** Prioritizes minimizing token spend. * **Balance (default):** Considers all factors for optimal model selection. * **Intelligence:** Sends more complex tasks to the most capable models. Admins can implement Cursor Router as a company-wide policy, leading to significant cost savings (30-60%) overnight. **Effective Prompting and Planning:** The biggest tip for efficient AI use is to "plan your shot." Avoid vague prompts like "fix the bug." Instead, be specific about the errors, the files involved, and expected outcomes. Vague prompts force the model to spend more tokens trying to understand and navigate the codebase, leading to longer, more expensive interactions and wasted context. Key prompting strategies include: * **Specificity:** Clearly define the task, error, or feature. * **Focused Context:** Use `@mention` to reference specific files or folders, rather than asking the agent to scan the entire codebase. * **Single Task per Turn:** Avoid multitasking in a single prompt; break down complex requests into smaller, manageable tasks. * **Success Criteria:** Define what success looks like (e.g., "all tests passing," "match the spec"). * **Concise Pasting:** When including logs or code snippets, paste only the most relevant parts to avoid bloating the context window. **Advanced Cursor Features and Best Practices:** * **Start New Chats per Task:** Avoid nursing a single chat for extended periods, as this leads to compacted context and limited space for new information. If you need to reference past context, use the `@mention` feature to point to previous chats efficiently. * **Short Rules:** Always-on rules should be concise, as they are loaded into every turn. * **Skills:** These are efficient ways to provide workflows or complex instructions to the agent. The agent only loads the skill's description until it's needed, similar to referencing a book from a shelf. * **MCPs and Sub-Agent Definitions:** Audit and remove unused MCPs. Sub-agent definitions are an advanced topic covered in other workshops. * **Golden Paths:** * **Small Direct Tasks:** Name the scope (file, folder, component) and often a single prompt is sufficient, especially with models like Composer. * **Vague Unfamiliar Areas:** Start in "Ask Mode" to gather context, then move to "Plan Mode." * **Building a Feature:** Pull in context, use "Plan Mode" to create a technical spec, then split into subtasks or use "Multitask Mode." * **Refactoring:** Clearly define success criteria. Test-driven development is recommended, ensuring tests pass before and after the refactor. * **Hard Bugs:** Use "Debug Mode," which employs deterministic testing to reproduce and fix bugs efficiently, saving tokens. Bugbot can also be used to check PRs or local changes for bugs. **Demo and Q&A:** The demo showcases Cursor's interface, including the model picker, effort levels, fast mode, and the Cursor Router. It demonstrates how to use "Ask Mode" to gather context for adding a custom color picker, referencing previous chats, and then moving to "Plan Mode" to create a technical specification. The session highlights the cost difference between using an expensive model like Fable for a vague prompt versus using Composer 2.5 with a well-planned prompt. The speaker also shows how to inspect the context window and audit rules and skills. A practical example illustrates how planning with a general reasoning model like Grok and then building with an efficient model like Composer can lead to substantial cost savings. During the Q&A, the impact of switching models mid-conversation on cache and cost is discussed. While the cache is rebuilt, the impact is generally minimal. The Cursor Router's efficiency in managing model selection and its positive impact on cost are emphasized. The strategy of using a more expensive, general reasoning model for planning and a cost-effective model like Composer for implementation is recommended.