WTF is a Software Factory?

This discussion with Adam Jacob, CEO of Swamp Club, explores the concept of software factories, AI agents, and their implications for the future of software development, testing, and organizational structures. **1. Swamp Club and AI Agents:** * Swamp Club is a platform for building AI agents with determinism and structure. It takes user intent and breaks it down into modular automation with typed inputs and outputs, wiring them into workflows, and refactoring them. * It can integrate with arbitrary APIs and internal systems, handling data storage in a shared data lake for querying. * Users apply Swamp Club for diverse tasks, including managing production infrastructure, controlling robot arms in wet labs, and personal automations like email processing. * Examples of personal use include keeping software extensions updated, organizing folder structures, cleaning up old builds, and automating Sentry bug reporting by cataloging issues, completing tasks, and creating PRs. * The key distinction from directly prompting LLMs like Claude is that Swamp Club converts the natural language intent into deterministic, modular automation. This means the automation runs consistently every time, unlike LLMs which may produce varied results and incur token costs for each execution. * Swamp Club allows users to describe tasks in natural language, which it then translates into a runnable script or modular automation. This process avoids burning "Frontier Intelligence tokens" repeatedly. * The system builds a "web of automation" that is inherently modular, self-factoring, and scalable, starting with small agents and growing into larger behaviors. * Swamp Club integrates with existing inference models (users bring their own inference), sitting underneath agents and harnesses to provide its deterministic automation capabilities. * It operates on a paid, per-user-per-month model, though initial experimentation for building automations may not incur immediate costs. **2. Understanding Software Factories:** * Software factories are a broad concept, often misunderstood, with varied interpretations across the industry. * Adam Jacob defines a software factory as a system that automates as much of the Software Development Life Cycle (SDLC) as possible, ultimately handling all aspects except initial inputs, final outputs, and potentially some intermediate human decision points. * This includes automating product definition, work breakdown, team workflow, and ensuring stable, high-quality, architecturally aligned, and non-regressing code. * The goal is to replace manual SDLC processes with agent-driven behaviors, accelerating the journey from intent to outcome. * Software factories are unique to each application, team, and project because the internal architecture and design choices that make a product good are inherently unique. * They are more than just prompting an LLM or a collection of skills; they involve encoding product knowledge and behavior into a structured system. * The inputs to a software factory can vary, from simple queues of tickets to more carefully standardized processes for software production. **3. Trust and Verification in Software Factories:** * Ensuring trust in agent-produced output is crucial, especially when humans are less involved in the intermediate steps. * **Blackbox Acceptance Testing:** This involves testing the product from an external perspective, without internal system knowledge, to confirm that features work as intended. This type of testing, previously deemphasized in continuous delivery due to its manual effort, becomes vital in a factory model to ensure contracts aren't broken by automated changes. * Software factories provide repeatable automation, ensuring consistent behavior regardless of the engineer or product person providing inputs. * The perception of software factories as "slop factories" arises when organizations simply use AI to increase code velocity without changing their SDLC or implementing robust validation. This leads to more, faster production of poorly aligned and understood code, creating a "process paradox." * The solution to this paradox is more AI, more automation, and more structure, shifting towards a systems design problem where the factory itself is designed to produce quality. **4. Organizational Transformation for Software Factories:** * Transitioning large organizations (hundreds or thousands of engineers) to software factories is a significant challenge, not primarily technological but organizational. * The shift breaks existing work processes and points of control. Product people, whose job is to direct engineers, may feel threatened as engineers with trustworthy factories can implement and refactor features rapidly without constant oversight. * Product roles evolve to reviewing outcomes, aligning them to themes, and leveraging the low cost of rework to make changes. * **Strategy for Large Organizations:** * Start with a small, empowered team (e.g., five people) given permission to break all existing organizational rules (org structure, planning, etc.). * This team's sole purpose is to build an end-to-end, trusted software factory for a real problem slice, experiencing life on the "other side." * These individuals then become "translators" who understand both old and new paradigms, guiding the larger organization. * This process, similar to "kaaku" in lean manufacturing, redesigns the factory around new technology that alters system bottlenecks. * It's critical to act quickly ("strike while the iron is hot") and demonstrate working proof in code, avoiding philosophical debates. * While many companies have started this journey, few have fully completed it, indicating it's an ongoing discovery phase for the industry. * The question of what 10,000+ engineers will do once productivity dramatically increases remains open. Historically, automation has always opened up new capacities and transformed jobs rather than eliminating them, suggesting an infinite demand for software creativity. **5. Testing in the Age of AI and Software Factories:** * **Blackbox Testing:** Highly valuable for confirming product functionality without internal system knowledge. It's the most important first step for organizations moving towards software factories, as it makes refactoring and factory building safe. * **Integration Tests:** Still relatively valuable for catching bugs earlier by ensuring different parts of the system work together. AI can help change integration tests to prove internal workings. * **Unit Tests:** Their value is an open question. AI-generated code is often functionally correct. The traditional test-driven development loop (writing tests before code) may not be necessary for AI, and some research suggests it can even hurt outcomes. * Unit tests might still be useful in languages without strong type systems, where they catch issues that type systems (like in TypeScript or Rust) handle effectively. * The industry is in an experimental phase regarding unit tests; some teams may consider deleting them to see the impact on regressions and performance, knowing that AI can regenerate them if needed. **6. Validating Trust Deterministically in a Factory:** * **Blackbox Testing:** (As mentioned, crucial for external validation). * **Software Architecture:** Instead of architecture emerging organically, it's now intentionally designed "from on high." Agents, if left to their own devices, may produce brittle code without a consistent architecture. * **Commit to a Software Architecture:** Define a fixed set of patterns and a rich vocabulary (e.g., Domain-Driven Design) that agents can understand and apply. Agents can mimic architectural patterns provided in "skills." * **Architectural Alignment Review:** All generated code should be reviewed against architectural principles (defined in a skill) unique to the application, language, and domain. * **Adversarial Review for Non-Architectural Requirements:** * Define "skills" for non-functional requirements like accessibility standards (e.g., WCAG AAA compliance). * These skills can be lean, leveraging an agent's existing knowledge of standards, but also include specific organizational requirements (e.g., color contrast rules). * Any change through the system is evaluated against these requirements. If it fails, the factory redoes the work. * Engineers "tweak the factory" whenever a new class of errors is found, rather than blaming the AI. This means improving the machine that builds the product. * **Adaptive Systems:** Use models (like Jev) to determine which tests or adversarial checks are necessary based on the nature of a specific code change, creating a highly adaptive and efficient system. * This approach is about "engineering the system whose job is to build the program," shifting the engineering focus to designing the machine that builds the product. **7. Evolution of Skills vs. God Machine:** * Early in the AI era, there was an expectation that LLMs would get smarter, and simply "throwing more stuff at it" would result in perfect outcomes ("god machine"). * Experience shows this is unreliable, inconsistent, and expensive. * Skills are crucial for applying judgment, taste, or reasoning to specific problems within the larger process. * Intelligence must be seen as a component of a larger system, with defined inputs, outputs, and measurable correctness (e.g., meeting accessibility standards). * Example: Swamp Club achieves a 100% Lighthouse score due to an adversarial skill that prevents it from dropping, demonstrating the power of consistent, automated enforcement of standards. **8. The Future of Software Development:** * The goal of software factories is not just volume but the output of *good* and *great* products. * The current era is a "golden age" for engineers, where collaboration and experimentation are key to discovering new patterns and techniques. * Engineers are not slaves to prompts but active participants in building the machines that build software. * Labs (like OpenAI) may have advanced AI, but they don't understand the intricacies of specific enterprise systems (e.g., LinkedIn, JP Morgan Chase). Engineers within those organizations are the ones who know their systems best and must drive the application of AI. * The capacity unlocked by AI means engineers can build things previously unimaginable, leading to "wild places" and entirely new operating systems or platforms. * The focus is shifting from building better, faster versions of existing software (e.g., a "better Adobe Creative Suite") to inventing entirely new forms of software that couldn't exist before. * This includes highly personalized software, tailored precisely to individual needs, potentially leading to "software getting smaller, not bigger." * Swamp Club provides the primitives and a foundational software factory to accelerate this personalized, intent-driven development. **9. Personal Enjoyments and Plugs:** * **Sick Picks:** * Dell XPS 14 (new model with Panther Lake) running Manjaro Linux. The polished experience is a delight. * Lorna Shore (heavy metal band) – rediscovering their latest album. * **Shameless Plugs:** * Swamp Club (check it out). * Swamp Discord: A community of like-minded individuals building software factories and sharing knowledge, providing immense value and fostering collective learning in this evolving field.