I made the personal assistant of my dreams

The speaker built a device called "Prompt Boy" to avoid excessive phone use, especially for tasks that could lead to distraction. Prompt Boy is a small, handheld device that records audio, transcribes it, and then parses it into actionable tasks. It functions as an "Async Siri" with access to the user's system and coding projects, acting as a coding agent. The project was built in four main parts: 1. **Handheld Device:** * **Hardware:** Includes a battery, microphone, 3D-printed case, and an ESP32S3 board with a 1.64-inch touchscreen. * **Features:** Built-in Wi-Fi, gyroscope, accelerometer, SD card slot for storage, and USB Type-C. * **Functionality:** Records audio in small chunks, saves to the SD card, and uploads to a Cloudflare R2 bucket upon completion. It does not perform transcription on the device itself. * **Case Design:** Initially struggled with case design, but used a tool called "NERB" (NERB.dev) to generate a functional case based on component dimensions. The case was then sanded extensively for a smooth finish. 2. **Cloudflare Infrastructure:** * **R2 Bucket:** Hosts the audio files from the device. * **Durable Objects:** Stores to-do data. * **Cloudflare Workers:** Hosts the web app and connects to durable objects. This ensures the app and data are cloud-based and accessible. 3. **Web App:** * **Technology:** Built with SvelteKit and runs on Cloudflare Workers. * **Purpose:** Functions as a to-do app that can initiate tasks on other systems. It provides a control interface for leaving to-dos, starting tasks, or providing feedback. 4. **Agent Runner (MacRunner):** * **Functionality:** Handles audio transcription, AI calls, and agent actions. * **Process:** Receives audio chunks from the R2 bucket, transcribes them locally on a Mac Mini (to avoid paying for cloud compute), and sends the transcript back to the app. * **Task Routing:** Uses a deterministic routing system to classify tasks. For example, keywords like "research" or "to-do" direct tasks to specific lanes. Mentions of project names trigger coding work. * **AI Integration:** Leverages "Swamp Club" for deterministic automations. If a task requires action, it runs pre-defined scripts rather than the AI deciding on tools or writing scripts on the fly. This includes actions like investigating bugs on Sentry or checking a Minecraft server. * **Execution:** For coding tasks, it sends information to "Pi," a local AI agent on the computer. This setup is considered somewhat insecure but is sandboxed by limiting the computer's access to accounts. The system allows for tasks to be classified as simple to-dos, coding work, research, or controlled actions. The use of Pi and Swamp Club enables AI to work on code, run repairs, and troubleshoot system issues through deterministic scripts. The speaker chose Pi over a more comprehensive system like Hermes for more control and to integrate with their existing coding workflow. **Areas for Improvement:** * **Case:** Magnets holding the case together are not strong enough. * **Firmware:** The device's interface and firmware require further development. * **Wi-Fi Connectivity:** Investigating Bluetooth integration for easier network setup, similar to smartwatches, to improve Wi-Fi reliability when on the go. The core motivation for Prompt Boy is to reduce phone usage by providing a single-purpose device for tasks, preventing the distraction associated with opening a smartphone. The speaker is proud of the project, which combines hardware and software development, and plans to continue working on it as their primary tool for to-dos and agent interaction.