AI Code Generation and Editing

The AI Code Generation and Editing system represents the intelligent core of Onlook, encompassing not only the sophisticated natural language to code translation engine, but also the comprehensive developer interface and tooling that makes AI-powered development seamless and professional. This system combines advanced AI capabilities with intuitive user interfaces and robust development tools to create a complete coding environment.

Product Specifications

User Experience Flow

The AI-powered development experience centers around the right panel interface, which provides both conversational AI interaction and direct code access. Users engage with the AI through natural language while having full visibility and control over the generated code, creating a transparent and educational development process.

Functional Requirements

Feature	End-to-End Business Logic and Rules
AI Persona and Core Rules	User Intent: To interact with an AI that behaves as a competent, expert-level software engineer. End-to-End Flow: 1. System - Persona Definition: The AI’s core identity is defined in `packages/ai/src/prompt/system.ts`. The `SYSTEM_PROMPT` instructs the AI to act as an “expert React, Next.js and Tailwind design-engineer.” 2. System - Rule Enforcement: The prompt enforces strict operational rules: refactor for maintainability, adhere to existing codebase conventions, provide concise expert-level answers, and ask clarifying questions when requests are ambiguous. 3. System - Tool Empowerment: The AI is explicitly granted a suite of tools to act as an autonomous agent: `grep` and search for codebase exploration, a web scraper for URL content, web search for current information, a `typecheck` tool for code validation, and a `terminal` tool for executing commands. User Display: The user experiences a professional and capable AI partner that provides high-quality code, asks intelligent questions, and autonomously performs development tasks. Key Rules & Constraints: - The AI’s persona is the foundational context for every interaction, ensuring consistent, high-quality output. - The system explicitly grants and lists the tools the AI is permitted to use.
Initial Page Creation	User Intent: To create a new webpage from a high-level, descriptive prompt on a blank canvas. End-to-End Flow: 1. Frontend - User Action: On a blank project, the user submits their first creative prompt (e.g., “create a playful landing page for a SaaS product”). 2. Backend - Specialized Prompt Invocation: The backend identifies this as an initial creation task and prepends the `CREATE_NEW_PAGE_SYSTEM_PROMPT` (from `packages/ai/src/prompt/create.ts`) to the main system prompt. 3. Backend - High-Level Goal Interpretation: This specialized prompt instructs the AI that the user’s input is a “blank slate” task. It must infer a distinct design style from the prompt’s language and generate the full code for the initial page. 4. Backend - Full-File Code Generation: The AI generates the complete code for the `page.tsx` file, including React components, JSX structure, and Tailwind CSS classes that align with the inferred visual style. 5. Backend - File System Integration: The generated code is written to the project’s virtual file system. 6. Frontend - Display: The new code is sent to the client, and the Real-time Preview renders the newly created page. Key Rules & Constraints: - This flow is triggered only for the first prompt in a project, treating it as a conceptual task to be turned into a complete webpage.
Context-Aware Editing	User Intent: To make a precise modification to a specific UI element using natural language. End-to-End Flow: 1. Frontend - User Action: The user selects a UI element in the preview and submits a modification command (e.g., “make this button blue”). 2. Frontend - Context Gathering: The client gathers multiple, distinct context types: the user’s raw text, the full content of relevant files, specific code snippets for selected elements (`highlight`), project-level information (`project`), error messages (`error`), and user-provided images (`image`). 3. Backend - Prompt Hydration: The backend receives this context. The `getHydratedUserMessage` function in `packages/ai/src/prompt/provider.ts` systematically wraps each piece of information in a specific XML tag to create a single, highly-structured prompt. For example: - `<instruction>`: Contains the user’s raw text. - `<context>`: Wraps a collection of file contexts. - `<file>`: Contains the code, path (`<path>`), and branch (`<branch>`) for a specific file. - `<highlight>`: Contains a specific code snippet the user selected. - `<errors>`: Wraps reported error messages. 4. Backend - LLM Processing & Structured Output: The hydrated prompt is sent to the LLM. The LLM returns a structured `CodeDiffRequest` object, which precisely defines the required change with properties like `oid` (the target element’s ID), `attributes`, and `textContent`. 5. Backend - Safe Code Application via AST: This structured object is passed to the code parser (`packages/parser`), which applies the change safely to the code’s Abstract Syntax Tree (AST). 6. Frontend - Feedback Loop: The modified code, regenerated from the AST, is sent back to the frontend, and the live preview updates. Key Rules & Constraints: - The prompt hydration process is systematic and relies on a well-defined XML-like structure to provide clear, unambiguous context to the LLM. Each context type is handled and tagged separately.
Code Manipulation via AST	User Intent: To have code modifications applied in a way that is safe, accurate, and structurally sound, ensuring that AI-driven changes do not break the page or application. End-to-End Flow: 1. Backend - Transforming Code into a Structural Blueprint: To operate on code safely, the system first parses the entire code file from its plain text format into a machine-readable “structural blueprint” (an AST). This blueprint precisely maps out every element on the page (e.g., buttons, text fields), its properties (e.g., color, size), and its hierarchical relationship to other elements (e.g., which element contains which). 2. Backend - Pinpointing the Target Element: When a user wants to modify a specific element, the system uses a unique internal ID attached to each element to locate it precisely. The modification instruction issued by the AI explicitly specifies which ID to operate on, eliminating any ambiguity. 3. Backend - Executing Standardized Modification Commands: Instead of returning raw code, the AI generates and sends one or more structured commands. The system possesses a predefined, standardized set of capabilities, and each command tells the system exactly what action to perform on the target element. Examples include: - Styling Operations: “Replace the class attribute of the target element with a new style” or “Append a new style to the target element.” - Content Operations: “Update the text content of the target element.” - Structural Operations: “Insert a new element inside the target element,” “Move the target element next to another element,” or “Delete the target element.” 4. Backend - Regenerating Code from the Blueprint: After all modifications have been safely applied to the structural blueprint, the system uses this updated blueprint to regenerate a clean, well-formatted, and syntactically correct code file. Key Rules & Constraints: - This blueprint-first modification strategy is the cornerstone of the system’s reliability. It fundamentally ensures that any AI-generated code modification is syntactically valid and structurally whole. Direct text replacement is strictly forbidden. - All page elements must have a unique internal ID, which is a prerequisite for enabling precise and unambiguous operations.
AI-Powered Design Suggestions	User Intent: To receive proactive, expert guidance on the next logical steps in the design and development process, especially when unsure what to do next. End-to-End Flow: 1. System - Suggestion Mode Trigger: The system activates a “suggestion mode” either when the user explicitly asks for ideas or based on internal triggers (e.g., detecting user inactivity). 2. Backend - Shifting AI Persona: In this mode, the system instructs the AI to shift its persona from a direct code implementer to an expert web designer and UX specialist. Its primary goal is no longer to write code, but to provide insightful recommendations. 3. Backend - Holistic Project Analysis: The AI is directed to perform a holistic analysis of the project. It must evaluate the conversation history to understand the user’s original goals and review the current state of the webpage to identify what has been built and what is logically missing or could be improved. 4. Backend - Formulating High-Quality Suggestions: The AI must generate suggestions that are specific, actionable, and feel like a natural next step. The core criteria for its suggestions include enhancing the user experience, aligning with the user’s vision, considering mobile responsiveness, and prioritizing high-impact improvements. 5. Frontend - Presenting Choices: The generated suggestions are presented to the user in the interface as clear, clickable options. 6. Frontend - User Action: The user can choose to accept one of the suggestions, which then seamlessly transitions into a new instruction for the AI code generation module to execute. Key Rules & Constraints: - This feature transforms the AI from a passive tool that only follows commands into an active, collaborative partner that helps guide the creative process.
Autonomous Environment Management	User Intent: To have the AI assistant manage project dependencies and other environment-related tasks automatically, creating a seamless development experience. End-to-End Flow: 1. Backend - Task Identification: During its reasoning process, the AI identifies a situation where a command-line action is necessary. The most common scenario is when its generated code requires a new software library (dependency) that is not yet installed. 2. Backend - Command Formulation: The AI, acting as an autonomous agent, formulates the appropriate command to resolve the situation (e.g., it formulates the command to install the missing library). 3. Backend - Automated Execution: The system safely executes the formulated command within the project’s environment on the user’s behalf. This happens automatically, without requiring the user to manually open a terminal and type the command. 4. Frontend - User Notification: The user is informed via a message in the chat interface that the action has been taken (e.g., “I’ve installed the ‘framer-motion’ package for you.”). Key Rules & Constraints: - This capability enhances the AI’s role as a capable agent, allowing it to autonomously manage the development environment. - The system must adhere to the project’s specific technical environment, such as exclusively using the `bun` package manager instead of `npm`. - To prevent unintended side effects, the AI is explicitly forbidden from executing commands that run or restart the main application (e.g., `bun run dev`).
Conversation Context Optimization	User Intent: To maintain a fast, responsive, and cost-effective interaction with the AI, even during long and complex design conversations. End-to-End Flow: 1. System - Prioritizing the Current Task: When a user sends a new request, the system treats it as the highest priority. To ensure the AI has the most complete information to work with, the system attaches the full source code of all relevant files to this latest request. 2. System - Intelligently Summarizing History: To keep the overall conversation efficient and concise, the system applies an optimization rule to all previous messages in the chat log. For these older messages, it automatically strips out the bulky source code content, leaving only lightweight file path references to indicate which files were being discussed at that point in time. 3. Backend - Proactive Cost and Performance Check: Before sending the final assembled prompt (containing the full-context latest message and the summarized history) to the AI, the system pre-calculates its total size to ensure it fits within the AI’s processing limits and helps control costs. 4. Backend - Efficient AI Processing: The AI receives this optimized prompt, allowing it to focus on the fully-detailed current request while still having a lightweight historical record of the conversation for background context. Key Rules & Constraints: - This dynamic context strategy is core to the application’s performance and economic viability, balancing the need for accuracy on the current task with the cost of transmitting historical data. - The full code context is only provided for the most recent user message. All other messages in the conversation history must have their file content removed. - The system uses an algorithm compatible with OpenAI’s GPT models to accurately calculate the final size of every request.
Right Panel Interface System	User Intent: To have a comprehensive, organized interface that provides seamless access to both AI conversation and direct code editing capabilities within a single, cohesive workspace. End-to-End Flow: 1. Frontend - Dual-Tab Architecture: The right panel features two primary tabs: “Chat” for AI interaction and “Code” for direct code editing. Users can switch between these modes fluidly, maintaining context and workflow continuity. 2. System - Intelligent Tab Coordination: When AI generates or modifies code, the system automatically updates the Code tab to reflect changes, allowing users to review, understand, and manually refine the AI’s work before applying it. 3. Frontend - Contextual Controls: Each tab provides contextual controls appropriate to its function. The Chat tab includes conversation management, suggestion controls, and chat history access, while the Code tab offers syntax highlighting, direct editing capabilities, and code validation. 4. System - Seamless Workflow Integration: The panel integrates seamlessly with the overall editor workflow, maintaining synchronization with the main canvas, left panel selections, and project state changes. Key Rules & Constraints: - The interface must provide clear visual distinction between AI-generated and user-modified content. - Tab switching must be instant and preserve the current state of both AI conversation and code editing. - All panel interactions must maintain consistency with the overall editor’s visual and interaction design language.
AI Chat Interface and Conversation Management	User Intent: To engage in natural, productive conversations with an AI assistant that understands development context and provides helpful, accurate assistance. End-to-End Flow: 1. Frontend - Conversational Interface: The chat interface provides a familiar messaging experience with user messages, AI responses, typing indicators, and conversation threading that makes complex development discussions easy to follow. 2. System - Conversation History Management: The system maintains comprehensive conversation history with intelligent organization, search capabilities, and the ability to reference previous discussions or decisions made during the development process. 3. Backend - Context-Aware Response Generation: Each AI response is generated with full awareness of the current project state, selected elements, recent changes, and conversation history, ensuring responses are relevant and actionable. 4. Frontend - Interactive Message Features: AI responses can include interactive elements like code snippets that can be directly applied, suggestions that can be accepted with one click, and links to relevant documentation or resources. Key Rules & Constraints: - All AI responses must be clearly attributed and timestamped for accountability and reference. - The conversation interface must handle both short, quick interactions and long, complex development discussions effectively. - Message threading and organization must scale to support extensive development sessions without becoming cluttered.
Integrated Code Editor and Review System	User Intent: To have direct access to view, understand, and modify the code that the AI generates, ensuring full transparency and control over the development process. End-to-End Flow: 1. Frontend - Professional Code Editing: The Code tab provides a full-featured code editor with syntax highlighting, auto-completion, error detection, and all the features developers expect from modern code editing tools. 2. System - Real-time Code Synchronization: Any changes made in the code editor are immediately reflected in the live preview, and any AI-generated changes are instantly visible in the editor, maintaining perfect synchronization between all views. 3. Frontend - Code Review and Validation: The editor provides clear indicators for AI-generated changes, allows users to review modifications before applying them, and includes validation tools to ensure code quality and correctness. 4. System - Version Control Integration: All code changes, whether AI-generated or user-modified, are properly tracked and integrated with the project’s version control system, maintaining a clear history of development progress. Key Rules & Constraints: - Users must always have the ability to review and approve AI-generated changes before they are applied to the project. - The code editor must provide the same level of functionality and user experience as standalone code editors. - All code modifications must maintain proper formatting, syntax validity, and adherence to project coding standards.
Development Tools and Environment Management	User Intent: To have comprehensive development tools that handle project dependencies, error resolution, and environment management automatically, creating a seamless development experience. End-to-End Flow: 1. System - Automated Dependency Management: When the AI generates code that requires new packages or libraries, the system automatically detects these dependencies and installs them using the appropriate package manager (Bun), without requiring manual intervention. 2. Backend - Intelligent Error Detection and Resolution: The system continuously monitors the development environment for errors, build issues, and compatibility problems, providing intelligent suggestions and often resolving issues automatically. 3. Frontend - Comprehensive Error Reporting: When issues occur, the system provides clear, actionable error messages with specific guidance on resolution, including direct links to problematic code locations and suggested fixes. 4. System - Environment Health Monitoring: The development environment is continuously monitored for performance, stability, and proper configuration, with proactive notifications and automatic optimization when possible. Key Rules & Constraints: - All automated actions must be clearly communicated to the user with explanations of what was done and why. - The system must provide fallback options when automatic resolution is not possible. - Environment management must be robust enough to handle complex project configurations and dependencies.
Advanced Developer Assistance and Tooling	User Intent: To access professional-grade development tools including terminal access, debugging capabilities, and advanced project management features directly within the AI-powered environment. End-to-End Flow: 1. System - Integrated Terminal Access: The AI assistant has access to terminal commands and can execute development tasks like running tests, building projects, or managing git operations on behalf of the user when requested. 2. Backend - Code Quality and Analysis: The system provides continuous code quality analysis, including type checking, linting, performance optimization suggestions, and security vulnerability detection. 3. Frontend - Debugging and Inspection Tools: Users have access to debugging tools, console output, network inspection, and other development utilities that help them understand and troubleshoot their applications. 4. System - Project Optimization and Suggestions: The AI analyzes the overall project structure and provides suggestions for improvements in architecture, performance, accessibility, and best practices. Key Rules & Constraints: - Terminal access must be secure and limited to safe, project-related commands. - All development tools must integrate seamlessly with the AI assistant’s capabilities. - Advanced features must be accessible to both novice and expert developers through appropriate interface design.

Technical Solution

Natural Language Technical Solution

The AI Code Generation system operates through a sophisticated multi-layered architecture that seamlessly integrates conversational AI, code intelligence, and development tooling within a unified interface.

The Right Panel Interface serves as the primary interaction hub, featuring a dual-tab system that provides both AI conversation capabilities and direct code editing access. The Chat tab enables natural language interaction with the AI assistant, while the Code tab provides professional-grade code editing with syntax highlighting, validation, and real-time synchronization.

When a user submits a request through the chat interface, the AI Service performs comprehensive context gathering, collecting information about selected elements, current file contents, project structure, and conversation history. This context is then systematically organized using a structured XML-based format that provides the AI with rich, unambiguous information about the user’s intent and current project state.

The Prompt Hydration System enriches the user’s simple request with extensive context, including the AI’s expert persona definition, project-specific information, error states, and relevant code snippets. This hydrated prompt is then sent to advanced Large Language Models through the Vercel AI SDK, with support for multiple providers including Anthropic’s Claude and OpenRouter.

The AI responds with structured modification instructions rather than raw code, which are then processed by the Code Parser Service using Babel’s Abstract Syntax Tree capabilities. This ensures all modifications are syntactically valid and structurally sound, maintaining code quality and preventing errors.

The system also includes comprehensive Development Tools that provide automated dependency management, intelligent error detection and resolution, terminal command execution, and continuous environment monitoring, creating a complete development ecosystem within the AI-powered interface.

End-to-end Execution Sequence Diagram

AI Replication & Reference

AI Code Generation Replication Guide

Prompt

You are an expert AI engineer and full-stack developer. Your task is to create a comprehensive AI-powered development environment that combines conversational AI, code generation, and professional development tools in a unified interface.
 
**Core Feature: Complete AI Development Environment**
 
**1. Right Panel Interface System:**
- Build a tabbed interface with "Chat" and "Code" tabs that work seamlessly together
- Implement real-time synchronization between AI conversation and code editing
- Create contextual controls for each tab (conversation management, code editing tools)
- Support resizable panels with intelligent width management
 
**2. Advanced Prompt Construction:**
- Create a sophisticated context gathering system that collects multiple data types:
  - User instructions, file contents, selected element context, project information, error states, and images
- Implement XML-based context wrapping with specific tags for each data type:
  - `<instruction>`, `<context>`, `<file>`, `<highlight>`, `<errors>`, `<project-info>`
- Build intelligent context truncation for long conversations to manage costs and performance
 
**3. Multi-Modal AI System:**
- Implement multiple AI personas: code generator, design consultant, UX specialist
- Support different prompt strategies for different tasks (creation vs. editing vs. suggestions)
- Integrate with multiple LLM providers (Anthropic, OpenRouter, Morph, Relace)
- Build conversation history management with smart context optimization
 
**4. Professional Code Editor Integration:**
- Create a full-featured code editor with syntax highlighting, auto-completion, and error detection
- Implement real-time code synchronization between AI generation and manual editing
- Build code review capabilities that highlight AI changes and allow user modifications
- Support version control integration for tracking all changes
 
**5. Development Tools and Environment Management:**
- Build automated dependency detection and installation using Bun package manager
- Implement intelligent error detection with actionable resolution suggestions
- Create terminal command execution capabilities for the AI assistant
- Build environment health monitoring and optimization systems
 
**6. AST-Based Code Transformation:**
- Use `@babel/parser` for parsing code into Abstract Syntax Trees
- Implement comprehensive transformation capabilities for all code modification types:
  - Style operations (class updates, property changes)
  - Content operations (text updates, attribute modifications)
  - Structural operations (insert, move, remove, group, ungroup)
  - Media operations (image insertion and management)
- Use `@babel/generator` for converting modified ASTs back to clean code
 
**7. Conversation and Suggestion Systems:**
- Build intelligent suggestion generation based on project analysis
- Implement conversation threading and history management
- Create interactive message features (clickable suggestions, code application)
- Support multi-turn conversations with context preservation
 
**Technical Requirements:**
- **Frontend Framework:** Next.js with React and TypeScript
- **State Management:** MobX for reactive state management
- **AI Integration:** Vercel AI SDK with multiple provider support
- **Code Processing:** Babel for AST manipulation and code transformation
- **Communication:** tRPC for type-safe API communication
- **Package Management:** Bun for dependency management and script execution
 
**Acceptance Criteria:**
1. Users can switch seamlessly between Chat and Code tabs while maintaining context
2. AI generates structured modification instructions that are applied via AST manipulation
3. The system handles multiple context types and conversation optimization automatically
4. Code editor provides professional development capabilities with real-time synchronization
5. Development tools automatically manage dependencies, errors, and environment health
6. All AI interactions are transparent, reviewable, and user-controllable
7. The system supports both novice and expert developers through adaptive interface design

Functional Role	Code File	Description
AI System Prompt	`packages/ai/src/prompt/system.ts`	Defines the master set of instructions, rules, and persona for the AI assistant.
Prompt Construction	`packages/ai/src/prompt/provider.ts`	Contains the logic for building context-rich prompts, wrapping user instructions and code in XML tags.
Specialized AI Prompts	`packages/ai/src/prompt/create.ts`, `packages/ai/src/prompt/suggest.ts`, `packages/ai/src/prompt/shell.ts`	Specialized prompts for different AI modes: page creation, design suggestions, and shell command execution.
Right Panel Interface	`apps/web/client/src/app/project/[id]/_components/right-panel/index.tsx`	The main right panel component that houses both Chat and Code tabs with resizable functionality.
AI Chat Interface	`apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/index.tsx`	The conversational interface for AI interaction with history management and controls.
Code Editor Integration	`apps/web/client/src/app/project/[id]/_components/right-panel/code-tab/index.tsx`	Professional code editor with syntax highlighting and real-time synchronization.
Chat Manager	`apps/web/client/src/components/store/editor/chat/index.ts`	Manages AI conversation state, history, and message processing.
LLM Provider Factory	`packages/ai/src/chat/providers.ts`	Initializes and configures clients for different LLM providers (Anthropic, OpenRouter).
Code Transformation Engine	`packages/parser/src/code-edit/transform.ts`	The core engine that applies AI-generated changes to the code’s Abstract Syntax Tree (AST).
AST Parsing/Generation	`packages/parser/src/parse.ts`	Provides the fundamental functions for converting code to an AST and back again, using Babel.
Code Change Application	`packages/ai/src/apply/client.ts`	A client that orchestrates applying code changes using different AI providers (Morph, Relace).
Token Management	`packages/ai/src/tokens/index.ts`	Handles token counting and conversation context optimization for cost management.
Error Management	`apps/web/client/src/components/store/editor/error/index.ts`	Manages error detection, reporting, and resolution suggestions for development issues.
IDE Integration	`apps/web/client/src/components/store/editor/ide/index.ts`	Provides IDE-like features including terminal access and development tool integration.
API Communication	`packages/rpc/src/index.ts`	Defines the tRPC router that exposes the AI editing functionality to the frontend.

Global Overviews Visual Editor