2.1 AI Code Generation and Editing: From Natural Language to Structured Code Conversion

Onlook’s AI system enables users to generate and modify web application code through natural language commands. This capability moves beyond simple text-based editing, relying on sophisticated mechanisms to ensure that all changes are precise, structurally sound, and syntactically correct.

This chapter details the fundamental processes that allow Onlook’s AI to perform these actions. We will explore three key areas: first, how the AI’s operational rules and specialized abilities are defined; second, the technical method Onlook uses to safely and accurately alter your code’s structure; and third, the complete workflow from your initial natural language instruction to the final application of those code modifications in your live project.

2.1.1 Defining AI Behavior and Capabilities: The Role of System Prompts

For Onlook’s AI to effectively assist in web development, it needs a clear understanding of its role and how to execute tasks accurately. This understanding is established through the System Prompt, which acts as the AI’s foundational instruction manual. It defines the AI’s professional persona, its core operational guidelines, and the specific tools it has access to.

Based on Onlook’s internal configuration (rooted in packages/ai/src/prompt/system.ts), the AI is precisely instructed to operate as an “expert React, Next.js, and Tailwind design-engineer.” This specialized role ensures that the AI’s actions—its responses, suggestions, and code modifications—are consistently aligned with modern web development standards and best practices.

The System Prompt includes several key directives that shape the AI’s behavior and enhance its effectiveness:

• Commitment to Code Quality: The AI is programmed to prioritize code that is maintainable. This means it is instructed to refactor code to keep files and functions concise and manageable. It must also respect and utilize existing coding conventions, libraries, and styles already present in your project, ensuring consistency and seamless integration of its changes.
• Precision in Execution and Communication: The AI’s outputs are designed to be precise, concise, and reflect expert-level judgment. When describing any changes it has made, the communication is direct and to the point, avoiding unnecessary verbosity.
• Equipped with Development Tools: Beyond processing language, Onlook’s AI is empowered with a suite of operational tools that extend its capabilities beyond simple text generation. These tools enable the AI to:
  • Explore Project Files: Utilize internal search and grep-like functionalities to efficiently navigate and understand the project’s codebase.
  • Access External Knowledge: Perform web searches to gather current information, research specific topics, or understand external references (e.g., URLs) that you might provide.
  • Interact with the Development Environment: Execute terminal commands within its operational environment (as defined by SHELL_PROMPT integrated in packages/ai/src/prompt/provider.ts). This allows the AI to perform actions such as installing necessary dependencies or running build scripts.
  • Verify Code Integrity: Employ a typecheck tool to automatically verify its code modifications, helping to prevent the introduction of type errors and ensuring the stability of the application.
• Clarification for Ambiguity: If a user’s instruction is unclear or lacks sufficient detail, the AI is programmed to ask clarifying questions rather than making assumptions. This proactive approach ensures that its actions are always aligned with your intent.

Furthermore, Onlook’s system (packages/ai/src/prompt/provider.ts) can dynamically combine this core System Prompt with other specialized prompts. This allows the AI to adapt its “mindset” for specific tasks, whether it’s generating a new page from scratch, providing design suggestions, or handling a direct query. By meticulously defining the AI’s role, rules, and access to these operational tools, Onlook establishes a foundation for intelligent, reliable, and context-aware assistance throughout the web development process.

2.1.2 Application of Abstract Syntax Tree (AST) in Code Modification: Ensuring Structure and Syntax

When Onlook’s AI modifies code, a critical technical challenge is ensuring that these changes are not only functionally correct but also maintain the code’s underlying structure and avoid introducing syntax errors. Simply treating code as plain text and performing search-and-replace operations is inherently unreliable; it can easily break a program by altering crucial structural components or introducing mismatched elements.

Onlook addresses this by utilizing an Abstract Syntax Tree (AST). Imagine an AST as the detailed architectural blueprint or skeletal structure of your code. Instead of a flat piece of text, your code is represented as a hierarchical, tree-like data structure where each “node” in the tree represents a specific grammatical construct in the programming language—such as a variable, a function definition, a JSX element, or an attribute. This blueprint provides a deep, structural understanding of the code’s organization and its inherent syntax rules.

Here’s how Onlook leverages the AST for precise and safe code modification:

• Structural Understanding: When Onlook’s AI needs to modify your application, the original code is first converted into this AST (a process managed by parsing capabilities, primarily in packages/parser/src/parse.ts). This allows Onlook to “see” your code not as raw lines of text, but as a structured collection of components, properties, and their logical relationships.
• Targeted Instructions: The AI does not directly generate new lines of code to be pasted into a file. Instead, it produces structured instructions that specify what change to make to which specific part of the code’s AST. For example, an instruction might be: “On the JSX element with ID ‘X’, update its ‘className’ attribute to ‘new-style’” or “Insert a new button element inside JSX element ‘Y’.” These instructions are defined as structured data (ee.g., CodeAction types in packages/models/src/actions/index.ts).
• Safe and Precise Execution: The core logic for applying these instructions resides in the transformAst function (packages/parser/src/code-edit/transform.ts). This function meticulously traverses the AST to locate the exact node (e.g., a specific button, a div, a text block) targeted by the AI’s instruction. It then programmatically modifies that specific node within the tree. This AST-driven approach ensures:
  • Syntactic Correctness: Because changes are made at the structural level of the AST, Onlook can guarantee that the resulting code remains grammatically valid and free of syntax errors. This is crucial for maintaining a functional application.
  • Structural Integrity: Modifications respect the inherent hierarchy and relationships within the code, preventing accidental disruptions to the application’s layout or logic.
  • Diverse Operations: Onlook’s AST engine supports a range of complex structural changes, including updating element attributes, modifying text content, and even advanced operations like inserting new elements, removing existing ones, moving elements within the DOM, or grouping/ungrouping components.

By working directly with the AST, Onlook’s AI functions less like a blunt text editor and more like a skilled architect or surgeon, making precise and informed adjustments to the very blueprint of your application. This foundational mechanism is what allows Onlook to deliver reliable, production-ready code modifications from natural language instructions.

2.1.3 End-to-End Process of AI Code Modification: Instruction Generation and Application

The journey from a user’s natural language idea to a visible, functional change in a live application within Onlook is a carefully orchestrated, multi-step process. This integrated workflow ensures that AI-driven modifications are accurate, context-aware, and seamlessly applied.

Here’s how Onlook’s AI code modification process works, from the initial instruction to the final visual update:

1. User Input and Comprehensive Context Collection:
   • The process begins when a user provides an instruction. This can be a natural language prompt (e.g., “Add a dark mode toggle to the settings component”) or a direct visual interaction within the real-time preview (e.g., clicking an element and then asking to “change its color to blue”).
   • Concurrently, Onlook gathers extensive contextual information. This includes:
     • Code Snippets: Relevant sections of code from the affected files.
     • Visual Highlights: Any specific parts of the code that the user has highlighted.
     • Error Messages: Current technical errors in the project that the AI should be aware of.
     • Project Information: General details about the active project.
     • Visual Inputs: Even images can be part of the input, showcasing Onlook’s potential for multimodal understanding (as evidenced by packages/ai/src/prompt/provider.ts handling various MessageContext types, including ImageMessageContext).
2. Intelligent Prompt Construction:
   • All this collected information is then meticulously assembled into a structured prompt by Onlook’s prompt provider (packages/ai/src/prompt/provider.ts). This is more than a simple concatenation of text; the system strategically wraps different pieces of information (user instructions, code context, errors) within specific XML tags. This structured format is crucial as it helps the underlying Large Language Model (LLM) clearly distinguish between various types of input and accurately understand the user’s intent.
   • The prompt also dynamically incorporates the System Prompt (as detailed in Section 2.1.1), which guides the AI’s persona and operational rules based on the specific task.
   • To manage the LLM’s token limits and optimize processing costs, Onlook intelligently handles the volume of information sent. For instance, in longer conversational threads, it might truncate the content of older file contexts, sending full code content only for the most recent interactions.
3. AI Analysis and Structured Instruction Generation:
   • The meticulously constructed, structured prompt is sent to the configured LLM provider (packages/ai/src/chat/providers.ts), which could be a service like Anthropic or OpenRouter.
   • The LLM processes this comprehensive prompt, leveraging its extensive knowledge base and the precise directives from Onlook’s System Prompt. Its output is not directly executable code. Instead, it generates a set of precise, structured modification instructions. These instructions are designed to be machine-readable and specify what changes to apply to which specific elements of the code. For example, an instruction might be to “replace the className attribute” of a particular JSX element or “insert a new element at a given location.”
4. AST-Based Code Transformation:
   • The AI’s structured instructions are then passed to Onlook’s robust code parsing and transformation engine. The original application code is first converted into an Abstract Syntax Tree (AST) using Babel’s parsing capabilities (packages/parser/src/parse.ts).
   • The transformAst function (packages/parser/src/code-edit/transform.ts) takes these AI-generated instructions and meticulously applies them directly to the AST. This involves traversing the tree to find target nodes (often identified by unique Onlook IDs, or OIDs, attached to elements) and then programmatically altering their attributes, content, or structural position within the code’s blueprint.
   • This AST-driven approach is paramount: it guarantees that all modifications maintain the code’s syntactic correctness and structural integrity, preventing the common errors that arise from simple text-based replacements.
5. Updated Code Generation and Real-time Visual Feedback:
   • Once the modifications are successfully applied to the AST, the transformed tree is converted back into a clean, formatted code string (packages/parser/src/parse.ts).
   • This updated code is then transmitted back to the user’s browser. The real-time preview environment (an iframe managed by penpal, as will be discussed in Section 2.2) receives this new code and instantly re-renders the application.
   • This immediate visual feedback allows users to see the AI’s changes in real-time, completing the iterative development loop and enabling continuous refinement.

This comprehensive, end-to-end process, rooted in intelligent prompting, AST-based precision, and real-time feedback, is how Onlook transforms abstract natural language ideas into concrete, functional, and syntactically sound code modifications, making advanced web development accessible.

Related Technical Files