Tau Voice: Voice-Enabled Conversational CAD Design

What It Does

Tau Voice adds hands-free voice interaction to Tau, a browser-based CAD platform. Users speak design requests and receive voice responses while watching 3D models generate in real-time. Perfect for designers working with their hands or away from keyboards.

Flow: Voice input (ElevenLabs) → Multi-agent orchestration (LangGraph) → CAD generation → Voice response → Visual preview

How It Works

• Multi-Agent System: LangGraph supervisor routes between specialized agents (research expert, CAD expert)
• Voice Integration: ElevenLabs provides bidirectional speech-to-text and text-to-speech
• Browser Tools: Agents can search the web and browse pages for reference materials
• CAD Generation: Supports multiple kernels (Replicad, OpenSCAD, JSCAD, KCL) for real-time 3D model creation
• Real-Time Updates: Event stream communication keeps UI synchronized with agent actions

Judging Criteria

Working Prototype: Fully functional end-to-end with voice I/O, real-time CAD generation, and error handling.

Technical Complexity & Integration: Multi-agent orchestration (LangGraph), ElevenLabs voice API, browser tool agents (Tavily), multi-kernel CAD support, and robust state management.

Innovation & Creativity: First voice-enabled conversational CAD platform combining hands-free design with sophisticated multi-agent AI.

Real-World Impact: Accessibility for mobility-limited users, hands-on prototyping workflows, democratizes CAD through natural language.

Theme Alignment: Voice + browser + tool agents seamlessly integrated into cohesive autonomous assistant.

Tech Stack

Voice: ElevenLabs API
AI Orchestration: LangGraph, LangChain
Frontend: React 19, Three.js
Backend: NestJS, PostgreSQL (Drizzle ORM)
CAD Kernels: Replicad, OpenSCAD, JSCAD, KCL
Tools: Tavily (web search), OpenAI/Anthropic (LLMs), Event Streams

Setup & Demo

Prerequisites: Node.js 24+, pnpm, Docker

# 1. Install dependencies  
pnpm install  
  
# 2. Start database  
pnpm db:up  
  
# 3. Configure environment  
cp apps/ui/.env.example apps/ui/.env.local  
cp apps/api/.env.example apps/api/.env.local  
# Add ElevenLabs API key and other required keys  
  
# 4. Start servers  
pnpm dev  

Access: UI at http://localhost:5173, API at http://localhost:3000

Demo:

1. Open web interface → Enable voice mode in chat
2. Speak: “Create a parametric box, 50mm wide, 30mm tall, 20mm deep”
3. Watch agent generate CAD model with voice feedback
4. Continue conversation: “Make it taller” or “Add rounded corners”
5. Model updates in real-time with voice responses

Prior Work Description

This project builds upon Tau, an existing open-source CAD platform. All code outside of the elevenlabs branch represents prior work.

Existing Platform

The Tau platform includes:

• Basic frontend with React and code editor
• Backend API with database
• CAD kernel integrations (Replicad, OpenSCAD, JSCAD, KCL)
• Text-based chat interface with AI agents
• File management and 3D visualization

The hackathon work adds voice interaction capabilities on top of this existing infrastructure.