# babylonjs-mcp

**Repository Path**: jaypro/babylonjs-mcp

## Basic Information

- **Project Name**: babylonjs-mcp
- **Description**: https://github.com/VibeCAD/babylonjs-mcp
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-08-05
- **Last Updated**: 2026-08-05

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# VibeCAD - AI-Powered 3D Scene Manipulation with MCP

VibeCAD is an innovative 3D scene manipulation system that combines natural language processing with the Model Context Protocol (MCP) to create and manage 3D objects in a Babylon.js environment. Users can create 3D scenes using simple text commands through either a web interface or Claude Desktop.

## Core Functionality

VibeCAD enables users to:
- Create 3D objects using natural language commands
- Manipulate objects (move, rotate, scale, color)
- Manage scene objects through an intuitive GUI
- Use AI assistants (OpenAI or Claude) to interpret commands
- Real-time 3D visualization with Babylon.js

## Current Capabilities
This version of the MCP is able to create primitive shapes cylinder, cube, and sphere are working - but torus is not for some reason. 
The create, delete, select tools are all working. The list function is not working. 
The browser console logs in the GUI show that a babylon.js Torus is being generated, but it is not being rendered. 
Next steps would involve increasing the functionality of the MCP server to handle more complex shapes. 

## Architecture Overview

```mermaid
graph TB
    subgraph "Input Sources"
        User1[User via Browser]
        User2[User via Claude Desktop]
    end
    
    subgraph "AI Processing"
        OpenAI[OpenAI API]
        Claude[Claude Desktop]
    end
    
    subgraph "MCP Server"
        STDIO[STDIO Interface]
        HTTP[HTTP API :8081]
        WS[WebSocket Server :8080]
        Core[MCP Core Logic]
    end
    
    subgraph "Frontend"
        GUI[React GUI :5173]
        Babylon[Babylon.js Scene]
        WSClient[WebSocket Client]
    end
    
    User1 -->|Text Input| GUI
    GUI -->|API Call| OpenAI
    OpenAI -->|Structured Commands| HTTP
    
    User2 -->|Text Input| Claude
    Claude -->|MCP Protocol| STDIO
    
    HTTP --> Core
    STDIO --> Core
    Core -->|Commands| WS
    WS -->|Real-time Updates| WSClient
    WSClient --> Babylon
    
    style GUI fill:#e1f5fe
    style MCP Server fill:#fff3e0
    style Babylon fill:#c8e6c9
```

## MCP Server Capabilities

### Available Tools

#### 1. **create_object**
Creates a new 3D object in the scene.
```json
{
  "shape": "box|sphere|cylinder|cone|torus",
  "name": "unique_object_name",
  "position": { "x": 0, "y": 1, "z": 0 },
  "size": 2,
  "color": { "r": 1, "g": 0, "b": 0 }
}
```

#### 2. **delete_object**
Removes an object from the scene.
```json
{
  "name": "object_name"
}
```

#### 3. **select_object**
Selects an object for manipulation.
```json
{
  "name": "object_name"
}
```

#### 4. **list_objects**
Lists all objects currently in the scene.
```json
{}
```

## Project Structure

```
VibeCAD/
├── babylonjs-mcpV2/          # MCP Server
│   ├── src/
│   │   ├── core/             # Core MCP logic
│   │   │   ├── mcp-server.ts
│   │   │   ├── types.ts
│   │   │   ├── babylon-scene.ts
│   │   │   ├── command-parser.ts
│   │   │   └── websocket-client.ts
│   │   └── mcp-stdio-server.ts  # Main server file
│   ├── dist/                 # Compiled JS
│   └── package.json
│
└── gui/gui/                  # Frontend Application
    ├── src/
    │   ├── App.tsx           # Main React component
    │   ├── App.css           # Styles
    │   └── main.tsx          # Entry point
    ├── public/
    └── package.json
```

## Technical Architecture

### WebSocket Connection
- **Purpose**: Real-time bidirectional communication between MCP server and GUI
- **Port**: 8080 (configurable via `MCP_WS_PORT`)
- **Protocol**: Standard WebSocket with JSON message format
- **Features**:
  - Auto-reconnection (5 attempts, 3-second intervals)
  - Connection status monitoring
  - Message acknowledgment system

### HTTP API Wrapper
- **Purpose**: Bridge between web-based AI services and MCP server
- **Port**: 8081 (configurable via `MCP_HTTP_PORT`)
- **Endpoints**:
  - `GET /health` - Server status and connection info
  - `POST /tools/:toolName` - Execute MCP tools
- **Features**:
  - CORS enabled for browser access
  - RESTful design
  - Structured JSON responses

## Communication Paths

### Browser Path (OpenAI Integration)
1. User enters natural language command in GUI
2. GUI sends text to OpenAI API
3. OpenAI interprets and returns structured tool calls
4. GUI makes HTTP POST to MCP server's HTTP API
5. MCP server processes command and sends via WebSocket
6. GUI receives WebSocket message and updates 3D scene

### Claude Desktop Path
1. User enters command in Claude Desktop
2. Claude interprets using MCP tools directly
3. Claude calls MCP server via STDIO interface
4. MCP server processes command and sends via WebSocket
5. GUI receives WebSocket message and updates 3D scene

## Architecture Justification

### Why HTTP API?
- **Browser Security**: Browsers cannot directly access STDIO interfaces
- **CORS Support**: Enables cross-origin requests from web applications
- **RESTful Design**: Familiar pattern for web developers
- **Stateless**: Each request is independent, improving reliability
- **Debugging**: Easy to test with tools like curl or Postman

### Why WebSocket for GUI Updates?
- **Real-time**: Instant updates without polling
- **Bidirectional**: Allows for future features like state sync
- **Efficient**: Lower overhead than HTTP for frequent updates
- **Event-driven**: Natural fit for UI updates

### Why Dual Interface (STDIO + HTTP)?
- **Flexibility**: Supports both desktop and web clients
- **MCP Compliance**: STDIO for standard MCP protocol
- **Web Compatibility**: HTTP for browser-based AI services
- **Unified Logic**: Single codebase serves both interfaces

## Configuration

### Environment Variables

#### MCP Server
- `MCP_WS_PORT` - WebSocket port (default: 8080)
- `MCP_HTTP_PORT` - HTTP API port (default: 8081)

#### GUI
- `VITE_MCP_WS_URL` - WebSocket URL (default: ws://localhost:8080)
- `VITE_MCP_HTTP_URL` - HTTP API URL (default: http://localhost:8081)

## Getting Started

### Prerequisites
- Node.js 18+
- npm or yarn
- OpenAI API key (for browser path)
- Claude Desktop (for MCP path)

### Installation

1. **Install MCP Server**
```bash
cd babylonjs-mcpV2
npm install
npm run build:server
```

2. **Install GUI**
```bash
cd gui/gui
npm install
```

### Running the Application

1. **Start MCP Server**
```bash
cd babylonjs-mcpV2
node dist/mcp-stdio-server.js
```

2. **Start GUI**
```bash
cd gui/gui
npm run dev
```

3. **Open Browser**
Navigate to http://localhost:5173

## Usage Examples

### Via Browser (OpenAI)
- "Create a red cube"
- "Make a blue sphere above the cube"
- "Delete the cube"
- "Create three green cylinders in a row"

### Via Claude Desktop
Configure Claude Desktop with:
```json
{
  "mcpServers": {
    "babylonjs": {
      "command": "node",
      "args": ["/path/to/babylonjs-mcpV2/dist/mcp-stdio-server.js"]
    }
  }
}
```

Then ask Claude to create 3D objects using the available tools.

## Future Enhancements

- Additional MCP tools (move, rotate, scale, change color)
- Advanced object grouping and hierarchies
- Scene persistence and loading
- Multi-user collaboration
- Export to 3D file formats
- Physics simulation integration

## License

MIT License - See LICENSE file for details