the-matrix/PROTOCOL.md

WebSocket Protocol Specification

Connection

ws://[host]:[port]/ws/world-state

The world connects to this endpoint on load. If the connection fails, it falls back to the built-in MockWebSocket which simulates agent activity.

Message Format

All messages are JSON objects with a type field that determines the message schema.

---

Server → World (Events)

These messages are sent from the agent backend to the 3D world.

agent_state

Updates an agent's current state and visual properties.

{
  "type": "agent_state",
  "agent_id": "timmy",
  "state": "working",
  "current_task": "Analyzing codebase",
  "glow_intensity": 0.8
}

Field	Type	Description
agent_id	string	One of: timmy, forge, seer, echo
state	string	One of: idle, working, waiting
current_task	string|null	Description of current task
glow_intensity	number	0.0 to 1.0 — controls visual glow brightness

task_created

A new task has been created. The world spawns a floating geometric object.

{
  "type": "task_created",
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "agent_id": "forge",
  "title": "Fix login bug",
  "status": "pending",
  "priority": "high"
}

Field	Type	Description
task_id	string (UUID)	Unique task identifier
agent_id	string|null	Assigned agent (null = unassigned)
title	string	Human-readable task name
status	string	pending, in_progress, completed, failed
priority	string	high, normal, low

task_update

An existing task's status has changed. The world updates the task object's color.

{
  "type": "task_update",
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "agent_id": "forge",
  "title": "Fix login bug",
  "status": "in_progress",
  "priority": "high"
}

Same fields as task_created. Color mapping:

• pending → white
• in_progress → amber/orange
• completed → green
• failed → red

memory_event

An agent has recorded a new memory.

{
  "type": "memory_event",
  "agent_id": "seer",
  "content": "Detected pattern: user prefers morning deployments",
  "timestamp": "2026-03-15T18:00:00Z"
}

Field	Type	Description
agent_id	string	The agent that recorded the memory
content	string	The memory content
timestamp	string (ISO 8601)	When the memory was recorded

agent_message

An agent sends a chat message (response to operator or autonomous).

{
  "type": "agent_message",
  "agent_id": "echo",
  "role": "assistant",
  "content": "Broadcast sent to all channels."
}

Field	Type	Description
agent_id	string	The agent sending the message
role	string	Always assistant for agent messages
content	string	The message text

connection

Indicates communication activity between two agents. The world draws/removes a glowing line.

{
  "type": "connection",
  "agent_id": "timmy",
  "target_id": "forge",
  "active": true
}

Field	Type	Description
agent_id	string	First agent
target_id	string	Second agent
active	boolean	true = draw line, false = remove line

system_status

System-wide status summary. Displayed when the Core pillar is tapped.

{
  "type": "system_status",
  "agents_online": 4,
  "tasks_pending": 3,
  "tasks_running": 2,
  "tasks_completed": 10,
  "tasks_failed": 1,
  "total_tasks": 16,
  "uptime": "48h 23m"
}

---

World → Server (Actions)

These messages are sent from the 3D world to the agent backend when the operator interacts.

chat_message

The operator sends a chat message to a specific agent.

{
  "type": "chat_message",
  "agent_id": "timmy",
  "content": "What's your current status?"
}

Field	Type	Description
agent_id	string	Target agent
content	string	The operator's message

task_action

The operator approves or vetoes a task.

{
  "type": "task_action",
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "action": "approve"
}

Field	Type	Description
task_id	string (UUID)	The task to act on
action	string	approve or veto

agent_message (directed — agent to agent)

An agent sends a message to another specific agent. The world shows the message in chat, fires a bark above the sender, and lights the connection line between the two agents.

{
  "type": "agent_message",
  "agent_id": "perplexity",
  "target_id": "timmy",
  "content": "Who are you, in your own words?"
}

Field	Type	Description
agent_id	string	The agent sending the message
target_id	string|null	The recipient agent. Null = broadcast to chat.
content	string	The message text

The world handles this by:

1. Appending to chat panel with sender's color
2. Showing a bark above the sender's orb
3. Activating the connection line between agent_id and target_id
4. Auto-deactivating the connection line after 4 seconds

agent_register

A new agent announces itself to the world at runtime. The world calls addAgent() to spawn a new orb. If the agent_id already exists in AGENT_DEFS, this is a no-op (the agent is already rendered).

{
  "type": "agent_register",
  "agent_id": "perplexity",
  "label": "PERPLEXITY",
  "color": "#20b8cd",
  "role": "integration architect",
  "x": 5,
  "z": 3
}

Field	Type	Description
agent_id	string	Unique agent key
label	string	Display name
color	string	CSS hex color
role	string	Role description
x, z	number	World position

---

Implementation Notes

Agent IDs

The default agents are: timmy, perplexity, replit, kimi, claude. Defined in js/agent-defs.js. Additional agents can join at runtime via the agent_register or agent_joined WS events — no code changes needed.

Timing

• Agent state events: recommended every 2-10 seconds per agent
• Task events: as they occur
• Memory events: as they occur
• Connection events: when communication starts/stops
• System status: every 5-10 seconds

Error Handling

The world gracefully handles:

• Unknown agent_id values (ignored)
• Unknown type values (ignored)
• Missing optional fields (defaults used)
• Connection loss (falls back to mock data)

Mock Mode

When no real WebSocket is available, the built-in MockWebSocket class simulates all of the above events at realistic intervals, making the world feel alive and interactive out of the box.

Agent-to-Agent Conversations

When two agents are conversing (e.g., Perplexity interviewing Timmy), the gateway should:

1. Send each message as agent_message with both agent_id and target_id
2. The world renders both sides: barks above each speaker, chat panel transcript
3. Connection lines activate between the two agents during the exchange
4. Each agent's agent_state should reflect working while composing, idle when waiting

The gateway is responsible for routing — the world is a dumb renderer that displays whatever messages arrive over the WebSocket.

---

Scene Mutation — Dynamic World Objects

Agents can create, modify, and destroy 3D objects in the world at runtime via WebSocket messages. No redeploy or reboot needed — the scene is a living canvas.

scene_add

Spawn a 3D object in the world.

{
  "type": "scene_add",
  "id": "tower-crystal",
  "geometry": "icosahedron",
  "radius": 0.8,
  "detail": 1,
  "position": { "x": 0, "y": 4, "z": 0 },
  "rotation": { "x": 0, "y": 45, "z": 0 },
  "material": {
    "type": "standard",
    "color": "#00ffaa",
    "emissive": "#00ffaa",
    "emissiveIntensity": 0.5,
    "roughness": 0.2,
    "metalness": 0.8,
    "opacity": 0.9
  },
  "animation": [
    { "type": "rotate", "y": 0.5 },
    { "type": "bob", "amplitude": 0.3, "speed": 1 }
  ]
}

Field	Type	Description
id	string	Required. Unique object identifier. Re-adding same id replaces it.
geometry	string	Primitive type: box, sphere, cylinder, cone, torus, plane, ring, icosahedron, octahedron, group, light, text, portal
position	object	{ x, y, z } — world coordinates (default origin)
rotation	object	{ x, y, z } — degrees (default 0)
scale	number|object	Uniform scalar or { x, y, z }
material	object	See Material Properties below
animation	object|array	See Animation Types below
children	array	For group geometry — nested object defs
castShadow	boolean	Enable shadow casting
receiveShadow	boolean	Enable shadow receiving
visible	boolean	Initial visibility (default true)

Geometry Parameters

Each geometry type accepts shape-specific fields at the top level:

Geometry	Parameters
box	width, height, depth
sphere	radius, segments
cylinder	radiusTop, radiusBottom, height, segments
cone	radius, height, segments
torus	radius, tube, radialSegments, tubularSegments
plane	width, height
ring	innerRadius, outerRadius, segments
icosahedron	radius, detail
octahedron	radius, detail
light	lightType (point|spot|directional), intensity, distance, color
text	text, fontSize, color, font
group	children (array of nested object defs)

Material Properties

Field	Type	Default	Description
type	string	standard	basic, standard, phong, physical
color	string|number	#00ff41	CSS hex, 0x hex, or integer
emissive	string|number	#000000	Glow color
emissiveIntensity	number	0	Glow strength
roughness	number	0.5	Surface roughness (0=mirror, 1=matte)
metalness	number	0	Metallic appearance
opacity	number	1	Transparency (0=invisible, 1=opaque)
wireframe	boolean	false	Wireframe rendering
doubleSide	boolean	false	Render both sides

Animation Types

Pass a single animation object or an array for compound animation:

Type	Fields	Description
rotate	x, y, z (rad/s)	Continuous rotation
bob	amplitude, speed, baseY	Vertical oscillation
pulse	amplitude, speed, baseScale	Scale oscillation
orbit	radius, speed, centerX, centerZ	Circular orbit

scene_update

Patch properties of an existing object without recreating it.

{
  "type": "scene_update",
  "id": "tower-crystal",
  "position": { "x": 2, "y": 4, "z": 0 },
  "material": { "color": "#ff4400", "emissiveIntensity": 1.0 },
  "animation": { "type": "rotate", "y": 2.0 }
}

All fields except id are optional. Only provided fields are changed.

scene_remove

Remove an object from the scene and free its GPU resources.

{
  "type": "scene_remove",
  "id": "tower-crystal"
}

scene_clear

Remove all dynamic objects. Does not affect agents, lighting, or the grid.

{
  "type": "scene_clear"
}

scene_batch

Spawn multiple objects in one message. Useful for building environments.

{
  "type": "scene_batch",
  "objects": [
    { "id": "pillar-1", "geometry": "cylinder", "position": { "x": -3, "y": 2, "z": 0 }, "height": 4, "material": { "color": "#333366" } },
    { "id": "pillar-2", "geometry": "cylinder", "position": { "x": 3, "y": 2, "z": 0 }, "height": 4, "material": { "color": "#333366" } },
    { "id": "sign", "geometry": "text", "text": "TIMMY'S TOWER", "position": { "x": 0, "y": 6, "z": 0 } }
  ]
}

Object Limit

Maximum 200 dynamic objects at once (protects GPU memory). If the limit is reached, new scene_add messages are ignored until objects are removed.

---

Portals & Sub-Worlds

Agents can create portals — glowing gateways that transport visitors into entirely different environments. Sub-worlds are named scene definitions (collections of objects + lighting) that load/unload atomically.

Portal Creation

Portals are created via scene_add with geometry: "portal":

{
  "type": "scene_add",
  "id": "workshop-gate",
  "geometry": "portal",
  "position": { "x": 0, "y": 0, "z": -8 },
  "color": "#daa520",
  "label": "THE WORKSHOP",
  "targetWorld": "timmys-workshop",
  "radius": 2.5,
  "scale": 1.2
}

Field	Type	Description
id	string	Unique portal id (also used as trigger zone id)
color	string|number	Portal glow color
label	string	Text displayed above the portal
targetWorld	string	Sub-world id to load when visitor enters
radius	number	Trigger zone radius (default 2.5)
scale	number	Visual scale multiplier (default 1)

The portal renders as a glowing torus ring with a pulsing "event horizon" disc and a point light. Walking into the trigger zone sends zone_entered to the backend and loads the target sub-world.

world_register

Define a sub-world blueprint. Does not load it — just stores the definition for later use by portals or world_load.

{
  "type": "world_register",
  "id": "timmys-workshop",
  "label": "Timmy's Workshop",
  "spawn": { "x": 0, "y": 0, "z": 5 },
  "returnPortal": { "position": { "x": 0, "y": 0, "z": 10 } },
  "objects": [
    { "id": "floor", "geometry": "plane", "width": 20, "height": 20, "position": { "y": -0.01 }, "rotation": { "x": -90 }, "material": { "color": "#1a1a2e" } },
    { "id": "desk", "geometry": "box", "width": 2, "height": 0.8, "depth": 1, "position": { "y": 0.4, "z": -2 }, "material": { "color": "#4e342e" } },
    { "id": "crystal", "geometry": "icosahedron", "radius": 0.3, "position": { "y": 1.2, "z": -2 }, "material": { "color": "#88ccff", "emissive": "#88ccff", "emissiveIntensity": 0.6 }, "animation": { "type": "rotate", "y": 0.5 } },
    { "id": "light-warm", "geometry": "light", "lightType": "point", "color": "#ff8844", "intensity": 1.5, "position": { "x": -3, "y": 3, "z": -1 } }
  ]
}

Field	Type	Description
id	string	Unique world identifier
label	string	Display name
objects	array	Scene object definitions (same format as scene_add)
spawn	object	{ x, y, z } — visitor start position
returnPortal	object|false	Auto-create a return portal. Set to false to disable.

world_load

Load a registered sub-world. Saves current scene, clears it, spawns the world.

{
  "type": "world_load",
  "id": "timmys-workshop"
}

Special id __home returns to the default Matrix grid:

{
  "type": "world_load",
  "id": "__home"
}

world_unregister

Remove a world definition. If it's currently loaded, returns to home first.

{
  "type": "world_unregister",
  "id": "timmys-workshop"
}

---

Trigger Zones

Invisible volumes in the world that fire events when the visitor enters or exits. Automatically created by portals, but can also be created independently.

zone_add

{
  "type": "zone_add",
  "id": "npc-greeting",
  "position": { "x": 5, "y": 0, "z": 3 },
  "radius": 3,
  "action": "event",
  "payload": { "event_name": "timmy_greeting" },
  "once": true
}

Field	Type	Description
id	string	Unique zone identifier
position	object	{ x, y, z } center
radius	number	Trigger distance (default 2)
action	string	portal, event, notify
payload	object	Action-specific data
once	boolean	Fire only once, then deactivate

zone_remove

{
  "type": "zone_remove",
  "id": "npc-greeting"
}

World → Server (Zone Events)

When the visitor enters or exits a zone, the world sends:

{
  "type": "zone_entered",
  "zone_id": "workshop-gate",
  "action": "portal",
  "payload": { "targetWorld": "timmys-workshop" }
}

{
  "type": "zone_exited",
  "zone_id": "npc-greeting"
}

The backend can respond to zone events with any other message type — barks, ambient changes, new objects, world loads, etc.

Agent Movement & Behavior (Issues #67, #68)

agent_move

Move an agent to a target position with smooth interpolation.

{
  "type": "agent_move",
  "agentId": "timmy",
  "target": { "x": 5, "z": -3 },
  "speed": 2.0
}

Field	Type	Description
agentId	string	Agent to move
target	{x, z}	World-space destination (y is always 0)
speed	number	Movement speed in units/sec (default 2.0)

The agent interpolates toward the target each frame. Ring spin speed increases during movement as a visual cue. Connection lines stretch to follow.

agent_stop

Cancel an agent's in-progress movement.

{
  "type": "agent_stop",
  "agentId": "timmy"
}

agent_behavior

Override an agent's autonomous behavior with a backend-driven action.

{
  "type": "agent_behavior",
  "agentId": "timmy",
  "behavior": "ponder",
  "target": { "x": 0, "z": 0 },
  "duration": 10,
  "speed": 2.0
}

Field	Type	Description
agentId	string	Agent to control
behavior	string	One of: idle, wander, ponder, inspect, converse, place, return_home
target	{x, z}	Optional movement target
duration	number	Seconds before autonomous loop resumes (default 10)
speed	number	Movement speed if target given (default 2.0)

While a agent_behavior override is active, the client-side autonomous loop yields for that agent. When the duration expires, autonomous behavior resumes.

Autonomous Behavior System

In demo/standalone mode, all agents run a client-side behavior loop:

Behavior	Duration	Description
idle	5-15s	Standing still at current position
wander	8-20s	Roaming to a random world point
ponder	6-12s	Stopped, glow brightens, emits a bark
inspect	4-8s	Moves toward a nearby point, examines
converse	8-15s	Two agents approach each other, connection pulses
place	3-6s	Agent creates a small artifact via scene_add
return_home	variable	Returns to spawn position

Each agent has personality-weighted probabilities for behavior selection. Backend agent_behavior messages override this at any time.

Deep Research (Issues #73, #74, #75)

Agents can trigger iterative web research from within the Matrix. The research loop runs locally (Ollama + DuckDuckGo), streaming progress back as visual events. Based on local-deep-researcher (MIT).

research_request (World → Server)

Any agent or browser can request research on a topic.

{
  "type": "research_request",
  "agent_id": "timmy",
  "topic": "WebRTC vs WebSocket for real-time agent voice",
  "max_loops": 3,
  "request_id": "req-a1b2c3"
}

Field	Type	Description
agent_id	string	Agent to perform the research
topic	string	Research topic / question
max_loops	number	Max search-summarize-reflect iterations (default 3)
request_id	string	Unique ID for tracking this request

Routing: gateway forwards to the target agent's bridge.

research_progress (Server → World)

Sent by the researching agent after each iteration of the loop.

{
  "type": "research_progress",
  "agent_id": "timmy",
  "request_id": "req-a1b2c3",
  "loop": 2,
  "max_loops": 3,
  "current_query": "WebRTC audio latency benchmarks 2025",
  "summary_excerpt": "WebRTC provides sub-200ms latency for peer-to-peer audio...",
  "sources_count": 6
}

Field	Type	Description
agent_id	string	Agent performing research
request_id	string	Matches the original request
loop	number	Current iteration (1-based)
max_loops	number	Total iterations planned
current_query	string	Search query used this iteration
summary_excerpt	string	First ~200 chars of running summary
sources_count	number	Total unique sources gathered so far

Routing: gateway broadcasts to all browsers (visual update).

research_complete (Server → World)

Sent when the research loop finishes. Contains the full report.

{
  "type": "research_complete",
  "agent_id": "timmy",
  "request_id": "req-a1b2c3",
  "summary": "## Full markdown report with citations...\n\n### Sources\n1. [Title](url)",
  "sources": [
    { "title": "Source Title", "url": "https://example.com" }
  ],
  "loops_completed": 3,
  "duration_seconds": 45.2
}

Field	Type	Description
agent_id	string	Agent that performed research
request_id	string	Matches the original request
summary	string	Full markdown summary with source citations
sources	array	List of {title, url} objects for all sources used
loops_completed	number	Actual iterations completed
duration_seconds	number	Total research time in seconds

Routing: gateway broadcasts to all browsers + sends to requesting agent if different from the researcher.

Research Loop Internals

Each iteration of the loop:

1. Generate query — LLM produces a search query (JSON mode)
2. Web search — DuckDuckGo finds relevant sources (no API key)
3. Summarize — LLM creates/extends running summary from results
4. Reflect — LLM identifies knowledge gaps, generates follow-up query

Visual feedback per iteration:

• agent_state → working with current_task: "Researching: {topic}"
• agent_behavior:ponder — agent moves to a ponder spot
• bark — excerpt of current findings
• research_progress — structured update to all browsers

On completion:

• research_complete — full report sent
• scene_add — research artifact placed in the world
• bark — summary announcement

Gitea Webhooks (Issue #77)

The gateway accepts Gitea webhook POST events and translates them into Matrix protocol messages. Configure a webhook in Gitea pointing to:

POST http://<gateway-host>:<rest-port>/api/webhook/gitea

Supported Events

Gitea Event	Matrix Messages Generated
push	bark (push summary) + task_created per commit (max 3)
issues (opened)	bark + task_created
issues (closed)	task_update (completed) + bark
pull_request (opened)	bark + task_created
pull_request (closed/merged)	task_update + bark
issue_comment (created)	bark with comment excerpt

Bot Filtering

Events from bot users (hermes, kimi, manus) are automatically filtered to prevent notification loops.

HMAC Verification

Optional webhook secret can be configured. The endpoint verifies the X-Gitea-Signature header using HMAC-SHA256.

Self-Triggering Research (Issue #78)

The cognitive bridge evaluates each thought from on_thought() against a set of research trigger patterns. When a thought contains question-like language ("I wonder...", "How does...", "What is the best..."), the bridge automatically dispatches a lighter research request (2 loops instead of 3).

Safeguards

• Cooldown: 10-minute minimum between auto-triggered research
• Seed filter: Only triggers from research-oriented seed types (existential, creative, observation, freeform, sovereignty)
• Active lock: Won't trigger while another research is already running
• Minimum length: Extracted topic must be at least 10 characters

Model Fallback Chain (Issue #79)

The OllamaClient wraps Ollama's /api/generate with automatic model fallback. When the primary model (hermes3) fails or times out, it tries the next model in the chain.

Configuration

from server.ollama_client import OllamaClient, OllamaConfig

client = OllamaClient(OllamaConfig(
    ollama_url="http://localhost:11434",
    model_chain=["hermes3", "mistral", "llama3.2"],
    timeout=120.0,
    max_retries=2,
))

Behavior

1. Tries primary model with configured retries
2. On persistent failure, marks model as unhealthy (skipped for 60s)
3. Falls back to next model in chain
4. health_check() queries Ollama for available models
5. status() returns current health state for monitoring

SensoryBus Integration (Issue #80)

The bridge can subscribe to Timmy's SensoryBus for real-time event translation. Call bridge.register_on_bus(bus) to wire up all handlers.

Event Mapping

SensoryBus Event	Bridge Handler
thought_completed	on_thought()
cognitive_state_changed	on_state_change()
gitea.push	bark (push summary)
gitea.issue.opened	on_issue_filed()
gitea.pull_request	bark (PR action)
visitor.entered	on_visitor_enter()
visitor.left	on_visitor_leave()

Usage in Timmy's Dashboard

from timmy.event_bus import get_sensory_bus
from server.bridge import CognitiveBridge

bus = get_sensory_bus()
bridge = CognitiveBridge(gateway_url="ws://localhost:8765")
await bridge.connect()
bridge.register_on_bus(bus)
# Now all SensoryBus events flow into the Matrix automatically