Import AITURK IDE 1.0.0-beta.1 from Hermes 63279301; preserve MIT license
This commit is contained in:
@@ -0,0 +1,364 @@
|
||||
# Unreal MCP — Tool Surface Reference
|
||||
|
||||
How Epic's editor-embedded MCP server organizes, advertises, and executes
|
||||
tools, and how to extend the surface when the shipped tools run out.
|
||||
Everything here is against UE 5.8's experimental plugin (id
|
||||
`ModelContextProtocol`); expect drift between engine versions — the live
|
||||
`describe_toolset` schema always outranks this file.
|
||||
|
||||
## Architecture in one paragraph
|
||||
|
||||
The **Unreal MCP** plugin hosts the HTTP server inside the editor process
|
||||
(default `http://127.0.0.1:8000/mcp`, loopback-only, no auth, HTTP + SSE
|
||||
only — no stdio/WebSocket). It implements the protocol but ships no tools of
|
||||
its own. Tools come from **Toolsets** — classes deriving from
|
||||
`UToolsetDefinition` (C++) or `unreal.ToolsetDefinition` (Python) — collected
|
||||
at startup by the **Toolset Registry** subsystem (sibling plugin,
|
||||
auto-enabled; the registry itself ships no toolsets either). The shipped
|
||||
tools live in per-domain plugins under `Engine/Plugins/Experimental/
|
||||
Toolsets/` — the workhorse is **EditorToolset** (core editor toolsets,
|
||||
Python + C++) — and **AllToolsets** is a one-checkbox aggregator plugin
|
||||
that depends on ~21 of them (verified from `AllToolsets.uplugin`, 5.8):
|
||||
AIModule, AnimationAssistant, AutomationTest, ConfigSettings, Conversation,
|
||||
DataRegistry, DataflowAgent, Editor, GameFeatures, GameplayTags, GAS,
|
||||
MCPClient, Niagara, PCG, Physics, Plugin, SemanticSearch, SlateInspector,
|
||||
StateTree, UMG, WorldConditions. Project plugins and Game Feature Plugins
|
||||
can contribute more. Unreal MCP wraps every registered tool call as an MCP
|
||||
Tool. Execution is **serialized onto the game thread** — one tool call at a
|
||||
time, editor UI blocked while each runs.
|
||||
|
||||
## Tool-search mode (the default contract)
|
||||
|
||||
With `Enable Tool Search` on (default), `tools/list` advertises exactly three
|
||||
meta-tools:
|
||||
|
||||
| Meta-tool | Args | Returns |
|
||||
|---|---|---|
|
||||
| `list_toolsets` | — | Registered toolset names + descriptions |
|
||||
| `describe_toolset` | toolset name | JSON Schemas for every tool in that toolset |
|
||||
| `call_tool` | toolset/tool name + arguments object | The tool's result, same turn |
|
||||
|
||||
Discipline:
|
||||
|
||||
- `list_toolsets` once per session; re-run only after `RefreshTools`, plugin
|
||||
changes, or reconnect.
|
||||
- `describe_toolset` before first use of any toolset. Parameter names, types,
|
||||
and required fields come from the schema — never from memory or this file.
|
||||
- Results: primitive results arrive wrapped as `{"result": ...}` (CVar
|
||||
`ModelContextProtocol.WrapPODToolResultsInObject`, default true).
|
||||
Structured results serialize with field-level schema.
|
||||
- Errors come back as tool-call errors with the engine-side message — read
|
||||
them; they usually name the offending parameter or missing asset.
|
||||
|
||||
Eager mode (`Enable Tool Search` off) advertises every tool individually.
|
||||
Under Hermes that means each tool becomes `mcp_unreal_engine_<tool_name>` at
|
||||
session start, and `hermes mcp configure unreal-engine` can prune the list.
|
||||
Schema payload grows with every registered toolset, and tool authors are told
|
||||
NOT to rely on eager advertising — stay in tool-search mode unless a very
|
||||
small fixed surface is wanted.
|
||||
|
||||
## call_tool dispatch semantics (live-verified, 5.8)
|
||||
|
||||
Verified against a running 5.8 server; these details are where naive
|
||||
clients die:
|
||||
|
||||
- `list_toolsets` returns **fully-qualified** toolset names — Python:
|
||||
`editor_toolset.toolsets.scene.SceneTools`; C++:
|
||||
`EditorToolset.EditorAppToolset`. Epic's prose says "SceneTools"; the
|
||||
registry speaks qualified names. Use them verbatim in `describe_toolset`
|
||||
and `call_tool`'s `toolset_name`.
|
||||
- `tool_name` must be the **short** name (`get_current_level`,
|
||||
`CaptureViewport`). Passing the fully-qualified tool name fails with
|
||||
"Unknown tool" even though `describe_toolset` displays qualified names.
|
||||
- `call_tool` args: `{"toolset_name": ..., "tool_name": ..., "arguments":
|
||||
{...}}`; result returns on the same turn (the HTTP response blocks until
|
||||
the game thread finishes the call).
|
||||
- **`TOptional` parameters must be passed explicitly as `null`** — omitting
|
||||
them errors with `input param "X" needs a default value`. E.g.
|
||||
`CaptureViewport` minimal call is `{"captureTransform": null,
|
||||
"annotations": null, "bShowUI": false}`.
|
||||
- **Schema `required` is literal.** `find_actors` marks `name`, `tag`,
|
||||
`collision_channels` required even though they're semantically optional —
|
||||
pass `""` / `[]` to mean "any".
|
||||
- **Property names are camelCase with UE's `b` prefix intact** at this
|
||||
reflection layer: `bUseTemperature`, `bAtmosphereSunLight`, `fogDensity`,
|
||||
`bRealTimeCapture`, `mobility`. Writing `useTemperature` does NOT error
|
||||
the whole call — the response names each property that could not be set
|
||||
(schema-in-error style; READ error text, it lists the exact failures and
|
||||
often the full input schema).
|
||||
- **Object references travel as `{"refPath": "<soft object path>"}`**
|
||||
everywhere (actors, classes, components). Class refs use
|
||||
`/Script/Module.Class` (e.g. `/Script/Engine.PointLight`); actor refs are
|
||||
the full path (`/Temp/Untitled_1.Untitled_1:PersistentLevel.DirectionalLight_UAID_...`).
|
||||
Spawn/find tools RETURN refPaths — capture and reuse them.
|
||||
- **`ObjectTools.set_properties` takes `values` as a JSON *string***, not
|
||||
an object: `{"instance": {"refPath": ...}, "values":
|
||||
"{\"intensity\": 10.0}"}`. `get_properties` likewise returns a JSON
|
||||
string inside `returnValue`. Double-encode/decode accordingly.
|
||||
- Primitive results arrive wrapped as `{"returnValue": ...}` inside the
|
||||
text content block.
|
||||
|
||||
### HTTP wire behavior (for raw clients / debugging)
|
||||
|
||||
- `initialize` → plain JSON response + `Mcp-Session-Id` header you must
|
||||
echo on every subsequent request; `notifications/initialized` → 202
|
||||
empty; `tools/call` → **`text/event-stream`**: the result arrives as an
|
||||
`event: message` + `data: <jsonrpc>` frame only when the game thread
|
||||
finishes. A client that treats the response as plain JSON reads an empty
|
||||
body. Send `Accept: application/json, text/event-stream` always.
|
||||
|
||||
## Shipped toolsets
|
||||
|
||||
The registry is project-dependent; `describe_toolset` on the live server is
|
||||
the only source of truth for schemas. The core surface below is verified
|
||||
against EditorToolset's source in the 5.8 install (Python:
|
||||
`.../EditorToolset/Content/Python/editor_toolset/toolsets/`; C++:
|
||||
`EditorAppToolset.h`).
|
||||
|
||||
**EditorToolset plugin (the core), Python toolsets** (live-verified on 5.8;
|
||||
qualified prefix `editor_toolset.toolsets.<module>.<Class>`):
|
||||
|
||||
| Toolset | Verified tools (subset) |
|
||||
|---|---|
|
||||
| `scene.SceneTools` | `load_level`, `get_current_level`, `find_actors` (by name/type/tag/bounds), `add_to_scene_from_class`, `add_to_scene_from_asset`, `remove_from_scene`, `save_actor`, `create_level_instance`, folders |
|
||||
| `actor.ActorTools` | `get_label`/`set_label`, tags, `get_actor_transform`/`set_actor_transform` (`xform` fields optional = "don't change"), parenting, components |
|
||||
| `primitive.PrimitiveTools` | `add_cube` (dimensions), `add_sphere` (radius), `add_cylinder`/`add_cone` (radius+height) — adds StaticMeshComponents with `local_transform` to a host actor: spawn `/Script/Engine.Actor`, then compose. The fastest blocking path, zero asset dependencies |
|
||||
| `object.ObjectTools` | `list_properties` (returns full JSON schema of every property), `get_properties`/`set_properties` (JSON-string `values`), `reset_properties` (restore defaults — also your rollback), `get_class`, `search_subclasses` |
|
||||
| `material_instance.MaterialInstanceTools` | `create`, `list_parameters`, `get/set_scalar_parameter`, `get/set_vector_parameter` |
|
||||
| `asset.AssetTools` | `find_assets`, `load_asset`, `exists`, `save_assets`, `is_dirty`, `get_dependencies`/`get_referencers` (check before delete!), `delete`, `move`, `duplicate`, folders, `read_file`/`write_file` (project-scoped) |
|
||||
| `blueprint.BlueprintTools` (+ dsl/layout/node) | Blueprint authoring |
|
||||
| `material.MaterialTools`, `static_mesh.StaticMeshTools`, `texture.TextureTools`, `data_table.DataTableTools`, … | per-asset-type operations |
|
||||
| `programmatic.ProgrammaticToolset` | **the batching escape hatch** — see below |
|
||||
|
||||
**`EditorToolset.EditorAppToolset` (C++, same plugin) — the agent's eyes
|
||||
(full live list):** `CaptureViewport`, `CaptureEditorImage`,
|
||||
`CaptureAssetImage`, `GetCameraTransform`/`SetCameraTransform`,
|
||||
`GetSelectedActors`/`SelectActors`/`FocusOnActors`/`GetVisibleActors`,
|
||||
`WorldPosToScreenCoords`/`ScreenCoordsToWorld`,
|
||||
`GetSelectedAssets`/`SelectAssets`,
|
||||
`GetContentBrowserPath`/`SetContentBrowserPath`, `OpenEditorForAsset`,
|
||||
`GetOpenAssets`, `SearchCVars`, `StartPIE`/`StopPIE`/`IsPIERunning`.
|
||||
|
||||
`CaptureViewport` specifics (live-verified): args `{"captureTransform":
|
||||
<transform-or-null>, "annotations": <config-or-null>, "bShowUI": false}`.
|
||||
Returns base64 PNG (decode + save it yourself) plus camera
|
||||
location/rotation/FOV. `captureTransform` captures from any pose WITHOUT
|
||||
moving the user's viewport — use it as a virtual camera. Annotation config
|
||||
`{"gridSpacingCm": 500, "gridExtentCm": 3000, "gridHeight": <ground Z>,
|
||||
"labelActors": true}` overlays a projected ground grid and actor callouts;
|
||||
**grid coordinate labels are in METERS** (world cm ÷ 100). Use annotated
|
||||
captures for placement work, clean ones for beauty checks.
|
||||
|
||||
Also confirmed live: `ToolsetRegistry.AgentSkillToolset`,
|
||||
`EditorToolset.LogsToolset` (read Output Log + set verbosity — useful for
|
||||
self-debugging), `SemanticSearchToolset` (hybrid vector+BM25 asset search),
|
||||
five `NiagaraToolsets.NiagaraToolset_*` groups, `PCGToolset` (+Spatial),
|
||||
`UMGToolSet`, three `GASToolsets.*`, `AutomationTestToolset`,
|
||||
`ConfigSettingsToolset` (read/write Project Settings & Editor Preferences
|
||||
sections by schema — the remote path to exposure defaults, rendering
|
||||
settings, etc.), `SlateInspectorToolset`, `PluginToolset`,
|
||||
`animation_toolset.toolsets.sequencer.SequencerTools` + keyframing/
|
||||
controlrig/outliner siblings, `aimodule_toolset` BehaviorTreeTools,
|
||||
`state_tree_toolset` StateTreeTools, and more — 67 toolsets on a blank
|
||||
project with AllToolsets enabled.
|
||||
|
||||
Known gap: no mesh-modelling tools — spawn/place/instance existing meshes,
|
||||
yes; author new geometry, no. The supported route to parametric geometry is
|
||||
a custom Python toolset wrapping **Geometry Script** (`UDynamicMesh`:
|
||||
append box/cylinder/sphere, booleans, then `Create New Static Mesh Asset
|
||||
from Mesh` to bake an `SM_` asset). For organic/sculpted meshes, model in
|
||||
Blender and import.
|
||||
|
||||
First-session move: `list_toolsets`, then `describe_toolset` each group you
|
||||
plan to use, and keep those schemas in working memory for the session.
|
||||
|
||||
## ProgrammaticToolset — sanctioned batching
|
||||
|
||||
The serial-call rule makes N-step edits slow over the wire. The shipped
|
||||
answer is `ProgrammaticToolset` (verified in `programmatic.py`):
|
||||
|
||||
1. `get_execution_environment` — **mandatory first call** (the tool's own
|
||||
docstring requires it); returns the allowed modules, script constraints,
|
||||
and usage instructions.
|
||||
2. `execute_tool_script(script)` — runs a **sandboxed** Python script that
|
||||
defines `run() -> dict`. Inside, you call other registered tools
|
||||
programmatically and glue them with logic — one MCP round-trip for a
|
||||
whole loop (e.g. spawn 20 actors with computed transforms).
|
||||
|
||||
Sandbox facts (from source): allowed imports are `json`, `math`,
|
||||
`datetime`, `copy`, `re`, `time` only; `open()` is restricted to
|
||||
project-contained paths; scripts run inside an editor **transaction scope**
|
||||
(undo-friendly); it is tool orchestration, NOT general Python — arbitrary
|
||||
`unreal.*` calls are not the contract. Data returns via `run()`'s dict.
|
||||
|
||||
Use it whenever a recipe loop exceeds ~5 homogeneous calls; keep one-off
|
||||
edits as plain `call_tool`.
|
||||
|
||||
## Project Agent Skills (AgentSkillToolset)
|
||||
|
||||
Projects and plugins can register **Agent Skills** — named instruction
|
||||
bundles for project-specific conventions and workflows (naming schemes,
|
||||
folder layout, canonical multi-step sequences). They are NOT listed by
|
||||
`list_toolsets`; reach them through `call_tool`:
|
||||
|
||||
1. `AgentSkillToolset.ListSkills` → names + descriptions of registered
|
||||
skills.
|
||||
2. If one matches the task, `AgentSkillToolset.GetSkills` on it → full
|
||||
instructions, then FOLLOW THEM — a project skill exists precisely
|
||||
because the project's way differs from the obvious way, and it takes
|
||||
precedence over this skill's generic defaults.
|
||||
|
||||
Check at the start of unfamiliar work in any project, not just once ever.
|
||||
|
||||
## Seeing your work: screenshots and captures
|
||||
|
||||
An agent that can't see the viewport is flying blind. In order of preference:
|
||||
|
||||
1. **`EditorAppToolset.CaptureViewport`** (confirmed shipped) — returns the
|
||||
image through MCP as base64 PNG with camera metadata; supports capturing
|
||||
from an arbitrary transform without disturbing the user's viewport, and
|
||||
an optional annotation overlay (world-space meter grid + actor callouts)
|
||||
for spatial-placement work. This is the default verification tool.
|
||||
2. **Console `HighResShot` via any console/exec tool** when you need
|
||||
resolutions beyond the viewport: `HighResShot 3840x2160` writes to
|
||||
`<Project>/Saved/Screenshots/<Platform>/` on the EDITOR host's
|
||||
filesystem; read the file back (same machine) with `vision_analyze`.
|
||||
3. **Custom toolset escape hatch** for anything else (e.g. camera-actor
|
||||
framed captures with MRQ-quality settings).
|
||||
|
||||
Always `vision_analyze` the capture and art-direct against the brief before
|
||||
declaring a milestone done.
|
||||
|
||||
## Plugin configuration reference
|
||||
|
||||
Editor Preferences > General > Model Context Protocol:
|
||||
|
||||
| Property | Default | Notes |
|
||||
|---|---|---|
|
||||
| Auto Start Server | `false` | Turn on for frictionless sessions |
|
||||
| Server Port Number | `8000` | Change on conflict; mirror in Hermes config url |
|
||||
| Server URL Path | `/mcp` | Same |
|
||||
| Enable Tool Search | `true` | Keep on (see above) |
|
||||
|
||||
Console commands (editor console, backtick):
|
||||
|
||||
| Command | Effect |
|
||||
|---|---|
|
||||
| `ModelContextProtocol.StartServer [port]` | Start server (optional port override) |
|
||||
| `ModelContextProtocol.StopServer` | Stop server, close all sessions |
|
||||
| `ModelContextProtocol.RefreshTools` | Re-poll toolset providers — run after authoring/hot-reload/Game-Feature activation |
|
||||
| `ModelContextProtocol.GenerateClientConfig <Client\|All>` | Write client config files (ClaudeCode/Cursor/VSCode/Gemini/Codex) — NOT used for Hermes |
|
||||
|
||||
Command-line flags for launching the editor pre-configured:
|
||||
`-ModelContextProtocolStartServer` (force start regardless of preference),
|
||||
`-ModelContextProtocolPort=N`.
|
||||
|
||||
Console variables:
|
||||
|
||||
| CVar | Default | Notes |
|
||||
|---|---|---|
|
||||
| `ModelContextProtocol.WrapPODToolResultsInObject` | `true` | Primitive results wrapped as `{"result": ...}` |
|
||||
| `ModelContextProtocol.AudioResultOggFormat` | `false` | OGG instead of WAV for audio results |
|
||||
| `ModelContextProtocol.ProgressIntervalSeconds` | `1.0` | Min interval between progress notifications |
|
||||
| `ModelContextProtocol.PaginationPageSize` | `0` | 0 = no pagination of list results |
|
||||
| `ModelContextProtocol.EnableAnalytics` | `true` | Epic telemetry gate |
|
||||
|
||||
## Debugging the connection
|
||||
|
||||
- **Output Log** at editor startup logs bind address/port/path — first stop
|
||||
when the server seems absent. Port-in-use and missing-dependency failures
|
||||
surface here.
|
||||
- **Log verbosity:** `Log LogModelContextProtocol Verbose` in the editor
|
||||
console.
|
||||
- **MCP Inspector** (`npx @modelcontextprotocol/inspector`, point at
|
||||
`http://127.0.0.1:8000/mcp`, transport "Streamable HTTP") lists every
|
||||
advertised tool with schemas and offers form-based invocation — isolates
|
||||
"server broken" from "agent calling it wrong".
|
||||
- **After Live Coding / authoring:** connected clients can hold stale
|
||||
schemas. `ModelContextProtocol.RefreshTools`, then reconnect (new Hermes
|
||||
session) if schemas still look stale.
|
||||
|
||||
## Extending the surface: custom toolsets
|
||||
|
||||
When shipped tools don't cover an operation, the supported path is authoring
|
||||
a project toolset — NOT trying to smuggle arbitrary code through unrelated
|
||||
tools. Python toolsets are first-class and hot-loadable, so prefer them.
|
||||
|
||||
### Python toolset (recommended)
|
||||
|
||||
Any enabled plugin's `Content/Python/` directory (or the project's) can hold
|
||||
toolset modules; the registry discovers them at startup. Shape (mirrors
|
||||
Epic's shipped `ActorTools`):
|
||||
|
||||
```python
|
||||
import unreal
|
||||
import toolset_registry
|
||||
|
||||
@unreal.uclass()
|
||||
class MySceneTools(unreal.ToolsetDefinition):
|
||||
"""One-line toolset description — surfaces to the agent in list_toolsets."""
|
||||
|
||||
@toolset_registry.tool_call
|
||||
@staticmethod
|
||||
def take_viewport_screenshot(filename: str, width: int, height: int) -> str:
|
||||
"""Capture the active viewport to Saved/Screenshots.
|
||||
|
||||
Args:
|
||||
filename: Base filename without extension.
|
||||
width: Output width in pixels.
|
||||
height: Output height in pixels.
|
||||
|
||||
Returns:
|
||||
Absolute path the screenshot will be written to.
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
Conventions that matter (they generate the schema the agent sees):
|
||||
|
||||
- `@unreal.uclass()` on the class; inherit `unreal.ToolsetDefinition`.
|
||||
- Class docstring = toolset description; write it for an agent audience.
|
||||
- Each advertised function: `@toolset_registry.tool_call` + `@staticmethod`.
|
||||
Functions without the decorator stay private.
|
||||
- Type hints (`str`, `bool`, `list[str]`, `unreal.Actor`, dataclasses) drive
|
||||
the JSON Schema; Google-style docstrings (`Args:`/`Returns:`) become the
|
||||
parameter descriptions. Write them with API-surface care.
|
||||
- Small, single-responsibility tools with structured return types beat
|
||||
mega-tools returning prose. Data leaves the tool via its RETURN VALUE —
|
||||
`print()`/stdout go to the UE log, not back over MCP.
|
||||
|
||||
After authoring: `ModelContextProtocol.RefreshTools` in the editor console,
|
||||
then re-`list_toolsets` from Hermes. Users on Claude Code can scaffold with
|
||||
the `create-toolset` skill from Epic's `unreal-mcp` plugin pack; the
|
||||
conventions above still apply.
|
||||
|
||||
### C++ toolset
|
||||
|
||||
Derive from `UToolsetDefinition`, mark the class `UCLASS(BlueprintType,
|
||||
Hidden)`, expose static `UFUNCTION(meta = (AICallable))` methods; doc
|
||||
comments reflect into schemas. Use only when Python can't reach the API,
|
||||
when reflected `USTRUCT` signatures are needed, or when the Python boundary
|
||||
cost matters. Exclude a function with `meta = (AIIgnore)`. Live Coding
|
||||
propagates edited function bodies, but NEW `UFUNCTION`s require a full
|
||||
editor restart. There is also a direct-registration path
|
||||
(`IModelContextProtocolTool` + `IModelContextProtocolModule::AddTool()`) for
|
||||
runtime-shaped tools; caller owns deregistration.
|
||||
|
||||
## Runtime and cooked builds
|
||||
|
||||
The server is editor-hosted by default but not editor-only: runtime modules
|
||||
can host it in cooked builds via `IModelContextProtocolModule::StartServer()`.
|
||||
The Toolset Registry adapter (and the three tool-search meta-tools) are
|
||||
editor-only, though — cooked-build tools must be registered explicitly
|
||||
through `AddTool()` and are advertised eagerly. MCP Resources and Prompts are
|
||||
not advertised by any shipping toolset.
|
||||
|
||||
## Known limitations (5.8, experimental)
|
||||
|
||||
- HTTP + SSE transports only; loopback-only listener; non-loopback `Origin`
|
||||
headers rejected; no auth layer. Not safe beyond the local machine.
|
||||
- Serial game-thread execution: overlapping calls unsupported; editor UI
|
||||
blocks during each call.
|
||||
- Feature-incomplete by Epic's own labeling; APIs and data formats subject
|
||||
to change without notice.
|
||||
- Live Coding does not propagate new `UFUNCTION` declarations.
|
||||
Reference in New Issue
Block a user