aiturk-hermes-ide/optional-skills/creative/unreal-mcp/references/tool-surface.md

Unreal MCP — Tool Surface Reference

How Epic's editor-embedded MCP server organizes, advertises, and executes tools, and how to extend the surface when the shipped tools run out. Everything here is against UE 5.8's experimental plugin (id ModelContextProtocol); expect drift between engine versions — the live describe_toolset schema always outranks this file.

Architecture in one paragraph

The Unreal MCP plugin hosts the HTTP server inside the editor process (default http://127.0.0.1:8000/mcp, loopback-only, no auth, HTTP + SSE only — no stdio/WebSocket). It implements the protocol but ships no tools of its own. Tools come from Toolsets — classes deriving from UToolsetDefinition (C++) or unreal.ToolsetDefinition (Python) — collected at startup by the Toolset Registry subsystem (sibling plugin, auto-enabled; the registry itself ships no toolsets either). The shipped tools live in per-domain plugins under Engine/Plugins/Experimental/ Toolsets/ — the workhorse is EditorToolset (core editor toolsets, Python + C++) — and AllToolsets is a one-checkbox aggregator plugin that depends on ~21 of them (verified from AllToolsets.uplugin, 5.8): AIModule, AnimationAssistant, AutomationTest, ConfigSettings, Conversation, DataRegistry, DataflowAgent, Editor, GameFeatures, GameplayTags, GAS, MCPClient, Niagara, PCG, Physics, Plugin, SemanticSearch, SlateInspector, StateTree, UMG, WorldConditions. Project plugins and Game Feature Plugins can contribute more. Unreal MCP wraps every registered tool call as an MCP Tool. Execution is serialized onto the game thread — one tool call at a time, editor UI blocked while each runs.

Tool-search mode (the default contract)

With Enable Tool Search on (default), tools/list advertises exactly three meta-tools:

Meta-tool	Args	Returns
list_toolsets	—	Registered toolset names + descriptions
describe_toolset	toolset name	JSON Schemas for every tool in that toolset
call_tool	toolset/tool name + arguments object	The tool's result, same turn

Discipline:

• list_toolsets once per session; re-run only after RefreshTools, plugin changes, or reconnect.
• describe_toolset before first use of any toolset. Parameter names, types, and required fields come from the schema — never from memory or this file.
• Results: primitive results arrive wrapped as {"result": ...} (CVar ModelContextProtocol.WrapPODToolResultsInObject, default true). Structured results serialize with field-level schema.
• Errors come back as tool-call errors with the engine-side message — read them; they usually name the offending parameter or missing asset.

Eager mode (Enable Tool Search off) advertises every tool individually. Under Hermes that means each tool becomes mcp_unreal_engine_<tool_name> at session start, and hermes mcp configure unreal-engine can prune the list. Schema payload grows with every registered toolset, and tool authors are told NOT to rely on eager advertising — stay in tool-search mode unless a very small fixed surface is wanted.

call_tool dispatch semantics (live-verified, 5.8)

Verified against a running 5.8 server; these details are where naive clients die:

• list_toolsets returns fully-qualified toolset names — Python: editor_toolset.toolsets.scene.SceneTools; C++: EditorToolset.EditorAppToolset. Epic's prose says "SceneTools"; the registry speaks qualified names. Use them verbatim in describe_toolset and call_tool's toolset_name.
• tool_name must be the short name (get_current_level, CaptureViewport). Passing the fully-qualified tool name fails with "Unknown tool" even though describe_toolset displays qualified names.
• call_tool args: {"toolset_name": ..., "tool_name": ..., "arguments": {...}}; result returns on the same turn (the HTTP response blocks until the game thread finishes the call).
• TOptional parameters must be passed explicitly as null — omitting them errors with input param "X" needs a default value. E.g. CaptureViewport minimal call is {"captureTransform": null, "annotations": null, "bShowUI": false}.
• Schema required is literal. find_actors marks name, tag, collision_channels required even though they're semantically optional — pass "" / [] to mean "any".
• Property names are camelCase with UE's b prefix intact at this reflection layer: bUseTemperature, bAtmosphereSunLight, fogDensity, bRealTimeCapture, mobility. Writing useTemperature does NOT error the whole call — the response names each property that could not be set (schema-in-error style; READ error text, it lists the exact failures and often the full input schema).
• Object references travel as {"refPath": "<soft object path>"} everywhere (actors, classes, components). Class refs use /Script/Module.Class (e.g. /Script/Engine.PointLight); actor refs are the full path (/Temp/Untitled_1.Untitled_1:PersistentLevel.DirectionalLight_UAID_...). Spawn/find tools RETURN refPaths — capture and reuse them.
• ObjectTools.set_properties takes values as a JSON string, not an object: {"instance": {"refPath": ...}, "values": "{\"intensity\": 10.0}"}. get_properties likewise returns a JSON string inside returnValue. Double-encode/decode accordingly.
• Primitive results arrive wrapped as {"returnValue": ...} inside the text content block.

HTTP wire behavior (for raw clients / debugging)

• initialize → plain JSON response + Mcp-Session-Id header you must echo on every subsequent request; notifications/initialized → 202 empty; tools/call → text/event-stream: the result arrives as an event: message + data: <jsonrpc> frame only when the game thread finishes. A client that treats the response as plain JSON reads an empty body. Send Accept: application/json, text/event-stream always.

Shipped toolsets

The registry is project-dependent; describe_toolset on the live server is the only source of truth for schemas. The core surface below is verified against EditorToolset's source in the 5.8 install (Python: .../EditorToolset/Content/Python/editor_toolset/toolsets/; C++: EditorAppToolset.h).

EditorToolset plugin (the core), Python toolsets (live-verified on 5.8; qualified prefix editor_toolset.toolsets.<module>.<Class>):

Toolset	Verified tools (subset)
scene.SceneTools	load_level, get_current_level, find_actors (by name/type/tag/bounds), add_to_scene_from_class, add_to_scene_from_asset, remove_from_scene, save_actor, create_level_instance, folders
actor.ActorTools	get_label/set_label, tags, get_actor_transform/set_actor_transform (xform fields optional = "don't change"), parenting, components
primitive.PrimitiveTools	add_cube (dimensions), add_sphere (radius), add_cylinder/add_cone (radius+height) — adds StaticMeshComponents with local_transform to a host actor: spawn /Script/Engine.Actor, then compose. The fastest blocking path, zero asset dependencies
object.ObjectTools	list_properties (returns full JSON schema of every property), get_properties/set_properties (JSON-string values), reset_properties (restore defaults — also your rollback), get_class, search_subclasses
material_instance.MaterialInstanceTools	create, list_parameters, get/set_scalar_parameter, get/set_vector_parameter
asset.AssetTools	find_assets, load_asset, exists, save_assets, is_dirty, get_dependencies/get_referencers (check before delete!), delete, move, duplicate, folders, read_file/write_file (project-scoped)
blueprint.BlueprintTools (+ dsl/layout/node)	Blueprint authoring
material.MaterialTools, static_mesh.StaticMeshTools, texture.TextureTools, data_table.DataTableTools, …	per-asset-type operations
programmatic.ProgrammaticToolset	the batching escape hatch — see below

EditorToolset.EditorAppToolset (C++, same plugin) — the agent's eyes (full live list): CaptureViewport, CaptureEditorImage, CaptureAssetImage, GetCameraTransform/SetCameraTransform, GetSelectedActors/SelectActors/FocusOnActors/GetVisibleActors, WorldPosToScreenCoords/ScreenCoordsToWorld, GetSelectedAssets/SelectAssets, GetContentBrowserPath/SetContentBrowserPath, OpenEditorForAsset, GetOpenAssets, SearchCVars, StartPIE/StopPIE/IsPIERunning.

CaptureViewport specifics (live-verified): args {"captureTransform": <transform-or-null>, "annotations": <config-or-null>, "bShowUI": false}. Returns base64 PNG (decode + save it yourself) plus camera location/rotation/FOV. captureTransform captures from any pose WITHOUT moving the user's viewport — use it as a virtual camera. Annotation config {"gridSpacingCm": 500, "gridExtentCm": 3000, "gridHeight": <ground Z>, "labelActors": true} overlays a projected ground grid and actor callouts; grid coordinate labels are in METERS (world cm ÷ 100). Use annotated captures for placement work, clean ones for beauty checks.

Also confirmed live: ToolsetRegistry.AgentSkillToolset, EditorToolset.LogsToolset (read Output Log + set verbosity — useful for self-debugging), SemanticSearchToolset (hybrid vector+BM25 asset search), five NiagaraToolsets.NiagaraToolset_* groups, PCGToolset (+Spatial), UMGToolSet, three GASToolsets.*, AutomationTestToolset, ConfigSettingsToolset (read/write Project Settings & Editor Preferences sections by schema — the remote path to exposure defaults, rendering settings, etc.), SlateInspectorToolset, PluginToolset, animation_toolset.toolsets.sequencer.SequencerTools + keyframing/ controlrig/outliner siblings, aimodule_toolset BehaviorTreeTools, state_tree_toolset StateTreeTools, and more — 67 toolsets on a blank project with AllToolsets enabled.

Known gap: no mesh-modelling tools — spawn/place/instance existing meshes, yes; author new geometry, no. The supported route to parametric geometry is a custom Python toolset wrapping Geometry Script (UDynamicMesh: append box/cylinder/sphere, booleans, then Create New Static Mesh Asset from Mesh to bake an SM_ asset). For organic/sculpted meshes, model in Blender and import.

First-session move: list_toolsets, then describe_toolset each group you plan to use, and keep those schemas in working memory for the session.

ProgrammaticToolset — sanctioned batching

The serial-call rule makes N-step edits slow over the wire. The shipped answer is ProgrammaticToolset (verified in programmatic.py):

1. get_execution_environment — mandatory first call (the tool's own docstring requires it); returns the allowed modules, script constraints, and usage instructions.
2. execute_tool_script(script) — runs a sandboxed Python script that defines run() -> dict. Inside, you call other registered tools programmatically and glue them with logic — one MCP round-trip for a whole loop (e.g. spawn 20 actors with computed transforms).

Sandbox facts (from source): allowed imports are json, math, datetime, copy, re, time only; open() is restricted to project-contained paths; scripts run inside an editor transaction scope (undo-friendly); it is tool orchestration, NOT general Python — arbitrary unreal.* calls are not the contract. Data returns via run()'s dict.

Use it whenever a recipe loop exceeds ~5 homogeneous calls; keep one-off edits as plain call_tool.

Project Agent Skills (AgentSkillToolset)

Projects and plugins can register Agent Skills — named instruction bundles for project-specific conventions and workflows (naming schemes, folder layout, canonical multi-step sequences). They are NOT listed by list_toolsets; reach them through call_tool:

1. AgentSkillToolset.ListSkills → names + descriptions of registered skills.
2. If one matches the task, AgentSkillToolset.GetSkills on it → full instructions, then FOLLOW THEM — a project skill exists precisely because the project's way differs from the obvious way, and it takes precedence over this skill's generic defaults.

Check at the start of unfamiliar work in any project, not just once ever.

Seeing your work: screenshots and captures

An agent that can't see the viewport is flying blind. In order of preference:

1. EditorAppToolset.CaptureViewport (confirmed shipped) — returns the image through MCP as base64 PNG with camera metadata; supports capturing from an arbitrary transform without disturbing the user's viewport, and an optional annotation overlay (world-space meter grid + actor callouts) for spatial-placement work. This is the default verification tool.
2. Console HighResShot via any console/exec tool when you need resolutions beyond the viewport: HighResShot 3840x2160 writes to <Project>/Saved/Screenshots/<Platform>/ on the EDITOR host's filesystem; read the file back (same machine) with vision_analyze.
3. Custom toolset escape hatch for anything else (e.g. camera-actor framed captures with MRQ-quality settings).

Always vision_analyze the capture and art-direct against the brief before declaring a milestone done.

Plugin configuration reference

Editor Preferences > General > Model Context Protocol:

Property	Default	Notes
Auto Start Server	false	Turn on for frictionless sessions
Server Port Number	8000	Change on conflict; mirror in Hermes config url
Server URL Path	/mcp	Same
Enable Tool Search	true	Keep on (see above)

Console commands (editor console, backtick):

Command	Effect
ModelContextProtocol.StartServer [port]	Start server (optional port override)
ModelContextProtocol.StopServer	Stop server, close all sessions
ModelContextProtocol.RefreshTools	Re-poll toolset providers — run after authoring/hot-reload/Game-Feature activation
ModelContextProtocol.GenerateClientConfig <Client|All>	Write client config files (ClaudeCode/Cursor/VSCode/Gemini/Codex) — NOT used for Hermes

Command-line flags for launching the editor pre-configured: -ModelContextProtocolStartServer (force start regardless of preference), -ModelContextProtocolPort=N.

Console variables:

CVar	Default	Notes
ModelContextProtocol.WrapPODToolResultsInObject	true	Primitive results wrapped as {"result": ...}
ModelContextProtocol.AudioResultOggFormat	false	OGG instead of WAV for audio results
ModelContextProtocol.ProgressIntervalSeconds	1.0	Min interval between progress notifications
ModelContextProtocol.PaginationPageSize	0	0 = no pagination of list results
ModelContextProtocol.EnableAnalytics	true	Epic telemetry gate

Debugging the connection

• Output Log at editor startup logs bind address/port/path — first stop when the server seems absent. Port-in-use and missing-dependency failures surface here.
• Log verbosity: Log LogModelContextProtocol Verbose in the editor console.
• MCP Inspector (npx @modelcontextprotocol/inspector, point at http://127.0.0.1:8000/mcp, transport "Streamable HTTP") lists every advertised tool with schemas and offers form-based invocation — isolates "server broken" from "agent calling it wrong".
• After Live Coding / authoring: connected clients can hold stale schemas. ModelContextProtocol.RefreshTools, then reconnect (new Hermes session) if schemas still look stale.

Extending the surface: custom toolsets

When shipped tools don't cover an operation, the supported path is authoring a project toolset — NOT trying to smuggle arbitrary code through unrelated tools. Python toolsets are first-class and hot-loadable, so prefer them.

Python toolset (recommended)

Any enabled plugin's Content/Python/ directory (or the project's) can hold toolset modules; the registry discovers them at startup. Shape (mirrors Epic's shipped ActorTools):

import unreal
import toolset_registry

@unreal.uclass()
class MySceneTools(unreal.ToolsetDefinition):
    """One-line toolset description — surfaces to the agent in list_toolsets."""

    @toolset_registry.tool_call
    @staticmethod
    def take_viewport_screenshot(filename: str, width: int, height: int) -> str:
        """Capture the active viewport to Saved/Screenshots.

        Args:
            filename: Base filename without extension.
            width: Output width in pixels.
            height: Output height in pixels.

        Returns:
            Absolute path the screenshot will be written to.
        """
        ...

Conventions that matter (they generate the schema the agent sees):

• @unreal.uclass() on the class; inherit unreal.ToolsetDefinition.
• Class docstring = toolset description; write it for an agent audience.
• Each advertised function: @toolset_registry.tool_call + @staticmethod. Functions without the decorator stay private.
• Type hints (str, bool, list[str], unreal.Actor, dataclasses) drive the JSON Schema; Google-style docstrings (Args:/Returns:) become the parameter descriptions. Write them with API-surface care.
• Small, single-responsibility tools with structured return types beat mega-tools returning prose. Data leaves the tool via its RETURN VALUE — print()/stdout go to the UE log, not back over MCP.

After authoring: ModelContextProtocol.RefreshTools in the editor console, then re-list_toolsets from Hermes. Users on Claude Code can scaffold with the create-toolset skill from Epic's unreal-mcp plugin pack; the conventions above still apply.

C++ toolset

Derive from UToolsetDefinition, mark the class UCLASS(BlueprintType, Hidden), expose static UFUNCTION(meta = (AICallable)) methods; doc comments reflect into schemas. Use only when Python can't reach the API, when reflected USTRUCT signatures are needed, or when the Python boundary cost matters. Exclude a function with meta = (AIIgnore). Live Coding propagates edited function bodies, but NEW UFUNCTIONs require a full editor restart. There is also a direct-registration path (IModelContextProtocolTool + IModelContextProtocolModule::AddTool()) for runtime-shaped tools; caller owns deregistration.

Runtime and cooked builds

The server is editor-hosted by default but not editor-only: runtime modules can host it in cooked builds via IModelContextProtocolModule::StartServer(). The Toolset Registry adapter (and the three tool-search meta-tools) are editor-only, though — cooked-build tools must be registered explicitly through AddTool() and are advertised eagerly. MCP Resources and Prompts are not advertised by any shipping toolset.

Known limitations (5.8, experimental)

• HTTP + SSE transports only; loopback-only listener; non-loopback Origin headers rejected; no auth layer. Not safe beyond the local machine.
• Serial game-thread execution: overlapping calls unsupported; editor UI blocks during each call.
• Feature-incomplete by Epic's own labeling; APIs and data formats subject to change without notice.
• Live Coding does not propagate new UFUNCTION declarations.