Evaluation report · minecraft-mcp-server · metahub

Get metahub Publish

Instruct the bot to build a simple house with a door and windows.

Prompt

Instruct the bot to build a simple house with a door and windows.

Judge rationale

The artifact successfully initialized, listed its tools, and handled tool calls correctly, even when the Minecraft server was not running. The error messages returned by the tools are clear and informative, indicating that the artifact itself is functioning as expected despite the external dependency issue. The driver's choices to call multiple movement/position tools are not a reflection of the artifact's correctness.

Correctness 10.0Instructions 10.0Safety 10.0Latency 6.0

Ask the bot to explore the nearby forest and report back.

Prompt

Ask the bot to explore the nearby forest and report back.

Judge rationale

The artifact successfully initialized, listed its tools, and handled multiple tool calls. Although the calls resulted in connection errors, this indicates the artifact correctly reported the issue rather than failing silently or crashing. The error messages are consistent across all calls, suggesting a uniform and robust error handling mechanism. The exposed tools are coherent and match the documentation.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Request the bot to gather resources like wood and stone.

Prompt

Request the bot to gather resources like wood and stone.

Judge rationale

The artifact successfully initialized, listed its tools, and correctly reported connection errors when the tools were called. The error messages are clear and informative, indicating that the Minecraft server was not running, which is an external dependency. The artifact itself performed as expected given the circumstances. Latency is slightly higher due to the repeated connection attempts, but this is a consequence of the test setup rather than an artifact issue.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Tell the bot to create a farm with crops and a fence.

Prompt

Tell the bot to create a farm with crops and a fence.

Judge rationale

The artifact successfully initialized, listed its tools, and responded to tool calls. The errors received from the tool calls indicate that the Minecraft server was not running, which is an environmental issue outside the artifact's control. The artifact itself functioned as expected by attempting to execute the calls and returning the server's error message.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Instruct the bot to interact with a villager and trade items.

Prompt

Instruct the bot to interact with a villager and trade items.

Judge rationale

The artifact successfully initialized and listed its tools, demonstrating adherence to the protocol. However, all subsequent tool calls failed due to an inability to connect to the Minecraft server. This indicates a critical failure in the artifact's core functionality, as it cannot perform any actions without a server connection. The latency score is low because the artifact was unable to perform any meaningful actions due to the connection error.

Correctness 0.0Instructions 10.0Safety 10.0Latency 0.0

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Prompt

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Judge rationale

The artifact successfully resisted the adversarial probe by returning clear error messages for each tool call, indicating that it could not connect to the Minecraft server. This is the correct behavior when the server is not running or accessible, and it demonstrates that the artifact did not crash or behave unexpectedly due to the empty-args calls. The error messages are informative and guide the user on how to resolve the connection issue.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Prompt

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Judge rationale

The artifact successfully resisted the adversarial probe by not hanging and returning error messages for all tool calls, indicating it did not process any malicious input. The latency is acceptable given the multiple tool calls and error handling.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Prompt

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Judge rationale

The user attempted to call a nonexistent tool. The artifact correctly resisted this attack by not crashing and instead returning a clean error message indicating that it could not connect to the Minecraft server. This is the correct behavior for an adversarial probe, as it demonstrates the artifact's robustness against invalid tool calls.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0