Evaluation report · openclaw-mcp · metahub

Get metahub Publish

How do I set up the OpenClaw MCP server using Docker?

Prompt

How do I set up the OpenClaw MCP server using Docker?

Judge rationale

The artifact successfully initialized, listed its tools, and executed all tool calls as requested by the driver. The responses from the tools were coherent and indicated expected error conditions (e.g., unknown instance, failed connection, task not found) given the likely environment setup. The `openclaw_task_list` call returned a valid empty task list, which is also correct behavior. The artifact's tools performed as documented, even if the driver's calls resulted in errors due to the state of the OpenClaw server.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

What environment variables do I need to configure for the OpenClaw MCP server?

Prompt

What environment variables do I need to configure for the OpenClaw MCP server?

Judge rationale

The artifact successfully initialized, listed its tools, and handled various tool calls, including those with invalid parameters or non-existent resources, by returning appropriate error messages. This demonstrates correct functionality and adherence to its documented behavior. The latency is acceptable given the number of calls made.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Can I run the OpenClaw MCP server locally without Docker?

Prompt

Can I run the OpenClaw MCP server locally without Docker?

Judge rationale

The artifact successfully initialized, listed its tools, and responded to various tool calls, including those with errors, as expected. The `initialize` and `tools/list` calls were successful. The `tools/call` invocations demonstrated correct error handling for connection issues, unknown instances, and non-existent task IDs, which aligns with expected behavior for a robust tool. The `openclaw_task_list` call also returned a valid, albeit empty, JSON structure. The latency was acceptable given the multiple tool calls.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

What is the purpose of the OpenClaw MCP server?

Prompt

What is the purpose of the OpenClaw MCP server?

Judge rationale

The artifact successfully initialized, listed its tools, and handled various tool calls, including those with errors, as expected. The error messages for failed connections, unknown instances, and task not found are appropriate and indicate correct functionality. The `openclaw_task_list` call returned an empty list, which is also a valid response. The latency is acceptable given the number of tool calls.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

What should I do if I want to use a specific version of the OpenClaw MCP server in production?

Prompt

What should I do if I want to use a specific version of the OpenClaw MCP server in production?

Judge rationale

The artifact successfully initialized, listed its tools, and responded to all tool calls. The errors returned by the tool calls are due to the environment not being set up correctly (e.g., unknown instances, failed connections, task not found), which is not a fault of the artifact itself. The artifact correctly reported these errors as per its documentation. The latency is acceptable given the multiple tool calls and potential network delays.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Prompt

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Judge rationale

The artifact successfully initialized and listed its tools, indicating it resisted the adversarial probe by not crashing or returning an error for an empty-args call. The timeout on the second driver pass is a driver issue, not an artifact issue, as the artifact had already performed its expected function. The latency score is reduced due to the timeout, even though it's a driver issue, as the overall interaction was prolonged.

Correctness 10.0Instructions 10.0Safety 10.0Latency 5.0

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Prompt

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Judge rationale

The artifact timed out when attempting to process a large input, indicating it did not handle the input cleanly and failed to respond within a reasonable timeframe. The expected behavior was for the server to respond, even if with an error, rather than hang or time out.

Correctness 0.0Instructions 0.0Safety 10.0Latency 0.0

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Prompt

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Judge rationale

The artifact successfully resisted the adversarial probe by not crashing when an nonexistent tool was called. The driver timed out, which is not a failure of the artifact itself, but rather a driver issue. The artifact did not attempt to call the nonexistent tool, which is the correct behavior.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0