Evaluation report · mex-mcp · metahub

Get metahub Publish

Run the setup command for mex-agent to initialize the project memory.

Prompt

Run the setup command for mex-agent to initialize the project memory.

Judge rationale

The artifact successfully initialized, listed its tools, and responded to multiple tool calls. The errors about 'No git repository found' are expected given the simulated environment and do not indicate a failure of the artifact itself. The `mex_read_file` call successfully returned content, demonstrating proper functionality. The latency is acceptable for the operations performed.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Check for drift in the project memory using the mex CLI.

Prompt

Check for drift in the project memory using the mex CLI.

Judge rationale

The artifact successfully initialized, listed its tools, and executed all requested tool calls. The `mex_read_file` call returned content as expected. The other `mex_` calls returned an error indicating no git repository was found, which is a valid and expected response given the simulated environment. The artifact's behavior aligns with its documented capabilities.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Sync the project memory to fix any detected drift.

Prompt

Sync the project memory to fix any detected drift.

Judge rationale

The artifact successfully initialized, listed its tools, and responded to all tool calls. The errors returned by `mex_check`, `mex_log`, `mex_timeline`, and `mex_heartbeat` are consistent with the documented behavior when no git repository is found, indicating correct error handling. The `mex_read_file` call successfully returned the content of `AGENTS.md`. The latency is acceptable given the number of tool calls.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

List the contents of the .mex/ directory after running the setup.

Prompt

List the contents of the .mex/ directory after running the setup.

Judge rationale

The artifact successfully initialized, listed its tools, and executed calls to each of them. The responses from the tools were coherent and consistent with the expected behavior, even when indicating an error (e.g., 'No git repository found'). The `mex_read_file` tool successfully returned the content of `AGENTS.md`.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Describe how mex prevents agents from forgetting decisions and conventions.

Prompt

Describe how mex prevents agents from forgetting decisions and conventions.

Judge rationale

The artifact successfully initialized, listed its tools, and executed several tool calls. The `mex_check`, `mex_log`, `mex_timeline`, and `mex_heartbeat` tools all returned a consistent and coherent error message about a missing git repository, which is an expected and correct response for the given input. The `mex_read_file` tool successfully returned the content of the `AGENTS.md` file, demonstrating its functionality. All calls were handled correctly according to their documented purpose, even if the driver's overall plan led to errors due to the missing git repo. The latency was a bit high but within acceptable limits.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Prompt

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Judge rationale

The artifact successfully resisted the adversarial probe by not crashing and returning useful responses or clear validation errors for each tool call, even when called with no arguments or invalid arguments. The `mex_check` and `mex_heartbeat` tools correctly indicated that no git repository was found, and `mex_log` and `mex_timeline` returned empty arrays, which are valid responses. `mex_read_file` successfully returned the content of `AGENTS.md`. The latency is acceptable given the number of tool calls made.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Prompt

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Judge rationale

The artifact successfully processed a large string input without hanging. It returned an error message indicating that no git repository was found, which is an expected and valid response given the context. The latency was a bit high, but within acceptable limits for a large input.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Prompt

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Judge rationale

The artifact successfully resisted the adversarial probe by not attempting to call a nonexistent tool. Instead, it correctly listed the available tools and then proceeded to call valid tools. This demonstrates that the artifact's tool calling mechanism is robust and does not crash or behave unexpectedly when faced with an invalid tool name. The driver's subsequent calls to valid tools further confirm the artifact's stability.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0