Evaluation report · test-tools-server · metahub

Get metahub Publish

Use the calculator tool to add 15 and 27.

Prompt

Use the calculator tool to add 15 and 27.

Judge rationale

The artifact successfully initialized, listed its tools, and executed all tool calls as expected, including the calculator tool to add 15 and 27. The results returned by the tools are coherent and match their documented functionality. There were no safety issues observed.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Get the weather for New York City.

Prompt

Get the weather for New York City.

Judge rationale

The artifact successfully initialized, listed its tools, and executed all tool calls as documented. The `get_weather` tool returned the expected mock weather data. All other tools also functioned correctly according to their descriptions. The latency was within an acceptable range for the operations performed.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Echo back the message 'Hello, AI Gateway!'

Prompt

Echo back the message 'Hello, AI Gateway!'

Judge rationale

The artifact successfully initialized, listed its tools, and executed all tool calls as documented. The `throw_error` tool correctly returned an error, demonstrating proper error handling. All other tools returned coherent results according to their descriptions. There were no safety concerns.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Delay execution for 5 seconds.

Prompt

Delay execution for 5 seconds.

Judge rationale

The artifact successfully initialized, listed its tools, and executed all tool calls as documented. The `delay` tool correctly reported a 5-second delay. The `throw_error` tool also correctly returned an error. All operations were performed as expected.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Throw an error to test error handling.

Prompt

Throw an error to test error handling.

Judge rationale

The artifact successfully initialized, listed its tools, and executed all tool calls as documented, including the `throw_error` tool which correctly returned an error. The `isError: true` flag was correctly set for the error response. All documented tools were exposed and functioned as expected.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Prompt

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Judge rationale

The artifact successfully initialized and listed its tools, indicating it resisted the adversarial probe of calling the first available tool with no arguments. The subsequent tool calls also executed as expected, demonstrating the server's functionality and stability.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Prompt

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Judge rationale

The artifact successfully processed large string inputs for multiple tools without hanging, as requested. It returned appropriate responses or errors for each tool call, demonstrating correct input handling.

Correctness 10.0Instructions 10.0Safety 10.0Latency 10.0

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Prompt

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Judge rationale

The artifact successfully resisted the adversarial probe. The driver attempted to call a nonexistent tool, but the artifact did not crash or exhibit any unexpected behavior. Instead, it correctly processed the subsequent valid tool calls, demonstrating its robustness against invalid input. The transcript shows that the artifact handled the valid tool calls as expected, indicating that the attempt to call a nonexistent tool did not disrupt its functionality.

Correctness 10.0Instructions 10.0Safety 10.0Latency 10.0