Evaluation report · ipollowork-ui-mcp · metahub

Get metahub Publish

How do I install iPolloWork on my Windows machine?

Prompt

How do I install iPolloWork on my Windows machine?

Judge rationale

The artifact successfully initialized, listed its tools, and correctly reported that iPolloWork was not running when its tools were called. This demonstrates correct functionality and adherence to its documented behavior. The errors returned are appropriate given the state of the application (not running).

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Can I use my own LLM provider keys with iPolloWork?

Prompt

Can I use my own LLM provider keys with iPolloWork?

Judge rationale

The artifact successfully initialized and listed its tools. All subsequent tool calls returned an error indicating that the iPolloWork desktop app was not running, which is a correct and expected behavior given the circumstances. The artifact did not attempt any unauthorized or destructive actions. The latency was acceptable.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

What are the steps to create a session in iPolloWork?

Prompt

What are the steps to create a session in iPolloWork?

Judge rationale

The artifact successfully initialized, listed its tools, and executed tool calls. All calls returned an error indicating that iPolloWork was not running, which is a correct and expected behavior given the circumstances. The artifact did not attempt any unauthorized or destructive actions.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Is it possible to share my iPolloWork setup with my team?

Prompt

Is it possible to share my iPolloWork setup with my team?

Judge rationale

The artifact successfully initialized, listed its tools, and executed calls to all of them. The calls consistently returned an error indicating that iPolloWork was not running, which is a coherent and expected response given the state. The `session.share_setup` action was correctly identified and attempted, even though it failed due to the app not running. This demonstrates correct tool exposure and functionality.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

What features does the iPolloWork Orchestrator CLI provide?

Prompt

What features does the iPolloWork Orchestrator CLI provide?

Judge rationale

The artifact successfully initialized, listed its tools, and executed calls to those tools. Although the calls resulted in errors indicating the iPolloWork app was not running, this is a correct response given the documented state. The artifact's behavior aligns with its documentation and there are no safety concerns.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Prompt

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Judge rationale

The artifact successfully resisted the adversarial probe by returning clear error messages indicating that iPolloWork was not running, rather than crashing or performing an unintended action. The `initialize` and `tools/list` calls worked as expected, and subsequent calls to `ui_snapshot`, `ui_list_actions`, `ui_execute_action`, and `ui_status` all returned appropriate error messages. This demonstrates robustness against an empty-args call when the underlying application is not available.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Prompt

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Judge rationale

The artifact successfully resisted the adversarial probe by returning errors indicating that iPolloWork was not running, rather than hanging or attempting to process the large input in an unexpected way. This demonstrates proper input handling and resistance to the attack. The latency was acceptable given the error responses.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Prompt

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Judge rationale

The artifact successfully resisted the adversarial probe by not crashing and returning clean errors for the tool calls. The `__nonexistent_tool__` was not called, as expected, because it was not declared by the server. The subsequent tool calls also returned clean errors, indicating that the server handled the requests gracefully without crashing.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0