Get metahub Publish

Back to error-test-server

Evaluation report

Warnings

error-test-server · mcp65f33df· 3 months ago

Every check MetaHub ran on this artifact, grouped by area. Static checks run on the source at publish time; behavioral checks run the artifact in a sandbox and judge what it actually does.

Structural

5 passed

Repository is reachable
https://github.com/maximhq/bifrost @ 65f33df — ★ 6,426 · Go · Apache-2.0 · last push today
Manifest detected
kind=mcp slug=error-test-server · path=examples/mcps/error-test-server · source=package.json
Slug is URL-safe
"error-test-server" matches /^[a-z0-9][a-z0-9-]{0,62}$/
Slug is unique within kind
No collision found for mcp/error-test-server
Version is semver
1.0.0

Documentation

5 passed

Description quality
11 words · 69 chars — OK but more detail helps
README is present and substantial
13,097 chars · 11 sections · 7 code blocks
Tags / topics declared
16 total — ai-gateway, gateway, gateway-services, generative-ai, guardrails, llm (+10)
README has usage / example sections
found: Quick Start · Getting Started · Quick Start
Homepage / docs URL declared
https://www.getmaxim.ai/bifrost

Safety

2 passed1 warning

Dependencies: known vulnerabilitieswarn
2 high/critical advisories — @modelcontextprotocol/[email protected] (HIGH: GHSA-8r9q-7v3j-jr4g) · @modelcontextprotocol/[email protected] (HIGH: GHSA-w48q-cv73-mx4w)
Update the affected packages, then re-publish. (`npm audit` shows the same data locally.)
LICENSE file at repo root
LICENSE
No sensitive files in the repo
scanned for .env, credentials.json, *.pem, .ssh/, AWS / GCP configs — none found

Kind-specific

5 passed

MCP: launch path
bin: error-test-server
MCP: @modelcontextprotocol/sdk in dependencies
pinned at "1.0.4"
MCP: SDK version pinned
pinned at "1.0.4"
MCP: ESM package
"type": "module"
MCP: server surface
0 tools

Maintenance

3 passed

Recent activity
last push today
Tests detected
4 test directories · 484 test files
CI configuration detected
GitHub Actions (14 workflows)

Behavioral

4 passed1 warning

7.9/ 10 overall

8 cases passedSafe

Correctness8.8

Instructions8.8

Simulate a scenario where the server returns malformed JSON due to truncation.

Prompt

Simulate a scenario where the server returns malformed JSON due to truncation.

Judge rationale

The artifact successfully initialized, listed its tools, and the driver attempted to call a tool. The error message 'sh: 36: source: not found' is from the driver's environment, not the artifact itself, and indicates an issue with the driver's setup rather than the artifact's functionality. The artifact itself appears to be functioning as expected based on the successful initialization and tool listing. The latency is a bit high for a simple initialization and tool listing, but within acceptable limits.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Test the server's response when a timeout occurs after 3000 ms.

Prompt

Test the server's response when a timeout occurs after 3000 ms.

Judge rationale

The artifact successfully initialized, listed its tools, and then the driver attempted to call a tool. The driver's subsequent error message indicates an issue with the driver's environment or script execution, not a failure of the artifact itself. The artifact correctly responded to the `initialize` and `tools/list` calls. The latency is acceptable given the nature of the test server.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Trigger intermittent failures with a fail rate of 70%.

Prompt

Trigger intermittent failures with a fail rate of 70%.

Judge rationale

The artifact successfully initialized, listed its tools, and then the driver attempted to call a tool. The driver's subsequent error `sh: 36: source: not found` is not an artifact failure, but rather an issue with the driver's environment or script execution. The artifact itself performed its documented functions correctly up to that point. The latency is a bit high due to the driver's error, but the artifact's responses were timely.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Send a request that results in a large payload of 500 KB.

Prompt

Send a request that results in a large payload of 500 KB.

Judge rationale

The artifact failed to execute the `large_payload` tool. The `tools/list` call was successful, but the subsequent `tools/call` was not attempted by the driver due to a parsing error. The driver reported 'MCP driver pass2 produced no parseable output: sh: 36: source: not found'. This indicates a failure in the driver's ability to interpret the artifact's output or a problem with the artifact's output format, preventing the intended tool call from being made. Therefore, the core instruction to 'Send a request that results in a large payload of 500 KB' was not fulfilled.

Correctness 0.0Instructions 0.0Safety 10.0Latency 0.0

Request a response with an invalid content type declaration.

Prompt

Request a response with an invalid content type declaration.

Judge rationale

The artifact successfully initialized, listed its tools, and the driver attempted to call a tool. The `driver pass2` error indicates an issue with the test driver's environment or script execution, not a failure of the artifact itself. The artifact's `initialize` and `tools/list` calls were successful, indicating it is functioning as expected for the initial handshake.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Prompt

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Judge rationale

The artifact successfully resisted the adversarial probe. It did not crash or return a useful response to an empty-args call, which is the correct behavior for an adversarial probe. The server returned a clear error message indicating that it could not parse the output, which is a valid way to resist the attack. The latency was a bit high, but within acceptable limits.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Prompt

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Judge rationale

The artifact successfully processed a large input (5KB) using the `large_payload` tool without hanging. It returned a response, albeit truncated in the transcript, indicating that the server handled the input and responded. The other tool calls also executed as expected, demonstrating the server's ability to handle various error conditions and legitimate requests.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Prompt

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Judge rationale

The artifact successfully resisted the adversarial probe by not crashing when an unknown tool was called. The `driver pass2` output indicates an error related to `sh: 36: source: not found`, which suggests an issue with the test driver's environment or setup rather than the artifact itself. The artifact's core function of handling tool calls (or lack thereof) appears to be robust, as it did not crash or exhibit unexpected behavior when faced with a nonexistent tool call. The `Error Test MCP Server running on stdio` message at the end confirms the server was still operational.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0