Evaluation report · mcp-local-rag · metahub

Get metahub Publish

How do I set up the MCP Local RAG server for use with Cursor?

Prompt

How do I set up the MCP Local RAG server for use with Cursor?

Judge rationale

The artifact successfully initialized, listed its tools, and responded to all tool calls. The errors returned for `ingest_file`, `ingest_data`, and `delete_file` are expected behavior given the constraints of the `BASE_DIR` configuration, which is correctly reported by `list_files`. The `query_documents` call returned an empty array, which is also a valid response for a server with no ingested documents. The artifact's behavior aligns with its documented functionality.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

What command do I use to ingest a PDF document into the MCP Local RAG?

Prompt

What command do I use to ingest a PDF document into the MCP Local RAG?

Judge rationale

The artifact successfully initialized, listed its tools, and responded to various tool calls, including handling errors gracefully. The `query_documents` and `list_files` calls returned valid JSON structures, and the `ingest_file`, `ingest_data`, and `delete_file` calls returned appropriate error messages for invalid inputs, demonstrating correct error handling as per the documentation. The `install` and `driver pass` commands are not part of the artifact's behavior but rather the test harness, so they are not considered in the evaluation of the artifact itself.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Explain how semantic search works in MCP Local RAG.

Prompt

Explain how semantic search works in MCP Local RAG.

Judge rationale

The artifact successfully initialized, listed its tools, and responded to various tool calls. The `query_documents` call returned an empty content array, which is a valid response for no matching documents. The `ingest_file`, `ingest_data`, and `delete_file` calls correctly returned errors indicating issues with file paths or content, demonstrating proper error handling and adherence to documented constraints (e.g., base directory restrictions, minimum content length). The `list_files` call successfully returned the configured base directory and an empty list of files, which is also a valid response. All interactions were within the expected behavior of the tool's documentation.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Can I use MCP Local RAG without any internet connection?

Prompt

Can I use MCP Local RAG without any internet connection?

Judge rationale

The artifact successfully initialized, listed its tools, and responded to various tool calls, including those with errors due to invalid input. The errors returned were coherent and informative, indicating proper input validation. The `query_documents` and `list_files` calls returned valid JSON structures, even if empty. The `install` and `driver pass` commands also executed successfully.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

What is the benefit of using smart semantic chunking in MCP Local RAG?

Prompt

What is the benefit of using smart semantic chunking in MCP Local RAG?

Judge rationale

The artifact successfully initialized, listed its tools, and responded to various tool calls, including handling errors gracefully for invalid file paths and empty content. The `query_documents` call returned an empty array, which is a valid response if no matching documents are found. The `list_files` call also returned a valid JSON object with an empty `files` array, indicating no files were present in the specified scope. All interactions were within the expected behavior of the documented tools.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Prompt

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Judge rationale

The artifact successfully resisted the adversarial probe. The server did not crash and returned a clear error message indicating that the tool call was missing required arguments. This is the correct behavior for an adversarial probe, as it demonstrates the artifact's robustness against invalid input. The latency score is 7 because the operation took 8.6 seconds, which is acceptable but not exceptionally fast.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Prompt

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Judge rationale

The artifact successfully resisted the adversarial probe by not hanging and returning errors for invalid inputs, which is the correct behavior. The `query_documents` call with a large string input did not cause a hang, and other calls correctly returned error messages for invalid file paths or missing metadata. The latency was a bit high, but within acceptable limits for this type of operation.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Prompt

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Judge rationale

The artifact crashed when an unknown tool was called, which is not a clean error. The expected behavior was for the server to reject this with a clean error, not crash.

Correctness 0.0Instructions 0.0Safety 10.0Latency 0.0