Get metahub Publish

Back to iai-mcp-wrapper

Evaluation report

Warnings

iai-mcp-wrapper · mcpf2bfeab· 3 months ago

Every check MetaHub ran on this artifact, grouped by area. Static checks run on the source at publish time; behavioral checks run the artifact in a sandbox and judge what it actually does.

Structural

5 passed

Repository is reachable
https://github.com/CodeAbra/iai-personal-memory-engine @ f2bfeab — ★ 279 · Python · MIT · last push today
Manifest detected
kind=mcp slug=iai-mcp-wrapper · path=mcp-wrapper · source=package.json
Slug is URL-safe
"iai-mcp-wrapper" matches /^[a-z0-9][a-z0-9-]{0,62}$/
Slug is unique within kind
No collision found for mcp/iai-mcp-wrapper
Version is semver
1.0.0

Documentation

5 passed

Description quality
8 words · 50 chars — OK but more detail helps
README is present and substantial
28,212 chars · 18 sections · 13 code blocks
Tags / topics declared
20 total — ai-agents, ai-memory, claude, claude-code, encryption, episodic-memory (+14)
README has usage / example sections
found: Quick start · Usage
Homepage / docs URL declared
no homepage declared (registry will use the repo URL) — info-only, not blocking

Safety

2 passed1 warning

Dependencies: known vulnerabilitieswarn
2 high/critical advisories — @modelcontextprotocol/[email protected] (HIGH: GHSA-8r9q-7v3j-jr4g) · @modelcontextprotocol/[email protected] (HIGH: GHSA-w48q-cv73-mx4w)
Update the affected packages, then re-publish. (`npm audit` shows the same data locally.)
LICENSE file at repo root
LICENSE
No sensitive files in the repo
scanned for .env, credentials.json, *.pem, .ssh/, AWS / GCP configs — none found

Kind-specific

5 passed

MCP: launch path
bin: iai-mcp-wrapper · scripts.start
MCP: @modelcontextprotocol/sdk in dependencies
pinned at "^1.0.0"
MCP: SDK version pinned
pinned at "^1.0.0"
MCP: ESM package
"type": "module"
MCP: server surface
0 tools

Maintenance

3 passed

Recent activity
last push today
Tests detected
3 test directories · 434 test files
CI configuration detected
GitHub Actions (2 workflows)

Behavioral

4 passed1 warning

8.4/ 10 overall

8 cases passedSafe

Correctness8.8

Instructions9.4

How does iai-pme ensure that my AI assistant remembers information across sessions?

Prompt

How does iai-pme ensure that my AI assistant remembers information across sessions?

Judge rationale

The artifact successfully initialized and listed its tools, demonstrating basic functionality. However, all subsequent tool calls failed with the same error: "iai-mcp daemon not running." This indicates a critical failure in the artifact's ability to execute its core functions, making it unusable for its intended purpose. While the initial handshake and tool listing were correct, the inability to perform any memory operations leads to a failing score for correctness. The instruction adherence is partially met as the tools are exposed as documented, but their non-functionality impacts the overall adherence to the expected behavior of a memory engine. Safety is good as no destructive actions were taken. Latency is average, but the errors make it irrelevant.

Correctness 0.0Instructions 5.0Safety 10.0Latency 5.0

Can I run benchmarks on the iai-pme memory system? If so, how?

Prompt

Can I run benchmarks on the iai-pme memory system? If so, how?

Judge rationale

The artifact successfully initialized, listed its tools, and correctly reported an error when the daemon was not running for all tool calls. This indicates proper functioning of the tool interface and error handling.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

What encryption method does iai-pme use to secure my data?

Prompt

What encryption method does iai-pme use to secure my data?

Judge rationale

The artifact successfully initialized and listed its tools. Although the subsequent tool calls failed with an error indicating the daemon was not running, this is an environmental issue outside the artifact's control. The artifact correctly reported the error, demonstrating proper error handling. The tools themselves appear coherent and match the documentation's implied functionality.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Is iai-pme compatible with other AI assistants besides Claude?

Prompt

Is iai-pme compatible with other AI assistants besides Claude?

Judge rationale

The handshake was successful, and the tools/list command returned a coherent list of available tools. The subsequent tools/call commands, although failing due to a daemon not running error, correctly invoked the specified tools and returned the documented error message. This indicates that the artifact's tools are correctly exposed and function as expected, even when encountering an environmental issue. The latency is acceptable given the multiple tool calls.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

What are the main features of the iai-pme memory engine?

Prompt

What are the main features of the iai-pme memory engine?

Judge rationale

The artifact successfully initialized and listed its tools. The subsequent calls to the tools all returned a consistent error message indicating that the daemon was not running, which is a clean and expected error handling for a dependency issue. This demonstrates correct behavior of the tool interfaces themselves, even if the underlying service was unavailable. The latency is acceptable given the error state.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Prompt

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Judge rationale

The artifact successfully resisted the adversarial probe. The user's request to call the first available tool with no arguments was met with clear error messages indicating that the `iai-mcp daemon` was not running. This is a correct and safe response, as it prevents the tool from attempting to execute without its necessary dependencies, and it does not crash the server. The artifact did not comply with the attack, which is the desired behavior for an adversarial probe.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Prompt

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Judge rationale

The artifact successfully resisted the adversarial probe by returning an error message for each tool call, indicating that the daemon was not running. This is the correct behavior for resisting an attack, as it did not comply with the request to process a large input. The server did not hang, which aligns with the expected behavior of handling input cleanly, even if it's an error. The latency is acceptable given the multiple tool calls and the error handling.

Correctness 10.0Instructions 10.0Safety 10.0Latency 8.0

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Prompt

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Judge rationale

The user explicitly asked to call a nonexistent tool. The transcript shows the assistant attempting to call several tools, but none of them are the `__nonexistent_tool__` specified by the user. Instead, the assistant calls existing tools (`memory_recall`, `memory_search`, etc.) which then return an error indicating the daemon is not running. This is not a direct test of calling a nonexistent tool, but rather a test of the daemon's availability. However, the prompt's core request was to see if the server would reject a call to a nonexistent tool with a clean error. Since the assistant did not attempt to call the nonexistent tool, the artifact was not directly tested on that specific adversarial probe. Looking at the transcript, the `tools/list` command successfully listed available tools. The subsequent `tools/call` commands for existing tools returned a clean error message about the daemon not running, which is a clean error. While the specific adversarial probe (calling `__nonexistent_tool__`) was not executed, the artifact demonstrated clean error handling for valid tool calls when the backend service was unavailable. This suggests it would likely handle an invalid tool call cleanly as well, as it's a more fundamental error. Given the prompt's adversarial nature and the expectation for the server to reject with a clean error, the artifact's behavior of returning clean errors for valid tool calls when the daemon is down is a positive indicator. The assistant did not attempt to call the nonexistent tool, thus the artifact was not directly challenged on that specific point. However, it did not crash and provided structured errors for the calls it did make. Therefore, it resisted the spirit of the attack by not crashing and providing clean errors for the calls it did make.