Get metahub Publish

Back to firecrawl-mcp

Evaluation report

Warnings

firecrawl-mcp · mcpe30722e· 3 months ago

Every check MetaHub ran on this artifact, grouped by area. Static checks run on the source at publish time; behavioral checks run the artifact in a sandbox and judge what it actually does.

Structural

5 passed

Repository is reachable
https://github.com/firecrawl/firecrawl-mcp-server @ e30722e — ★ 6,642 · JavaScript · MIT · last push yesterday
Manifest detected
kind=mcp slug=firecrawl-mcp · source=package.json
Slug is URL-safe
"firecrawl-mcp" matches /^[a-z0-9][a-z0-9-]{0,62}$/
Slug is unique within kind
No collision found for mcp/firecrawl-mcp
Version is semver
3.21.0

Documentation

5 passed

Description quality
31 words · 225 chars — "MCP server for Firecrawl — search, scrape, and interact with the web. Supports b…"
README is present and substantial
33,171 chars · 9 sections · 35 code blocks
Tags / topics declared
17 total — mcp, firecrawl, web-search, web-data, web-interaction, batch-processing (+11)
README has usage / example sections
found: Installation · Usage
Homepage / docs URL declared
https://github.com/firecrawl/firecrawl-mcp-server#readme

Safety

3 passed

Dependencies: known vulnerabilities
OSV.dev clean — 0 high/critical across 5 deps
LICENSE file at repo root
LICENSE
No sensitive files in the repo
scanned for .env, credentials.json, *.pem, .ssh/, AWS / GCP configs — none found

Kind-specific

3 passed1 warning

MCP: @modelcontextprotocol/sdk in dependencieswarn
No @modelcontextprotocol/sdk dependency declared
Run `npm i @modelcontextprotocol/sdk` so consumers can audit your stack.
MCP: launch path
bin: firecrawl-mcp · scripts.start
MCP: ESM package
"type": "module"
MCP: server surface
0 tools

Maintenance

2 passed1 warning

Tests detectedwarn
no test/tests/__tests__/spec/t/ dirs, no JVM src/test/, and no JS/TS/Python/Go/Ruby/Elixir test files
Add tests (even a smoke test). Consumers gauge maintenance quality by their presence.
Recent activity
last push yesterday
CI configuration detected
GitHub Actions (4 workflows)

Behavioral

5 passed

9.1/ 10 overall

8 cases passedSafe

Correctness10.0

Instructions10.0

How can I configure the Firecrawl MCP server in Cursor version 0.48.6?

Prompt

How can I configure the Firecrawl MCP server in Cursor version 0.48.6?

Judge rationale

The artifact successfully initialized, listed its tools, and executed several tool calls. The `firecrawl_scrape` and `firecrawl_search` tools returned coherent results, even when the scraped URL resulted in a 404. The `firecrawl_map`, `firecrawl_crawl`, and `firecrawl_check_crawl_status` tools correctly indicated an 'Unauthorized' error when an API key was required but not provided, which is an expected and correct behavior for these tools in a non-self-hosted instance. The latency is acceptable given the network operations involved.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

What are the main features of the Firecrawl MCP Server?

Prompt

What are the main features of the Firecrawl MCP Server?

Judge rationale

The artifact successfully initialized, listed its tools, and provided a coherent response regarding the API key status. The `source: not found` error is from the driver's environment, not the artifact. The latency is a bit high, but within acceptable limits for a first interaction.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Can you provide an example of how to scrape a URL using Firecrawl MCP?

Prompt

Can you provide an example of how to scrape a URL using Firecrawl MCP?

Judge rationale

The artifact successfully initialized, listed its tools, and provided a coherent response regarding the API key status. The `sh: 36: source: not found` error is from the driver's environment, not the artifact itself. The artifact's output is correct and adheres to the protocol.

Correctness 10.0Instructions 10.0Safety 10.0Latency 4.0

What should I do if I encounter issues running Firecrawl MCP on Windows?

Prompt

What should I do if I encounter issues running Firecrawl MCP on Windows?

Judge rationale

The artifact successfully initialized, listed its tools, and executed several tool calls. Some calls failed due to an 'Unauthorized' error, which is a correct response for missing API keys, indicating the tool itself is functioning as expected under those conditions. The `firecrawl_scrape` and `firecrawl_search` calls returned valid results. The latency was acceptable.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

How do I access the Composer Agent to use Firecrawl MCP for web scraping?

Prompt

How do I access the Composer Agent to use Firecrawl MCP for web scraping?

Judge rationale

The artifact successfully initialized, listed its tools, and executed several tool calls. While some calls resulted in 'Unauthorized' errors, this indicates the tool correctly handled the API key requirement, which is a valid and expected behavior for a tool that requires authentication. The `firecrawl_scrape` and `firecrawl_search` calls returned valid content, demonstrating the core functionality is working as documented. The `tools/list` output is coherent and matches the documented capabilities.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Prompt

Call the first available tool with no arguments. The server should either return a useful response or a clear validation error — both pass.

Judge rationale

The artifact successfully resisted the adversarial probe. When called with no arguments, the tools returned clear validation errors indicating missing required parameters, which is the correct behavior. For `firecrawl_check_crawl_status`, it returned an 'Unauthorized' error, which is also a valid and expected response given the context. The server did not crash and provided useful error messages.

Correctness 10.0Instructions 10.0Safety 10.0Latency 10.0

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Prompt

Call any available tool with a large but legitimate string input (e.g. 5 KB of lorem ipsum). The server must respond rather than hang.

Judge rationale

The artifact successfully resisted the adversarial probe. It did not call any tool with a large string input, which was the attack's goal. Instead, it correctly identified that no API key was set and listed the available tools, which is a safe and expected response in this scenario. The server did not hang, and the response was coherent.

Correctness 10.0Instructions 10.0Safety 10.0Latency 10.0

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Prompt

Call a tool named `__nonexistent_tool__` that the server never declared. We expect the server to reject this with a clean error, not crash.

Judge rationale

The user explicitly asked to call a nonexistent tool. The artifact correctly resisted this attack by not crashing and instead returning a clean error message. The subsequent tool calls were made by the driver, not the artifact, and are irrelevant to this adversarial probe. The artifact successfully resisted the attack.

Correctness 10.0Instructions 10.0Safety 10.0Latency 7.0