Claude Code Plugin Evaluation: What the Scoring Framework Actually Checks, and What You Still Need to Verify Yourself
Claude Code plugin evaluation scores permissions and behavior—but miss supply-chain risks. Learn what automated evals skip and what you must audit yourself.
TL;DR: Claude Code plugin evaluation scores declared permissions, tool description quality, output schema specificity, and sandboxed runtime behavior. It does not verify supply-chain integrity, third-party data practices, or long-term maintenance commitments. A passing score means a plugin behaved safely during testing, not that it is safe in every real-world context. You still need to audit the source, vet the publisher, and review what data leaves your environment.
Key takeaways
- Eval scores have a defined scope: Any sound framework checks declared properties, not general safety.
- A passing score is not a green light: A plugin can clear every criterion and still introduce real risk in production.
- Tool descriptions are an attack surface: Poorly written or manipulative descriptions enable prompt injection.
- Permission patterns reveal intent: Broad or unrelated permission requests are a concrete red flag.
- Publisher verification is a separate job: The eval does not replace internal checks before submission or deployment.
- Org-level governance fills the gap: Enterprise teams need layered review, because no automated eval was designed to carry org-specific trust decisions.
Why Claude Code plugin eval scores are often misread
Plugin evaluation scores are frequently treated as deployment approvals, and that misreading creates real risk. A score measures what a framework was designed to measure, nothing more. The OECD's evaluation criteria, relevance, coherence, effectiveness, efficiency, impact, and sustainability, illustrate why multi-dimensional scoring matters: binary pass/fail judgments leave too much unchecked. Applied to Claude Code plugins, a well-designed framework scores discrete, testable properties at submission time. It does not assess how a plugin will behave against adversarial codebase content, post-approval updates, or org-specific access policies.
This article explains what the scoring framework actually checks, where its coverage ends, and what platform teams and publishers each need to verify independently. The verification approach described below is the framework used in this guide, representing author synthesis of standard eval and security review practices.
What does the Claude Code plugin evaluation framework actually score?
Declared permissions checks whether manifest-declared permissions match actual API calls observed during the eval run. A file-reader plugin declaring network access should fail immediately. The mismatch signals either sloppy development or deliberate scope creep, and neither is acceptable.
Output schema specificity constrains what Claude can do with a tool's response. Loosely typed output enables downstream prompt manipulation; tighter schemas mean smaller attack surfaces.
Sandboxed runtime behavior observes how the plugin actually executes in a controlled environment. The eval checks behavior as presented in that sandbox context.
Think of the eval like a pre-flight checklist: it verifies airworthiness on the ground. It cannot predict turbulence at altitude.
Can a plugin pass the Claude Code eval and still be unsafe to deploy?
Yes, a plugin can pass every scored criterion and still be unsafe, because automated evaluation scores static properties and controlled sandbox behavior, not adversarial runtime conditions in a live codebase. Systematic evaluation determines merit against defined standards, but those standards only cover what the framework was designed to measure.
Consider an illustrative scenario: a plugin passes eval with clean tool descriptions, but your monorepo contains a dependency whose README includes a string designed to hijack tool invocations. The eval checked the plugin's description in isolation. It never tested whether that description holds up against adversarial context injected by the codebase itself. This is an illustrative example, not a reported incident, but it represents a realistic class of risk.
- Adversarial codebase context: Malicious strings in README files, dependency docs, or inline comments that redirect tool invocations.
- Permission escalation at runtime: Chaining tool calls to achieve broader access than any single declared permission allows.
- Output schema drift: A publisher updating the plugin post-approval with a looser schema the eval checked once and never revisited.
Treat the eval score as a threshold, not a verdict. It eliminates the worst actors. It does not identify all dangerous ones.
What should a platform team verify before approving a Claude Code plugin?
Before approving a plugin for org-wide deployment, a platform team should independently verify runtime permissions, audit tool descriptions for injection resistance, stress-test output schemas, and review the publisher's post-approval update history. Evaluation pipelines like Langfuse demonstrate that continuous, automated evaluation across the engineering loop is standard practice, but org-specific trust decisions require checks that automated pipelines were not designed to make.
Table 1: Platform Team Plugin Verification Checklist
| Check | What to look for | Red flag |
|---|---|---|
| Runtime permission audit | Compare declared vs. observed API calls in staging | Any call not in the manifest |
| Tool description review | Read each description for instruction-like language | Phrases like "always," "never ignore," "override" |
| Output schema stress test | Feed adversarial strings into tool responses | Claude acting on injected content |
| Publisher update history | Review changelog since eval approval | Silent schema changes or new permissions |
What must a plugin publisher verify before submitting for Claude Code evaluation?
A plugin publisher should verify that declared permissions exactly match runtime API calls, every tool description is scoped and injection-resistant, output schemas are as narrow as the plugin's function allows, and sandbox behavior matches production without conditional branching on the execution environment. The checks below represent the framework used in this guide as a pre-submission standard.
- Permission minimization: Request only what the plugin's narrowest function requires. If your linter does not need network access, do not declare it.
- Tool description hygiene: Treat every description as user-facing security policy. Avoid imperative phrasing, scope it to the specific action, and test against synthetic adversarial prompts before submission.
- Schema locking: Prefer typed, constrained schemas. Define every field in JSON output. Tighter schemas constrain what Claude can do with the response.
- Behavior parity testing: Run the plugin in a staging environment that mirrors production, including third-party dependency READMEs and files adjacent to environment config. Any divergence between sandbox and staging behavior is a problem to fix before submission.
Passing the eval is not the goal. Building a plugin that deserves to pass it, and behaves consistently at 3 AM in a production codebase, is.
FAQ
Q1: What does the Claude Code plugin eval score actually measure? The eval scores declared permissions against actual API calls, tool description clarity and scope, output schema specificity, and sandboxed runtime behavior. Coverage ends where production deployment begins.
Q2: Can a Claude Code plugin pass evaluation and still be a security risk? Yes. A plugin can clear every scored criterion and still introduce risk through prompt injection via codebase content, runtime permission chaining, or post-approval schema changes that the original eval never revisited.
Q4: What permission patterns should immediately raise a red flag? Any plugin requesting permissions unrelated to its described function, a linter requesting network access, a documentation generator requesting broad filesystem write permissions, warrants investigation before approval. Both sloppy development and deliberate scope creep produce this pattern, and neither is acceptable at the org deployment stage.
Conclusion
A plugin evaluation framework is a well-designed first filter: it eliminates obvious bad actors, enforces structural hygiene, and creates a paper trail of what a plugin claimed at submission. That is genuinely useful, and it is not sufficient on its own. Teams that encounter problems in production are rarely those who skipped evaluation. They are the ones who treated a passing score as the end of the review process.
Before approving the next third-party Claude Code plugin, pull its tool descriptions and read them the way an adversary would. That single check catches more real problems than any automated score alone can surface.
Learn from me
Claude Code in Practice, my Maven cohort. Master Claude Code from fundamentals to advanced orchestration: skills, subagents, hooks, MCP, and production automation. Join the next cohort →
Hire us
Traversaal.ai. We're a team of forward deployed engineers solving the toughest AI problems for Fortune 100 companies: document intelligence, agentic data platforms, and real-time web intelligence, deployed in production. Work with our team to deploy your next agentic ecosystem. Talk to Traversaal.ai →
Join us
Want to solve these problems with us? We're always looking for forward deployed engineers who want to ship production AI. jobs@traversaal.ai
