How to Evaluate AI Agent Safety: 5 Signals That Actually Matter

Open-source AI agents are being adopted at an unprecedented rate. Developers are integrating coding assistants, research agents, and workflow automation tools into production systems that handle sensitive data and make real decisions.

But how do you know if an AI agent is safe to use? GitHub stars measure popularity, not trustworthiness. A project with 50,000 stars can still have unsigned releases, no security policy, and dependencies riddled with known vulnerabilities.

At HVTracker, we track trust signals for 170+ open-source AI agents daily. Here are the 5 evidence-based signals that actually predict whether an agent is safe to adopt.

1. OpenSSF Scorecard

This is the single most informative signal for supply-chain safety. A project scoring 7+/10 on the Scorecard has branch protection enabled, pins its dependencies, uses SAST tools, and has a vulnerability disclosure process.

The problem? Most AI agent projects don't score well. In our dataset, the median Scorecard score across tracked agents is below 5/10. The agents that score 7+ tend to be backed by established organizations with mature security practices.

2. Package Provenance

This matters because the supply-chain attack surface for AI agents is enormous. When you run pip install or npm install, you're trusting that the published package matches the source code you reviewed on GitHub. Without provenance, there's no way to verify this.

Provenance adoption is still early. In our tracking, only a fraction of AI agents publish packages with verified provenance attestations. But the ones that do are signaling a level of security maturity that goes beyond the basics.

3. Signed Commits

A high signed-commit ratio (80%+) indicates that a project's maintainers take code integrity seriously. It's not a perfect signal — some legitimate maintainers don't sign commits — but when combined with other signals, it's a strong indicator of security culture.

We measure this by sampling recent commits and calculating the ratio that are cryptographically verified. Projects backed by organizations with security policies tend to score highest here.

4. Activity Patterns

An unmaintained AI agent is a security liability. Vulnerabilities in dependencies go unpatched. Breaking changes in APIs go unaddressed. Users who report bugs get silence.

But very high activity can also be a warning sign. A project with hundreds of commits per week from a single contributor might be moving too fast for proper review. The healthiest pattern is consistent, multi-contributor activity with a reasonable cadence.

5. Transparency Indicators

A project that lacks a security policy is telling you something: they haven't thought about what happens when a vulnerability is discovered. A project with no license leaves you legally exposed. Missing documentation makes it harder to audit what the agent actually does.

We also look at whether the project uses public GitHub Actions (visible CI/CD) versus private pipelines. Public CI/CD means anyone can verify how the software is built, tested, and released.

Putting It All Together

No single signal tells the full story. An agent with a perfect Scorecard score but no activity in 6 months might be abandoned. An agent with thousands of commits but no provenance might have supply-chain risks.

The most trustworthy agents score well across multiple dimensions: they have reasonable Scorecard scores, some form of provenance or signing, active maintenance, and transparent governance.

This is exactly what HVTracker's Trust Score measures — a composite of activity, adoption, transparency, safety, and identity signals, weighted to reflect real-world risk. Each agent gets a score from 0-100, updated daily.

Further Reading

• HVTracker Methodology — how trust scores are computed
• Security & Guardrails Agents — agents specifically built for AI safety
• Coding Agents — ranked by trust score
• OpenSSF Scorecard — check any repo's supply-chain security
• SLSA Framework — supply-chain integrity standard