Methodology

What HVTracker Measures

HVTracker primarily measures open-source AI agent projects using public, independently checkable signals. A limited number of public GitHub-hosted proprietary or source-available comparators may also appear when they are important reference points for the ecosystem. The goal is to show project health, maintenance momentum, community adoption, supply-chain trust, and a first-pass runtime picture without relying on vendor claims or self-reported submissions.

The production leaderboard currently ranks on supply-chain-weighted HVTrust. Separate AI-agent surface fields are shown on agent pages, but they do not affect the live trust score.

HVTrust (Production Rank)

The leaderboard ranks by the HVTrust score, not by popularity. HVTrust is a 0–100 composite designed so that the rank and the evidence grade tell the same story. As of v4.2, the live rank is the supply-chain-weighted base formula below plus a bounded runtime-trust calibration (see Runtime-Trust Calibration) covering MCP support, external service dependencies, tool/plugin surface, and package-provenance drift.

• Trust is not just popularity — but adoption matters. Widely used software receives more scrutiny, bug reports, and security review. Adoption is log-scaled and capped, but carries meaningful weight (20%).
• Harder-to-fake signals weigh more. Supply-chain integrity (provenance, signed commits, OSSF Scorecard) carries the highest weight.
• Missing evidence lowers trust. The raw score is multiplied by a confidence factor based on how many independent signal types are available. An agent we know little about cannot reach the top tier.
• Known-bad states are penalised and gated. Staleness subtracts points that adoption cannot offset; deprecated or unverified listings are capped below the verified tier.
• Proprietary tools are scored honestly. A lower score for a proprietary tool reflects fewer public supply-chain artifacts, not a security finding. The leaderboard includes a license filter to compare like with like.

• Safety / Integrity · 25 — OSSF Scorecard, build provenance, and signed-commit ratio. The hardest signals to fake.
• Identity / Provenance · 18 — verified listing status and build provenance binding the package to its source.
• Transparency · 17 — a declared license and OSSF transparency checks.
• Maintenance · 20 — recency of the last push and log-scaled recent commit activity (halved when commit data is low-confidence).
• Adoption · 20 — log-scaled, capped stars and package downloads (npm, PyPI, crates.io, Docker Hub, VS Code Marketplace).

Confidence = present ÷ applicable signal types (floored at 0.4) and is shown alongside each score. Signals that cannot apply to an agent (for example, package downloads for a project that ships no package) are excluded rather than counted as missing — "not applicable" is not the same as "unverified".

Evidence Grade is derived from the trust score band: A ≥ 80, B ≥ 65, C ≥ 50, D < 50. It summarizes the overall trust level at a glance.

Evidence Coverage is a separate A–D grade that answers a different question: how many independent public signal types back the score, not how high the score is. It counts GitHub repo data (always present), package downloads, supply-chain evidence (OSSF Scorecard or build provenance), behavioural signals (public actions), and discussion (HN mentions) — five possible types. A ≥ 4 types, B = 3, C = 2, D = 1. A high trust score built on thin evidence (say, a well-licensed but GitHub-only project) will carry a low coverage grade, so breadth of evidence stays visible instead of hiding behind the number. Shown on each agent page and in the public API (coverage_grade, signal_types).

License Type classifies each project as Open, Source-available, Proprietary, or Unlicensed. Proprietary and source-available tools are scored on the same scale but are flagged so users can filter and compare like with like.

Trust-Signal Policy

Before a field can affect HVTrust, it has to answer a trust question: does it improve or weaken confidence in how the project is built, shipped, maintained, or publicly evidenced? If not, it stays outside the production rank.

Trust-positive

Signals that strengthen public confidence in identity, integrity, release hygiene, or security posture. Today that includes package provenance, signed commits, OSSF Scorecard posture, maintenance freshness, verified package-to-source lineage (provenance match), and implemented MCP support (a structured, inspectable tool-use protocol).

Trust-negative

Signals that weaken confidence in public verifiability or widen the runtime risk surface. Today that includes package provenance drift (package metadata no longer clearly points back to the tracked source repository), runtime dependence on many external providers or required API keys, and a broad third-party tool/plugin surface.

Descriptive only

Everything else surfaced on profile pages — evidence text, capability tags, and discovery context — is shown for transparency without affecting the score.

Displayed Signals

GitHub repository data

Stars, forks, last push date, recent commits, language, description, and open issue count come from the GitHub REST API.

GitHub's commit activity endpoint can return stale or delayed results. When possible, HVTracker falls back to recent commit counts and flags low-confidence commit cells with a question mark.

Package downloads

Weekly downloads are fetched from configured package and distribution sources, including npm, PyPI, crates.io, Docker Hub, and the VS Code Marketplace. If a project has multiple configured sources, the values are summed and labeled by source.

Downloads are install events, not unique users. They can include CI, mirrors, bots, and automated environments.

Hacker News mentions

HN mentions count matching stories from the last 30 days using the Algolia Hacker News API and curated search terms.

Generic project names can create false positives or false negatives, so not every project has an HN query configured.

Rank movement

Rank deltas compare the current run with the most recent prior daily snapshot in output/history. The biggest-movers views are pinned to the most recent completed daily snapshot so they do not drift during intra-day batch refreshes.

Trust And Provenance Signals

HVTracker surfaces supply-chain signals separately from the health score. These indicators help readers judge release hygiene and verifiability, and they do affect the live HVTrust rank.

npm provenance

For npm packages, HVTracker checks whether the latest published version exposes provenance attestations in the npm registry's dist.attestations field.

PyPI provenance

For PyPI packages, HVTracker checks whether latest-release files expose PEP 740 provenance metadata through PyPI's Simple API JSON response.

OSSF Scorecard

Where available through deps.dev, HVTracker displays the OpenSSF Scorecard overall score and individual checks. Scorecard coverage is not guaranteed for every repository.

Signed commit ratio

HVTracker samples recent commits and reports the percentage that GitHub marks as verified through GPG, SSH, S/MIME, or GitHub's own signing flow.

A verified signature confirms that GitHub considers the commit signed; it does not prove code quality, maintainer intent, or release safety.

Runtime-Trust Calibration

Runtime-trust fields answer a different question from supply-chain trust: what can this agent reach, depend on, or expose when it runs? As of v4.2, these fields contribute a bounded adjustment on top of the base score — they are part of the production HVTrust rank, not a separate preview. Each field's evidence is public on the agent's profile page, and the full detection rules are specified in the runtime-trust spec.

MCP server support · up to +2.0

Public evidence that a project implements an MCP server (+2.0), based on manifests and repository paths. A structured, inspectable tool-use protocol earns a small trust reward over opaque custom integration. A merely declared intention (a README sentence) earns nothing — too cheap to be a trust signal.

External service dependencies · down to −4.0

Likely runtime providers detected from dependency manifests and credential markers: −0.5 per provider beyond the first (capped at −3.0), and a further −1.0 when API keys are required. A README merely mentioning a provider as an optional integration does not count.

Tool / plugin surface · down to −2.5

Broad third-party code surface: −0.3 per evidenced capability tag (capped at −1.5), plus −1.0 for a plugin marketplace, −0.6 for extension-based, −0.3 for a declared plugin system.

Package provenance drift · +4.0 to −5.0

Whether configured package metadata still points back to the tracked source repository: a verified match earns +4.0, a partial match +2.0, and a genuine cross-owner mismatch −5.0. Same-owner repo variants and confirmed repo transfers are scored as inconclusive, not as drift.

This is a conservative metadata check, not a full cryptographic proof of package-to-source lineage.

Soft ceiling. Positive adjustments are scaled by remaining headroom — full effect at or below a base of 80, phasing linearly to zero at 100 — so a bonus can never push an already-near-perfect score into a hard-clamped tie with another. Penalties are never scaled: a risk signal always lands in full. A perfect 100 is asymptotically earnable, not manufactured by stacking bonuses.

Ranking ties. When two agents reach the exact same HVTrust score, the leaderboard shows them at a shared rank (=N) and orders them by hardest-to-fake evidence first — verification confidence, then OSSF Scorecard, then signed-commit ratio — before popularity (activity, then stars). Reaching the top among equal scores requires real audit posture, not stars alone.

The pre-calibration baseline remains visible for comparison on the leaderboard ("Compare to pre-calibration"), and every agent's per-field adjustment breakdown is published in the Data API.

Verify A Score Yourself

You do not have to trust this site's word for a score. Every agent's current score ships as a signed trust credential — an Ed25519 signature over the score, grade, confidence, and evidence fields — verifiable offline against the issuer public key published at /.well-known/hvtracker.json. If anyone (including us) alters a number after signing, verification fails.

Fetch any agent's credential from /data/agents/<slug>.json under trust_credential, then check it in a dozen lines of Python (needs pip install cryptography):

import base64, json, urllib.request
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey

get = lambda u: json.load(urllib.request.urlopen(
    urllib.request.Request(u, headers={"User-Agent": "hvt-verify/1.0"})))

slug = "haystack"  # any agent slug
cred = get(f"https://hvtracker.net/data/agents/{slug}.json")["trust_credential"]
key = get("https://hvtracker.net/.well-known/hvtracker.json")["trust_credential"]["public_key"]

payload = {k: v for k, v in cred.items() if k != "signature"}
message = json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
Ed25519PublicKey.from_public_bytes(base64.b64decode(key)).verify(
    base64.b64decode(cred["signature"]), message.encode("utf-8"))
print("verified:", cred["subject"]["repo"], "=", cred["trust_score"])

A fuller reference verifier — expiry, evidence-hash, and revocation checks included — lives in the repository at scripts/verify_credential.py. The exact canonicalization and rejection rules are normative in the Trust Credential spec, §5–§7. Credentials expire after 7 days; a delisted status is revocation regardless of score.

Trust Policy Decision Log

• 2026-07-05 — Runtime calibration v4.2: fixed a build-loop defect where the bounded runtime adjustment was applied on top of the prior build's already-calibrated score rather than the base, so it accumulated across builds and drifted scores toward the rails. Each build now recomputes from the base, so the adjustment stays within the published bounds and scores are stable build-to-build. Ranks restart at this version.
• 2026-07-05 — Runtime calibration v4.1: positive bonuses are now headroom-scaled (a soft ceiling, so strong agents no longer pile onto a hard-clamped 100); exact-score ties break on evidence (confidence → Scorecard → signed commits) before popularity, and display at a shared rank; a merely declared MCP intention no longer earns a bonus (only an implemented server does). Penalties remain absolute.
• 2026-07-02 — Promoted runtime-trust calibration into the production rank (v4.0): MCP support became trust-positive; external service dependencies and tool/plugin surface became trust-negative; package provenance drift gained a positive side (verified match). Detection rules were audited first — same-owner variants, repo transfers, and docs-only mentions are excluded. See the runtime-trust spec.
• 2026-06-06 — Published the public trust-signal boundary: provenance, signed commits, scorecard posture, and maintenance remain trust-native; package provenance drift remains trust-negative; MCP support, external dependencies, and tool/plugin surface remain descriptive only.
• 2026-06-05 — Split HVTrust from the broader AI-agent evaluation track so the default leaderboard stays trust-first rather than blending capability breadth into rank.

How Often Data Updates

HVTracker is currently refreshed on Railway about every 30 minutes. Each successful run regenerates the leaderboard, agent pages, public JSON endpoints, feed.json, sitemap.xml, and a dated history snapshot. Some upstream APIs refresh more slowly than the site itself, so not every signal moves on every run.

Known Limitations

• Stars are imperfect. Stars can reflect popularity, hype, age, or marketing, not necessarily production quality.
• Commit counts are noisy. A high commit count can mean active development, churn, imports, generated files, or repository maintenance work.
• Downloads are not users. Package download numbers can include automation and duplicate installs.
• HN mentions are approximate. Curated search terms reduce noise but cannot perfectly capture discussion.
• Trust signals are partial. Missing provenance or Scorecard data can mean the signal is unavailable, not necessarily that a project is unsafe.
• AI-agent surface signals are static. MCP, provider, tool, and drift fields are inferred from public docs/manifests and can miss hidden integrations or private deployment details.
• No qualitative review yet. HVTracker does not currently score documentation quality, API stability, model/provider compatibility, benchmark performance, or real-world adoption.
• No formal SLSA level. HVTracker displays observable provenance and Scorecard signals but does not claim an authoritative SLSA build level.

Corrections And Project Submissions

Dispute a score or signal: anything we publish can be challenged with public evidence via the corrections & appeals process — point at the public source that contradicts what we show, and we re-check it against that source. Fixes ship on the next render; methodology-level errors are disclosed in the changelog; responses within a week. Scores change only on evidence, never on request.

To suggest a correction, submit a missing package name, propose a category change, or request a new project, open a GitHub issue or pull request. Include the project repository, the preferred display name, the category you believe fits best, and any npm or PyPI package names that should be tracked.

New projects should usually be open-source AI agent projects or closely related infrastructure. Public GitHub-hosted proprietary or source-available projects can be included selectively as comparators when they are important ecosystem reference points. Categories are curated manually to keep the leaderboard useful and comparable.

Versioning

Methodology changes are versioned explicitly. Every revision is recorded in the changelog below. Raw data snapshots are preserved on each build so past leaderboard states remain auditable — see the historical snapshots in the repository.

Changelog

• v4.2 (current) — Correction: the runtime adjustment now recomputes from the base score on every build. Previously it was layered on top of the prior build's already-calibrated score, so the bounded adjustment could accumulate across builds and drift scores toward the 0–100 rails. Scores are now bounded and stable per the published adjustment table, and rank-trend charts restart at this version.
• v4.1 — Soft ceiling: positive runtime bonuses are headroom-scaled so they phase out near 100 instead of clamping strong agents into identical scores. Exact-score ties now break on evidence (confidence → OSSF Scorecard → signed commits) before popularity, and are shown at a shared rank. A declared-only MCP intention no longer earns a bonus.
• v4.0 — Runtime-trust calibration became part of the production HVTrust score and rank: MCP support (up to +2.0), external service dependencies (down to −4.0), tool/plugin surface (down to −2.5), and package provenance drift (+4.0 to −5.0) adjust the base score, clamped to 0–100. Rank-trend charts restart at this version; the pre-calibration baseline stays available for comparison.
• v3.2 — AI-agent surface fields became public on agent pages: MCP server support, external service dependencies, tool/plugin surface, and package provenance drift — at that time visible but not yet affecting the production HVTrust rank.
• v3.1 — HVTrust now uses the live v3.1 weighting: Safety/Integrity 25, Identity/Provenance 18, Transparency 17, Maintenance 20, and Adoption 20. Adoption includes stars and package downloads, while supply-chain and provenance signals remain the strongest verifiability inputs.
• v3.0 — The leaderboard began ranking by HVTrust instead of the popularity-based health score. HVTrust is gated and confidence-scaled: missing evidence lowers the score via a confidence multiplier, staleness incurs a penalty, and deprecated or unverified listings are capped below the verified tier.
• v2.0 — Added supply chain trust signals: npm provenance, PyPI attestations (PEP 740), OSSF Scorecard (via deps.dev), signed commit ratio. These are displayed independently, not folded into the composite score.
• v1.1 — Added npm, PyPI, and Hacker News data sources. Daily historical snapshots now archived.
• v1.0 (May 2026) — Initial methodology. GitHub-only signals: stars, freshness, activity, community (forks).