Methodology v2026.09 · edition 2026-09 · evidence to 26 Sept 2026
How a rating is made
Every number on this site is computed from recorded, dated, cited evidence by the formulas below. Nothing is typed in by hand, and no one can pay to change a score.
Principles
- Evidence first. Each input is a recorded fact with a source URL and the date it was observed.
- No invented numbers. If a pillar lacks the minimum evidence, it is shown as "not enough data" and left out of the composite. If fewer than two pillars have enough data, the agent has no composite score.
- Weights are public. The weights below are the weights the site uses.
- Dated. Every profile shows the date its evidence was last collected.
The AgentsWisdom Score (0–100)
Three pillars, each 0–100.
Pillar Q — Adoption and momentum
Reach is the mean of whichever of these public signals exist, each mapped onto 0–100 on a log scale between a floor and a ceiling:
| Signal | Floor (=0) | Ceiling (=100) |
|---|---|---|
| GitHub stars (main repo) | 100 | 200,000 |
| npm weekly downloads (main package) | 1,000 | 5,000,000 |
| PyPI monthly downloads (main package) | 5,000 | 20,000,000 |
| VS Code Marketplace / Open VSX installs | 10,000 | 50,000,000 |
| Chrome Web Store users | 1,000 | 10,000,000 |
| Officially reported users (weekly/monthly active, or total, from the vendor or reputable press) | 10,000 | 50,000,000 |
| Reported customer organisations (for business products that report companies, not people) | 100 | 100,000 |
score = clamp(100 × (log10(x) − log10(floor)) / (log10(ceiling) − log10(floor)), 0, 100)
Momentum comes from releases or dated changelog entries in the 90 days before collection: 0 → 0, 1–2 → 40, 3–5 → 60, 6–11 → 80, 12+ → 100.
A reported figure counts only if it is specific to the product being rated. A whole platform's or company's user base (for example all Replit users, for Replit Agent) does not count.
Q = 0.7 × Reach + 0.3 × Momentum. Minimum evidence: one reach signal and a momentum count.
Pillar E — Expert reviews
Verdicts from named, independent experts (YouTube creators, newsletters, technical publications, respected practitioners' blogs) published in the last 12 months. Each is coded positive (100), mixed (50) or negative (0) from what the reviewer actually said.
E = (sum of verdicts + 3 × 50) / (number of reviews + 3): a Bayesian average that pulls small samples toward a neutral 50, so three friendly reviews cannot produce a perfect score. Minimum evidence: 3 reviews.
Who counts as an expert. Reviews must have a named author. AI-tool directories, listing and comparison sites, and AI-written review pages are not expert reviews and are not counted.
Analyst-firm placements are shown, not scored. Gartner, Forrester and similar placements are vendor-briefed rankings, not hands-on reviews.
Benchmarks are shown, not scored. Public agent leaderboards (Terminal-Bench, SWE-bench) pair each harness with a different model and a different submission date, so a placing mostly measures the model rather than the harness. We publish each product's leaderboard entries, with the model and the rank, on its profile and in comparisons, but they do not move the score. We will score them once a leaderboard compares harnesses on the same model.
Pillar C — Community sentiment
A sample of public community discussion from the last 12 months (Reddit, Hacker News, X, YouTube comments, GitHub Discussions, Discord/forum posts that are publicly readable). Each sampled item is coded positive, mixed or negative about the tool.
C = (100 × (positive + 0.5 × mixed) + 6 × 50) / (N + 6): the same Bayesian average, so a small sample stays near neutral. Minimum evidence: N ≥ 6 items from at least 2 different platforms. The site always shows N next to the score.
How items are chosen. Researchers sample in the order they find posts, including complaints, rather than picking favourable ones. At most 3 items are counted from any one discussion thread, so a single viral thread cannot dominate a sample. GitHub issues and pull requests are bug reports and change requests, not verdicts on the tool, so they are not counted as sentiment.
Composite
Score = Σ(weight × pillar) / Σ(weight) over the pillars that meet their minimum. Requires at least two pillars. Rounded to a whole number.
Weights by category (Q / E / C):
| Category | Q (adoption) | E (experts) | C (community) |
|---|---|---|---|
| Coding agents | 35 | 40 | 25 |
| Orchestrators and meta-harnesses | 40 | 30 | 30 |
| Personal and desktop agents | 30 | 35 | 35 |
| Browser and computer-use agents | 35 | 40 | 25 |
| Multi-agent frameworks | 45 | 30 | 25 |
| Agent-building platforms | 35 | 35 | 30 |
| Research agents | 35 | 40 | 25 |
| Business and sales agents | 25 | 40 | 35 |
Confidence is shown with every score: High when all three pillars meet their minimum and C has N ≥ 12; Medium when all three meet their minimum; Low when only two do. Rankings list High and Medium scores only. Low-confidence scores are shown as provisional and listed after the ranked products.
Trust and Safety grade
Seven dimensions, each graded A–D from documented facts, or "not enough data":
| Dimension | A | B | C | D |
|---|---|---|---|---|
| Permission model | Sandbox on by default and per-action approval modes documented | Approval modes documented, no default sandbox | Auto-executes by default; controls exist but are opt-in | No documented controls |
| Data access scope | Access is scoped (per-folder/per-repo/allowlists) and documented | Broad access, documented | Broad access, poorly documented | Undocumented |
| Data storage | Runs locally or self-hostable; no required cloud storage of code/content | Cloud storage with documented controls (region, deletion) | Cloud storage, controls unclear | Undocumented |
| Data retention / training | Documented no-training default and a stated retention period (or zero-retention option) | No-training default, retention unclear | Training opt-out only | Trains on user data by default or undocumented |
| Incident history (24 months) | No known security incidents or CVEs | Minor issues only, fixed promptly and disclosed | Serious incidents or CVEs (one or several), each fixed promptly and publicly disclosed | A serious issue still unresolved, or fixes that were slow or undisclosed |
| Compliance | Independent attestations (SOC 2 Type II / ISO 27001) published | In progress or partial | Claims without evidence | None / N/A for a hobby project |
| Transparency | Open source (OSI licence) with public changelog | Closed source with detailed public changelog and docs | Sparse docs or changelog | Opaque |
Overall grade = the rounded mean of graded dimensions (A=4 … D=1), requiring at least 4 graded dimensions.
Dated evidence only. A review or community item counts only when it has a date within the 12 months before collection. Undated items are kept on the page for context but are not scored.
How evidence is checked
Before anything is scored, every review and community item is checked against its source:
- Links:
- The link must point to one specific post, comment, video or article. Search-results pages, subreddit or repository front pages, and malformed video or Hacker News links are rejected.
- A Hacker News item or Reddit post or comment must exist, and its date must match the recorded date within 2 days. Its quoted excerpt must appear in it.
- Articles and videos must exist. A page that returns "not found" is rejected.
- Items that could not be checked: sometimes the check can't reach the source. The page may block automated requests, a Reddit item may be missing from the public archive, or a link to a Reddit post may quote a comment under it. These items are kept, because nothing shows them to be wrong. They are marked "unverified" on the profile, and each profile says how many there are.
- Untrusted research batches: these are held to a stricter standard.
- An item is used only when its excerpt was found at the source.
- The batch's trust grades, pros and cons, pricing and dated changes are withheld.
- Its description, features and status are shown, marked as unverified.
- Rejected items are neither scored nor published. Each profile lists how many items were removed and why, under "Evidence checks".
- Every item must have been checked. The site cannot be built if any review or community item was added after the last check.
Status labels
- Active or in beta.
- Dormant: not deprecated by its makers, but no releases or commits for months.
- Deprecated: its makers are winding it down.
- Acquired.
- Renamed, or discontinued.
Dormant and deprecated products are rated but never recommended on "best for" pages.
What is not scored
- Personal hands-on testing is not part of v2026.09. When we add it, it will be a separate, labelled pillar.
- Sponsored placement does not exist. Affiliate links, if any, never affect scores.
Changelog
- v2026.09, revision 3 (26 Sep 2026), after an independent pre-launch review:
- every item is now checked at its source (see "How evidence is checked"). Items from one research batch were found to have invented links, paraphrased quotes and shifted dates; those items were removed and the affected scores recomputed;
- reviews from tool directories, AI-written pages and unnamed authors are no longer counted;
- GitHub issues no longer count as sentiment;
- at most 3 items are counted per thread;
- the incident-history grades were redone under revision 2's rule;
- the "dormant" status was added;
- (after the second review) items that could not be checked are marked "unverified". A $0 plan is shown as "Free" only when it includes the product: trials and plans that exclude it are labelled as such. The incident grades that were kept now state their reason.
- v2026.09, revision 2 (26 Sep 2026): incident history no longer gives D for incidents that were fixed promptly and disclosed. Heavily scrutinised open-source tools attract more reports, so D is kept for issues that are unresolved or were handled slowly or without disclosure. Undated reviews and posts are no longer scored.
- v2026.09, revision 1 (26 Sep 2026): small-sample shrinkage for reviews and community; benchmarks shown but not scored (model-confounded); rankings exclude low-confidence scores. Made before launch, after the first evidence pass showed three-review products outranking far better-evidenced ones.
- v2026.09 (26 Sep 2026): first published methodology. Replaces the April 2026 draft (per-use-case scores with private weights), which was never populated.