How we assess vendors
Why there’s no numeric score
Most comparison sites give every product a number out of 5 or 10. We don’t, for a simple reason: nobody on this site has run every one of these systems hands-on in a live lab, so a precise numeric score would imply a level of tested confidence we don’t have. Instead, each fiche carries an assessment by dimension: a qualitative reading of the publicly available, sourced evidence, shown as color-coded cards rather than a bar chart.
There’s also no aggregate star rating anywhere on this site, and no aggregateRating in the page’s structured data. That kind of rating is meant for real user reviews, not editorial reading of vendor documentation.
The three levels
Each dimension is marked one of three ways:
- Strong — the evidence found is clearly positive and reasonably specific (a dedicated compliance document, a large, consistently strong independent review base, a concrete deployment option).
- Mixed — the evidence is genuinely split: some positive signal, some real limitation, or a small/uncertain sample size that shouldn’t be over-weighted.
- Limited — little is published on the point (no public price list, a standard listed without detail, few independent reviews), or the evidence points to a real constraint. The level describes what is publicly documented, not the quality of the vendor.
Every card also carries a short rationale explaining specifically why it landed where it did, with a link back to the fiche’s sources for anyone who wants to check the underlying evidence themselves.
On full profiles backed by an evidence file, a dimension with no independent evidence is left out and listed as not assessed, rather than given a level. Short profiles carry no assessment at all.
Independent coverage
Each full profile also states how much independent material backs it: Broad, Moderate or Limited. This measures what third parties have published about the product (public contracts, institutions describing their own use, peer-reviewed papers, user reviews), not our opinion of the vendor or of the product. A well-established product can have limited independent coverage simply because its customers rarely write about the software they use. Each profile gives the reason for its level at the top of its independent sources section.
The six dimensions
Each level follows the same written criteria on every fiche, so that two vendors with the same public record get the same level.
- Overall fit — judged for the teams the profile says the product suits (research discipline, size, deployment model). Strong: at least two independent sources document use in that segment (an organisation’s own records, a public contract, or user reviews from that segment). Mixed: one such source, or documented use that covers only part of the segment. Limited: none.
- Compliance documentation — how much the vendor documents publicly, not whether the software can be used compliantly; it rolls up the standard-by-standard table on each fiche. Strong: a dedicated vendor document describes how the current product, or its GxP edition, meets the main regulation for its buyers (for 21 CFR Part 11: the audit trail and the electronic signature workflow), and validation support is documented (IQ/OQ documents, a validation package or a validated environment). For an electronic lab notebook, a verifiable security certification (an ISO 27001 certificate with its certification body, or a SOC 2 report with a named auditor and audit period) is also required. Mixed: part of this is documented, or it is documented for an earlier version. Limited: standards are listed or stated, with neither controls nor validation support described.
- Deployment execution — Strong: several deployment options are documented, or one option is fully specified (provider and region, or the customer’s choice of region), with at least one concrete execution signal: a published or independently reported implementation time, an availability commitment, or an independent account of how an implementation was carried out. Mixed: deployment is documented but one or both of these conditions are missing (for example, the only hosted option has no published region), or at least a quarter of the reviews we read describe a demanding implementation. Limited: deployment is barely described.
- Pricing transparency — Strong: prices are published for the main commercial offer. Mixed: some prices are published (a free edition with its scope, add-on fees, starting prices for part of the offer), but not for the main commercial offer. Limited: the vendor publishes no price, or only a free edition whose limits are not published. Amounts found in public contracts are shown on the fiche but do not change the level.
- Usability — the themes that recur in independent user reviews about day-to-day use, assessed only when at least three reviews discuss it, and never from the average rating alone. Strong: at least five reviews praise it, complaints are no more than a quarter of the praise, and the sample is not dominated by reviews the vendor solicited. Mixed: praise and complaints both recur, or the sample is small or mostly solicited. Limited: complaints dominate.
- Support — the same approach applied to what reviewers say about support and service; two independent institutional evaluations that rate support favourably, such as a tender evaluation, also qualify for strong.
Compliance evidence levels, specifically
Because compliance is the dimension most likely to matter for a regulated lab, it gets its own table on every fiche, standard by standard, using four levels:
- Vendor-documented — the vendor publishes a specific, citable page or document mapping its product to that standard.
- Vendor-stated — the vendor states support for the standard without describing it in detail or in a dedicated source.
- Not mentioned — nothing on this was found during research; it does not mean the vendor lacks the capability, only that we couldn’t verify it.
- Not supported — the vendor or another credible source explicitly states the standard isn’t supported.
For an electronic lab notebook, two of these matter more than the rest. 21 CFR Part 11 and EU Annex 11 govern whether notebook entries and electronic signatures can stand as regulated records, which is the whole point of replacing a paper notebook in a GxP environment. ISO 27001 and SOC 2 matter because an ELN usually holds a lab’s unpublished research data in a vendor’s cloud. Where a vendor documents an audit trail and signature workflow specifically, that’s called out on the fiche.
See Methodology for the sourcing rules behind all of this.