Most SaaS procurement decisions get made on three things: feature coverage, sticker price, and whoever ran the demo most charismatically. The bill comes due eighteen months later when the vendor doubles their pricing during contract renewal, gets acquired by a competitor, or quietly degrades the SLA on the integration your business depends on. By then your team is locked in, the data is hard to extract, and the cost to switch dwarfs the cost to stay.
A vendor risk scorecard does not eliminate that risk. It makes the risk visible before the contract is signed, so the trade-off is at least deliberate. This guide walks through how to build a scorecard that fits in a single spreadsheet, gets filled in within an afternoon per vendor, and produces a numeric comparison your procurement committee can defend in a budget meeting.
The audience is mid-sized companies (50 to 500 employees) buying SaaS in the $10K to $250K annual range, where the procurement decision is significant enough to matter but not large enough to justify a dedicated vendor-risk-management team.
Why feature comparison alone is not enough
The default vendor comparison is a feature matrix. Vendor A has SSO and SCIM. Vendor B has SSO and audit logs. Vendor C is missing both but is 40 percent cheaper. The team picks based on what the features add up to and where the budget lands.
What that approach misses, consistently, are the dimensions that determine whether the vendor will still be a good partner three years later. A vendor with an excellent feature set but a six-month runway is a worse purchase than a vendor with a slightly weaker feature set and ten years of profitability. A vendor that lost their CISO last quarter is a worse purchase than the equivalent vendor whose security team has been stable for five years.
These dimensions exist in vendor RFPs but typically as text fields nobody scores. The scorecard's job is to force a number, on each dimension, that can be compared apples-to-apples across vendors.
Photo by Joachim Schnürle on Pexels
The eight dimensions worth scoring
Most useful scorecards cover six to ten dimensions. More than ten and nobody fills them in; fewer than six and important risk categories get bundled together to the point of uselessness. The eight below cover the major axes for typical SaaS purchases.
1. Security posture. Does the vendor have SOC 2 Type II or ISO 27001 certification? Are they current (within 12 months)? Have they had a breach in the last 24 months, and if so, how did they communicate it? Reference: the AICPA SOC 2 overview for what these certifications actually attest to.
2. Data handling and privacy. Where is data hosted (region, cloud provider)? Are subprocessors disclosed? Is GDPR or CCPA compliance documented? Can you export your data on demand, and in what format? The NIST privacy framework is the standard reference for what good data-handling practice looks like.
3. Reliability and SLA. What is the documented uptime SLA, and what are the credits if missed? Does the vendor publish a status page with historical incident data? How transparent are they about postmortems? A vendor that publishes detailed postmortems is signaling a healthier engineering culture than one that hides them.
4. Financial health. Is the vendor profitable, or are they burning runway? When did they last raise, and from whom? What is their announced employee headcount, and is it growing, flat, or shrinking? Public companies disclose this in 10-Ks; private companies require LinkedIn-headcount research and Crunchbase. For the smallest vendors, the answer might be "we cannot tell," which is itself information.
5. Integration depth. How does this vendor integrate with your existing stack? Is it API-first, with documented stable endpoints? Webhook support? Pre-built integrations with your identity provider, your data warehouse, your finance system? Pre-built integrations save weeks of engineering. Their absence does not disqualify a vendor but does add hidden cost to the deployment.
6. Switching cost. If you need to leave this vendor in two years, what is the realistic effort? Can you export all your data? In what format? Are there contractual notice periods or early-termination penalties? Are critical workflows hard-coded against this vendor's API in ways that would require months to refactor?
7. Roadmap alignment. Does the vendor's publicly announced roadmap match the direction your team is heading? Has the vendor shipped what they said they would ship in the last 12 months? Roadmap velocity and reliability matter as much as roadmap content.
8. Support quality. Do they offer phone, email, or chat support? What are the response time SLAs? Is the support team in-house or outsourced? Reviews on independent platforms (Gartner Peer Insights, G2, TrustRadius) reveal the gap between marketing-stated support quality and lived experience.
Photo by Sora Shimazaki on Pexels
The scoring rubric
Each dimension gets scored from 1 to 5. Specific anchor points for each score (avoiding "what does a 3 mean?" debates):
- 5 - exceeds standard practice in this category; the vendor differentiates here
- 4 - meets standard practice with strong evidence
- 3 - meets standard practice with adequate evidence
- 2 - partial evidence or known weaknesses
- 1 - fails standard practice; significant risk
For each dimension, the team writes a one-sentence justification next to the score. This is the most-skipped step and the most important. A 4 with the justification "SOC 2 Type II current, no breaches in last 36 months, CISO in role since 2022" is defensible. A 4 with no justification is just a number, and in a procurement debate it loses to a competing 4 with no justification, leaving the room arguing about who feels more strongly.
"The scorecard is mostly a forcing function for the conversations you would have had after the contract anyway. Doing them before signing means the deal price reflects the risk you are taking, not just the price the vendor wanted." - Dennis Traina, founder of 137Foundry
Weighting matters more than people expect
Not all dimensions are equal. For a vendor that will hold customer PII, security posture and data handling are existential; a 2 on either should disqualify the vendor regardless of other scores. For a vendor that will own a non-critical workflow (an internal scheduling tool, a meeting transcription service), security still matters but financial health and switching cost might be weighted lower.
A common weighting for typical mid-market SaaS:
- Security posture: 20%
- Data handling and privacy: 15%
- Reliability and SLA: 15%
- Financial health: 10%
- Integration depth: 10%
- Switching cost: 15%
- Roadmap alignment: 10%
- Support quality: 5%
Adjust per purchase. A scorecard for a marketing automation tool weights integration depth higher; a scorecard for a payroll system weights security and data handling higher.
The weighted score per vendor is the sum across dimensions of (score x weight). A vendor scoring 4s across the board ends up with a 4.0 weighted score. A vendor with 5s on security and reliability but 2s on financial health and roadmap ends up closer to 3.5, which is appropriate because the financial-health 2 is a real risk even if the security 5 is impressive.
Where the data comes from
Most of the data points needed for a scorecard come from four sources:
- The vendor's trust center, status page, and security questionnaire response. These are usually the fastest source for security, data handling, and reliability.
- The vendor's contract redlines and master service agreement. SLA specifics, data export rights, termination clauses, and pricing-escalation caps live here. Lawyers should be involved; this is where switching cost is determined.
- Independent review platforms (G2, Gartner Peer Insights, TrustRadius, Capterra). Support quality, real-world reliability, and roadmap execution show up in user reviews more reliably than in vendor marketing.
- LinkedIn, Crunchbase, news search. Financial health, key-personnel turnover, recent acquisitions, recent layoffs. The 137Foundry team typically spends 30 to 60 minutes per vendor on this category alone for any purchase above $25K annual.
For sensitive categories (security, financial), it is fair to ask the vendor for direct evidence: the SOC 2 audit report under NDA, a redacted summary of their financial position, a reference call with a customer of similar size. Vendors who refuse these requests after a serious procurement conversation are signaling something about how the relationship will go.
Photo by Pixabay on Pexels
How the scorecard plugs into procurement governance
A scorecard is most useful when it is embedded in the procurement workflow, not bolted on at the end. Three checkpoints:
- Pre-RFP: agree on the dimensions and weights for this purchase. This forces alignment on what actually matters before vendors start pitching.
- Mid-evaluation: after each vendor demo and reference call, fill in the scorecard. Doing it incrementally beats trying to score four vendors from memory at the end.
- Pre-signature: review the scorecard with the committee. Any vendor with a score below 2 on a category you weighted above 10% is a vendor you need a deliberate exception decision for.
For larger purchases (over $100K annual), the scorecard moves up to a risk-committee review. For the smallest purchases (under $10K annual), a simplified three-dimension scorecard (security, switching cost, financial health) is enough, with the full eight-dimension version reserved for higher-stakes deals.
Worth knowing: a useful scorecard is one the procurement committee actually fills in. A perfect 14-dimension scorecard nobody completes is worth less than a scrappy 6-dimension scorecard the team uses every time. The 137Foundry team has helped several mid-market companies build internal scorecards and the pattern is consistent: simpler scorecards with stronger weighting discipline outperform sophisticated scorecards that nobody maintains. For broader context on how technology buying decisions tie into engineering team productivity and operational risk, see the 137Foundry services overview and the 137Foundry company background.
A few useful external references
- AICPA SOC 2 overview for understanding what SOC 2 actually certifies
- NIST cybersecurity framework for the security-posture dimension
- Cloud Security Alliance CCM for cloud-vendor-specific risk dimensions
- Gartner Peer Insights for independent vendor reviews
What this scorecard does not do
Two things the scorecard explicitly does not handle:
The scorecard does not replace legal review. Contract language matters and a vendor who scores 5 across the board can still write a master service agreement that makes them effectively un-fireable. Have a lawyer read the contract.
The scorecard does not predict acquisition. A small SaaS company can be acquired by anyone at any time, and the acquirer's intentions may not match the acquired company's roadmap. The scorecard surfaces signals (vendor headcount, last funding round, founder activity) but cannot eliminate the risk.
What the scorecard does is force structured thinking about real risks before they become real problems. That is usually enough.
