How to choose a Cypress testing agency: scorecard, pilot and RFP
Updated
By Ronald Renaud, Analyst writing on QA vendors and test automation
Editorial ranking published by Ronald Renaud. Personal project of the publisher. No advertising, no vendor payments.
To choose a software testing company for Cypress work, score each shortlisted vendor on the same eight criteria before any sales call, then run a paid pilot of three to four weeks on one real user flow inside your own repository and CI. Weight Cypress-specific public work and code ownership highest. Read only the client reviews that describe automation. Write ownership of test code, fixtures and CI configuration into the contract before the first commit. Decide on pilot evidence (run time, flake rate, handover quality) rather than on the proposal.
The ranking applies these criteria to twelve agencies using public evidence; this guide turns them into a process for your own shortlist.
How to choose a software testing company: the evaluation scorecard
A scorecard keeps the comparison honest when proposals use different formats. The weights are the ones the ranking uses; they are set out on the methodology page. Copy the table, score each vendor from 0 to 4 per row, multiply by the weight and add up. Score only what the vendor shows you in writing or in the pilot.
| Criterion | Weight | What to ask for | What earns a 4 |
|---|---|---|---|
| Cypress depth and public work | 22 | Cypress cases, articles, public repositories, component testing work | Several Cypress cases with metrics and at least one code sample you can read |
| Test code ownership and handover | 16 | Draft contract clause, handover plan | Tests in your repository from day one plus written handover steps and training |
| CI/CD integration and maintenance | 14 | Pipeline example, flake-rate target, maintenance routine | Parallel runs in your CI, a stated flake ceiling and a named maintenance cadence |
| Migration experience | 12 | A migration case with named frameworks | A public migration case with before-and-after run time or stability figures |
| Client review quality and volume | 12 | Links to verified reviews that mention automation | Recent verified reviews describing framework or CI work, not manual testing only |
| Named clients and published outcomes | 9 | Named clients and a reference call | Named clients with measured automation results and a reference who used Cypress |
| Engagement flexibility and onboarding | 8 | Models offered, start date, pilot terms | Several models, a stated start date and written pilot terms |
| Pricing transparency | 7 | Rates by role, minimum project size | Rates by role in writing plus the post-build maintenance cost |
Two adjustments are legitimate: with no legacy suite, move migration points to CI/CD; when replacing Selenium or Protractor, raise migration. Make the change before reading any proposal, so the weights do not bend toward a vendor you already like.
Cypress test automation process: what a vendor engagement should look like
A sound Cypress test automation process has five stages, and a proposal should name each one with an owner and an exit condition.
- Audit. The vendor reviews the application, the current tests (if any), the CI setup and the release rhythm. Output: a written risk map of user flows.
- Framework setup. Folder layout, configuration per environment, test data approach, selector convention and reporting. Output: a pull request into your repository.
- First flows. Three to five critical flows automated and running on every pull request. Output: green runs in your CI with a measured run time.
- Expansion. Coverage grows in agreed batches, each reviewed like application code. Output: a coverage list mapped to the risk map.
- Maintenance or handover. Either a steady maintenance routine with a flake-rate report, or a handover to your team with documentation and paired sessions.
A vendor that quotes a test count before an audit is pricing volume, not risk. The cost guide shows how each stage tends to be billed, and the engagement models guide covers when a dedicated team fits better than a fixed-scope project.
How do you decide what to automate first
Rank candidate flows by two numbers you already have: how often the flow changes and how much a failure costs. A checkout or sign-up path that changes weekly and blocks revenue when broken goes first. A settings page edited twice a year goes last, or stays manual.
A practical order:
- Smoke path. Log in, reach the main screen, perform the core action. This becomes the pull-request gate.
- Revenue and data-loss paths. Payment, subscription changes, record deletion, exports.
- Flows with a history of regressions. Pull these from your bug tracker, not from memory.
- Everything else, only once the first groups run reliably.
Ask each vendor to build this ranking from your bug tracker during the audit.
Cypress testing best practices to require in the contract
The official Cypress best practices page lists habits you can turn into acceptance criteria for vendor code. The ones worth writing into a statement of work:
| Practice (per Cypress docs) | How to check it in vendor code |
|---|---|
Select elements by dedicated data-* attributes | Search the specs for long CSS chains or text-based selectors |
| Log in programmatically, not through the UI in every test | Look for a login command or cy.session instead of repeated form fills |
| Each test passes when run on its own | Run one spec file in isolation; it should pass |
| No arbitrary waits; wait on route aliases or assertions | Search for cy.wait( followed by a number |
| Test only sites you control | Third-party logins and payment pages are stubbed or covered by API checks |
Retries deserve their own clause. Cypress test retries are off by default, and a vendor can switch them on to hide flaky tests. Ask for the retry setting in writing and for a report of tests that passed only on retry.
How to set up Cypress so the vendor's work stays yours
Set up the base project yourself, or have the vendor do it in your repository under your accounts. Per the Cypress installation docs, Cypress installs as a dev dependency with npm install cypress --save-dev and opens with npx cypress open.
A setup that survives a vendor change has:
- Cypress pinned in your
package.jsonand the CI job in your pipeline file, run by your CI account. - Secrets in your CI secret store, never in a vendor-held config.
- Test data created by scripts in the repository and reports written to storage you control.
Parallel runs need a decision early. The Cypress parallelization docs state that the --parallel flag requires --record, which means Cypress Cloud. The alternative is splitting spec files across CI machines yourself or through a self-hosted dashboard. Testrig Technologies documents one such case: a retailer's Cypress tests running in parallel in GitHub Actions with a Sorry-Cypress dashboard (vendor-published).
Cypress pilot project: a four-week plan
A pilot tests working habits on your code. Keep it paid, short and scoped to one flow group. Several ranked vendors publish trial terms: BetterQA states a two-week trial without a contract, Testrig Technologies offers a free proof of concept, and TestDevLab lists one-time engagements from one week.
| Week | Vendor delivers | You measure | Pass condition |
|---|---|---|---|
| 1 | Audit notes, risk map, framework pull request in your repository | Quality of questions asked; review comments needed on the PR | Framework merged; risk map matches your bug history |
| 2 | Three to five smoke flows running on every pull request | PR run time; number of red runs not caused by real defects | Smoke run within your agreed time limit |
| 3 | Next flow group; one API-level check; flaky-test report | Flake rate over the week; time from failure to diagnosis | Flake report lists cause and fix for each flaky test |
| 4 | Documentation, a paired session with your developer, a written estimate for full scope | Can your developer add a test alone after the session? | Your team adds and merges one test without vendor help |
Agree the pass conditions in writing before week 1. If two vendors reach the final round, run the same pilot with both.
RFP questions that go past the sales call
The questions on the ranking page cover the first call. An RFP can ask for artefacts instead of answers:
- Send one anonymized Cypress spec file and its support commands from a past project.
- Describe your selector policy and what you do when the application has no
data-*attributes. - How do you create and clean up test data, and which parts need database access?
- Show a flaky-test report from a past engagement, with names removed.
- How do you run tests in parallel if we do not buy Cypress Cloud?
- Name the engineers proposed for the pilot and their Cypress experience; what is the replacement policy?
- Which parts of your delivery depend on proprietary tools, libraries or licences?
- What is the monthly cost once the framework is built and work shifts to maintenance?
- What does handover include, and when does it start before the contract ends?
Where a vendor declines to send artefacts for questions 1 and 4, record the refusal in your scorecard.
Red flags when evaluating a Cypress agency
| Red flag | Why it matters | What to ask instead |
|---|---|---|
| Cypress appears only in a tool list | No evidence of framework work | A Cypress case, article or code sample |
| Tests would run in the vendor's repository or cloud account | You lose the suite when the contract ends | Your repository and CI from the first commit |
| A test-count promise before any audit | Volume replaces risk-based coverage | A risk map built from your bug tracker |
| Retries enabled with no report | Flaky tests stay hidden | Retry setting and a pass-on-retry report |
| Reviews that describe only manual testing | Ratings do not reflect automation work | Reviews mentioning Cypress, CI or framework builds |
| Handover priced separately and undefined | The exit cost appears at the worst moment | Handover scope and price in the main contract |
How to verify what a vendor claims
Reviews. Clutch checks reviewer identity and project details. Its help center says a review it cannot confirm but still finds useful is published as "Not Verified". Count only verified reviews, then filter them by the words automation, Cypress, CI and framework. Check the review date and the reviewer's company size; the comparison table shows review counts per ranked vendor.
Case studies. Treat vendor case figures as vendor-published until a reference confirms them. DeviQA's Compass case, for example, reports pull-request testing cut from over 40 minutes to 10 minutes (vendor-published). A reference call with the client is the way to confirm a figure like that.
Code ownership clauses. Ask for these in the master agreement or statement of work:
- Assignment of all rights in test code, fixtures, configuration and documentation to you on creation, not on final payment.
- Work happens in repositories and CI accounts you own; vendor access is granted and revoked by you.
- A list of any vendor libraries used, with a perpetual licence to keep using them after the contract.
- Handover deliverables named in the contract: documentation, recorded walkthroughs, paired sessions.
- A notice period long enough to complete handover.
Public ownership statements vary widely across the ranked vendors. ThinkSys publishes explicit terms on repository ownership and handover, which is why it scores 100 on that criterion. For TestFort, Abstracta and Testrig Technologies no public statement on test-code ownership was found, so the clause has to come from your own draft. The scenario table maps buyer priorities to vendors.
FAQ
Which Cypress agency should I choose?
Choose the agency that scores highest on your own weighted scorecard and then passes a pilot on your code. The ranking gives a starting shortlist scored on public evidence across eight criteria. A team replacing Selenium should weight migration more; a team on a strict budget should weight pricing transparency more. Fix the weights before reading proposals, then pilot the top two vendors on the same scope.
What is a Cypress pilot project and how long should it last?
A Cypress pilot project is a short paid engagement on one real flow group, run in your repository and CI, before a long contract. Three to four weeks is enough to see framework setup, pull-request runs, a flaky-test report and a handover session. Agree pass conditions before the start: a run-time limit, a flake-rate ceiling and one test added by your own developer without vendor help.
Who owns the Cypress code after the project?
You own it only if the contract says so. Ask for assignment of all rights in test code, fixtures, CI configuration and documentation on creation, with the work done in repositories and CI accounts you control. Get a perpetual licence for any vendor library that stays in the suite. Several ranked vendors publish no statement on test-code ownership, so the clause usually comes from your own draft.
Can an agency take over an existing Cypress test suite?
Yes, and the audit matters more in that case. Ask the vendor to run the suite as it is, report run time and failure rate, and classify each failing test as a real defect, a flaky test or an obsolete one. The plan should fix selectors, remove arbitrary waits and set a retry policy before adding coverage. DeviQA's QIMA case describes upgrading an outdated Cypress version (vendor-published).
How should a vendor handle flaky Cypress tests?
A vendor should measure flakiness, not hide it. Cypress retries are off by default, and switching them on without a report masks unstable tests. Ask for a weekly list of tests that passed only on retry, with the cause and fix for each. Common causes are arbitrary waits, selectors tied to styling and tests that depend on data created by other tests. Put a flake-rate ceiling in the statement of work.
Can I run Cypress in parallel without Cypress Cloud?
Yes, but not with the built-in flag alone. Cypress documentation states that --parallel requires --record, which sends results to Cypress Cloud. Without it, teams split spec files across CI machines themselves or use a self-hosted dashboard. Ask each vendor which route it proposes. If it chooses Cypress Cloud, the subscription should be in your name so the run history stays with you.
How long does it take to build a Cypress test suite?
It depends on the number of flows, the state of test data and whether the application exposes stable selectors. A pilot gives the best estimate: after four weeks you know how many flows the team automates per week on your code. Ask the vendor to extrapolate the full scope from that measured rate, in writing, with its assumptions listed. The cost guide covers how time converts into budget.
When does test automation not make sense?
Automation pays back slowly on flows that change every sprint during early discovery, on screens edited once or twice a year, and on checks that need human judgement, such as visual design review or exploratory testing. It also struggles without a stable test environment or reproducible test data. A vendor that proposes to automate everything without discussing these limits has not looked closely at your product.