Skip to content
cypressagencies.com

How to choose a Cypress testing agency: scorecard, pilot and RFP

Updated

By Ronald Renaud, Analyst writing on QA vendors and test automation

Editorial ranking published by Ronald Renaud. Personal project of the publisher. No advertising, no vendor payments.

To choose a software testing company for Cypress work, score each shortlisted vendor on the same eight criteria before any sales call, then run a paid pilot of three to four weeks on one real user flow inside your own repository and CI. Weight Cypress-specific public work and code ownership highest. Read only the client reviews that describe automation. Write ownership of test code, fixtures and CI configuration into the contract before the first commit. Decide on pilot evidence (run time, flake rate, handover quality) rather than on the proposal.

The ranking applies these criteria to twelve agencies using public evidence; this guide turns them into a process for your own shortlist.

How to choose a software testing company: the evaluation scorecard

A scorecard keeps the comparison honest when proposals use different formats. The weights are the ones the ranking uses; they are set out on the methodology page. Copy the table, score each vendor from 0 to 4 per row, multiply by the weight and add up. Score only what the vendor shows you in writing or in the pilot.

CriterionWeightWhat to ask forWhat earns a 4
Cypress depth and public work22Cypress cases, articles, public repositories, component testing workSeveral Cypress cases with metrics and at least one code sample you can read
Test code ownership and handover16Draft contract clause, handover planTests in your repository from day one plus written handover steps and training
CI/CD integration and maintenance14Pipeline example, flake-rate target, maintenance routineParallel runs in your CI, a stated flake ceiling and a named maintenance cadence
Migration experience12A migration case with named frameworksA public migration case with before-and-after run time or stability figures
Client review quality and volume12Links to verified reviews that mention automationRecent verified reviews describing framework or CI work, not manual testing only
Named clients and published outcomes9Named clients and a reference callNamed clients with measured automation results and a reference who used Cypress
Engagement flexibility and onboarding8Models offered, start date, pilot termsSeveral models, a stated start date and written pilot terms
Pricing transparency7Rates by role, minimum project sizeRates by role in writing plus the post-build maintenance cost

Two adjustments are legitimate: with no legacy suite, move migration points to CI/CD; when replacing Selenium or Protractor, raise migration. Make the change before reading any proposal, so the weights do not bend toward a vendor you already like.

Cypress test automation process: what a vendor engagement should look like

A sound Cypress test automation process has five stages, and a proposal should name each one with an owner and an exit condition.

  1. Audit. The vendor reviews the application, the current tests (if any), the CI setup and the release rhythm. Output: a written risk map of user flows.
  2. Framework setup. Folder layout, configuration per environment, test data approach, selector convention and reporting. Output: a pull request into your repository.
  3. First flows. Three to five critical flows automated and running on every pull request. Output: green runs in your CI with a measured run time.
  4. Expansion. Coverage grows in agreed batches, each reviewed like application code. Output: a coverage list mapped to the risk map.
  5. Maintenance or handover. Either a steady maintenance routine with a flake-rate report, or a handover to your team with documentation and paired sessions.

A vendor that quotes a test count before an audit is pricing volume, not risk. The cost guide shows how each stage tends to be billed, and the engagement models guide covers when a dedicated team fits better than a fixed-scope project.

How do you decide what to automate first

Rank candidate flows by two numbers you already have: how often the flow changes and how much a failure costs. A checkout or sign-up path that changes weekly and blocks revenue when broken goes first. A settings page edited twice a year goes last, or stays manual.

A practical order:

  • Smoke path. Log in, reach the main screen, perform the core action. This becomes the pull-request gate.
  • Revenue and data-loss paths. Payment, subscription changes, record deletion, exports.
  • Flows with a history of regressions. Pull these from your bug tracker, not from memory.
  • Everything else, only once the first groups run reliably.

Ask each vendor to build this ranking from your bug tracker during the audit.

Cypress testing best practices to require in the contract

The official Cypress best practices page lists habits you can turn into acceptance criteria for vendor code. The ones worth writing into a statement of work:

Practice (per Cypress docs)How to check it in vendor code
Select elements by dedicated data-* attributesSearch the specs for long CSS chains or text-based selectors
Log in programmatically, not through the UI in every testLook for a login command or cy.session instead of repeated form fills
Each test passes when run on its ownRun one spec file in isolation; it should pass
No arbitrary waits; wait on route aliases or assertionsSearch for cy.wait( followed by a number
Test only sites you controlThird-party logins and payment pages are stubbed or covered by API checks

Retries deserve their own clause. Cypress test retries are off by default, and a vendor can switch them on to hide flaky tests. Ask for the retry setting in writing and for a report of tests that passed only on retry.

How to set up Cypress so the vendor's work stays yours

Set up the base project yourself, or have the vendor do it in your repository under your accounts. Per the Cypress installation docs, Cypress installs as a dev dependency with npm install cypress --save-dev and opens with npx cypress open.

A setup that survives a vendor change has:

  • Cypress pinned in your package.json and the CI job in your pipeline file, run by your CI account.
  • Secrets in your CI secret store, never in a vendor-held config.
  • Test data created by scripts in the repository and reports written to storage you control.

Parallel runs need a decision early. The Cypress parallelization docs state that the --parallel flag requires --record, which means Cypress Cloud. The alternative is splitting spec files across CI machines yourself or through a self-hosted dashboard. Testrig Technologies documents one such case: a retailer's Cypress tests running in parallel in GitHub Actions with a Sorry-Cypress dashboard (vendor-published).

Cypress pilot project: a four-week plan

A pilot tests working habits on your code. Keep it paid, short and scoped to one flow group. Several ranked vendors publish trial terms: BetterQA states a two-week trial without a contract, Testrig Technologies offers a free proof of concept, and TestDevLab lists one-time engagements from one week.

WeekVendor deliversYou measurePass condition
1Audit notes, risk map, framework pull request in your repositoryQuality of questions asked; review comments needed on the PRFramework merged; risk map matches your bug history
2Three to five smoke flows running on every pull requestPR run time; number of red runs not caused by real defectsSmoke run within your agreed time limit
3Next flow group; one API-level check; flaky-test reportFlake rate over the week; time from failure to diagnosisFlake report lists cause and fix for each flaky test
4Documentation, a paired session with your developer, a written estimate for full scopeCan your developer add a test alone after the session?Your team adds and merges one test without vendor help

Agree the pass conditions in writing before week 1. If two vendors reach the final round, run the same pilot with both.

RFP questions that go past the sales call

The questions on the ranking page cover the first call. An RFP can ask for artefacts instead of answers:

  1. Send one anonymized Cypress spec file and its support commands from a past project.
  2. Describe your selector policy and what you do when the application has no data-* attributes.
  3. How do you create and clean up test data, and which parts need database access?
  4. Show a flaky-test report from a past engagement, with names removed.
  5. How do you run tests in parallel if we do not buy Cypress Cloud?
  6. Name the engineers proposed for the pilot and their Cypress experience; what is the replacement policy?
  7. Which parts of your delivery depend on proprietary tools, libraries or licences?
  8. What is the monthly cost once the framework is built and work shifts to maintenance?
  9. What does handover include, and when does it start before the contract ends?

Where a vendor declines to send artefacts for questions 1 and 4, record the refusal in your scorecard.

Red flags when evaluating a Cypress agency

Red flagWhy it mattersWhat to ask instead
Cypress appears only in a tool listNo evidence of framework workA Cypress case, article or code sample
Tests would run in the vendor's repository or cloud accountYou lose the suite when the contract endsYour repository and CI from the first commit
A test-count promise before any auditVolume replaces risk-based coverageA risk map built from your bug tracker
Retries enabled with no reportFlaky tests stay hiddenRetry setting and a pass-on-retry report
Reviews that describe only manual testingRatings do not reflect automation workReviews mentioning Cypress, CI or framework builds
Handover priced separately and undefinedThe exit cost appears at the worst momentHandover scope and price in the main contract

How to verify what a vendor claims

Reviews. Clutch checks reviewer identity and project details. Its help center says a review it cannot confirm but still finds useful is published as "Not Verified". Count only verified reviews, then filter them by the words automation, Cypress, CI and framework. Check the review date and the reviewer's company size; the comparison table shows review counts per ranked vendor.

Case studies. Treat vendor case figures as vendor-published until a reference confirms them. DeviQA's Compass case, for example, reports pull-request testing cut from over 40 minutes to 10 minutes (vendor-published). A reference call with the client is the way to confirm a figure like that.

Code ownership clauses. Ask for these in the master agreement or statement of work:

  • Assignment of all rights in test code, fixtures, configuration and documentation to you on creation, not on final payment.
  • Work happens in repositories and CI accounts you own; vendor access is granted and revoked by you.
  • A list of any vendor libraries used, with a perpetual licence to keep using them after the contract.
  • Handover deliverables named in the contract: documentation, recorded walkthroughs, paired sessions.
  • A notice period long enough to complete handover.

Public ownership statements vary widely across the ranked vendors. ThinkSys publishes explicit terms on repository ownership and handover, which is why it scores 100 on that criterion. For TestFort, Abstracta and Testrig Technologies no public statement on test-code ownership was found, so the clause has to come from your own draft. The scenario table maps buyer priorities to vendors.

FAQ

Which Cypress agency should I choose?

Choose the agency that scores highest on your own weighted scorecard and then passes a pilot on your code. The ranking gives a starting shortlist scored on public evidence across eight criteria. A team replacing Selenium should weight migration more; a team on a strict budget should weight pricing transparency more. Fix the weights before reading proposals, then pilot the top two vendors on the same scope.

What is a Cypress pilot project and how long should it last?

A Cypress pilot project is a short paid engagement on one real flow group, run in your repository and CI, before a long contract. Three to four weeks is enough to see framework setup, pull-request runs, a flaky-test report and a handover session. Agree pass conditions before the start: a run-time limit, a flake-rate ceiling and one test added by your own developer without vendor help.

Who owns the Cypress code after the project?

You own it only if the contract says so. Ask for assignment of all rights in test code, fixtures, CI configuration and documentation on creation, with the work done in repositories and CI accounts you control. Get a perpetual licence for any vendor library that stays in the suite. Several ranked vendors publish no statement on test-code ownership, so the clause usually comes from your own draft.

Can an agency take over an existing Cypress test suite?

Yes, and the audit matters more in that case. Ask the vendor to run the suite as it is, report run time and failure rate, and classify each failing test as a real defect, a flaky test or an obsolete one. The plan should fix selectors, remove arbitrary waits and set a retry policy before adding coverage. DeviQA's QIMA case describes upgrading an outdated Cypress version (vendor-published).

How should a vendor handle flaky Cypress tests?

A vendor should measure flakiness, not hide it. Cypress retries are off by default, and switching them on without a report masks unstable tests. Ask for a weekly list of tests that passed only on retry, with the cause and fix for each. Common causes are arbitrary waits, selectors tied to styling and tests that depend on data created by other tests. Put a flake-rate ceiling in the statement of work.

Can I run Cypress in parallel without Cypress Cloud?

Yes, but not with the built-in flag alone. Cypress documentation states that --parallel requires --record, which sends results to Cypress Cloud. Without it, teams split spec files across CI machines themselves or use a self-hosted dashboard. Ask each vendor which route it proposes. If it chooses Cypress Cloud, the subscription should be in your name so the run history stays with you.

How long does it take to build a Cypress test suite?

It depends on the number of flows, the state of test data and whether the application exposes stable selectors. A pilot gives the best estimate: after four weeks you know how many flows the team automates per week on your code. Ask the vendor to extrapolate the full scope from that measured rate, in writing, with its assumptions listed. The cost guide covers how time converts into budget.

When does test automation not make sense?

Automation pays back slowly on flows that change every sprint during early discovery, on screens edited once or twice a year, and on checks that need human judgement, such as visual design review or exploratory testing. It also struggles without a stable test environment or reproducible test data. A vendor that proposes to automate everything without discussing these limits has not looked closely at your product.