In the world of software development, we’ve long held onto the belief that a separate Quality Assurance (QA) or staging environment is essential for delivering reliable software. But what if I told you that this might not be the case anymore? Let’s explore why some modern development practices are challenging this conventional wisdom and how we can ensure quality without a dedicated QA environment.

Rethinking the Purpose of QA

Traditionally, QA environments have been used for various types of testing:

• Integration Testing
• Manual Testing (by developers)
• Cross-browser Testing
• Device Testing
• Acceptance Testing
• End-to-End Testing

But do we really need a separate environment for all of these? Let’s break it down.

The Pros and Cons of Mocks vs. End-to-End Testing

When we talk about testing, we often debate between using mocks and real systems. Both approaches have their merits and drawbacks.

Cons of Mocks

• Need frequent updates to match new versions
• May miss breaking changes that affect your system
• Can’t guarantee full system compatibility

Cons of Real Systems (QA/Staging)

• Not truly representative of production
• Require maintenance
• May lack proper alerting and monitoring
• Often have less hardware, resulting in slower performance

As Cindy Sridharan, a testing engineer and blogger, puts it:

“I’m more and more convinced that staging environments are like mocks – at best a pale imitation of the genuine article and the worst form of confirmation bias. It’s still better than having nothing – but ‘works in staging’ is only one step better than ‘works on my machine’.”

Consumer-Driven Contract Testing: A Replacement for End-to-End Testing

Consumer-Driven Contract Testing (CDCT) is more than just a bridge between mocks and real systems – it’s a powerful approach that can effectively replace traditional end-to-end testing. This method allows for “distributed end-to-end tests” without the need for a full QA environment. Let’s explore how this process works in detail.

The CDCT Process

1. Defining and Recording Pact Contracts
   • Consumers write tests that define their expectations of the provider’s API.
   • These tests generate “pacts” – JSON files that document the interactions between consumers and providers.
   • Pacts include details like HTTP method, path, headers, request body, and expected response.
2. Using Mocks for Consumer-Side Testing
   • The generated pacts are used to create mock providers.
   • Consumers can now run their tests against these mocks, simulating the provider’s behavior.
   • This allows consumers to develop and test their code without needing the actual provider service.
3. Publishing Contracts by API Consumers
   • Once generated and tested locally, these pact files are published to a shared location, often called a “Pact Broker”.
   • The Pact Broker serves as a central repository for all contracts in your system.
4. Verifying Contracts in Provider Pipelines
   • Providers retrieve the relevant pacts from the Pact Broker.
   • They run these contracts against their actual implementation as part of their CI/CD pipeline.
   • This step ensures that the provider can meet all the expectations set by its consumers.
   • If a provider’s changes would break a consumer’s expectations, the pipeline fails, preventing the release of breaking changes.
5. Continuous Verification
   • As both consumers and providers evolve, the process is repeated.
   • New or updated pacts are published and verified, ensuring ongoing compatibility.

How CDCT Replaces End-to-End Testing

Consumer-Driven Contract Testing (CDCT) changes the testing process by enabling teams to conduct testing independently of other systems. This approach allows developers to use mocks for testing, eliminating the need for a fully integrated environment and providing fast feedback early in the development process.

The key advantage of CDCT lies in its solution to the stale mock problem. The same pact contract that generates the mock also publishes a test that verifies the assumptions made in the mock. This test is then run on the backend system, ensuring that the mock remains an accurate representation of the actual service behavior.

As systems grow in complexity, CDCT proves to be more scalable and maintainable than traditional end-to-end testing. It covers the same ground as end-to-end tests but in a more modular way, basing scenarios on real consumer requirements. This approach not only eliminates environment dependencies but also ensures that testing reflects actual use cases, making it a powerful replacement for traditional end-to-end testing in modern development practices.

In my opinion, you need end to end test to verify a feature works. But we know end-to-end test are flakey, so pact is the only viable solution I have found that gives you the best of both worlds.

Dark Launching: Enabling UAT in Production

Dark launching is a powerful technique that allows development teams to conduct User Acceptance Testing (UAT) directly in the production environment, effectively eliminating the need for a separate QA environment for this purpose. Let’s explore how this works and why it’s beneficial.

Dark launching, also known as feature toggling or feature flags, involves deploying new features to production in a disabled state. These features can then be selectively enabled for specific users or groups, allowing for controlled testing in the real production environment.

By leveraging dark launching for UAT, development teams can confidently test new features in the most realistic environment possible – production itself. This approach not only removes the need for a separate QA environment but also provides more accurate testing results and faster time-to-market for new features. It’s a key practice in modern development that supports rapid iteration and high-quality software delivery.

But it takes me a long time to deploy to production, it’s much faster to deploy to QA, right?

Your production deployment should be as fast as QA; there’s no reason for it not to be. Normally if it is, you have a CI pipeline that isn’t optimized. Your CI should take less than 10 minutes…

The Ten-Minute Build: A Development Practice from Extreme Programming

Kent Beck, in “Extreme Programming Explained,” introduces the concept of the Ten-Minute Build. This practice emphasizes the importance of being able to automatically build the whole system and run all tests in ten minutes or less. If the build takes longer than ten minutes, everyone stops working and optimizes it until it takes less.

He also says: “Practices should lower stress. An automated build becomes a stress reliever at crunch-times. ‘Did we make a mistake? Let’s just build and see’.”

But I didn’t write my tests yet, so I don’t want to go to production yet…

Test-First Development: Building Confidence for Production Releases

In the realm of modern software development, Test-First Development practices such as Behavior-Driven Development (BDD) and Acceptance Test-Driven Development (ATDD) have emerged as powerful tools for building confidence in code quality.

At its core, Test-First Development involves writing tests before writing the actual code. This might seem counterintuitive at first, but it offers several advantages. By defining the expected behavior upfront, developers gain a clear understanding of what the code needs to accomplish. This clarity helps in writing more focused, efficient code that directly addresses the requirements.

The power of these Test-First Development practices lies in their ability to instill confidence in the code from the very beginning. As developers write code to pass these predefined tests, they’re essentially building in quality from the ground up. This approach shifts the focus from finding bugs after development to preventing them during development.

By embracing Test-First Development, it will not only enhance your development process but makes practices like dark launching safe for UAT.

When to Use (and Not Use) Dark Launching

Dark launching is great for:

• Showing feature progress to designers or Product Owners
• Allowing stakeholders to use incremental UI changes

However, it’s not suitable for manual testing. Your automated tests should give you confidence in your changes.

Addressing Cross-Browser Testing

Cross-browser testing can be handled through automation tools like Playwright or by using local environments for fine-tuning and inspection.

The Case for Eliminating QA Environments

What I find most commonly is engineers who can’t run their systems locally. If this is the case for you, in order to see your changes, you need to wait for a CI pipeline and deployment to QA. This means your inner loop of development includes CI, and this will slow you down A LOT.

Our goal is to make the inner loop of development fast. QA environments, in my experience, are a crutch that engineers use to support a broken local developer experience. By taking them away, it forces people to fix the local experience and keep their production pipeline lean and fast, both things we want.

While it might be tempting to keep a QA environment “just in case,” this can lead to falling back into old habits.

Conclusion

Embracing modern development practices without a QA environment might seem daunting at first, but it can lead to faster, more reliable software delivery. By focusing on practices like consumer-driven contract testing, dark launching, and test-first development, teams can ensure quality without the overhead of maintaining a separate QA environment. Remember, as with any significant change, it requires commitment and a willingness to break old habits. But the rewards – in terms of efficiency, quality, and speed – can be substantial.