Part of the F5 Experience is also running tests. Can I open my IDE and “just run the tests” after cloning a project?

Unit tests generally yes, but pretty much every other kind of test that engineers create these days (UI, Integration, end-to-end, etc.) needs complex environments managed either manually, or spawned in Kubernetes or a Docker Compose that brings your laptop to a crawl when running.

End-to-end tests I’ll leave for another day, and focus mainly on the ones with fewer hops such as UI and integration tests.

So what’s the problem here? The problem is if tests are hard to run, people won’t run them locally. They’ll wait for CI, and CI then becomes a part of the inner loop of development. You rely on it for dev feedback for code changes locally.

Even the best CI pipelines are looking at at least 5-10 minutes, the average ones even longer. So if you have to wait 10-15 minutes to validate your code changes are OK, then it’s going to make you less effective. You want the ability to run the test locally to get feedback in seconds.

Let’s first measure the problem. Below are open-source repos for Jest, Vitest, NUnit, and xUnit collectors:

• dotnet-build-metrics
• testresults-collector

These allow us to fire the data at an ingestion endpoint to get it to our Hadoop. They can be reused as well by anyone that sets up an endpoint and ingests the data.

They will also send from CI, using the username of the person that triggered the build when running in CI and the logged-in user from the engineer’s local. This allows us to compare who is triggering builds that run tests vs. if they are running on their locals.

Looking into this data on one of our larger repos, we found that there was a very low number of users running the integration tests locally, so it was a good candidate for experimentation.

When looking at the local experience, we found a readme with several command-line steps that needed to be run in order to spin up a Docker environment that worked. Also, the steps for local and CI were different, which was concerning, as this means that you may end up with tests that fail on CI but you can’t replicate locally.

Looking at this with one of my engineers, he suggested we try Testcontainers to solve the problem.

So we set up the project with Testcontainers to replace the Docker Compose.

The integration tests would now appear and be runnable in the IDE, the same as the unit tests. So we come back to our zero setup goal of the F5 Experience, and we are winning.

Also, instead of multiple command lines, you can now run dotnet test and everything is orchestrated for you (which is what the IDE does internally). Some unreliable “waits” in the Docker Compose were able to be removed because the Testcontainers orchestration takes care of this, knowing when containers are ready and able to be used (such as databases, etc.).

It did take a bit of time for our engineers to get used to it, but we can see over time the percentage of engineers running CI vs. Local is increasing, meaning our inner loop is getting faster.

Conclusion

The F5 Experience in testing is crucial for maintaining a fast and efficient development cycle. By focusing on making tests easy to run locally, we’ve seen significant improvements in our team’s productivity and the quality of our code.

Key takeaways from our experience include:

1. Measure First: By collecting data on test runs, both in CI and locally, we were able to identify areas for improvement and track our progress over time.
2. Simplify the Setup: Using tools like Testcontainers allowed us to streamline the process of running integration tests, making it as simple as running unit tests.
3. Consistency is Key: Ensuring that the local and CI environments are as similar as possible helps prevent discrepancies and increases confidence in local test results.
4. Automation Matters: Removing manual steps and unreliable waits not only saves time but also reduces frustration and potential for errors.

The journey to improve the F5 Experience in testing is ongoing. As we continue to refine our processes and tools, we should keep in mind that the ultimate goal is to empower our engineers to work more efficiently and confidently. This means constantly evaluating our testing practices and being open to new technologies and methodologies that can further streamline our workflow.

Remember, the ability to quickly and reliably run tests locally is not just about speed—it’s about maintaining the flow of development, catching issues early, and fostering a culture of quality. As we’ve seen, investments in this area can lead to tangible improvements in how our team works and the software we produce.

Let’s continue to prioritize the F5 Experience in our development practices, always striving to make it easier and faster for our engineers to write, test, and deploy high-quality code.