Home›
Blog›
What are software regressions?
Subscribe

What are software regressions?

Prevent your software from decaying with automated testing and essential KPIs.

July 23, 2024

We all know managing software projects is challenging and one of the most common issues a development team faces besides implementing new features is dealing with bugs. Now bugs come in many shapes and sizes and should be reduced by as much as possible before new code reaches production. One problem every team faces is software regressions.

What are software regressions?

Software regressions are a type of bug where a previously working feature breaks.

There are several ways regressions enter into a software product. One of them is when a previously unknown or inaccessible bug enters into production because of a code change. This can happen when a codebase is updated and some function is used in a way the original author didn't intend for it to be used. You will often see these kinds of regressions happen in utility functions since they may have been built with a single purpose and many undocumented assumptions.

Another way software regressions creep into production is when code changes are made without understanding the original intention for its implementation. This is a classic example of [Chesterton's Fence]. Many experienced developers will use defensive programming to mitigate these kinds of problems. Some of these techniques include:

  • Adding comments in the code about assumptions made while writing core functionality.
  • Making additional checks, such as null pointer checks, when a piece of code may be re-used in unknown ways in the future.
  • Using a robust typing system.

but the list goes on. Even if developers are doing a great job and following best practices, it's still challenging to find every defect. Instead, it is ultimately the job of the QA team to mitigate against regressions from entering into production by developing automated checks or manually testing edge cases developers wouldn't have thought of.

How regressions impact UX

Besides regressions being frustrating for your development team, they are even more frustrating for users. This is because they typically expect their workflows to always function correctly. When you open up Google Docs and something isn't working, do you instinctively blame your internet connection, do you blame Google, or do you blame the browser itself? I'm guessing you'd blame your connection because it's generally less reliable than these other products.

Now imagine navigating through a complex multi-step workflow only to find out there's a critical juncture that's broken near the end. If your users cannot recover from the bug they've encountered they'll be left frustrated and looking for alternative paths to complete their work. Not only that, they will have wasted part of their workday on something they will have to redo later. This kind of waste adds up and certainly factors into spending decisions for many companies.

How can I manage regressions?

Managing regressions is a multifaceted process because many vectors may cause them. One of the most effective tools for dealing with regressions is by writing automated tests called regression tests. These let your developers not have to keep a list of previously encountered bugs in the back of their minds while writing code since these problems can be taken into account by a testing suite.

Besides automated testing, you can keep track of the general performance around regressions by using the correct KPIs.

Core regression KPIs

Here is a list of core KPIs you can use to get a picture of how regressions are impacting your project.

Bug Repopen Rate

(Count of bugs being reopened) / (Total number of bugs resolved) * 100%

One of the most direct measurements of regressions in software projects is the bug reopen rate. This is because it measures how often reported bugs are being reopened as issues. Having a high bug reopen rate indicates that your team is spending more time fixing problems they previously thought were finished and less on implementing new features.

There are multiple factors that can cause a high bug reopen rate, including

  • Your software isn't effectively tested when a bug is labeled as fixed, so the bug may still exist in cases that weren't tested.
  • Code changes could be consistently breaking previously working functionality.
  • There may be a large amount of interdependency in your codebase. This means if you update one part of your code, it could impact functionality in many other parts of your codebase.
  • Unclear requirements can cause your developers to implement the wrong functionality.

Planned vs Executed Testing Progress

(Number of executed tests) / (Number of planned tests) * 100%

If your team is unable to execute every planned test before releasing software into production, the likelihood of new regressions increases. This metric helps you determine how optimized your development workflow is for catching problems before release. A low score indicates your team is unable to perform all testing in a timely manner.

Defect Density

Create a heatmap by partitioning your project into different categories or components and count the number of defects with respect to their severity.

Figuring out the defect density in your project helps you understand where the main problems align. For example, if you manage a large software project split up into several components, it's useful to measure which components are giving the most problems. This helps prioritize which areas need improvement and gives you a better idea of where to allocate more testing resources.

Risk Management Efficiency

(The number of high-severity bugs in high-risk areas)/(Total number of severe bugs) * 100%

Another area you should consider when dealing with regressions is understanding how they are impacting critical areas of your project. For example, if you manage personal identifying information, such as in a FinTech or HealthTech product, then you should keep track of how many bugs exist within risky sections of your product. This way you can see if the bugs with the highest potential for upending your startup are being dealt with effectively.

Customer-oriented regression KPIs

The other side of understanding regressions is measuring how they could be impacting customers.

Escaping Defects Rate

(Number of Defects Found in Production) / (Total Number of Defects) * 100%

An easy way to measure the customer-facing impact of bugs is by looking at the percentage of bugs entering production. By keeping track of this value your organization will be better informed on how well its software development cycle can find and remove bugs before releasing to production.

Post-Release Support Tickets

Track the number of support tickets post-release for a new version of your software.

Another way to measure your product development performance is by measuring the number of support tickets created after a new release. This can help give insight into how updates to QA processes are impacting your customer experience. If QA is being tightly integrated into your development process and post-release support tickets are trending downward, then your process changes are working as intended.

Net Promoter Score

If you survey your users asking how likely they are to recommend your product to a friend, on a 0-10 scale, then

NPS = %(Users giving an 8-10) - %(Users giving a 0-5)

One of the most effective metrics for measuring customer satisfaction is the Net Promoter Score (NPS). By tracking this value you are understanding how likely your customers will evangelize your product, making it more likely for their friends and colleagues to convert. If you see your NPS is low and your QA metrics are doing poorly, it's probably time to start investing more time and effort into effective QA processes.

References