Technical Debt: Why it'll ruin your software

As you can see, this is a well-known monument in Italy, more specifically in Pisa, hence it’s name: Tower of Pisa.

This tower has a very interesting history, it was built around the 12th century, but due to the soft ground it was built on, it started tilting. The structure was only stabilized in 2001 after 8 years of remedial work.

Actually, one of the corrections made during these 8 years made the tilting even worse! Let those who never made a refactoring that made the problem worse throw the first stone. I can’t really blame anyone haha.

John, the purrgrammer

• This is John.
• John is a Senior Developer always with a lot on his plate.
• John is also an amazing programmer with a lot of years of experience.

Suddenly, a new project arrived at John’s desk, and it needed:

• Payment System
• Social Network Authentication
• Delivery services integration

Since John is an excellent professional that always delivers his activities, he tackled the new project head-on and delivered it in time.

After John’s delivery, the rest of his team went on to review his code, and they discovered a few problems, such as:

• Bugs and inconsistency on payments
• Sometimes the deliveries weren’t processed
• Authentication was too simple when checking the social networks used

John knew his code had a few bugs. But nearing the deadline, another project appeared and it took a lot of John’s time, so he couldn’t go back and fix the problems.

John and his team had to deliver the project riddled with bugs and problems.

But John kept a positive attitude.

“In the future, I’ll go back and fix those bugs”

SPOILER

C’mon, did you really think that this was going to happen?

Let’s dive deeper into the problems of John’s code:

• Payments couldn’t be processed in different currencies
• If the delivery system is offline, the code wouldn’t work
• Users with deactivated accounts could still access the system
• No automated testing

And this, my friends, is what we know as Technical Debt. Why?

Because what happened was that John chose the first solution he could think of in that short deadline, this affected the code’s quality and the team accepted this. In the future, if any logistics of the business, such as accepting different kinds of currency or changing the delivery service, needs to be changed, the Changing Cost of the code is going to be enormous. The moment John chose the faster and easiest solution for him was the moment that the Technical Debt was inserted in the code.

What happened with the Tower of Pisa is a lot like what we understand as Technical Debt.

It probably started off as a couple of small mistakes and problems, but the constructors decided to ignore them and build and scale on top of these problems. The tower was built so fast that these little “bugs” in the construction jeopardized the whole structure.

And the same thing can happen with software. The difference is that the bugs and problems are easier to spot in the developing process, because, after a while, new features become impossible to deliver before um fix these bugs.

Why is Technical Debt always bad?

We are used to understanding any Technical Debt in software as a very bad thing. And it usually is.

But we can also see it as a strategic and/or economic decision since a project riddled with Technical Debt is still a project that was delivered fast. It can be a very positive thing for the product.

The problem with this approach starts when we start to forget what we have done in the past. Imagine that Technical Debt is a really greasy and tasty junk food. To eat it once or twice may be wonderful, but after the eighth ou ninth meal, it might be time to acknowledge you have a problem.

By accepting that it is a tradeoff, we can move faster to pay the debt in the future. It really something very normal to do.

I don’t know how many of you have paid the cost of too much junk food. Just remember that this cost, as well as in software, becomes higher with time.

If you identified some of these situations in your day-to-day tasks, don’t worry, especially if you have some time dedicated to remedying the problem in the future. This is a really common scenario when dealing with software delivery.

Technical Debt Quadrant

If we look at Martin Fowler’s Technical Debt Quadrant, we can see John in the upper quadrants, between Reckless and Prudent.

He knew he didn’t have enough time to design the best solution, but he also knew that: In the future, he had to come back and improve his code.

Ok, accepting this as something ordinary, we can try to understand who is really responsible for the software’s debt.

Who’s responsible for the debt?

If we focus on the name (Technical Debt), we’re lead to think that it’s a problem made by technical people, namely: the developers, who are responsible for writing the code that ultimately caused the debt.

The problem is that software is the result of the structure and processes of a whole company.

Also, the whole software industry changed since the term “Technical Debt” was born. In John’s scenario, for example, the problem with the not-so-good solution that he crafted is more connected to a management issue, or even to the product team, than to himself. We know John is a good developer, but he had to ignore a lot of problems in order to deliver within the established deadline.

Sometimes the bottlenecks that are creating debt come from different parts of the company, such as:

• Management team vs Developers speed
• Product team not having a long-term plan
• UI/UX team too far from the developers

But we can’t forget that, sometimes, we are indeed to blame for our own bad decisions.

• Our bad decisions counts
• Framework smells
• Low test coverage

Like when we choose a new and exciting framework, without investing the necessary time to understand if that is the best tool for the job at hand, or even when we stop paying attention to the signs that a framework is being abandoned, or when the developers aren’t testing and refactoring the application as they should… All of this is our fault.

After discovering which problems we have in our code, we need to look for the tools we can use to help us deal with the situation.

How to deal with the debt?

• Rewriting everything?
• Hire more people just to deal with the debt?
• Stop delivering until we can fix some of the debt?
• Creating Technical Debt tasks in a separate Technical Debt board?

Imagine if people had simply stopped building the Tower of Pisa and destroyed everything to rebuild. The chances of the same mistakes happening, or even new mistakes, were really high. And consider the amount of money necessary for demolishing and rebuilding everything.

What if the company simply hired more people to fix the debts? In the case of the Tower, we would have people trying to build it higher working alongside people trying to make it straight and steady. This really doesn’t solve the main problem.

In software, hiring a new team to deal with existing problems would mean that the old team would have to take the time to explain the whole context of the software to the new team. And, believe me, this would take a lot more time than you’d think.

And what if we created a new place to store our Technical Debt task? After a while, this place would become completely invisible to the team.

If an ordinary developer’s backlog already has its own old and invisible tasks, imagine creating a whole different board with separate tasks. However, these are the solutions most people usually look for.

So, Luan, if these options aren’t the most viable, what can we do?

After what we already know, we must try to find truly viable solutions to deal with the Technical Debt.

Sustainable solutions

1. I.M.P.A.C.T

IMPACT is a process to:

• Identify
• Mark
• Plan
• Act
• Test

As we can see, the first step is about finding the debt spots and, in the second step, we need to make sure that these spots are noticeable for the whole team. You can assure that by creating new tasks flagged as Technical Debt, for example. We can also choose the priorities of these tasks based on their relevance. Tools like Jira have mechanisms to increase the priority of certain tasks over time, which is amazing, because, as we have seen before, the cost of a Technical Debt increases over time.

After that, we need to create a plan, and really follow that plan, for the marked tasks. This way, we will effectively Act on the debts and, after the fixes and improvements being implemented, we need to Test and assure that no behaviors were damaged, be it new bugs, features, improvements, or anything that could hinder the final experience.

You can find more detailed information about this in a book called “Refactoring for Software Design Smells - Managing Technical Debt”.

2. Pareto Principle

If you don’t know how many tasks you should have for each team iteration.

We can use the Pareto Principle to assign 20% of the team’s productivity to Technical Debt tasks, and the 80% that remains is used in ordinary tasks, as new features, bug fixing, and everything else.

3. Technical Debt Review Reunions

If you have big deliveries in your process after a few iterations, it might be a good thing to have post-mortem reunions to discuss everything that’s been introduced to the code that has the potential to increase the Technical Debt.

By doing this, we can identify the debts earlier.

These are three examples of possible solutions, but I’d advise you to be extra critical and choose the one that fits you and your team the best.

Let’s probe the economical side of this problem. Is it lucrative to work with a system that already has a big Technical Debt cost?

Is it lucrative

I’m going to use a lecture by J.B. Rainsberger to illustrate my point. In this lecture, Rainsberger explains the cost of the next new feature for your software and how to diminish your error rate when estimating this cost.

In this chart, he compares two different costs over time in a project.

As we can see, the cost of a software built without thinking in design, refactoring and good practices raises much more exponentially than it would if you started the project by thinking in design.

Despite the higher initial cost of starting a project by thinking in architecture design, we can also note that this cost is mitigated throughout the project’s lifetime if compared to an approach without these concerns.

Without design, the cost of new features in the software would become so high that stopping everything and rebuilding the whole thing would be financially a good call.

If you’re currently writing your software this way, you’re actually betting that the project is going to be over before the dotted line.

The hardest part about these estimates is that we never know when the dotted line will arrive. It could be in 3 months, 1 year, 10 years… We simply don’t know.

The cost of low-quality software

Low-quality software has become one of the most expensive things in human history.

In 2018, CISQ released a report on software costs in the USA, the calculation took into consideration: legacy systems, catastrophic failures, problematic and canceled projects cost. In comparison, the cost in which they arrived is equivalent to 2/3 of all the health expenses in the US in the same year.

Programmers spent an average of 3.8 hours/day on debugging bad or low-quality code. So imagine all that money being used for other things, like better wages, social initiatives, training, and personal improvements?

Beyond the profit

Beyond the bleeding money and slower delivery, there are other problems of software development directly linked to high levels of Technical Debt.

It brings a lot of weight to our lives. If you’re constantly dealing with problematic codes, a relatively easy task could become a real monster.

Anxiety and Depression

It’s very common to be afraid to make changes, especially if you’re new to the project. Plan meetings could become endless because the development team needs to discuss with project management to explain the real cost of delivering new features.

We start to get anxious because we don’t know if the next challenge will be a big problem or a hard-to-fix bug in the middle of a sprint.

Who among us hasn’t had an out of date or no longer maintained code?

Technical atrophy

It’s a common scenario for us. And it is really complicated to deal with because we need to accept the consequences and problems that come with it.

Usually what happens is that you start copying and pasting code from an out of date library or framework and fixing the problems yourself. Or even, sometimes, making a copy of the whole project and start only using your clone.

Furthermore, we may still find incompatibilities with new tools. You may want to use new platforms, but not being able anymore since you’re stuck with some old version of a tool no longer supported by the new platform.

Imagine how hard it may be for a developer to be relocated in the industry after working about 5 years into a javascript framework that isn’t maintained anymore. The personal cost this would have to him, needing to start from scratch with React or Vue.js, for example.

A lot of the concepts we’re used to be working with may even be absent in old tools.

Problematic growth

It makes no sense for the company you work in to grow 300% or 400% each year if your team doesn’t have the mental health and time to work in the best way possible into the new size. This scenario is pretty common in the startup scene, glamorizing excessive workhours after some unreal goal or some sales spike in the middle of the night when you need to wake up to get some downed service up and running.

In a realistic and respectable world, machines should take care of these situations, and not us.

After talking about all of this, the Tower of Pisa is finally sustainable.

Actually, in the Tower’s case, it was a bug that became a feature, haha!

It won’t fall, even if it keeps leaning every now and then. The situation of the Pisa’s landmark is controlled, even if tourists insist on “holding” it for pictures.

The same thing can happen with John’s project. If we apply all of the viable solutions, we may easily have sustainable software in our hands.

All of the references I used in this talk are on an Awesome List on Labcodes’ Github.