The Painter and the Paint Bucket: A new explanation for Technical Debt

roadpaint

Technical Debt is a metaphor that explains about the consequences of doing an incomplete work, in which software development problems are compared with a debt acquired by means a loan, relating those problems to both the principal and the interests of that debt. This analogy will be understood if the target audience have somewhat common management abilities. If the public does not fully understand those financial concepts, this awareness will be more difficult to transfer.

To cope with this problem, a simpler metaphor is proposed, based on the joke of a painter and the paint bucket, which involves more ordinary elements and situations of daily life, and therefore should be easer to transfer to a more diverse audience.

Introduction

The Technical Debt metaphor describes the consequences of the decisions taken during software development, in terms that a manager can understand. This way, Ward Cunningham explained the problems they had in the project using a concept from the financial world: a loan.

In his explanation, Cunningham commented that for each new release, a debt was generated, in the same way that when requesting a loan. Each release allowed the development team to show progress and generate business value, but this debt must be paid back, otherwise they will be paying interests on that debt while they keep elaborating on the code base that were “not-quite-right” because of the loss of productivity they may incur.

Cunningham’s metaphor worked to sensitize managers, because the audience was specialized in financial concepts. The message was understood. However, what if the audience where not versed in finance? Not all persons, even being specialized professionals in some given discipline are literate enough to financial concepts, even those relative to loans.

In this post a new explanation for the technical debt metaphor is proposed, based on the joke of the painter and the paint bucket. This exposes the audience to concepts that are both easier to visualize and assimilate, and by those means, helping the software professional to increase awareness on the subject.

The Painter and the Paint Bucket

The new explanation is based on a joke that highlights by the absurd the productivity problems associated to technical debt:

A man is hired to paint the lines on the road.

On the first day he paints ten miles, and his employers are amazed.

But, the second day he painted just five, and on only the third day, he painted only a mile of the road.

Disappointed his boss asks what the problem was.

The man replies, “Well sir, every day I have to walk farther and farther to get back to the paint bucket.”

Analysis

The joke presents a worker with a clear objective: to paint the center lines of a road. To accomplish this objective he counts with a paintbrush and a bucket with the necessary paint in it. The worker is given freewill to accomplish his job the way he want. The business process that describes his job consists of several activities with different degrees of physical and mental difficulty, some of which he may choose not to do:

  1. Soak the brush in the paint bucket.
  2.  Measure the width of the road to determine the center.
  3.  Paint a line in the road.
  4. Walk to the next road sector that needs to be painted.
  5.  Walk to the paint bucket.
  6.  Lift the paint bucket.
  7.  Move the paint bucket to a new sector.
  8.  Drop the paint bucket.

Activities 1 to 4 are part of the painting process, and are essential to fulfill the objective. Activities 5 to 8 are part of the bucket management process and can be executed optionally, however, some activities imply others. For example, moving the paint bucket implies previously lifting it and dropping it afterwards.

As in any other work, it is composed of activities that are required, either because they represent the essence of it (like painting a new strip) or because it is imposed by the company culture (like ethical behavior). Other activities are important part of the job even if not always necessarily executed (like walking to a new road sector), and others are at discretion of the worker.

The reasons of why the painter decides not to carry with him the paint bucket may be varied. In terms of the technical debt classification quadrants proposed by Fowler, they could be:

  • Inadvertent and reckless: “I didn’t know I could carry it with me”.
  •  Inadvertent and prudent: “I could be saved a lot of time by carrying the bucket with me”.
  •  Deliberate and reckless: “Top priority is to paint only”.
  •  Deliberate and prudent: “Can’t move the bucket because it’s too heavy”.

According to Fowler’s classification, the joke represents an example of an inadvertent and reckless technical debt, taking for granted that the man is not experienced enough to perform the task at hand.

Technical Debt

The whole process suffers from scalability problems because of the decision on leaving the bucket always in the position where he started painting, as he needs to walk back and forth with a loaded paintbrush to the new road sector to paint. While the painting process can be executed with different degrees of efficiency and quality, it is the bucket management process what introduces technical debt, that is, there are activities that are not done quite right that affects the final result.

Under the decision of not carrying the bucket with him, suppose that the worker is pressed to complete the job in certain time, he would have to make concessions about the way the work is carried on: running instead of walking to the paint bucket, knowing this is a short term solution as he could not do this the whole day. Another option would be to decrease time in the painting process, maybe affecting the final quality of his work.

This problems are easily extrapolated to software engineering. Think as if the road defines a new release of the software under maintenance, where each meter of the road to be painted represents a new feature to add (functionality, bug fix, etc.). All the activities carried on to reach the release contribute to the final quality and the development schedule.

Principal

Following the definition in [1], the Principal of the technical debt is “the cost of repairing quality issues in software systems to achieve an ideal quality level”. As you may have guessed, if the painter makes the necessary effort to carry with him the paint bucket all the time, there is no need to walk to the bucket, therefore no time is spent in it and represents the ideal quality level. Conversely, as the bucket is farther away from the painter, he needs to spend time to walk to the bucket each time.

The effort to keep the paint bucket near the painter is what correspond to the notion of Principal of the technical debt, and it is proportional to the distance between the painter and the bucket. In order to pay back technical debt, actions must be taken to bring both near each other.

Interest

Consider the Interest as “the extra maintenance cost spent for not achieving the ideal quality level” [1]. The increasing distance between the painter and the bucket implies an extra effort, that is, walking back and forth to the next road section to paint.

As a result, the Interest is represented by the effort of the walks, which will increase over time, unless some of the Principal is paid back, that is, unless the bucket be carried over closer to the next sector to paint.

Conclusion

As the concept of technical debt permeate to other professional disciplines besides software engineering, the need to reach a wider audience to sensitize about their effects require us to find simpler metaphors. The joke of the painter and the paint bucket is an example of an explanation that is simpler to visualize than the original one, which involves be acquainted with financial concepts related to loans.

Technical debt will increase as the painter make progress and the paint bucket is farther away from the next road sector to paint (the principal of the debt), and this distance will increase over time (the interest of the debt) unless something is done to decrease it. This analogies are simple enough to be able to extrapolate them to other business processes or professional disciplines, including software engineering.

References

[1] Nugroho, A., Visser, J., & Kuipers, T. (2011, May). An empirical model of technical debt and interest. In Proceedings of the 2nd Workshop on Managing Technical Debt (pp. 1-8). ACM.

A Semantic Versioning convention of Corporate or Parent POMs

Maven allows us to inherit from parent POMs to allow reuse of project or corporate standards, like:

  • Forcing a given minimum maven version.
  • Using a specific library version.
  • Defining an exact plugin version (for maximum build reproducibility).

If the IT organization share some basic configurations across all their project, it usually represent it as a corporate POM, that is a top-level pom.xml file that is used as the parent of all the projects in their project portfolio.

For some reason, commonly this corporate POMs are versioned with a single integer number. I don’t know if it is either just a convention, or it had some origins on some specific constraint on older Maven versions which are no longer present, or if there is some logical rationale behind it.

On the other hand, I like the Semantic Versioning reasoning, and I try to comply with it in my own projects. So the simple question is: Why would we not follow the same versioning technique for corporate POMs too?

We can follow the Semantic Versioning for corporate POMs, using a Major.Minor.Patch versioning format, as any of the applications and libraries we daily use.

The key to define a semantic versioning, is that POMs are not just configuration files: They define a public API, just like any library have classes with public methods (which comprises its public API).

The difference in the interpretation is in the point of view of the person using the API: In the case of the class library, it is a developer trying to use the API to build a bigger functionality; in the case of the build tool API (pom.xml file), it’s a developer trying to use the API to generate that previously written piece of functionality. And as happens with a class library, the magnitude of the change from version to version also can be measured.

Which part of the X.Y.Z version format to change?

Let’s see some basic rules for the case when you need to change the corporate POM, being X the Major part or the version, Y the Minor part, and Z the Patch part, as defined in the Semantic Versioning page:

Patch

7. Patch version Z (x.y.Z | x > 0) MUST be incremented if only backwards compatible bug fixes are introduced. A bug fix is defined as an internal change that fixes incorrect behavior.

When applied to a corporate POM, we can interpreted it as:

Patch version Z MUST be incremented if the generated artifacts built are functionally exact the same as the generated by the previous version.

This means that a rebuild with the new POM will generate the artifacts exactly as the previous one, so the result will be a functionally equivalent version of it.

For example, if you are building a web application, where the final artifact is a .WAR file, then adding a new <developer> to the <developers> section of the corporate/parent POM will produce exactly the same web application with its original functionality untouched (except maybe some resource filter that add the new developer to some file).

Minor

8. Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backwards compatible functionality is introduced to the public API. It MUST be incremented if any public API functionality is marked as deprecated. It MAY be incremented if substantial new functionality or improvements are introduced within the private code. It MAY include patch level changes. Patch version MUST be reset to 0 when minor version is incremented.

We can interpret it as:

Minor version Y MUST be incremented if the POM is modified in a way that it generates artifacts that are functionally equivalent, but essentially it produce a new different artifact. It MAY be incremented if substantial new functionality or improvements are introduced within the way the build is done, and may affect the final artifact. In both cases the build MUST work. Patch version MUST be reset to 0 when minor version is incremented.

For example, if your project use the Apache commons-lang library, defined in the section of its parent POM, and you want to update it from version 2.4 to 2.6, then you MUST change the minor version, because the generated artifact (.war file) will be different (a dependency has changed), though probably at runtime the application works the same.

It MAY be incremented if new plugin versions (commonly defined in a <pluginManagement> section) want to be used. It can happen that the plugin execution produces a different result, so is probable that the final artifact is modified too.

One thing is for sure: If you are given a new corporate/parent POM with a higher Minor number, then it is very likely the build will work out of the box, but you MUST RE-TEST the generated artifact.

Major

9. Major version X (X.y.z | X > 0) MUST be incremented if any backwards incompatible changes are introduced to the public API. It MAY include minor and patch level changes. Patch and minor version MUST be reset to 0 when major version is incremented.

Interpretation for a POM file:

Major version X MUST be incremented if the build (probably) will break, or the generated artifacts will break. It MAY include minor and patch level changes. Patch and minor version MUST be reset to 0 when major version is incremented.

For example, you may introduce the use of a plugin which requires Maven 3+ to work, then breaking the current build (possibly done with Maven 2.2+). Another example is the update from an old version of a library to a new one, where the public API changed, and this will obviously break the build.

Sometimes it will not break the build but the generated artifacts: consider the case where you upgrade your application presentation layer code from JSTL 1.0 to JSTL 1.1 in a parent POM, the build will not break but the web application will not work properly since the JSP pages will declare <%@ taglib uri=”http://java.sun.com/jstl/core” prefix=”c” %> instead of <%@ taglib uri=”http://java.sun.com/jsp/jstl/core” prefix=”c” %>.

Of course, when in doubt, just go back to the old convention: change the Major version number and you are ready to go.

Conclusion

There is no need to use the current convention of “a single major version number” in your corporate/parent POMs. You just need the discipline to use the X.Y.Z format right, which have the added benefit of implicitly show us the magnitude of the required change to upgrade to a newer parent version: this information is not conveyed in the current convention.

In teams where their members are not very experimented with Maven, there may be a big initial number of parent POMs before stabilizing, so the X.Y.Z format may help greatly (specially if the organization has high people turnover).

As a rule of thumb consider this guidelines, regarding changes in your parent POMs:

  • A Patch version change: A small change is required to upgrade. Artifacts should work out of the box.
  • A Minor version change: A moderate size change is required to upgrade. Artifacts need to be re tested in their original runtime environment.
  • A Major version change: A bigger effort is required to upgrade. Build probably break, and code may need to be upgraded too.

Things to consider before doing big refactorings in small businesses

When someone in the team advice to carry out a big refactoring, seems like a revelation made present into the room. There are the ones that look really surprised and find the proposition somewhat bold and there are others that may think “finally! someone brings this subject to the table!”

If you are lucky, the refactoring was detected on time and everybody is happy that it was discovered. Sometimes it come too late to the discussion and no one want to dig any further into this problem, and sometimes the refactoring is just a different way of doing things but no a necessarily better.

The question is which indicators we can watch to make a sound decision on putting the whole team to work on a refactoring or not. Here are some issues to have into account when that moment finally arrive.

Here I present some questions you can ask yourself and to your team to help you, which basically are based on pure common sense. You will find it ordered looked from the most technical view-point to the most business related:

Make sure the refactoring make sense

Your first task should be to dig into the technical issues surrounding the refactoring:

It is a necessary change to do?

Think (and ask to the team if they agree) if it would be better to refactor or would it better off leaving the code base as it is now. Having some smart people to throw an idea on the table doesn’t mean everybody leave the room running to implement it. Sometimes the refactor comes into play too late in the schedule and you may want to release as it is now, and schedule the refactoring on a future iteration.

Decide if the refactoring it is indeed an improvement that would unleash some new functional or non-functional requirement? or it is just a different way of doing the same? For example a big and important relational table where one of their attributes is a blob. You may want to split that table in two for better performance and improve memory consumption of server resources, but are really those non-functional requirements what need to be optimized? or do you want to do it because the software “would just work better that way”?

Will the refactoring leave the project in a pivot position?

A pivot position is a state of your project in which it is situated on a crossroads where all the paths leads you to a success scenario. So you must always prefer the project to be in a pivotal position for maximizing maneuverability, because you will never reach a dead-end, or be forced to follow a specific path to success. You must always have your options open in the presence of problems.

To illustrate the concept, some refactorings are needed because of unfilled holes into the functional specifications. To present an example in a much minor scale, suppose there is an algorithm to perform some financial calculation that you are not sure if it apply or not to a use case, then you can model it using an Strategy pattern. Here the pivotal position is having implemented the Strategy, so change will be easier when the unfilled specification hole gets actually filled. However, just proposing to change that every algorithm in your project has to be modeled as a Strategy pattern is an overkill: it is just a speculation to think that every algorithm is subject to change, even plain if-then-else logic, so a big refactoring like this is not appealing at all.

Remember the 3Ps of software engineering

The 3Ps of software engineering are:

(People, Process, Product)

Sometimes called the 4Ps if you add Project to the tern. It basically states that Project depends on how well you can control three variables: they are done by People which uses a Process to develop a Product. So for the project to be successful, you must keep healthy the people, the process used by people, and the product you are developing.

Suppose that the three variables are in harmony when no changes are made to the project, I like to think the 3Ps as a perfectly balanced 3 plate scales hanging from a triangle. When a new requirement or functionality arrives to implement, a “disturbance in the force” is generated by the environment that the team must “sense” and start working to balance it again.
Considering that you realize you need to refactor a big piece of the code base, and this will impact the whole team, you must plan wisely to avoid missing the target. First of all start with the most important, unstable and unpredictable of the three variables: the People.

People

  • Is there consensus that the refactoring must be carried over?
  • Will the team be more confident on making non trivial changes to the code base after that?
  • Will they deliver more quickly?
  • Will they deliver with fewer bugs?
  • Will people have longer working days?

Product

  • Will the product improve in some aspect visible to the users (like performance or functionality) or on any other way (like extensibility or maintainability)?
  • Will the actual product or service degrade its quality of service until the refactoring is completed? For how long?
  • Will the refactoring be completed by stages, progressively? or will the product or service be modified in one shot?

Process

  • Will the process allow the team to deviate from the usual plan to perform the refactoring?
  • Is the structure of the team or business adequate to quickly communicate the needs and changes between different roles like developers, DBAs, sysadmins, testers, users?

Supply and demand

Remember that in the market economy, the fundamental truth are given by the laws of Supply and Demand, and for any business they are as inescapable as the laws of gravity in nature. Specially if you work on a small business, chances are that you must think from time to time in the business case of proposed changes. You must asses how the refactoring will impact your business in the medium and long-term future:

  • Will the refactoring improve your capacity to supply new functionality/services/products?
  • Will the refactoring impact positively on the demand of your services/products?

Stay aligned with management in this. Ask yourself “What would I do if it were my only decision? What would I do if I’d have to bet $100,000 of my own pockets on this change? You would be surprised what you would be your answers when you carry out yourself that exercise!

Conclusions

When faced to do big refactorings many things must be evaluated before allowing the team to reorganise the code base of the project. IMO, the best thing you can do is: search for consensus with the team to decide if the change is appealing or not, jointly plan the effort, and analyse the business case of it.