Tracking software bugs: keep the important ones; drop the rest

In ancient neolithic farming communities, the size of the average tribe or village was about 150 people. That’s also about the size of most military units dating back to Roman antiquity and earlier, and it’s a number that features frequently in a variety of social antropological examples. It turns out, as Robin Dunbar popularised with his research from the early nineties, that 150 is about the number of individuals a single person can maintain a stable social relationship – that is, where you know who each of the people are, and how they relate to all the other people in the ‘network’. Malcom Gladwell talks about this extensively in his book Tipping Point.

The same is true, I think, for bugs in a software system. We have the ability to understand, process and relate to a relatively small and definitely finite backlist of bugs. I don’t know if 150 is exactly the number – and it would probably depend on the product and the type of bugs – but I think it’s close enough.

Many software projects have the tendency to track hundreds or even thousands of defects in long, long lists, maintained and even driven by complicated tools. Having so many bugs open makes it very difficult to see what’s truly important. With 800 bugs in the system, answering the question “what’s important today?” is hard.

You also need to deal with the fact that a large portion of these bugs won’t ever get fixed. They just won’t. Not because we don’t want to – at least not exclusively – but because a team can only fix so many bugs at a time. With such a large bug backlog the world around you changes quicker than you can fix all the defects, so many of them will end up being invalid or no longer relevant long before they could have been actually fixed. So was it worth tracking them in the first place?

Software development blogger Gojko Adzic thinks we should do away with bug tracking completely. I don’t think I would go so far: I think recording bugs in some kind of tool is necessary to see the major open problems with the software, allow scheduling of fixes and provides an important communication channel especially with offsite or offshore teams. I don’t think, however, that it is valuable or efficient to record every one of 1001 defects that were ever discovered.

Just like the stone-age farming villages, I think a team has the ability to deal with about 150 open defects at a time. Any more is just noise, and is probably going to cost (read: waste) you a lot of time in adminstration and error management that you won’t get a meaningful return on. What if you kept a record of the top 150 defects, and just dropped the rest? Just don’t record them?

The natural question to ask at this point is rightly “what are the top defects?”. The word ‘top’ is all about priority, and priority is a very context sensitive thing. Priority is relative to the vision of the product, the maturity of the software, the type of users and other product related factors; but priority is also relative to the priority of the other defects in the system. If your payment system is dropping payments or sending cash to the wrong accounts, then that layout issue on the invoice page is not going to be so important. But if the payments are processed perfectly and the layout problem is the only blemish, then it’s ‘top’ priority, relative to the other open issues.

Here’s a proposal: Define a minimum bar for defects; one that keeps your bug count at about 150. As the highest priority items are cleared up, then the priority of the next ones increases relatively – so you raise the bar in terms of which defects can be added. If your bug list grows over 150 (hopefully it doesn’t!) then drop the bar to stop recording stuff that you don’t have time or headspace to care about.

If you don’t care about it, then just drop it. If part of you cares but you have a hundred other more important things, then drop it too. Spending effort and time recording stuff that you’re team is not able to deal with just creates noise, consumes effort and makes it hard to see what’s important.

Let the team decide…

In a typical agile development team a few hundred decisions get made every day, by everyone. Should we build this user story first or that one? Should I refactor this method now or do it later? Is this bug more important than that user story? Should I call that meeting for this week or next week?

Every decision that gets made, in the end, makes its mark on your product. Every decision impacts the team’s velocity or productivity, the product quality, the product features and the user experience.

If you’re planning a trip in your car, every decision you make along the way will impact where you arrive at, how long it takes and how many dents you have on your car when you get there. Do you want to get there quickly, or is time not so important? Do you want to avoid dirt roads, or do you prefer them? Is there a particular landmark you want to see on the way? You have an idea of what your journey will be like before you leave, and your decisions are based on that vision of your journey.

Back to the development team: every decision made will impact where and how you arrive at your destination. So the question is: does everyone in the team have the same vision of what your journey will look like?

One of the Product Owner’s most important roles is to define and communicate the product vision. The more the team understands and buys into the product vision, the more likely it is that each decision they make will lead the journey in the same direction you want to go.

Here’s a concrete example: a lot of POs spend a lot of time prioritising things: bugs, stories, meetings, etc. Many POs I know want to be involved in every decision the team makes. What makes one thing more important and/or urgent than another thing is all about vision and context. What would happen if the PO spent all the time she spent prioritising bugs on communicating the product vision? If the team understands the vision and context, then they should be able to make the right priority decision based on that vision and context.

Time invested making a single decision for the team has little return beyond the decision itself. Time invested in communicating vision will repay itself over and over with better decisions across the team, fewer bottlenecks and less wasted time.

Continuous delivery: release early, release often

If you’re in the business of building a car, you only really only have one chance to get it right. Those years of research and development, trials, testing, marketing and building culminate in a single, big release – your car launch. You only have one chance because by the time your customers drive your car away from the lot, it’s too late to fix that overheating problem with the brakes, or add another 0.5L engine capacity. Shame.

Software used to be the same way. A software project would be analysed, specified, analysed again, specified again, built and re-built and finally, after months or years of development, would be burned onto a CD or floppy disks, put in a box and sent to a store. Before the internet getting updates to software was tricky. You had just one shot to get it right.

Today, we live in a very different software world – especially true for those of us working on web products. With the internet we have a delivery immediacy unlike any other product manufacturing system in the world. We can make a change today and our customers see it in their product, today. You can add a new feature today and start generating new revenue or new customer acquisitions tomorrow.

It goes further. You can release a feature today, and watch, in real time, how your users interact with it. If your key metrics go down, you can roll the feature back out. If there’s a problem, you can either fix it quickly, or roll it back. You have the power to interact directly with your customers immediately over the best channel possible: the product itself.

This is what continuous delivery is all about. It’s like a beautiful gift for any product developer, and I think not making the most of it is a waste of that gift. Release early, release often.

Better SCRUM User Stories: Connect the story with the real user value

Everything you do to your product has a reason. There’s a reason that button is blue, or that message contains those words. There’s a reason you display that data or that notification.

Every reason is, at its heart, about the user.

This is why I like using user stories to talk about specific product developments and changes: writing stories reminds us that at every step of the way, everything you do is about your user. Every time you want to change something or add something, you need to understand ‘why’ – why are you adding what you’re adding, and why should the user care.

Connect the user story with the real user value. What the user wants. Not what you want, what marketing wants or what your boss wants – but what the user wants.

Here’s an example of a user story that came across my desk recently:

As a user, when I hover over search results in the search list, I want the pins on the map to change colour.

What’s wrong with this user story? Firstly, it’s a bit ambiguous – which pins?

As a user, when I over over search results in the search list, I want the pin representing that search result to change colour.

Ok, it’s a bit clearer. But is this what the user really wants? I don’t think users have a pin colour problem. What they have is a problem of identifying where the search result is located on the map. The colouring of the pins is a solution, which should be saved for the story specification or acceptance criteria. What the user wants, I think, is this:

As a user, I want to quickly understand where each search result is on the map relative to the other search results.

Clear. But why? Why does the user really want this? Understanding the why helps us understand how to solve their problem. That’s where the second, and often forgotten, part of the user story comes in: the ‘so that’ part:

As a user, I want to quickly understand where each search result is on the map relative to the other search results, so that I can get a feeling for where the place is relative to me or some other point.

Now we are at the core of what the user really wants, and we can go on to design a solution that solves this user problem. This process of story refinement gives us a better understanding of the true motives of the user: the ‘why’.

If you find that you’re looking at a user story that contains more problem specification than solution, keep refining until you get to the real user value; what the user really wants.

Darwin and the theory of software evolution

Bower bird

Female bower birds prefer males with colourful blue tail feathers and an impressive nest filled with lots of blue ornaments. To a bower bird, the brightness and quality of your tail, as well as your ability to gather a stunning assortment of blue nest decorations, indicates how healthy and strong you are, and how likely your genes are to produce equally strong and healthy offspring. In other words, a bigger tail and a cooler house equals more sex, which equals more children. Your genes live on.

Generation after generation the strongest genes survive, the weakest ones are killed off, and the species evolves – better and better. It’s survival of the fittest. Or, perhaps even more apt, survival of the most effective.

Good software product development tends to emulate Darwin’s evolution. Software is built, released, used and measured. The successful features, the heavily used features, the most often talked about features receive more development, more design, more attention; the least used are left alone, watered down or removed entirely. Survival of the most effective.

On the web we have the powerful ability to accelerate the evolution. We can release software updates multiple times per day. Design – code – release – measure – rinse. Repeat. Techniques such as A/B testing accelerate it even more: which is more effective? Text link or graphical link? Blue feathers or green feathers? Big nest or bigger nest? Survival of the most effective.

For the male bower bird, just as for software products, the audience (user) is critical. Every decision the bower bird makes will be judged by the female. It doesn’t matter if the nest is made of wire instead of twigs, or if the bower produced a new nest creation framework. The female bower doesn’t care; that’s not what she makes her decision based on.

Is your product evolving? Who is your audience, and who are you making your product decisions for? Are they the same?

The space between

Web developers can tell you that application speed is all about latency. It’s about the speed between component A and B; the distance between this server and that one; the response time of that API. In other words, it’s about the space between.

In organisations complexity is compounded in the spaces between – the spaces between teams, between components and between divisions. The trouble is the spaces between are often a no-mans land that is no-one’s responsibility in particular… but it’s the spaces between that matter if you want to get anything meaningful done.

Factories build detailed processes to help their employees navigate the spaces between teams to get things done. But what if there is no process for what you need to do? What if there is no roadmap to tell you how to get there?

If you understand the spaces between, you don’t need a playbook or a manual. When you step in and own the space between, you can make things happen and get stuff done.

The spaces between are your opportunity to do something new and valuable.

The sign is not the solution to the problem

The sign is not the solution to the problem

This sign reminds me of a presentation from Seth Godin at the 2006 Gel conference. This was one of his examples of stuff that is broken. “They’re building a device that can crush small children to death,” he says. “They oughta walk down the hall to the engineers and say, ‘The sign is not the solution to the problem‘.”

The sign, it turns out, is a workaround. It’s a hack. The real solution involves fixing the gate so it won’t mangle your hand or crush you to a pulp – but it takes someone to stand up and say “hey, guys, is this really the best we can do?” Even if it’s not her job to do so…

Another example: an internal team is delivering you an API that’s changing constantly. You find yourself rewriting large sections of code constantly to work around the changing API. What do you do? Do you happily (or unhappily) go on re-writing code every week when the API changes? Do you build an extra integration layer between your code and the API to manage the scope of the modifications you have to do? Or do you start a discussion with the team providing the API about why it is changing so often, and maybe implement an API contract between you and them?

Often, (every day), we see stuff that is complex, that is broken or that us just plain silly. When you encounter complexity or brokenness, I think you have three choices:

  1. You can accept the complexity or brokenness, and do nothing.
  2. You can accept the complexity or brokenness, and try to hack a workaround around it.
  3. You can ask, why is it complex? Why is it broken? And how can we fix it (properly)?

Doing #2 might help you navigate the problem, but it creates debt for you and your organisation that someone needssigh to repay some day.

Doing #1 is, I propose, at best depraved indifference; at worst malevolent and destructive.

Be like water

Flowing water

I remember reading about a concept from Taoism called Wei Wu Wei. Roughly translated it means “action without action”, or “effortless action”. In the same way that running water naturally and effortlessly flows around obstacles, so should our actions be thoughtful and mindful, but effortless.

Bruce Lee, when describing his martial art Jeet Kune Do, said:

“Be like water making its way through cracks. Do not be assertive, but adjust to the object, and you shall find a way round or through it. If nothing within you stays rigid, outward things will disclose themselves.”

Software development process is like this: like water. The path from concept to released software is a flowing stream, and as product creators it is our responsibility to ensure that it can flow easily and effortlessly.

Your product development process can either help the stream flow, or it can create dams. A broken development server, for example, is a dam. A bureaucratic and document-heavy change management process is a dam. Dams create tension and confusion. Dams create waste.

Focus on avoiding wasteful process. Tear down dams.

Be like water.

Maximise success

Many processes and organisational controls are set up to minimise failure. They are set up to minimise the number of times a team releases bad software; minimise the number of customer complaints; minimise the server crashes or minimise the failed product launches.

What if instead of trying to minimise failure, your organisation focused on maximising success?

What would it do for your product or consumer offering if instead of trying to minimise the bad software releases, you focused on maximising good ones? What would it do for your organisation if instead of focusing on minimising failed product launches, you focused on maximising good ones?

The problem with focusing on minimising the bad is that it creates roadblocks instead of opportunities. When a conversation starts with the question “do we have a reason to not ship this?” it sets the expectation that not shipping is okay. A focus on bad minimisation discourages risk taking and rewards the status quo.

The easiest way to minimise bad releases is to not release anything at all – so when you incentivise and reward the act of not shipping bad releases, either nothing will get shipped, or what does get shipped will be safe and unremarkable.

There will always be a reason not to ship if you look hard enough. Instead of focusing on trying to find it, try investing that energy into shipping it the best way you know how.

Not my job

Not my job
Not my job

One of my first jobs was as a bartender in a pub. My supervisor, Ted, had what I then thought was a compulsive neatness habit. If he saw a chair was out of place, he would straighten it. If there was an empty glass on the table, he would take it. A cigarette on the floor, he would pick it up. He couldn’t abide it if the other staff didn’t do this too. He’d say to us “don’t just turn a blind eye because you’re a bartender and not a cleaner. We work as a team here; your job is everything.”

The bigger your organisation becomes; the more distributed your team or the more defined the roles, the easier it becomes to think “that’s not by job”. Just like the guy driving the line-marking car in the photo above, who figured getting out of the truck to move the branch was not his job.

What would you think of a striker who stood by and let the ball roll into the goals because the goalkeeper was on the ground? It’s not his job to protect the goal, right?

If you are working on a team building a product or an experience for a customer, then your job is to bring them that experience, and to have it be the best experience it can be. Your job is not just testing or writing code; it’s not just writing the help copy or managing the DNS configs. They are just things that you’re good at – your job is the product. If you see something that’s broken, then it’s your job to fix it. Your job is everything.

So when you see that crooked stool or that empty glass – what will you do?