Software industry is peculiar in many senses
I've seen very good softwares and very bad ones. I've spent many hours looking at code: mine, other's; at work, at home; for money, for free, for fun. Unfortunately, good code does not mean good software. On the other hand, bad code hardly (although it might) produces good software. Also, softwares should be long maintained after they are created. This becomes very very hard when code isn't good.

I plan to gather my experiences and those of my friends and enemies and share code quality assurance techniques, tools and tricks with you.

I will make numbers, reproducible and falsifiable. I will use your opinions to evolve myself. No comment (unless very offensive) will be deleted. I will include documentation, sources and trusted references when available. But in the end I will give you my opinion, right or wrong I hope it will do you some good.

Go back to the basics

You wanna win? Always go back to the basics.
Dr. Richard Webber in Anatomy, Season 2, Episode 22

When in trouble companies try to adopt industry methodologies standards, managing techniques, coercive tactics. None usually work, at least not as good as expected. They always forget about the basics. A software is built of code. Code is the base.

I don't mean I don't agree with methodologies, managing techniques and specially coercive tactics. Just that I think they, alone, will not succeed. You need to strike at the root of the problem.

Many different things can be done to improve the code. Some, you might like, others you might hate, some others, might even require too much effort/cost to be feasible.

The quality of the code is a subjective concept. For me the most important feature is maintainability, which includes itself many features. The point is, a code takes a relatively short time to be written and released, then it takes a very long time being maintained.

Programming is craftsmanship, is an art.

This is an art but not everybody can appreciate it. Dr. Miguel Katrib (my greatest mentor), refering to Arts Appreciation

I made my first program 20 years ago. It was Basic on an MSX. It defined my life. Many years had to pass until I got to University and discovered most of the programming I think I knew, was wrong. Now every single day that passes I learn some of what I've done the day before was wrong. I learn much more from my failures than from my successes.

This will be a journey by misstakes and headaches not about success and enlightment.

Healthy software development cycle

for the developers

• Your code should compile with no errors at any time. Which means every change you made will not break local build. Every minute your code does not compile you are breaking the cycle.
• You should be able to run it often. Let's say every couple of minutes.
• Your changes should be integrable often, let's say 2 or 3 times a day you can check in changes without breaking the CI build and download latests changes without breaking local build. The longer you wait to integrate changes greater are the chances to break something when merging.
• You should be able to test everything let's say 2 or 3 times by release. If you only test once before a release and you find problems...then what? Delay the release?...customers and bosses will be furious! It will be better to delay one component/feature release than the whole bunch.

Quality checkpoints

There are many places to measure/improve the code quality. From the top, the user end till the bottom, the tiniest component. Each place has its features, challenges, pros and cons. I think a software cannot be quality assured only at the user end. Why? It's very hard to do: you cannot manually test everything; it's very slow, as software grows there more and more to test, there are new features that should be tested and also all existing ones should be constantly retested. We work automatizing computers, do we really think we can manually do everything they do? Do you? Well I don't. I think the human intervention should be the last barrier. Everything should be certified before reaching that point.

Embrace science not technology

I've seen a huge tendency to empower technology over science. Technology are the tools, the stick around for a relatively short time. Science stays forever. No body has ask me on an interview what an object is. Instead the wonder if I know Sql Server or MySql. I know relational DBs, sure there would be differences between me and the guy that really deeply knows the tool. What about when the tool get obsolete? it happens all the time. If you learn stone solid theory, you will learn the tools easily and get the most out of them.

Use documented and tested patterns

I love design patterns. There are tons of them these days. Trying to invent the solution to every single problem would be at best a waste of time. There are a lot of smart people which has analysed and published solutions for most of your problems. The greeks, 2 thousand years ago knew this. Do you think every architect invented the columns again and again? We, developers, constantly do. Use trusted sources: I love and write in code project, but it ain't, msdn and wikipedia either.

Standardize the code

Use code standards, naming standards, documentation standards. Setup tools to enforce them. Define them in assemblies, don't need to impose, get people's opinion. Evolve with time: C#2 standards might not be good enought for C#5.

Code analysis

Code analysis provides a way to know how bad your code is. As said before, it's not a measure that you could relate to software quality, but it helps. It is very interesting to see how it evolves in time. I plan to make some analysis with mostly open source programs from the internet and I will statistically try to relate bugs, costs and soo.

Testing

Nowadays we work with gigantic amounts of data. Huge problem domains. Components interdependencies reach from one point of the software till light years away. Testing is hard and exahusting. Event worst when maintaining. Every bug found by any stage should be tested for all the eternity.

Human testing

Humans are smart, brilliant and creative. But are slow(very) and expensive and lazy(even more). Machines do not get tired, ask Sarah Connor, are very fast, ask John. In every release cycle the whole software has to be manually tested, over and over. These tests should follow a checklist, a script. Testers should test everything that has been changed and everything the change feature connects to, which could be basically the whole software. We'll do the math some day, I promise.

Computers are much cheaper than people. If you think otherwise...this place is not for you.

Automatic testing.

There are many different ways to implement automatic tests. I will cover the ones I know and use. To start with them might seem like a waste of time. The good thing about automatic tests is once they are running they will keep running until the end of times. Regression tests will take longer and longer but we could just add more computing power.

Let's believe for a minute we build German cars. Those are the best cars in the world. Let's mentally go to one of those factories. Everything is in order, clean. There are no oil spills on the floor.  They certify everything: screws, engines and seats.

Let's see what it would feel like in our world.

Software factory

Welcome to Kode Meister GmbH

We are agile people and use some iterative methodology. Our iterations are a week long. We have 3 developers and 1 tester. The developers finish between 3 and 5 features and solve between 5 and 10 bugs per week. For each feature and bug between 5 and 10 tests are defined. The tester must execute them all before the week ends. Our numbers are:

Average	Value
Feature finished per iteration	3.6
Bugs solved per iteration	7.4
Tests per feature	6.92
Tests per bug	7.03
Tests per iteration	76.9
Minutes to perform a test	2

The numbers have been randomly generated with this.

Our developers constantly break existing features and previously solved bugs so in each iteration we must execute the tests for previous iterations (regression tests). Only 10 weeks after we started, this is what we have:

At this point the tester is overwhelmed by the number of tests. With this progression we will need more testers soon. Do you honestly believe your human testers and developers can keep up with this? You might even solve unemployment issues in Europe.

What's really bad is that it will keep growing and to make it worst, we are not being that rigorous.

You could say: that cannot be done, we cannot test this much, we'll just have to be more careful when developing. Ok, if you believe that: Go ahead to the comments and troll. Nothing from what comes next will be interesting for you. But keep in mind...you won't be making German cars no more.

If you wanna be spared of the mathematics jump to the solution.

The mathematics behind.

Let $f_{\mathrm{i\; j}}$ be the feature $j$ in the iteration $i$, $F_{\mathrm{i}}=\left\{f_{\mathrm{i\; j}}\right\}$ the set with all features of the iteration $i$, $b_{\mathrm{i\; j}}$ the bug $j$ in the iteration $i$, $B_{\mathrm{i}}=\left\{b_{\mathrm{i\; j}}\right\}$ the set with all the bugs of the iteration $i$, $t\left(x\right)$ the number of test needed for feature or bug $x$. For our first iteration we will have a total of $s_0$ tests, where:
$$s_0=\sum t\left(F_0\right)+\sum t\left(B_0\right)$$ The regression tests for the iteration $i$ will be:
$$r_{\mathrm{i}}=s_{\mathrm{i}-1}$$ Generalizing our simple model, the total number of tests:
$$s_{\mathrm{i}}=\sum t\left(F_{\mathrm{i}}\right)+\sum t\left(B_{\mathrm{i}}\right)+r_{\mathrm{i}}$$ Using the numbers from Kode Meister GmbH we have:

To make it worst

If you think you can pay for previous. Perhaps you can afford being more rigorous as well. Before closing a feature or a bug to ensure nothing has been broken the regression tests must be executed. This means $\left|F_{\mathrm{i}}\right|$ times more that regression tests should be executed. Same goes for the bugs: $\left|B_{\mathrm{i}}\right|$. Finally the testers must run all the tests again, that is one more time. This new formula would be:
$$s_{\mathrm{i}}=\sum t\left(F_{\mathrm{i}}\right)+\sum t\left(B_{\mathrm{i}}\right)+\left(\left|F_{\mathrm{i}}\right|+\left|B_{\mathrm{i}}\right|+1\right)r_{\mathrm{i}}$$
Using same numbers again we have this values:

And this graph, now in a ${Log}_{10}$ scale:

It has gone from an arithmetic progression to a geometric one. Still think you can afford it?

Automize your tests

Testing adds an additional cost to your product, a big one. But if you want quality there is nothing else that would provide as much. I believe that in the long run it even reduces costs. It is not easy, some times you'd have a lot of ideas you won't be able to take out of your head because testing gets in the way. Think of it this way: better testing in the way, than lots of broken features.

I plan to dive deeply into testing. It basically goes like this:

Integration Tests

This is where I would start. Developers would create small scenarios were they would test how components interact together. They would use actual databases, files, network connections. Some components might be mocked. This are very cost effective. Should be constantly executed but they are slow so maybe not that often. They should be executed from the CI.

End to end tests

This is the coarse grain tests. They test against the whole system like a final user would. There are tools which allow automatizing application usage. There are tools to record actions and then replay them. The downside is speed. They require the whole environment to be running. They made changes in the databases and take about a fraction of what a test user could take to perform the whole regression test list. You could try to perform them a couple of times a day.

Functional tests

Finer than previous. Developers, working together with testers would create testing scenarios (small versions of the program). Testers would create and modify test cases, which are input and expected results. This tests are fully automatic and should be faster than previous. They could be a table with input values and expected outputs. They could be run with the CI.

Boundary tests

Used to test all the boundaries of external libraries. That is all features you use. Use it as the learning process, then keep the tests. It's important to run them when upgrading to ensure nothing has been broken. Should not be executed very often.

Unit tests

My favorites. Unfortunately, they are very very expensive. Prerequisites are astonishing. In the end, they really provide very little information about the whole. They are the best support for refactoring and keep the technical debt at bay. They also provide component specification. Must be run constantly, on every single change. They must be blazing fast. Tiny tiny tiny. Only a single component should be tested at the time, all its dependencies must be mocked.

First step to a cure is acknowledging the problem. Thing that cannot be done if we don't know we have a problem at all. So let's diagnostics our selves shall we.

Master - Details

At Kode Meister GmbH we created a simple application with a several master -details screens. All of our features were either. Then we had trivial bugs about mainly about drawing issues (text overflows and so), medium size ones, about wrong data processing and some very hard ones with different origins.

After 10 iterations it looked like our performance was going down. If you look first iteration then last its kind of obvious.

Data was generated with this.

Estimate the performance waste

The actual performance ($p$) a team generates in all iterations is constant. People performance is always about the same. If your team is growing or shrinking this number would vary. But we could take per capita instead without loosing generality.

We first need to assign each feature and bug arbitrary weights based on relative effort needed to be completed or solved. e.g: a Grid is 5, a Details is 7, a Trivial is 1, a Medium is 3 and a Big is 5.

The Math

Let's say the effort (weight) needed to complete a feature is $e\left(f\right)$, the effort needed to solve a bug is $e\left(b\right)$. Let $f_{\mathrm{i\; j}}$ be the feature $j$ in the iteration $i$, $F_{\mathrm{i}}=\left\{f_{\mathrm{i\; j}}\right\}$ the set with all features of the iteration $i$, $b_{\mathrm{i\; j}}$ the bug $j$ in the iteration $i$, $B_{\mathrm{i}}=\left\{b_{\mathrm{i\; j}}\right\}$ the set with all the bugs of the iteration $i$ Then the first iteration could be expressed as:
$$p=\sum e\left(F_0\right)+\sum e\left(B_0\right)$$
As time passes and it becomes more and more difficult to complete features and solve bugs we want to measure how much more difficult it is now than before. We will call it ${\Delta p\mathrm{i}}_{}$. The general formula will be:
$$p=\sum e\left(F_{\mathrm{i}}\right)+\sum e\left(B_{\mathrm{i}}\right)+{\Delta p\mathrm{i}}_{}$$
If we put ${\Delta p\mathrm{i}}_{}$ on a graph we will see:

Although it might look pretty bad is not the worst. It's linear. Worry when it gets a higher degree, and call for help if it becomes exponential.

What next?

Grab your ITS and start collecting data of your code in the past. You could use VCS as well if it helps. If you don't have defined iteration, take segments of equal length of time. Create as many categories as you need for your features and bugs, the assign the weights accordingly.

Sometimes it's easier if you work backwards.

Mr. Jennings in Paycheck

Look at the past. See how you've been doing, a year ago, 6 month ago, now. If you are not doing good: Hang on, well get to that.