Skip to main content

bad science

My business is change. I like to think we help people to change for the better and improve by introducing tools and techniques we think are good practice. To do this, we rely on our own research but we also encourage our customers to do their own research, too, and for the most part what they find backs up our suggestions for improvement. Imagine my surprise, then, when a customer recently sent me a link to some 'research' from a company billing itself as a 'Research Lab.' To give it even more credence, the company trumpeted its very own 'Chief Scientist' complete with PhD.

The 'research' was pretty standard stuff, they'd done some static analysis on a bunch of systems from large companies and had found out pretty much what many of us already know - most computer systems aren't written very well and there is an awful lot of technical debt around. There are many reasons for this, which, again, many of us are aware of, so I won't bore you by evangalising our favoured solutions here and the main reason for publication of the 'research' anyway, was marketing of the company's products and services in this area.

However, there was one phrase in the findings that had caught the customer's  eye, "Scores for robustness, security, and changeability declined as the number of releases grew, with the trend most pronounced for security." In other words, the more releases you do, the more likely you are to have issues with security, robustness and changeability.

The reason this caught his eye is that we had been helping this customer with a transition to a model of Agile Software Development, of which some of the key principles are: "Early and frequent release of valuable software" and "Deliver working software frequently." We had told him that releasing as often as possible would lead to more satisfied customers and better quality software. Now he wanted to know how we could reconcile this with 'research' done by a 'Chief Scientist' from a 'Research Lab' that said the more releases a product has, the more likely it is to have security, robustness and changeability issues.

He had a very good point, moving to a continuous integration and delivery model is one of the first steps we recommend to all our customers but we also like to think we are scientists and that our methods are based on solid foundations. Should we continue with a continuous integration and delivery strategy when the 'research' tells us otherwise?

I'd like to report that we spent a lot of time an effort on this ourselves but it really only took us a few minutes to figure out what the real problem here was. Anyone who's undertaken any field of scientific study will know the first thing, and probably hardest, thing to learn is critical analysis. In other words, how to understand what the data is telling us. The first lesson in critical analysis is understanding that just because there is a relationship between two things doesn't necessarily mean the relationship is causal and even when it is a causal relationship, it is very important to identify which item is the cause and which item is the effect.

For example, every so often in the medical sector some bright young scientist will release research purporting to demonstrate that skinny people are more likely to die young than fat people are. Often this research is pounced upon by the more sensationalist newspapers and published under banner headlines exhorting us to abandon our diets.

The research in question is usually based on the measurement of weight and age at death for non-accidental mortalities and the researcher invariably finds the average weight of the fatalities to be less than the average weight of the population, concluding, therefore, that the less you weigh, the more likely you are to die.

What the researcher fails to account for:

  • People die young mostly because of illness.
  • Many of the illnesses that cause you to die young are wasting illnesses that cause massive and rapid weight loss before death.
  • Many people spend a long time in hospital before they die and are subject to dietary controls.



The truth is - it’s not being slim that makes you die, it’s dying that makes you slim.

Most medical professionals are aware of this phenomenon and simply ignore it as do all responsible medical journals. At the same time, many practitioners ready themselves for the barrage of questions shortly to be directed their way from concerned patients that have read the 'research.' After all, if it's published 'research' it must be true, mustn't it?

Let's go back to the 'research' that told us frequent releases lead to security, robustness and changeability issues and see what the researchers might have missed...

  • Anyone who's worked in a regulated industry would agree that security and robustness issues are mission-critical and need to be fixed and released ASAP. You don’t wait for the next scheduled release to deploy the fixes for them, you do a patch release. In other words, security and robustness issues force you to have more releases. Ergo, a system with security or robustness issues is very likely to have more releases.
  • Code that is difficult to change suffers from a similar problem in that software is difficult to change when it is difficult to understand.  If we understood it, it would be easy to change. When it's difficult to understand, it's also difficult to know when it's been fixed. You may think you've fixed it and do a release but you haven't. You may have to do several releases before it’s finally put to bed.Therefore, code that is difficult to change is likely to have more releases.
  • Code that has no issues doesn't have patch releases.



The truth here is - it's not frequent releases that cause code issues, it's code issues that cause frequent releases.

Even funnier to some people but actually quite tragic for our industry - a piece of research lamenting the poor analysis and design skills of software developers blighted by poor analysis skills itself. Most experienced professionals and journals will recognise this 'research' for what it is but, again,  the practitioners need to ready themselves for the barrage of questions from concerned customers. After all, if it's published 'research' it must be true, mustn't it?

Popular posts from this blog

The Death Knoll for the Agile Trainer

The winds of change blow fiercely, propelled by AI-driven virtual trainers, and I can't imagine for a minute that certification organisations have not already recognised the potential for a revolution in training. They may even already be preparing to embrace technology to reshape the Agile learning experience. Traditional face-to-face training and training organisations are on the verge of becoming obsolete as virtual tutors take the lead in guiding aspiring Agile practitioners through immersive digital experiences. The future of training and coaching lies in AI-driven virtual trainers and coaches. Trainers, powered by artificial intelligence engines such as ChatGPT, are set to revolutionise the learning experience. With AI-powered virtual trainers, learners can engage in immersive virtual environments, actively participate in simulations, collaborate with virtual team members, and tackle real-world scenarios. These trainers automatically analyse progress, provide instant feedback

Embracing AI - Augmented Intelligence

There is no denying that artificial intelligence (AI) has made significant strides over recent years, becoming more advanced and capable than ever before. With this progress, many have begun to wonder whether AI poses a threat to humanity, particularly our jobs, privacy, security, and overall well-being.  Some may argue that the rapid advancement of AI could lead to a dystopian world where machines rule supreme and humans become obsolete. However, it is important to remember that at its core, AI exists to serve us, not replace us. Instead of viewing AI as competition for human intelligence, we should consider it as an augmentation of our abilities.  This idea of 'Augmented Intelligence,' instead of Artificial Intelligence, highlights how powerful technology can enhance rather than impede human potential. Augmented Intelligence recognizes that humans and machines each possess their unique strengths, making them better together than apart. Humans excel in creativity, intuition, a

The Business Value of Telemetry

Dynamic technologies and infrastructure allow server failures and network issues to be quickly addressed, easily mitigated and, in many cases, reliably predicted. As a result, there’s a new venue opening for IT: end-user telemetry, which enables IT to determine how its internal users are consuming business resources, what type of application issues they are experiencing and how it impacts business performance. Gartner suggests that infrastructure and operations (I&O) leaders must change their approach and prioritize top-down business-oriented metrics. The research firm predicts that “60% of IT monitoring investments will include a focus on business-relevant metrics” by 2021, up from just 20% this year. Changing The Game Of course, it’s one thing to recognize the value of business-driven metrics and another to implement effective monitoring processes company-wide to overcome key barriers to effective digital transformation. The first step is understanding the fundamental shift requi