Our Thoughts on "Agile" and "Scrum"
Since computer programming began, humans have wrestled with how infernally difficult it is. They have responded to client and management pressure by devising "methodologies," one after the other after the other, all of which seek to reduce software development into a process which can be reliably replicated, as an industrial or manufacturing process might be. Two that "in vogue" today are called Agile, and Scrum.
"Agile" is directly based on manufacturing philosophy and has been reported to achieve notable success when applied in fields of endeavor other than computer programming. It is based on very short work-cycles that attempt to build the project in a somewhat exploratory fashion.
"Scrum" borrows its name from the sport of Rugby – from a very curious melee in which the players dive into a big pile and try to come out of that pile holding the ball. Scrum borrows heavily from the language of sports and, if we may say, "arm-chair quarterbacking."
The Real Problem – Software is an Autonomous Machine.
The essential problem with these two silver bullets – and, some might say, with all of them – is that they address the human process of software development, treating it as though improvements to that human process would produce better software. But very often they do not. And so, the obvious question is, "why not?"
A few years ago, we found a little e-book, Managing the Mechanism, by Vincent P. North ($7.99 on Amazon and also on iBooks), which changed our thoughts on these things very profoundly, and which helped us to articulate the reasons for the reservations that we had always had.
Mr. North's novel premise is that computer software is, by its intrinsic nature, fundamentally different from anything and everything else(!) that human beings endeavor to construct. It is an autonomous, self-directing, machine. This machine has a complexity that all physical devices know not of: it is composed of a virtually-unlimited number of moving parts, all interconnected. The teams which construct it have no control over it once it is set in motion – the computer hardware does exactly as it has been programmed to do. Nothing more, and nothing less.
So, to borrow from (for example) the Rugby analogy, we would have a rugby team composed entirely of robots, facing an opposing team composed entirely of human players. The team that constructed the robots would be in the locker room, unable to influence the robots in any way, and the robot team would be obliged to make not one single error when faced with human opponents, the conditions of the field, the weather, and so on.
We agree with Mr. North that "programming methodologies" categorically fail to live up to their expectations and promise precisely because they do not properly consider this angle. When the software is on the field playing the game, the team that built it is in the grandstands or the locker room, and there is no remote-control in their hands. The process of designing and then constructing it is over. The microprocessor now does what it has been programmed to do, nothing more or less. That software is either "complete and correct in all respects," or "it is not."
"We're Not 'Manufacturing' ... Anything." ... Therefore, "Manufacturing Wisdom" Does Not Apply.
The "real-world champion of Agile," as it turns out, is Toyota.
Toyota Corporation produces automobiles and other pieces of motorized equipment in factories that are occupied by robots, CNC machines, and human beings. The "lean and agile" processes which they successfully implemented in their plants, and which they now speak so highly of, are therefore human processes taking place in a field of human endeavor, even though that endeavor consists of building automobiles. At Toyota and elsewhere, humans see what is taking place and make adjustments in real time to keep the processes on course. But "a general-purpose software machine" is not a human process. It cannot reason. It cannot think.
The closest analog to "computer software" in the world of physical manufacturing is the "CNC = Computer Numerically-Controlled" manufacturing machine, but we would readily argue that it is a very poor comparison. The complexity of the task performed by these machines is vastly less than that of software in general. Furthermore, a CNC machine does not take input from the part that it is fabricating. A "general-purpose software machine," however, is constantly responding to external inputs over which it has no direct control. It does so "without thinking."
"Methodologies," such as those which attempt to map Toyota's factory-floor successes to the world of computer software, of course focus on the human process. They attribute software quality to the quality of the human process. But, does this analogy in fact hold water?
The outcome of a "general-purpose software machine" is not governed by humans! ... ... ... The machinery is merely constructed by them (entirely ahead of time). ... In the end, "humans have nothing to do with it."
The success or failure of a software machine is determined by the objective correctness of a set of pre-programmed and inflexible instructions given to that machine – yes, "to an over-glorified piece of sand, less than the size of your pinkie fingernail."
By extension, by the team's ability to construct a set of instructions that covers all(!) possible inputs, and all(!!) subsequent intermediate software states, under every(!!!) condition that could possibly occur.
("Good luck with that.")
- Although "computer software development" is obviously an entirely-human process, subsequent "computer software execution" is not.
- Literally, it does not matter "by what means your team attempted to develop this thing." The one, and only, thing that matters is: "did they succeed ... perfectly?"
At this time it is also appropriate to point out that "a general-purpose software machine," being composed of millions of "moving parts," is also very hostile to an "iterative" development approach, although many methodologies today espouse precisely such an approach. The basic problem here is that the development teams, being necessarily "heads-down to their work" (i.e. tree-bark level view), simply cannot tell what the next "iteration" best ought to be. (And, it's not their fault(!) that they can't: both perspectives are vital.)
Management must maintain a holistic viewpoint over both the total software system and the business unit environment in which it is intended to function. Teams which seek to maintain only a short-term perspective (as some methodologies explicitly encourage) easily become "[seat of the] pantsers," who appear to be very "busy" while nothing of permanence is actually being achieved by the group. Therefore, a judicious interplay of perspectives, and perhaps, a similarly judicious exercise of executive prerogatives, is essential to success.
A far better analogy might be a game of Chess, where each move is carefully considered before it is made. Or, the classic wooden-blocks game that challenges you to remove blocks (and then replace them somewhere else) without causing the whole thing to fall down in a heap. This is not the traditional "waterfall" approach but it is in fact very close to it, regarding both the short-term activities and their long-term interactions with the remainder of the system, possibly including portions not yet constructed.
This is what makes all the difference.
Success Factors That Do Work:
Software development is an immensely complex mental task, but we have been doing it for a long time and therefore we can readily point out a few basic principles that do benefit any project, consistently.
- Explicit, but Old-Fashioned, Project Management: The biggest problem, in any project of any size, is capturing and maintaining control of the big picture of the project, even as it is constantly moving. Computer programmers are not looking at the forest, nor at the trees: they have to be studying the bark of one tree at a time. Furthermore, as they continue to build the software structure, one piece-of-bark at a time, they must constantly ensure that their latest change didn't break something. (And, that "something" could be anywhere, because everything in a piece of software is more-or-less connected to everything else.) The workers are not in the proper position to perceive the total picture, so "self-directed teams" of any sort will not, in our experience, produce the quality of results that they believe they can.
- Simultaneous Focus on Short And Long-Term Perspectives: As programmers carefully construct the next piece and carefully put it into place, their focus must be extremely tight. They cannot reasonably be expected to realize when their actions have had negative impacts elsewhere – the total structure is simply too big. Another team must be tasked with ongoing quality assurance while a third team must be making a combination of tactical and strategic decisions. (When it is not possible to have multiple entirely-separate teams, members may necessarily "wear several hats." But each member should nonetheless focus on just one area at any time: forest, tree, or bark.)
- "Get the Balls Out of the Air and On the Ground, Quickly." Humans have a very limited capacity for dealing with several things simultaneously. Meanwhile, the number of interactions in a piece of software is essentially infinite. The project should be architected in stages that can be quickly brought to closure so that the stage – whether it must be re-opened or not – can be treated for future planning purposes as an atomic unit. And, each stage should be as self-contained as possible, minimizing the number of things that are known to be able to [adversely, or otherwise] affect it.
- "The Napkin Rule." If you can't describe the piece on the front side of a single bar napkin, try again. (It's okay to use two napkins, but that means two pieces.)
- "Discover Early." Like it or not, some aspect of software development will always be "a voyage of discovery." Your goal must be, however, to cause those discoveries to be made as early as possible. It's far better to be told, "a rock might be here, or there," and to pleasantly discover that in fact it is not, than to suffer the fate of a well-known ocean liner that was traveling just a little bit too fast in waters that contained a bit more frozen water than they thought.
Yes, "computer software is an autonomous, self-directing machine." Therefore, successful management strategies (like the ones we use) are ones that recognize this characteristic: leveraging its strengths while minimizing its weaknesses.
- You won't hear us using silly ersatz-sports terminology.
- We won't even pretend that the next day's work can actually be described using Post-It® Notes on a white-board. ("No, no, no ... that's nonsense.")
- You will never hear us suggesting that a team's problems are "its [human] fault." (They're not.)
- But we can advise you how to effectively (re-)structure your project to achieve what you want – productive, motivated teams who are motivated because they know that they're consistently producing robust, high-quality results. Results that, as they all can very-plainly see for themselves, will lead to success.