Fractals, Mother Nature, and Software Develpment

© by Mike Robinson

Fractals are an amazing, and amazingly useful, mathematical phenomenon that actually shows-up time and again in Nature. They exhibit self-referential detail, in the sense that you can “drill down” arbitrarily deep into a fractal-based image and continue to see just the same degree of apparent detail ... yet never exactly the same.

Fractals, as it turns out, are one of Mother Nature's greatest “house tricks.” Immensely complicated phenomena, such as the branching of trees and the growth of forests, can be described through a very small set of self-referential rules. Furthermore, these strategies are inherently self-adaptive. If you plunk down an unexpected obstacle, such as a large boulder, in time the tree will grow around it ... without changing any of its built-in rules. Given enough time, the tree can grow to surround and split the rock.

One key usefulness of “a fractal approach to problem-solving” is that with it you are able to focus your attention on as much, or as little, detail as you at-the-moment require. You approach the problem simultaneously from a macroscopic and a microscopic viewpoint, as the needs of the moment may require. Throughout the course of the projct, you maintain both viewpoints in tandem:   that is to say, you do not trade one for the other.

Software is a Fractal Endeavor:

Software-development is, by its very nature, extremely detail-oriented. When you finally drill-down to the capabilities of the actual hardware, it is all “ones and zeroes, and nothing more.” The hardware is racing through a twisting pathway at a rate of multiple billions of operations per second (per CPU!) blindly following decisions that always evaluate to either True or False. This is the very nature of the beast. At least today, this is the one-and-only way that computing hardware can operate.

Therefore, that's the only way that the software which finally goes into that machinery can be constructed.

But that's not how we humans view the computer-programs that we encounter every day! For instance, if you happen to be reading this web-page through the Firefox® browser, you're using a program that consists of well over ten million of these “True or False” instructions, and those instructions are doing their work with the help of many tens of millions more instructions. But nevertheless, in your own mind and from your own perspective, that is not what you are doing:   you are “reading an interesting article.” Those hundreds of millions of yes-or-no instructions are, to you, merely a means to an end. Your only concern as an end-user is to read the article. You don't care what the Firefox team did, as long as they did it well.

You are therefore taking a very high-level view ... the end-user's view ... of a phenomenon that is simultaneously and necessarily identical to the software-developer's viewpoint, even though the developer's viewpoint appears to be entirely focused upon tens-of-millions of computer instructions embodied in vast amounts of painstakingly handwritten source-code. The more you contemplate it, the more you realize just how enormous that distance really is. It boggles the mind.

“How different” these two viewpoints are, and yet, “how utterly and completely the same.”

How, then, did the many thousands of people who have worked on Firefox over the years achive their goal, and how do they continue to do it? The Firefox project is certainly not unusual in this respect, so let me generalize:   how do professional software developers do it? How do you get such an enormous undertaking to work at all, let alone make it look easy? And on the other hand, how do other teams engaged in the very same endeavor fail so utterly, leaving such a great-big smoking hole in the desert landscape as they fall out of the sky?

There's a clue to be found in those fractals ...

It's All The Same... It Must Be “All the Same” ...

If you display a common fractal such as a Julia Set (at right), at any resolution, you'll see that it exhibits a strikingly complex and intricate apparent structure. But if you dive into the underlying mathematics, you'll discover that they are simple ... and that they embody no concept at all of “zooming” or “resolution.” It is pure-and-simply a fundamental characteristic of fractals that, if you choose a particular point on the set and “drill down” into it, you'll see the same infinitely-complex structure repeated endlessly. Again. And again, and again and again. But here is why you can actually do that:   the “rules” of the Set have not changed. They are just being expressed again. You perceive it to be a much-greater level of detail (because you've “zoomed in”) but mathematically speaking, it's exactly the same. Therefore, it is possible for you to regard a Julia Set, or any other fractal, at whatever level of apparent detail you wish without “changing the rules.”

When you display a fragment of a fractal-based image, or if you consider a fractal-based natural phenomenon such as tree-branching or leaf-placement, you are able to limit the scope of what you have to “consider.” To fully understand the branching-behavior that you see on one arbitrary branch of one arbitrary tree, you do not have to understand the entire tree; much less the entire forest. Nor, realistically speaking, could you.

(Hold on to your “mental hats” here ... Ready? Here we go...)

Having just said that “you don't have to understand the forest,” let me now seem to utterly reverse myself and say that you cannot possibly “fully understand” what you see on one arbitrary branch of one arbitrary tree, without considering it in the context of the entire forest in which it appears, and even the greater climate which surrounds that forest! You can look at a natural phenomenon in isolation, at whatever level of detail you wish (such as examining the emergence of a single leaf), but you cannot separate that arbitrarily-isolated viewpoint from an equally arbitrarily-expansive one.

Per contra, you also cannot separate that arbitrarily-expansive “forest view” (or even “climate view”) from the characteristics of all those leaves and branches and trees.

You cannot sever one viewpoint from the other.

Why? Because the two are one-and-the-same.

The two viewpoints are all the same, and they must be “all the same.” Nevertheless, through the magic of self-referential detail, endless drilling-down and recursion is possible (and quite meaningful) without losing either detail or perspective. Any sort of micro-phenomenon (such as the particular division of individual cells that leads to the growth of a new leaf on a particular limb at a particular just-right angle and placement) can be meaningfully undertaken at any point, while considering no more details than what actually surrounds that micro-phenomenon. A leaf doesn't need to consider the entire forest to be able to grow. Likewise, any sort of macro-phenomenon, such as the advancement of the forest, can be meaningfully understood without considering “every leaf therein,” provided that the leaves are not ignored as the need to consider them may arise.

Scientists are able to generate tens of thousands of research papers (and to keep myriads of graduate-students gainfully unemployed) because it is possible to reposition one's viewpoint anywhere within the vast expanse of nature without losing detail, and without losing the connection between “a single aspect” and “the whole.”

And now, software ...

It is the studied opinion of this author that most software-development “methodologies” fall flat on their august faces because they do not properly consider this “this apparent duplicity,” and/or because they cannot accommodate it in their oh-so binary, oh-so procedural world-view. Like a binary-driven computer, classical models regard one at the exclusion of the other, or the other at the exclusion of the one. In my opinion, they either deny-outright the connection between them, or argue (unsuccessfully) that it's somehow okay to exclude such things.

Mother Nature, quite plainly, doesn't “exclude such things,” yet she accomplishes both. What might we learn from this? To begin, let me review and comment-upon three of the most common classical software-development methodologies, considering each one of them in this particular light.

The “Waterfall” ...

For instance, consider the classical Waterfall model, which made its first appearance in the 1970's. As you can (barely) see from the diagram at right, this approach postulates four discrete development steps: design, coding, verification and testing, and finally, deployment. The idea is that the group can focus on one step at a time before moving-on to the other. But you can immediately see that there must be a problem in this picture:   look at all those arrows! If there's only one set of arrows leading “progressively” forward, how can it be that there are so many arrows leading backward, even one that goes “all the way back to start?” (How can it be “a waterfall,” when it's mostly flowing up?)

Clearly, the real dynamics of the Waterfall model must exist, not in the boxes (which purport to be well-defined “destinations”), but rather in the arrows. So it must be fairly self-evident that those boxes in the Waterfall picture are not goal-points, and the picture is not one of real progress. If it were, the layout of “forward-facing” arrows would be similar to the “backward-facing” ones (and presumably there would be a lot fewer of each). This picture may well be descriptive, but it is not descriptive of success.

Continuing my thesis, I submit that an adequate explanation for this “unexplained” (and unmanageable) dynamic is that the Waterfall model is trying to turn a forest into a funnel. And before long, it runs into trouble. There's a boulder in the way, for example. Or, different subgroups of the same development team, all with their own set of horse-blinders on, start running into one another, somewhere down the track. As the “Waterfall” drills-down from Design to Deployment, it moves too-quickly to the Coding stage ... only to be kicked-back to the Design stage. Or, it proceeds to the third step, Testing ... only to be kicked-back to ... the Design stage. And so on.

Meanwhile, what's happening to the thing that is actually being created here? What's happening to the source-code? Nearly all of it is going right onto the cutting-room floor:   as useless scrap. Perhaps in a vain effort to save what they started with, the team inevitably begins to try to mash it all together ... to “make it work somehow.” The project disintegrates into cacaphony. The bugs that get mashed-in at this point will never be expunged.

How would we evaluate this in regard to Nature's way? Well, we rather-plainly see that “the Waterfall” can either consider “the whole,” at the Design stage, or “a single aspect,” in Coding and Testing, but it cannot consider both at the same time. So we seem to be a long way from Mother Nature, where an individual leaf can grow successfully, while the forest does the same, and nobody has to start-over. Nature is endlessly self-adaptive, but it does not regress.

Nature is holistic, applying one strategy to everything she does.

The “Agile” ...

There are other methodologies, of course. For example, let's consider Agile, which is still getting a lot of press these days. The idea here is that you start (at twelve o'clock in the diagram) with the high-level design and planning stages, then you drop into the circle of iterative refinement, “the loop,” devising and creating and deploying(!) your latest effort as quickly as possible, presumably so that the users and “early adopters” can help you with the testing. You might not hit the target, but you presumably hit something, and I guess you're supposed to be proud that you did “it” quickly. With the “lessons” you have thus learned, the cycle begins again ... until eventually it pops-out (at ten o'clock) with what is called “system testing” and then “release.”

Okay, maybe I'm just being unenlightened here, but to me that “loop” looks a whole lot like that little wheel that my neighbor's gerbil used to spend so much time on. That “exit point” at ten-o-clock looks like the point where management finally pulls the plug on the project, or where the team members finally get to “the FISI point.” (FISI = “F*** It, Ship It.”) Otherwise, it's just the Waterfall model all over again, just with a lot more arrows. The Agile model is flittering about the whole project like a dragonfly, zooming from one point to another but only in a reactive mode. It is, in this author's opinion, a great way to discover where all the rocks are ... because you keep crashing into them.

The “Extreme” ...

All right then, how about “Extreme Programming (XP)?” Well, to me, this is “same song, different verse.” Basically the same diagram, but with even more arrows. Instead of one “loop” in the diagram, we have two. Or more. In this school of thought we encounter disturbing words, like “uncertain estimates” (v.s. “confident estimates&rdquo ... which one would you prefer to base spending a whole lot of money on?); quaint words like “velocity” and “stories” and “metaphors;” and even non-words, like “productionizing.” This is progress?

What do we see in all of these pictures? We see... arrows. And those arrows mean far, far more than the boxes which connect them.

Look At The Arrows...

Those arrows mean real trouble.

The multitude of arrows in each of these diagrams belies the presence of a dynamic in each scenario ... a dynamic that is more-or-less unmanaged by that scenario. The diagrams might be intended to encourage you to discount this, as though it were somehow proper to regard it as “unimportant” in light of all that obvious “activity.” Yet you would be seriously mistaken to allow yourself to do so. The key characteristic of each diagram is that it is filled with many more arrows than boxes, and that most of those arrows are not pointing in the direction of the intended project goal. That is cause for alarm.

More damningly, again in my opinion, these scenarios paint the development project as though it were “a voyage of discovery,” sailing out upon the open sea with neither map nor compass, nor a satellite orbiting overhead to show you where the hurricanes are. You send back regular telegrams advising your financial backers of your “progress,” but progress is supposed to be much more than “sailing across the waters without sinking.” (Let's face it:   none of these big software applications that we're building are stunningly new and different... These are cruise-ships, not Christopher Columbus' Santa Maria.)

All is One...

Recalling my original thesis, I said that in the extremely “nature-al” world of fractals we see the ultimate manifestation of an extremely holistic strategy. It is possible to “drill down” to an arbitrary single-aspect without losing sight of, nor the connection with, the whole. Each aspect can be considered “in isolation” precisely because it is an apparent “isolation,” not a real one. The rules do not change, regardless. This is traditional human reasoning turned precisely on its head:   instead of “the whole is the sum of its parts,” we have “the parts are an identical manifestation of the whole.”

Software does, in fact, exhibit a similar structure. During all this time that hundreds-of-millions of “true-or-false” computer instructions embedded in your computer have been working so hard on your behalf, kindly notice you have been able to “just keep reading.” The entire goal of that content-delivery system has therefore been successfully achieved, here at the nearly-end of this story, because at no point did you have to be distracted away from what is to you the entire purpose and objective of that entire system:  reading. No doubt you clicked the “page down” button a few times to bring new content onto the screen, and each time the content-delivery system reacted appropriately. (“if the page-down signal is issued, then scroll the screen down a page.”) You had no reason to “drill down” into the actual software-code that handles that request, but if you did, you would encounter exactly the same endlessly-repeating pattern:   “if .. then.” You would find that that as you “descended,” your viewpoint was ever-more focused, but never truly isolated. The thread that you would follow through the code would consist of endless repetitions of the same pattern, always remaining within the context of the entire content-delivery system software.

All of the leaves in a forest grow at once. All of them grow, and live, and die within the space of just a couple of cubic inches. All of the trees grow at once, and, unless one falls into another one, they each grow in their own separate space. The forest expands and contracts, over years and centuries, and the dynamics of the growth and retraction of the forest are one-and-the-same as those of the leaf. They are all connected; they are all one ... the forest growing in response to the leaves and the leaves growing in a way that reflects the nature of the forest. It all works because, in that natural system, the dynamics are all one-and-the-same. This is what makes nature endlessly adaptive. That is what makes nature successful.

As long as you regard “the whole” as something that you must briefly-regard and then run away from as a group in pursuit of a particular “aspect,” you will quickly lose sight of the whole and not be able to integrate whatever you just “did” into it. Also, the more-often and the longer you stay away from regarding both viewpoints simultaneously, the more likely it is that some particular aspect of your “hard work” will come to ... nothing but “scrap on the floor.”

Virtually all of the arrows on all of those diagrams, in a forward direction, represent a change-of-focus of the entire group from “the whole” to “the aspect.” Likewise, the arrows that move in a backward direction represent ... scrap, abandoned effort, misplaced or wasted development resources, and lots of other unpleasant things. Finally, none of the diagrams truly represent the nested level of focus-change that takes place ... forest, tree, limb, leaf ... nor do they represent the fact that the workgroup, far from being centered on just one aspect at a time, must be simultaneously concerned with a great many.

A software project is like a collection of “Chinese Boxes,” each one nested inside the other, each one reflecting a very similar structure and obeying the same rules. (Obviously, this analogy is somewhat flawed because the structure is actually a lot more similar to that tangle of yummy noodles that's presumably inside one of those boxes, but the analogy of “repetitions of one-and-the-same thing” still holds.) A successful software project will start with “the whole,” then branch, in parallel to more-detailed consideration of one or more aspects, but it must always return, as soon as possible, toward “the whole.” Insofar as possible, the contents of each successive “box” can be properly considered without choosing to open it, or, if one does choose to open it, without undue concern for whatever is outside. Finally, as various members of the team have “drilled down” perhaps-deeply into one particular box, other members of the team are continuing to remain focused on the greater project as a whole. Somebody's watching the wheel of the ship, while somebody else is watching from the satellite.

Mother Nature has one thing going for her that we don't have yet:   massive parallelism. Millions of trees, and trillions of cells, all around the planet really do grow “at the same time.” We can't actually do that with our software yet ... we're constructors, not farmers ... but we can structure our projects in the same “self-referential” way, and in that way cause them to be manageable.

Any software development team must “accomplish many things ‘at once’ ” while actually working “one at a time.” Until and unless we figure out a way to clone humans by the billions, as Nature can do with all those individual cells, we're going to be dashing from one aspect to another and necessarily focusing a brief but highly-focused intensity upon each one. But we must always arrange our affairs such that, having done so, we can as quickly as possible “close the box” on that particular aspect, and return our viewpoint in the direction of the whole. We cannot drive the entire focus of the entire team upon any one of these aspects but must always maintain, and return to, that holistic view.

This has nothing at all to do with “release cycles,” or even “testing cycles.” Instead it has to do with exactly how we choose to place our focus, both as a team and as individuals. It has to do with what we say we value; with what we choose to make the guiding force of our short- and long-term decisions.

Each of these strategies, I think, rather-unabashedly emphasizes coding, on the premise that “a programmer is paid to ‘write code’ so if you're ‘writing code’ you must be ‘doing something.’ ” Yet how many times it is proved again that this is just not so! Wasn't all that film-on-the-floor “properly exposed” and “artfully composed,” not to mention “manufactured at great cost?”

Progress is not “progress” if what is “done” must be re-done. You are not advancing on your intended objective if the only product of your “voyage of discovery” is merely to discover that you're lost. If you can't see the big picture yet, on any particular thing, then don't try to proceed to the small with regard to that thing. A team-member might be able to proceed to-the-small on a different part of the task, but maybe not. Either way “it's all right.” A micro-focused embodiment (such as “source-code”) of a big-picture thing that is not yet understood cannot be something worth actually building... yet. So, don't waste the effort. Focus your individual and collective efforts, from moment to moment, at whatever level of apparent detail and at whatever aspect of the project is most-likely to advance the effort without producing scrap. And if you must “scrap,” do it at the highest level possible. “Chalk is cheap ... source code isn't.”

It is an unfortunate fact of software development that we so-easily lose sight of this. Any time we “write more code,” we are necessarily looking through a microscope, focusing our entire attention on just one thing and doing so most-intently. It demands our complete and focused mental effort. And this is why every member of the project should be working in a management environment that returns their view, even daily, out of this necessary-but-extreme myopia, back to this high-level view. Likewise, the project should always be managed from the highest level possible, and this management-plan should then be expanded as-necessary into a nesting of identical procedural parts ... just like those boxes of yummy noodles. The project management strategy, like (insofar as possible) the code itself, should reflect that self-referential, self-refocusing, endlessly recursive approach. Don't change your strategy.

Nature has chosen a simple strategy and applies it repeatedly, dealing with every level of apparent detail in the same way and therefore simultaneously. The detail is “apparent,” but it is actually the same. Nature can divide without changing. It can progress in a meaningful way both “in the grand” and “in the microscopic.”

So can we.

© Copyright 1996-2008 Sundial Services. All rights reserved. | Terms of Use; Copyright; Trademarks | Privacy Policy | Website Credits | Site last updated:   August 14, 2008.