Agile Rant Update

After you've read my Agile rant you may have though: "just another old-fashioned hacker who just doesn't get it...". Fair enough, I'm myself not that sure if that what I'm saying isn't just result of lack of experience with Agile? Maybe with even more exposure all my reservations would vanish?

So I'm happy to find out that there are other people thinking roughly along my lines. For example here: Agile Hokum, in DDJ reader mail. He (the reader)says exaclty what I said:

But the "TDD corrupted by writing the tests after coding" is about as extreme as the Taliban. Testing is good. Write them before, during, after, much after, etc., all have their place.

and doubts usefulness pair programming as well. The standard TDD answer of the expert goes than like that:

I agree that tests written at anytime will generally have value. However, TDD was not designed as a way of validating code — plain unit testing will do that. TDD is first and foremost a design practice. And so doing TDD by writing the tests after the code is simply a misunderstanding of what you’re actually doing.

Well, that opinon is already dealt with in my Agile rant. But taking the simplistic road I'd just say that's the way it is handled in agile projects: not as a design technique but rather as a testing one. OK, here seems to be some confusion on the term itself: it's either Test Driven Development or Design...

On the other side the author (i.e. the reader) openly admits that he's not an Agile practitioner, so that opinion doesn't give any earnest support for my criticism?.

But look what an ex Googler says here:

This is not to say that all unit testing is bad--of course not. The dispatch logic in Guice Servlet and Sitebricks benefit from rigorous, modular unit tests. If you have ever used Gmail, Blogger, Google Apps, AdWords or Google Wave (all use Guice Servlet to dispatch requests) you have seen the benefits of this rigor first-hand. But take it from me, we could have achieved the same level of confidence with a few well written unit tests and a battery of integration tests. And we'd have been in a much better position to improve the framework and add features quickly.

That strikes a chord with me, because it's exactly the way I tested my last system - a lot of integration and pre-integration tests, and a couple of unit tests for classes which couldn't be easily tested thorougly that way. Wouldn't that be allowed at your company? Because this isn't Agile?

A Little Agile nad XP Rant

The title of this blog is "On Languages ... etc etc ... Systems and Proceses", but utill now I never said anything about processes (or did you think I meant UNIX processes?) so let us write something about the latter now. As lately everybody seems to be happy with Agile/XP (like here and here), let's rant about that a little. So beware, it's (at least partially) a #RANT#, so don't try to kill or evangelize me!

1. The Gripes

The one single thing you encounter today in the process land is the Agile. Well, I have a few gripes with that, which I should have wrote up earlier (and I wanted to, really, for the last 3 years or so), but now the time has come. On the other side, now when Agile is mainstream, maybe my points aren't valid anymore? You know, "the normative force of the factual"* :(. Maybe, but let us begin from the start of all that:

As I read the "Agile Manifesto" and the XP explanations some 10 years ago or so, I was just amazed. It was the liberation! And what I particularly liked were things like pair programming, test driven design, and iterative deleopment. But today I must realize that I was somehow mistaken - maybe did I get it all wrong? Somehow it's all different from what I imagined it'd be, and I don't like it very much. Two examples:

a) the pair programming

In my eyes it was all about two people caring about a piece of code, discussing its design, reviewing main rewrites, just because discussing something with another person can show you another perspective on the problem.

And now? The reality? It is crazy, it's about keyboard swapping, and two people doing coding at the same time. For 15 minutes it's my keyboard, and then it's yours. Come on, are we in Henry Ford's car factory? Or were we supposed to be doing some creative work? #RANT#

But there are people who actually like that (here, here) aren't they? I'd say this could work, but only if you are paired up with someone on your level. Not just another beginner guy with a big ego, thank you very much! Even then I find it to be an overkill, 2 people doing work of 1? Aren't there people able to be doing decent work by themselves anymore? #RANT#

b) test driven development/design (TDD)

I liked this idea very much. OK, I've only seldom written the tests before code myself, but there always have been tests for a new feature: either before I coded it or after that. Because really, that doesn't matter if you know what a feature you are going to implement.

But now? The reality? You should always write tests first. Or you are not agile. And why that? It comes even better - TDD can mean test driven design! A good example of this was to find in a blog posting** of the venerable Uncle Bob where he develops a prime factorization algorithm by TDD, i.e. by writing a series of increasingly complicated tests and reworking the algorithm accordingly. That looks for me like crazy, groping in the dark for the right solution, excuse me?! #RANT#

Why should we waste time and rework our solution on and on? The received answer is: because it results in decoupled code. Come on, aren't there other ways to write decoupled code? Like old and unfashionable thinking about its design a little? #RANT#

2. Some Summary and More Gripes

So meanwhile the XP is for me not the freedom, bur rather the next horsewhip of the pointy haired boss types. It's more about following the rules and their religious observance than about anything else. As I read about XP/Agile first, I thought it was targeted at proficient developers, you know, at the responsible adults, which should be allowed to be adult and just do their job! But now I have more and more the impression that it's rather targeted at beginners. Obviously, they need clear rules and directions. So maybe it's about the overall prevalence (and importance) of beginners in our industry?

Well, mabe (just maybe) I'm opposing the abovementioned XP principles because they introduce a process-like quality into XP, and I feel kind of betrayed now? It was supposed to be about freedom and interacting free individuals? Maybe it was just an illusion, maybe the recipe of Agile Manifesto was too simple?

In this this excellent post about "post agile" development Gwaredd Mountain states that no methodology is pure "agile" or pure "process-oriented". Rather, each one of them contains "freedom" (agile) and "process" elements, as a mix of both is needed for the real world. So even the supposedly totally free XP has to have some emergency brakes. Well, so much for my illusions of a better world :(. Nonetheless, the emergency brakes are rather expensive, the double pair of eyes to watch over "Brownian motion" like TDD development? That's my private opinion, but arguably it's somehow taking things to the extreme.

But there's 3rd problem, and it's not easily dismissed:

c) evangelism

In the Agile world-view there's either agile or waterfall, stone age, and nuclear winter. A typical Twitter post says:

Agile change is change. Change is hard for people

thus, if you can't get enthusiastic about Agile it means that you are unreformable, and just can't see the light! I think the Agile trainer guild is responsible for that***. As their training courses target mainly the beginners, they must give them rules to be followed in each circumstances. But I cannot understand why this must be promulgated to the general public? You know, "Only a Sith deals in absolutes", it's not a sign of wisdom do divide the world in the only right way and all the others! #RANT#

Ok guys, you must earn a living, but doesn't engineering decency mean anything to you? Or can't you think for yourselfs and would rather become a follower? Silver bullet anyone? That irritates me, but at the same timee it's interesting - what could possibly be a result of a realy intensive full-night talk with an Agile trainer? #RANT#

It's a little known fact, but Dr. Royce's defining paper for the waterfall process already contained iterations (see figures 3 and 4)! Agile is not that original, people were able to think and use common sense before that! #RANT#

3. So What Do I Think Myself?

You may ask, how should we develop software then? We're not supposed to go back to "cowboy coding" are we? Steve Yegge' post describes an alternative: the pretty informal Google development practices at that time and praises it as the solution. Well I must admit it does sound good, but then even Google wanted to introduce some process not a long time after that (Shh, we are adding a process). In that article they describe how the Scrum process was introduced in the AdSense project (if I'm not mistaken), however not a full embodiment of Scrum. But it seems that somehow for time critical, heavy-money projects you need some kind of a whip in the end... :(

So what do we need a process for? Do we need it at all?

First let me quote Aristotle (via "Pragamtic Programmers" link, and no, I don't read him in original...) - "We are what we repeatedly do. Excellence, then, is not an act, but a habit." As I understand it, we have to kind of design the things which we repeatedly do, if we want to ge good at something. And that's a start of a process.

The 2nd insight comes from the field of avalanche avoidance (because I'm somehow of an offpiste skier). What the professionals always do, is to have a checklist in the car and to go through it every time they set off. Because people forget, people do make mistakes, and in that profession, when they do, it can cost them their life. That's a kind of process too - you won't believe that you could do a mistake, but in the grand scheme of things, you probably will. So you need some annoying routine, even if you don't think so!

So what would I take? A methodology I liked rather well was the "Tracer Bullet Development" from the abovementioned "Pragmatic Programmers" book. It's simple, it has a lot of common sense (what I seem to like more then religious fervor) and it goes along the lines of my own understanding of the agile development which I came to during the last 10 or so years.

Because I'm not an enemy of agile. More than taht, I think it's somehow a necessity! In real life there's constant change, and the original design**** of a system cannot be sustained - either it will be dying or it must adapt! So for me, agile has got it right in its basic proposition - you must always iterate. As the ancients used to say "iterare necesse est, vivere non est necesse" :).

And what about my Agile/XP gripes? Here I'm with Steve Yegge - there's "Agile" and "agile", Agile of the trainers and pointy-haired bosses, and agile of the pragmatics. And I'm with the latter.

Waiting for your comments.

---
* or "die normative Kraft des Faktischen", by Immanuel Kant. This means: all that is, is good, because it is (i.e. exists) or something like that.

** sorry, the original blogpost I read is vanished, this one is the current version reworked as a programming kata:
http://butunclebob.com/ArticleS.UncleBob.ThePrimeFactorsKata

*** Read Steve Yegge's post about the so called "Agile" experts: "Good Agile, Bad Agile ", http://steve-yegge.blogspot.com/2006/09/good-agile-bad-agile_27.html, or this one: "Agile Ruined My Life", http://www.whattofix.com/blog/archives/2010/09/agile-ruined-my.php and look out for the Fraud markers!

**** For example, take this fascinating and/or terryfying case of dying architecture (from the "Refactoring in Large Software Projects" book):

Alas, everything started out so well! The concepts were clear-cut. The technical prototype was a success, the customer was thrilled and Gartner Group decided that the architecture was flawless. Then real life began to take its toll: the number of developers went up, the average qualification dropped, the architect always had other things on his plate – and time pressure increased.

The consequence was that the architecture concepts were executed less and less consistently. Dependencies were in a tangle, performance dropped (too many client/server hops and too many single database queries), and originally small modifications turned into huge catastrophes. To a certain extent the architecture concepts were circumvented on purpose. For example, classes were instantiated via Reflection because the class was not accessible at compile-time.

C vs C++ and some celebrity gossiping

Every time I read a post of Linus "Linux" Torvalds I can't help thinking "what a smug, assumptuous, xxx-yyy-zzz!". Well, I don't know the man personally, but I certainly wouldn't like to have him as my boss in any project, betcha! The first quote is a couple of years old (I cite from memory as I cannot find it anymore) and was a reply to some proposal Linus didn't like :

....go and play in your little world...

It looks innoculous enough here and now, but in the context it was realy ugly. And now, for some time everyone seems to feel obliged to speak about Linus' C++-hating post*, so I had a look at it myself. OK, nothing changed, it goes in the same vein:

*YOU* are full of bullshit. ...... is likely a programmer that I really *would* prefer to piss off, so that he doesn't come and screw up any project I'm involved with. ...... the end result is a horrible and unmaintainable mess. But I'm sure you'd like it more than git.

... etc, etc, etc. OK, maybe it's only his personal creative writing coach who's to be blamed, or perhaps it's the macho Linux kernel developer culture? But, aside of personal dislike, what the man says got a bell ringing with me. Why? Read on:

C++ leads to really really bad design choices. You invariably start using the "nice" library features of the language like STL and Boost and other total and utter crap, that may "help" you program, but causes:

• infinite amounts of pain when they don't work (and anybody who tells me that STL and especially Boost are stable and portable is just so full of BS that it's not even funny)
  
• inefficient abstracted programming models where two years down the road you notice that some abstraction wasn't very efficient, but now all your code depends on all the nice object models around it, and you cannot fix it without rewriting your app.

In other words, the only way to do good, efficient, and system-level and portable C++ ends up to limit yourself to all the things that are basically available in C.

Whoa, that man is really hardcore! What he's actually saying is: don't trust any code you didn't write by yourself! And on a higher level: any abstraction we are using is a trap, lulling us in a false sense of security. And more: we can really build big, fast, complex systems without using OO abstractions!

Didn't I feel the same before? That for the efficient, near system level code we can take C, and that all the fancy object thing, where the is better done in Ruby or (even) Java? So no place for C++ here? Take Wireshark protocol analyzer as example?

Gossiping

Well, the story doesn't end here. First, there are some entertaining comments on digg**. My favourites are:

• Linus codes a kernel for a living, so its not that surprising that he hates C++.
  
• Linus is an *****, he lives here in Beaverton and the man has a big ego for someone nobody outside of the Linux community cares about.
  
• Ok... Who cares if Linus Torvalds hates C++. I don't really give a damn.
  
• another episode of "I am Linus and hate everything"


• funny that linus prefers kde when it is programmed in C++
  
• Personally I've evolved thinking in OO abstractions, so working with C++ is much more natural for me than C. Does that mean I'm a crappy programmer? Only Linux Torvals knows


• The STL is the biggest piece of crap I've seen in 40 years of programming. It's a graduate students project to prove one can write a totally orthogonal, yet totally inefficient, impossible to maintain, piece of crud.

Well, as it seems, first: people aren't taking Linus such seriously, and second: STL ist the culprit!

Living in the past?

So, as to begin with something, what's the matter with STL? As we are gossiping in this installment, it's perfectly fine for me to say that the prevalent opinion on the Web (for example ***) is that Linus is referring to a problem from the past (around 2001 or so), when he's speaking abot the non-portability of C++. At that time the support for the C++ standard, and especially tempaltes, was very unconsistet across the compilers, and so the STL implementations could be nonportable between compilers! But nowadays even Visual C++ is quite up to speed here!

Then the inefficiency allegation. I don't even want to discuss it here, because it's so old (back in time to 1998 or so). There's long refutation along classical lines from that time to be found****, if only not very entertaining, and a shorter one*****, from a practitioner's point of view - Steven Dewhurst actually wrote low level code with C++ and templates:

Just to annoy people like Linus, I've also used typelist meta-algorithms to generate exception handlers with identical efficiency to hand-coded C. In a number of recent talks given at the Embedded Systems conferences, I've shown that commonly-criticized C++ language features can significantly outperform the C analogs.

Who's incompetent?

Next comes the critique that C++ tends to attract substandard programmers, and that:

... limiting your project to C means that people don't screw that up, and also means that you get a lot of programmers that do actually understand low-level issues and don't screw things up with any idiotic "object model" crap.

The first thought that comes to mind is Linus' "software Darwinism": in 2000 he lambasted people wanting a debugger in the Linux kernel. His argument was: I don't need any sissy that needs a debugger! I want people who understand the code as a whole! Any higher level abstraction or language (i.e. STL or C++) will make you a wimp and not careful enough!

The fact is, that is *exactly* the kinds of things that C excels at. Not just as a language, but as a required *mentality*. One of the great strengths of C is that it doesn't make you think of your program as anything high-level...

But isn't this just another management whip for the programmers to keep them under pressure, so they are more obedient? A manager's trick? The Linus' software management process? I'm most hardcore of you all, so I'm the overlord ;-). In that light Linus' diatribes are only politics: he's defending the status quo.

There's also a diffrent response to the "substandard programmers" reproach I must mention here. Steven Dewhurst broght in the point, that for a C programmer C++ is so complex because there are alien idioms, methodologies, tricks, and so on*****. You would be tempted to say it's to difficult for the average C coder, but there's something else! C++ isn't just C, it only happens to be backwards compatible! When you switch from C to Common Lisp you won't be an expert instantly, but the C folks assume they can just come and start programming C++. And then they cry that it's too difficult and complex.

Summary

This time there's no summary. I was just gossiping...

---
* Linus original post (admittedly taken out of context!): http://thread.gmane.org/gmane.comp.version-control.git/57643/focus=57918, but I must admit, when he's speaking, he does make a much better impression!
** Digg gossiping: http://digg.com/linux_unix/Linus_Torvalds_hates_C
*** Hacker News discussion: http://news.ycombinator.com/item?id=51451
**** A typical reply: http://warp.povusers.org/OpenLetters/ResponseToTorvalds.html
***** Steven Dewhurst's reply: http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=411