Bug hunting or how dumb can you get, really?

As a programmer you sometimes can't help but feel really stupid! Really stupid-stupid! Look at this code snippet for example of a foolish bug that kept me busy for some time. The story behind this is that I added a feature to the product I'm writing for a customer, and was totally happy with it. There was a real customer benefit and it worked like a charm through my tests. However, when deployed to the internal test system, it crashed unexpectedly now and than.

That was a kind of shock, because the product was running stable for more than a year in this environment, and I was totally unsettled as I didn't want to believe that the new feature could have something to do with it. It was tested rock-solid. There must be a mysterious bug somewhere else in the program. And it hid for one year before it disclosed its identity! Bad!

However, after I ran the collected test data on my development machine, it turned out to be a pretty uncomplicated bug:

    for(size_t i = 0; i < a_prInput->frameCount(); i++)
    {
        // header
        size_t start = a_prInput->NettoDataParts[i].uOffset;
        size_t pos = start;

        DnsHeaderData header = parseHeader(a_prInput, pos);

        // interesting data there?
        if(!header.isResponse)
        {
            TRACE("dns: packet isn't a response, ignoring...");
            continue;
        }

        // skip the questions
        for(int i = 0; i < header.questions; i++)
        {
            if(!parseDnsName(a_prInput, start, pos))
            {
                TRACE_ERR("dns: cannot parse a question record");
                m_uGoodByteCnt = pos - a_prInput->NettoDataParts[i].uOffset;

                continue;
            }

You see? The code was referencing the data form the outer loop with the index of the inner one! And it did it only in error case, and crashed only if it was out of luck. Arrgh... There was no problem with fixing this, but it started me thinking: I'm programming for some 20 years, I know C++ in and out, why did I make this bug in the first place? Did I violate some programming rule or best practice? Is it possible to avoid such bugs by some sound practices? Some answers come to mind:

1. There should be a warning from compiler
   That would be the ideal solution. But what warning should that be? And what if I really wanted this that way? I'd need to use an ugly #pragma then? No, better let it be.


2. I should have used at()What should I say, er, well, at() is for wimps! And if that wouldn't be the case, we have a choice between at() and the [] operator. So we want to excercise our rights!


3. Use telling names
   Well, of course, I could name each loop variable accordingly, but come on, can you imagine the hassle? Because if this has to be of any good you have to be consequent. So maybe only do this for nested loops? Maybe, but if I start to do this as to avoid this error, and something else to avoid some other error, you guess it, it becomes a pain! We need something more general,
4. Use for_each(), as we don't do any addressing here
   That's more interesting, as logically speaking, the inner loop isn't indexing single elements, but rather wants only to skip over the uninteresting headers. But, on the other side, for_each() is annoying, as it requires start and end of the sequence, which isn't a problem if I'd use the iterator, but gets pointless if I never want to use the iterator directly!


5. use iterators to go high level
   Isn't that the same as the above? No, because it's not an application of a specific technique in a specific situation, but rather a conscious decition for rising of the abstraction level. And quite serendipitously we can avoid the above mentioned error! Or are we? What if both outer and inner loop are using the same name for the iterators? Here we are agan back on the square one and have to use for_each() for the inner loop as to be safe.

Alas, none of them is satisfactory. Well, I know, there's one more solution: code reviews (or pair programming). But first, my customer isn't doing them that often, and second, I doubt this bug would be found in a review/pair. But even assuming it would be found: the price is much to hight! It took me a couple of minutes to fix it, once I had the right data, and my customer expects "results just now".
What is the key point? What does this all mean? Are we doomed? I think that there are two possiblities here: either use iterators or some other high level constructs (which admittedly is somehow of an overkill), or be prepared for occasionally fits of stupidity. In the end we are only humans, and errare humanum est... And who wants to be safeguarded against everything?

--
PS: Sorry, I started this entry long ago, but in the meantime I was so distracted that I couldn't come round to write anything for the blog. Besides, microblogging isn't doing it any easier ;-).
PS2: Other possibility: write shorter functions, use a skipNameRecords() method and you are saved! But really, that function is under 1 page lenght, How long should functions be? Five lines or so? Or seven as adjust to humain brain's cognitive limitations?

Fooling the ODR (and the compiler)

I just discovered a technique that is prety convenient but rather not self-explaining, nonportable (?), and possibly a maintenance nightmare. So you should never use it, should you? Except, as it's sort of cool, if you'd like to go a little on the geeky side...

1. The need for speed

Imagine you are writing some quick classes in the Java style (i.e. no header files), locate them in some C++ source file, and forget about them. Then you discover that you need to refactor your program (in my case it was a custom test framework) and to move some classes out to a separate file (a.k.a. compilation unit). Now you shun to do the right thing (i.e. to refactor) because you'd have to cut up your classes in two: the .h file and the .cc file, and to duplicate some (non always trivial) amount of code. So yo aren't doing that...

But stop, following trick did it for me: I defined the classes in a separate .h file and kept the .cc file with the extracted classes unchanged. I included the new .h file in the rest of the old .cc file where the classes were removed from. Then I just linked the the whole thing, et voilá, everything worked like a breeze - a refactoring as we like it!

2. Why does it work?

But wait, we spared some editing work, but didn't we violate the One Definition Rule of C++? I'd say we did. So why didn't the compiler complain? Well, as to answer it, we've got to look at the example a little closer.

The extracted classes looked like that:

TestThreads.cpp:

  #include "WorkerThread.hpp"

  // test thread A
  class ThreadA : public WorkerThread
  {
  public:
    ...
  private:
    ...
  } g_workerA;

  // test thread B
  class ThreadB : public WorkerThread
  {
  public:
    ...
  private:
    ...
  } g_workerB;

Note that there's no TestThreads.hpp definition file here, thus we don't have a direct ODR violation for the compiler to complain! The newly created definitions file TestThreads.hpp looked like this:

  #include "WorkerThread.hpp"

  class ThreadA : public WorkerThread
  {
  public:
    virtual void addRequest(Request& r);
    ...
  };

  class ThreadB : public WorkerThread
  {
  public:
    virtual void addRequest(Request& r);
    ...
  };

The WorkerThread.hpp definition file looks like this:

  class WorkerThread
  {
  public:
    virtual void addRequest(Request& r);
    ...
  };

And the rest of the TestMain.cpp file where the TestThread classes were previously placed:

  #include "TestThreads.hpp"

  extern TestThreadA g_workerA;

  extern TestThreadB g_workerB;

  g_workerA.addRequest(req);

Can you see why we could fool the ODR? Well, in this special case we didn't used the constructors directly in the TestMain.cpp file, only the methods of the WorkerThread superclass which then call the appropriate virtuall methods defined in the ThreadA class.
The linker then generates a reference to the virtual ThreadA::addRequest() method. Well, guess what, a method with the very name can be found in TestThreads.cpp, and the linker can resolve it no probs!

Although we clearly have two definitions of the ThreadA class, we get away with it! The ultimate reason for that is that the C++ compiler doesn't have the global knowledge of the whole program - it works only on the compilation unit level, and the relinquishes the responsibility for the complete programm to the linker! I.e. there's no whole programm optimization possible (at least not in the standard model, although Visual C++ compiler can do it).

3. How to extend the "hidden" classes?

Suppose we'd like to call a new method in the TestThread.cpp file from the outside. You cannot call this method directly, as we do not include the real definitions of the TestThread classes, but only the "fake" ones. Alas, there is a price to pay for fooling the compiler and for beeing lazy! The price here is, that we have to modify the base class (TestWorker.hpp) by adding a new public virtual method, if we need a new TestThreadA method to be called in the TestMain.cpp file!

Not pretty, uh? You know, sometimes you can go away with breaking the rules, but in the end you'll have to face the consequences. In the art of software design, your skills are measured at your ability to estimate for how long you will go away with laziness and tricks, and when you have to resort to the known, sound and boring "best practices".

But nonetheless, the price I had to pay was OK for the deeper knowledge of the compiler-linker cooperation and the increased knowlede of my own code.

Do you GoF - the belated anniversary and the ladder

Last year we had the 15th (or so?) anniversary of the seminal "Gang of Four" book. Is it that old already? So maybe discussing the design patterns is all an old and boring stuff now? Well, shame on me, I wanted to write up my ideas on that matter for such a long time that I run the risk of them beeing irrelevant. But on the other side it's a kind of unfinished business for me, and I have a to kind of close it up.

Some time ago I wondered - why didn't I use the design patterns that much in my programming and design work? I came to the conclusion then, that the design patterns are useful as a common vocabulary as to speak about designs, but then some new ideas surfaced in my mind. By the way, why should I use a pattern vocabulary if I don't use any patterns in my designs?

1. Ideas from chess and math

If you're playing chess (need a Wikipedia link here? ;-) you are accustomed to seeing the board in patterns: check-mate patterns it is! You try to imagine what kind of a check-mate setup could be possible in a given situation, and then work towards that by moving your pieces and trying to chase away your opponent's pieces as to complete the pattern you have in your mind. I must admit that I never had this feeling when writing software. Some pattern in my mind which is the solution for some design problem? Nope.

Read the "Refactoring to Patterns" book by Joshua Kerievsky? When I first heard of the book I though he struck the right idea: only use patterns to refactor code which is a mess. But then when actually reading the book, it was rather a feeling (in most cases) of doing something wrong to the code, well, like overengineering it! So the book didn't reconcile me with patterns too, as it didn't give me the chess-like feeling of a pattern just matching a given situation.

The other problem with patterns is their presentation. This overly formal template with "Forces" (why not just problems?), "Intent", "Consequences", etc, etc... This makes me think about "The mathematicians lament"* - where a matematician reproaches his fellow scientists of over-formalization of mathematics with the effect of its incomprehensibility. What he is saying is basically the following: yes we need the formalism, but only in some very difficult cases where our intuitions don't work. But not in every and one case: "nobody's dying here!"*, you can relax a bit!

It's the same with patterns: the stern formality of presentation is both somehow comical and making the contents rather indigestible, and an dry, uninspiring read. I can only say, relax, we are not deliberating about the meaninig of infinity like Cantor did! We are just describing some not so complicated techniques of object composition! Could you stop to be so pompous?

2. Dreyfus et al.

Do you know the Dreyfus model of knowledge acquisition**? It defines several phases in learning a specific skill: beginner, advanced beginner, competent, proficient and expert. Beginners need rules and recipies, competents can continue learning on their own, proficients and experts see the big picture and fly by the seat of his pants.


Where do the design patterns fit in this scheme? Well, at the beginner level they aren't useful, as they are not "step by step" recipies. On the other side, for the proficients and experts, they are some kind of rules, and experts do not like rules and recipies by definition!

I personally think that patterns appeal rather to the lower ranges of the skill spectrum. Did you see the hopelessly overengineered software full of design patterns written by the beginners? I've seen it only too often!

So maybe they are for the competent? And the usage by competent and proficient users? Let me tell you a little story here. It's not entirely true that I don't use patterns! Lately, I thought that I could warrant the use of the visitor pattern in my current product, as to extract the reporting aspect out of several content managing classes. Well, it has a kind of worked at first, but now, as this whole aspect has to be redesigned, I noticed that this design is rather unintuitive and I cannot simplify the reporting without dumping visitor first. Ok, experiment failed, but my initial problem was that I didn't really search for a solution of my problem, but rather stopped at the point where I found the (as I thought then) matching design pattern! But I didn't think it trough as thoroughly as I should!

3. Summing up

So maybe the thing with patterns is that they are like the "Wittgenstein's Ladder"? You know, what he said in his "Tractatus Logico-Philosophicus" (I cite from memory):

...this book is like a ladder, use it, but then throw it away.

Isn't it the same with patterns? Use them to expand your experience, but don't use them in your software! And as to make the discussion complete, let us state a completely different point of view, a Lisper's critique***:

When I see patterns in my programs, I consider it a sign of trouble. The shape of a program should reflect only the problem it needs to solve. Any other regularity in the code is a sign, to me at least, that I'm using abstractions that aren't powerful enough-- often that I'm generating by hand the expansions of some macro that I need to write.

This is the typical smug Lisp programmer assertion, as they mean that all other languages will in limes converge to Lisp, but newly I tend to think along the lines of the Lispers: the design patterns are a fix for lacking feature in language A, which you know from language B. But as we cannot change the language in most cases, and we are doing OO in the mainstream, just mind the ladder parable!

Update (June 2010): There's another explanation which I didn't think of before. I came across that when reading some programming book (which I cannot come up with now, sorry for the author!). Recall the mid 90-ties: it was the time of the "object craze", everybody wanted to be part of OO. But nobody really understood OO, and as the book went "Stroustrup gave us the C++ but he didn't tell us how to use it"! Her come the patterns, a kind of missing OO manual! This would explain my feelin that half of them was trivial (or not so difficult to figure them up by yourself) and the other half was implemeting language features missing from C++. Very sober diagnose, but at the moment it's my favourite.

---
* Paul Lockhart, "A Mathematician’s Lament", www.maa.org/devlin/LockhartsLament.pdf

** I found it in the book of Andy Hunt, "Pragmatic Thinking and Learning", Pragmatic Bookshelf, Oct. 2008, but here's a blog post link, and here another one

*** Paul Graham, "Revenge of the Nerds", http://paulgraham.com/icad.html

Local language trends 2009

I just had a look on the statistics pages on our local IT freelancer website (http://www.freelancermap.de/) and found the results pretty interesting. First let's have a look at the languages which are most popular in the freelance projects. The picture below shows the languages, the supply of programmers (the blue bar) and the demand in projects (the red bar).

As it seems, here in Germany there are only two serious languages: C/C++ and Java. Well, ok, Visual Basic and C# are sough after rather strongly (red bars), but there seems to be too little supply. Is that the fear of a vendor lock-in or is it just because Microsoft is considered evil that the programmers don't like .NET? The same case of too little supply for PHP, but here I must mention that the hourly rates for PHP tend to be rather low - PHP still considered a hacker language?


And now look at the last but one language with a longwinded German name I won't retype here (too lazy). In fact it's not a language of its own, but rather denotes "all the other ones in there", which includes everything fancy (or just plain old). You see there is too much supply, the IT managers seem to be more conservative than programmers, no surprise here.

Which leaves us with C/C++ and Java. Java is unmistakeably the 500 pound gorilla here, but it's hopelessly over-supplied (blue bar)! For C/C++ the situation isn't so dramatic, and I'm glad I specialize in C++!

The proposed course of action here: go away from Java (or well, from everything else for that matter) and start on .NET!

Now the overview of technologies in the freelancer projects:

Software development comes first (still a little place for more work - red bar!), then system management (plus "service and support"), however the suply here is much too big (blue bar)! It's just as bad as for SAP programming - in the past the eternal cash cow with ridiculously high hourly wages, now it's a real "problem child" - the market seems to be breaking down. Banking crisis anyone? For Web development and "graphics, content and media" there's much too much demand, just like in the PHP case above. Then the ominous field of "IT consulting", where I don't have any idea what it is, sorry, no comments there :-/. One interesting field is "Executive consulting" with so much demand and so little supply, maybe everyone is just too shy to position theirself in that niche?

The proposed course of action here: either stay with software development or move into "Executive consulting"!

Having said that: Merry Christmas you all!

Even more new C++ features!

Hi everybody! As I had a look on the new C++ proposal the last time , I found many small interesting things apart form the big, important features. But this was only a first look. Recently I had another look there (and at the C++0x FAQ* by Bjarne Stroustrup as well) and discovered even more interesting crittures. So what did I miss?

1. Some Syntax

First of all the new function definition syntax! Now I can write something like*:

  using PF [](double) -> void;

along the lines of lambda function definition! Well, that's definitely a treat, what a relief from

  typedef void (*PF) (double);

To be honest, I don't know if I've got the last one right, I have always to look thes syntax up or copy and paste from my other code! The [] notation can be generally used to say to the compiler that "the function type will be specified later"! Thusly we can write the following when usign embedded type like List::Link

  [] List::erase(Link* l) -> Link* { ... } ;

We postpone the return type definition utill we have entered the scope of List, and thus no scope specifier is needed!

I liked this notation (it makes the code look a little Groovy-like), but alas, it seems that not everyone does so, as the related "auto arrow" notation proposal was rejected recently**! What is auto-arrow? Look at that***:

  auto f() -> int;
  typedef auto PF() -> int;

the first one is a function from void to int, and the second one is a function type definition for the same signature. Looks good enough, but consider this one:

  auto *pf(auto *p() -> int) -> auto *()->int;

I don't even want to discuss what it means, if you are curious look it up in ***! So after that proposal was justly rejected, I'm a little afraid of the fate of the [] notation - is it the next one to be scrutinized? I hope not!

2. Initalization and initializers

I though I've understood the new initalizer notation, but I was surprized how ubiquitous it became! At first I though it would be only used in assignment like

  vector<int> xs = {1, 2, 3 };

Not so, nowadays you can use it (i.e. the new {} notation) everywhere, even instead of classic parentheses!

  X x{a};
  f({a}); // initializes a temporary
  X* p = new X{a};

This all does look pretty strange to me, and to be honest I don't like it that much, but on the other side you can use it to avoid verbosity with pretty good results:

  map1.insert({a, b}); // sic!

The other improvement with initialization is that we can initialize the string and integer class members directly in the definition of the class!

  class X
  {
    int a = 7;
    string s{"abcd"}; // new syntax!
  };

I can only say AT LAST and amen!!!

Belonging to the "initialization" block of features, we must mention the access to the inherited constructors (in addition to the also new feature of delegating constructors, i.e. constructors which can be invoked of another constructors of the same class):

  class A { public: A(){} };
  class B : public A
  {
    using A::A; // imports all the A's constructors into current scope!
  };

  // now this is possible:
  B b;

At this place I'd like to mention another related feature: disallowing copy and assignment operators. You can write for example:

  X& operator= (const& X) = delete;

to disallow it! Well, if I consider that Google even has a macro exaclty for that reason (DISALLOW_COPY_AND_ASSIGN(), see here) and that I used an UML tool to generate the private version of it for my every class I think that the default-generating them was a bad language design idea! If we want a copy semantics we'd better tell the compiler so! Hindsight is 20/20.

3. Threading etc.

Of course thread, mutex, condition variables and atomics are there as basic support (as expected). On the higher level we have futures and a packaged_task class, which wraps features for simpler usage. Moreover, Bjarne does promise that an async task launcher (logically named async) will be included in C++0x as well. Then we could launch tasks simply like that:

  auto f1{async(funcX(100))};
  auto f2{async(funcX(200))};

  return f1.get() + f2.get(); // waits on both futures

If he's right it would be very nice!

As GC is connected to multithreading in an intimate way (at least in my eyes, sholud blog about it some time soon!), I must state here that the GC is still missing!

But not entirely, we've got some minimal support for third party garbage collectors in there (20.8.13.7 Pointer safety). It defines what a "safely drived" (i.e reacheable) or an "disgused" pointer is (3.7.4.3 Safely-derived pointers) and lets the programmer specify where such pointers can/cannot be found with declare_reachable()/declare_no_pointers() functions.

One thing that struck me however in the threading library classes is the flexibility of interfaces. Take the std::thread's constructor for example. You can use it like that ****:

  void hello_func() { ... }
  // say hello
  std::thread t{hello_func};

or like that, using a functor:

  class Greeting
  {
  public:
    explicit Greeting(std::string const& message) { ... }
    void operator()() const  { ... }
    void greet(std::string const& message) const  { ... }
  };
  // say goodbye
  std::thread t(Greeting("goodbye"));

or currying the thread function with std::bind:

  void greeting_func(std::string const& message) { ... }
  std::thread t(std::bind(greeting_func, "hi!"));

or without std::bind:

  std::thread t(greeting_func, "hi!");

or without std::bind and with more arguments:

  void write_sum(int x, int y){ ... }
  std::thread t(write_sum, 123, 456);

or using the member function pointer

  Greeting x;
  std::thread t(&Greeting::greet, &x, "greet");

  // or by a smart pointer
  std::shared_ptr<Greeting> ptr(new Greeting);
  std::thread t1(&Greeting::greet, ptr, "hello");

  // or by reference
  std::thread t2(std::ref(x));

  // or directly!
  std::thread t2(x);

See, you don't need to worry about the initalization, the new constructs will just take anything you throw at it!

4. Miscellanea

First the variadic templates. What a name for templates with viariable number of arguments! I guess it's an interpolation of the "dyadic" concept which just means "of two arguments" :-). But seriously, we can writhe things like

  template<typename Head, typename... Tail>

doing recursion and typematching like in Clojure or Haskell ;-) Well a kind of, but it looks like that at least. Look at the n-tuple definition using variadic templates in Bjarne's FAQ*.

Further, the new and improved enums (which are class based now) can be now forward defined ! But I'm doing this all the time with C++ Visual Studio 2005! Ouch...

No for the geeky ones: we have user defined literals! You can define new literal types using the following new operator type: operator""() (plus constexpr!), for example

  "abc"s; // s: string literal
  2i;     // i: imaginary literal

For details have a look at Bjarne's FAQ*. NIce feature, but will I use it? It is readable for the maintainer? We'll see.

The next 2 features look to me as if they came straight from Python. First the raw strings:

  string s = R"[ahjh/?=(%/$§%§%\w\\w///]";

Why not just R"..."? Dunno. Next the foreach variant of the old for() loop.Together with initilizers we can now have some Python feeling in C++:

  vector<string> xs = {"aba", "bab", "aab" };
  for(auto x:xs) cout << x; // x:xs looks even a little Haskell-like ;-))

So there are many new things, some of them making C++ look like entirely another language. No I wonder - will the C++ programmers use it, or will they stuck to the old, but known evil (like the old function definition syntax)?

---

*Bjarne's "C++0x - the next ISO C++ standard", http://www2.research.att.com/~bs/C++0xFAQ.html

** Danny Kalev: "The Rejection of the Unified Function Syntax Proposal", http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=460

*** Daveed Vandevoorde: "The Syntax of auto Declarations", http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2007/n2337.pdf

**** for completness' sake, the examples are partially taken from: http://www.justsoftwaresolutions.co.uk/threading/multithreading-in-c++0x-part-1-starting-threads.html

More about Java Closures

Surprize, surprize! Hear, hear!

One year ago, Mark Reinhold, Principal Engineer at Sun Microsystems, announced At the Devoxx conference in Antwerp, Belgium that the next major release of Java, JDK 7, would not include closures. At the same conference this year, however, Reinhold announced in a surprise turn around the Java would be getting closures after all in JDK 7.*

More specifically, they adopt the BGGA proposal (look here) but without the non-local return (look here) and control invocation statements. This leaves Java without any destructor or C# style using statements support!

What can I say? I can only reinstate what I said in my previous post: comparing to the development of the C# language (look here ** for an interesting inteview about LINQ and its supporting language features) the Java language looks increasingly mouldy (ouch!!!).

But maybe I'm mistaken? maybe there will be some support for modern ressource management in Java but it's decoupled from the closure proposal? Let's hope so.

Update: well, I didn't hope in vain, there will be a feature for this: Automatic Ressource Management (see here)!!! Moreover, we'll got a little bit of type inference for the right hand generic expressions with new (as Google Collections provides it) with a "diamond" operator (?) <>, and strings in the switch statement (as in Groovy). All in all you cannot complain, I'd say, and some bloggers (ehm...) shouldn't be too quick to bash Java 7!

--
* I found the news here: http://www.artima.com/forums/flat.jsp?forum=276&thread=274496
** http://www.infoq.com/interviews/LINQ-Erik-Meijer, BTW I found an interesting quote of general interest there:

Actually, if you look at where I'm going, I'm getting more and more into old school object orientation with interfaces and so on, going away from functional programming. If you are passing a high order function you are passing one thing, one closure. If you're passing an interface or a class, you are passing many because you are passing a whole V table. These can all be recursive and so on. I'm rediscovering object oriented programming and I feel that maybe I wasted some of my time doing functional programming and missed out on objects.

You see, it's not only the functional programminch which is gonna be the Big Next Thing! I think that we cannot just condemn OO and go all for the functional. OO has its merits when structuring code, and a hybrid OO/functional approach seems to be a good thing to me!

Songs in code

Won't explain what they are, because it's obvious. These two made me laugh:

Bob Marley's (and Groovy's):

  4.times{
  !woman == !cry
  }

and REM's (and SQL's):

  DROP TABLE MyReligion

but normally they are boring, uninspired or plainly wrong, like this one:

  while(young)
  {} // loops forever

Come on, the whole thing is that young is a volatile variable, which will somehow, sometimes be set to false! ;-)