On Software and Languages

Static vs Dynamic Debate Decided

2012-01-22T11:25:00.000-08:00

As I said, it's decided (at least for me) now!

A couple of years ago it became fashionable to be ecstatic about dynamic (i.e. dynamically typed*, aka scripting) languages, this sentiment is maybe best exposed by Steve Yegge here. People started to talk about "freedom languages", and it all started getting rather emotional. Like in this tweet:

"Strong types are the hobnailed boot of the Enterprise Man on the neck of the Agile Code Poet" http://goo.gl/aeEcn

It is understandable as an aftershock of the J2EE meltdown by Ruby and Rails, but I was always rather suspicious with that. I was always rather suspicious because of the following gut feeling - why should I throw away things which can possibly help me? If you are skiing, you know that: you were falling and you'll fall, the question is only when (hope not in an icy couloire or over a cliff...). It's the same in programming; you will fall.

1. Theory

A proof written as a functional program (Wikimedia)

As at that time I was interested in mathematical logic and automatic theorem proving, I happened to know about the Curry-Howard isomorphism. It roughly states that a proof in intuitionistic logic has its equivalent type expression and vice versa. So if type system is equivalent to a (constructive) proofs of correctness, that's cannot be bad. We are getting some limited guarantees from the compiler (because the type system doesn't describe all of the program's behaviour) and that's one thing less we have to worry about!

Well, OK, maybe that's all theoretical babbling - who knows if all that logic/foundational thing in mathematic isn't contradiction-free? And even if it's not, it's maybe not that relevant in practice...

Because when confronted with such a question (i.e. what about the correctness guarantees), the standard answer was: "You gotta write more tests!" or "Strong Testing not Strong Typing"**. And because tests are a good thing, you cannot say if that maybe could be true...

2. Own Experiences

So let me share some of my own experiences with dynamic typing.

A couple of years ago, as I read the excellent "Higher Order Perl" book and I got through the first and middle chapters and the code was understandable and clear.

At this point it's maybe appropriate to give the book some praise, as it's well written, and shows an uknown side of Perl - it's functional self. As the title reveals, the book is about functional programming in... Perl! Yes, that's possible, it even seems to match the language rather well. Of course the language wasn't designed for that, and you will miss all the compiler support and type inference of OCAML but then, Perl isn't statically typed, OK? The only fly in the oinment is that Perl isn't fashionable for quite a couple of years yet...

But I'm digressing here, so back to the main theme: at first it wasn't aproblem that you don't know the types of the parameters in Perl. It's all convention, you have to retrieve the parameters from the argv and use them according to their type like that:

   my($top, $code) = @_;  # @_ is Perl's argv!

So all the type information is so to say in the code that uses it later on. But in the last chapter, when the author is developing a linear equation DSL-based drawing system I sometimes got lost in code. I simply didn't know what the input is, and what the function will be doing with it. How I wished I could see the parameter types in the function declarations!

Another example: if you are writing Python code using a library you aren't that accustomed to, and you don't want first to read the domcumentation for an hour or so, but just quickly code something small you need just now, you'll very likely just to get a bunch of runtime exceptions and have to guess what is exactly going on now. What is the exact type the library is expecting here? I did everything right, didn't I? Nope, you didn't. Please look up the docs.

So AFAIC, the types are for human understanding, even if you don't need the type checking guarantees of the compiler. So common sense dictataes using it - you know 80% (my wild guess here) of programming is maintenance, the code is written once and read 1000 times, and so on. We all know these old programming adages, and they don't just apply to telling variable names and insightful commens. Type information is in my eyes another part of writing understandable (as opposed to "genius"-) code.

3. Other people's experiences

OK, I'm not that much experienced in dynamic languages, so maybe I've got the wrong impression, or didn't understand something correctly, or maybe I'm just not clever enough?

But then, when I started reading the "Beginning Scala" book (well, I didnt't finish reading it in the end because I got bored, but that's another story...) the following lines struck me:

As my Ruby and Rails projects grew beyond a few thousand lines of code and as I added team members to my projects, the challenges of dynamic languages became apparent. We were spending more than half our coding time writing tests, and much of the productivity gains we saw were lost in test writing. Most of the tests would have been unnecessary in Java because most of them were geared toward making sure that we’d updated the callers when we refactored code by changing method names or parameter counts.

As you see, that practical experiences are rather confirming my assertions about static typing: a) why write test which can be (kind of) automatically generated by compiler, and: b) why not let automatically document and check the interfaces?

But there's another angle on that problem which I didn't think of:

Also, I found that working on teams where there were mind melds between two to four team members, things went well in Ruby, but as we tried to bring new members onto the team, the mental connections were hard to transmit to new team members.

This goes more into the psychology of programming on the first sight, and maybe it's a cultural thing altogether, but it relates a little with: c) why not document the interfaces as you are writting code to make obvious things obvious? I mean, if the interfaces are better described then maybe the new team memebers will absorb the knowledege better?

Another Ruby guy tried the Go language and wrote an interesting post about his experiences with static (although implicit) typing he encountered in Golang:

I guarantee that no matter how much I look at the Ruby code, if it runs for any length of time I'll eventually encounter a typing error because of a bug in the code…

Ok, the same old story about interface documentation and checking as above. But read on, ther's more to come:

I've learned that I vastly prefer the cost of slower development time to gain reliability and safety, as long as the cost isn't too high. Haskell is even safer, but it pushes the boundaries of how much pain I'm willing to endure trying to get code to pass through the type checker

Here he's making a good point, kind of reiterating my main argument: there's a trade-off between the developer time neede to type a piece of code and it's reliability. In the past we opted for ease of construction, but if we can have the realiability for very small additional cost? With type inference we hit a sweet spot here!****

So it's decided, there are real problems with dynamic typing in real-world projects, and static typing can help out here. Why would anyone want to use a dynamic language (beside for scripting some small pieces of code) then?

But wait, what about Clojure? I like this language... and I'm somehow in 2 minds here - the reason says ML but the heart says Lisp. So is this an emotional thing in the end? But in business you have to follow the path of reason, not coolness, to deliver results to your clients. And there aren't that much Lisp projects out there... So the decision isn't difficult: I'll opt for statically typed languages in general, but when offered a Clojure project (which will happen very seldom I guess) I'll jump to it! Don't you think it's a good tradeoff?

Goodie:

PS: do you remember pre-ANSI C? Probably not, but it wasn't type safe:

Within the function, the declaration of a is visible, so the compiler knows it's a char*. For a call, the compiler doesn't know what type of argument the function expects, so it's up to the caller to pass the right type. It's similar to what happens with printf-like functions.***

Here an example for those not remembering the "stone-age":

int
ParseGLFunc (interp, argc, argv, nArg)
    Tcl_Interp *interp;
    int argc;
    char *argv [];
    int *nArg;
{
    ....
}

* I'm sure you know it aleady, but statically typed means for a programming language to check the types at compile time whereas dynamically typed means checking of type correctness at runtime.
** An article by Bruce Eckel: "Strong Typing vs. Strong Testing"

*** found it here: "pre-ansi declarations"
**** here I must admit that I'm just impressed with the ML family of languages (i.e. OCAML, F# and well, Haskell too) and the automatic type deduction the compiler is doing! The code is just flowing from your fingers as in Python or Ruby, but it's type checked!

ASSERT(AfxGetThread() == NULL) with VisualStudio 2010, or the DLL-hell, revamped

2012-01-11T12:50:00.000-08:00

Everybody thinks DLL-hell is a thing of the past. Me included: #WTF, just start the Dependency Walker, look what libraries got loaded, correct it and be happy*. Not so, with the advent of Visual Studio 2010 everything got a bit more complicated!

Let me describe the problem which I encountered working in a project for one of my customers. Let's start with a...

Disclaimer:

This is only a short technical note - for those who (like me) first check the Internet for solution to weird programming questions!

The Problem:

is maybe best described by this desperate post:

Hi all,
I have a question to solve, the situation is I use a exe to load a dll(MFC regular dll),but when I use LoadLibrary() to load the dll and in the constructor of the object whose base class is CWinApp in the dll, It have a assert error, and the code line is ASSERT (AfxGetThread() == NULL).Is there anyone have the similar situation with me and have thoughts about it can tell me ,we can talk about it.
thanks in advance.

Well, the same happened to me when I was porting a C++ application from Visual Studio 2008 to VS 2010. To be honest, the situation was a little more complicated: a mixed mode C++/C# program using C++/CLI as a glue. But back to the problem: the standard explanation for that ASSERT is that you have forgotten to include the AFX_MANAGE_STATE macro (Reason 1) in exported DLL functions as described here.

Sadly, this wasn't my problem. It applies to non-MFC DLLs only**, which mine wasn't, so the whole module state management business should be cared automatically in DLL initalization code the linker is generating. This again is controlled by #define macros used whith the compiler (like _USRDLL, _AFXDLL and their hellish likes). So maybe this could be the reason? Nope, everything was correct :-(.

After more web-searching and finding only tons of happless cries for help I stumbled upon the 2nd useful link

Sounds like you have two or more CWinApp objects, and they're using the same AfxGetThread data. I suppose this could happen if your libraries are linking to different versions of MFC, including mixing of debug and release versions. You might use Dependency Walker (depends.exe) to determine the DLLs you link to implicitly, or the Sysinternals Process Explorer to determine the DLLs you're using at runtime

Well, that sounds interesting, mybe I've mixed debug and release DLLs? Nope (although I made that error later on, so it's defintely a good point to keep in mind (Reason 2)). What could it possibly be then???

The Solution:

The solution is more complicated, but somehow both typical and symptomatic for real-life nontrivial system using Microsoft technology... As my client had a really old codebase growing steadily with the years, all possible Microsoft technologies could be found there; beginning with plain old DLL, through MFC-based ones, further using C++/CLI and at last arriving at C# and .NET. This last one was burdened with a constraint: we had to use .NET v.2.0. The problem is, that Visual Studio 2010 doesn't suppport .NET 2.0! Because of time shortage they decided that only .NET 4.0 will be supported out of the box, and that the 2008 VS-compiler will be inegrated in Visual Studio 2010 to be used for older .NET versions.

Well, in itself it wouldn't pose any problems, if there wouldn't be the C++/CLI code which we used as glue between pure C++ MFC and the newer C# parts. It has to be compiled for the .NET 2.0 as well, thus should use the 2008 VS-compiler. But it is put in a DLL which int turn is loaded by a traditional C++/MFC one (created with 2010 VS-compiler!). See it? Two different compilers, 2 diffrent runtimes, 2 CWinApp objects as in (Reason 2) (or maybe the loader gets misleaded, I'm not sure about it). In any case, Visual Studio 2010 is forcing us to use 2 compilers in one application here (Reason 3).

The Mores:

I'm not sure about that one. How about this: stay away from MFC and use Qt? Or: don't migrate your code to VisualStudio 2010 (as it's rather slow to boot)? You're damned if you have to be backwards compatible?

* see here for author's previous letter from DLL-hell

** more hellish details: 3 kinds of DLLs you can use on Windows: ... TODO: link (but that's just plain boring!)

DSLs not so pointless?

2011-12-15T14:19:00.000-08:00

... or: busting myths and fallacies, this time - the DSL-s!

Until recently I'd only sneer at the idea of DSL's: what the @!%*, either we have a soild API for developers, or a specialized higher level language for the specific task. And the higher level language best shouldn't be done at all, because the non-technical user just wants a smooth GUI to click his requests together! So spare your DSL hype on me, go and find another gullible middle management person.

But then I read the following passage and the fallacy behind this reasoning became apparent to me:

In other environments I could write tests together with the product owner. Together, we could create and discuss tests that expressed what we wanted our code to do without getting buried in the details. The tests I wrote in C++ were quite unreadable for a non-C++ developer.

What was my error? DSL isn't meant as a programming language, neither for the developer, nor for the notechnical user - it is meant to document things!

Look at the following example of a testing DSL in C++:

  #include <igloo.h>  // testing framework*
  #include <cell.h>   // tested class
 
  using namespace igloo;

  Describe(A_Cell)
  {
      It(should_have_coordinates)
      {
          Cell cell(3,4);
          Assert::That(cell.getX(), Equals(3));
          Assert::That(cell.getY(), Equals(4));
      }

      It(can_detect_if_its_a_neighbour)
      {
          Cell cell(1,5);
          Cell neighb(2,4);

          Assert::That(cell.isNeighbourTo(neighb), IsTrue());
      }
  };

You must say it's a piece of C++ code** you could show to anyone and discuss it as a kind of formal spec. You and your partner don't have to learn Z or some other specialized notation. What both of you know already is just enough to bridge the gap. And it's cool code as well!

--
* see: http://igloo-testing.org/
** it's not a joke, all this is possible in C++

Geeky Halloween costumes...

2011-11-11T02:15:00.000-08:00

Matching nicely my last post abot the Go language: dress up as a programming language mascot:

(as appeared in Golang blog).

Any other languages to follow? For Perl it's simple (a camel), for OCAML too (a supercharged camel), but what about C++? Or Assembler?

PS: My favourite C++ pic is this:

. But you really can't dress up like that, can you?

What I did like about Go.

2011-10-25T05:07:00.000-07:00

Last week I've read the "Learning Go" book. I did that by sheer coincidence: it happened that I had the (free) book loaded on my eBook reader and just looked for light but interesting read. Why not an intro to a new programming language? So I started to with it and I didn't regret my decision. It was an entertaining and interesting lecture. I'd like to describe here what I liked about the language.

See, normally reviewers concetrate on what they didn't like in a language, but I think that's way too easy! The things we don't like are most probably the things we aren't accustomed with, and with some experience they wouldn't annoy us any more. So don't gripe, just be easy on Go!

As I learned from the book, Plan 9 project*, the (supposed to be) successor of Unix, included a language called Limbo which was a lot like Go already, which built on an earlier langauge called Newsqueak. So surprisingly, there's a lot of history showing up here!

Go itself is like it's mascot (look to the right!): small and quick. It's compiled and statically typed but does type deduction (like OCAML or Haskell). It main claim to fame are (beside garbage collection) goroutines and channels, but I have too little experience with them, so I'd rather concentrate on basic day to day traits of the language. I mean, the small things are the things that annoy us the most, and should a language will be anoying, it will screw them. Additionally, this way we can somehow transmit the "feel" of the language.

So let's start with:

1. multiple assignement

...a la Python:

  x, y := 11, 12
  _, y := 22, 23

As you can see the semicolons are not needed too - gives you a nice Python / F# / Clojure feeling!
If you have mutliple assignament, you get multiple return values for free, as in the following function**:

  func Write(file *File, b []byte)(int, Error)

See? You just specify the returns in parens after the parameters. And that leads us to the next goodie connected to the multiple assignment:

1a. the ok-idiom

it goes like that:

  if ok, x = foobar(y); ok {
    // do sth
    ...
  }

So no more if(foo() == -1)! Instead you can write:

  if nr, err = write(f, buf); err == nil {
    // do sth
    ...
  }

You can assign and test a value inside of the "if "clause. Additionally the parens can be omitted too with "if" - more Python / F# feeling! Nice, isnt it?

2. named return parameters

We can assign names the return parameters:

  func write(file *File, b []byte)(n int, err Error){
    ...
    
    return
  }

When we name the returns, the compiler will initialize them with defaults (i.e. their zero-values)! We can eithet refer to them inside of the function (!!) or even omit them from the return statement, and then they will be used automatically with the initialized values. One thing less to worry about.

3. maps (and strings)

They are part of the language and not in the library. I wish they'll be that way in C++ too. Come on, D has done that!. Add this to the {} initalization syntax and you can write (almost LISP, Python or Perl style)***:

  var opmap = map[int]string{
    ADD: ”+”,
    SUB: ”-”,
    MUL: ”*”,
    DIV: ”/”,
  }

Well, speaking frankly, the {} initialization is possible in C++ too, alas only as of C++11.

4. defer for destructors

You can define Deferred actions, i.e. actions which are executed on the end of scope only:

  file.Open(”filename.in”)
  defer file.Close()

It's a nice replacement for destructors (see my previous post here). And it looks better that some "using" constructs from other languages.

5. duck typing

Go has duck typing, but it's not C++ type of template duck typing. You don't have to mention a specific interface (yes, Go has a notion of interfaces!), it suffices to implement them ***:

  type I interface {
    Get() int
    Put(int)
  }

And then you just implement the required functions for a data type (see note ** below), and it will automaticall comply with the interface I:

  type S struct { i int }
  func (p *S) Get() int { return p.i }
  func (p *S) Put(v int) { p.i = v }

like that:

  func f(p I) { p.Put(1) }
  var s S 
  f(&s)

OK, so whats the difference to C++ type duck typing? C++ doesn't care about types: a template with two parameters simply takes two things, and then tests if they have the methods the template code requires:

  template <type T> void foo(T& t)
  {
    if(t.canBark()) t.bark();
  }

How's that different from Go? C++ gets problems with overloading in that context: it sees only a function that takes 1 thing. A totally generic thing! So you cannot define a second template with the same name and a different argument type, because at the definition level there aren't types, only things! That's why there's no range or complete container overloads for SLT algorithms (see H.Sutter's blogpost).

6. generic switch

Just like Groovy, Go has a very flexible switch statement:

  switch c {
    case ’ ’, ’?’, ’&’, ’=’, ’#’, ’+’:   
      return true
    case c < 'a': fallthrough
    case c > 'A':
      return true
    default:
      return false
 }

I mean, ever language needs something like that: case switch for strings and other types! And did you notice that fallthrogh must be stated explicitly? I like that.

7. the go keyword.

I liked the idea of introducing the word "go" as keyword to start a parallel "goroutine":

  int i
  go foobar(i)

I just find it funny :), that's all...

Note:
I admit, I didn't describe the features that make Go to stand out, only a couple of those which particularly appealed to me. For more info on: function definitions, val notation, control structures, function literals (i.e. lambdas), goroutines, etc please read the book by yourself!

---
* a note aside: as pointed out several times on the web, there's a notable similarity between the logos of Plan 9 and Go. BTW: don't you think they are cute?

** more precise, you'd rather define a function like this:

  func (file *File) Write(b []byte)(int, Error)

i.e. a function working on the File type, and invoke it like:

  nr, err = f.Write(buf)

But this is the part of the OO stuff so I skipped it in this post.

*** code example from the "Learning Go" book.

2009 Languages Overload

2011-06-26T15:26:00.000-07:00

This year (or more to the point, that year, i.e. 2009) was for me a year of languages. Or maybe better, of languages overload! While I'm earning my bacon mainly with C++ (and a bit of Python), this year I had a look at a couple of new languages (adding to my last year's Groovy adventures) like: C# 3.0, Go, Scala, Clojure and even a bit of Haskell. In case of Haskell I'm feeling rather guilty, as I wanted to keep myself busy with Haskell this year but didn't manage more than some superficial reads about monads, and only becaue I needed to explain some Clojure code! Shame!

1. Java

But let me first comment on another language, which is ly dying these days: Java. Is it really dying? Looking at this post by codemonkeyism you'd say so. Why don't we like Java anymore? Let me cite:

Inheritance: outside of frameworks, inheritance is inflexible, leads to tight
coupling and is plain bad. Composition is most often a better choice. As are
mixins and traits

Noisy syntax: Lately there has been the enlightenment that too much noise in a language is a bad thing. Java is especially noisy in closures (anonymous inner classes) and generics.

Null / NPE: Null as the default for object references was a billion dollar mistake. An object should by default need a value. Otherwise NPEs will proliferate through your code. Newer languages prevent nulls or make the null behavior the non default one

List processing: As shown by functional languages, list processing should not be done
in loops. ... Java should have native support for easy list processing, not via the – best we have – constructs in Google Collections.

For me the 2nd problem is the worst, i.e. the noise. It's appalling how much of the Java boilerplate can be thrown away - check this presentation to see how much can be done with Groovy! But each of the new languages does a pretty good job here. Well, to be true, my biggest problem wasn't stated above at all mentionae above - lack of operator overloading! Whatever, the fact is that there's a host of new languages out thre, and the times seem to be more exciting than ever since the 70-ties!

2. Scala

Scala seems to be a language having every single feature from any language known to man. The overall impression was: OK, I know that one from ... One single feature I liked is the solution to the multiple inheritance conundrum (you know, the Deadly Diamond of Death) by linearization of the base classes. Clean, efficient, impressive! From the above list it's solving the Null/NPE, noise and inheritance problems. But the syntax is ML derived! Hate it! if this is a Java replacement, why cannot we stay in the old good C-lands? Another problem with Scala is poor tool support: I wasn't able to get the Eclipse plugin running, despite applying various workarounds found on Scala's homepage and upgrades of both the platform and the plugin. Even Martin Odersky (Scala's creator) is using Emacs for work!

On the other side, I think Groovy solves the list processing more elegantly, and it's DSL capabilites are more convincing too! (example to come)

3. Clojure

I like Clojure! I don't know why. It's not typesafe, has Lisp-y syntax and very complicated mechanism for integration of Java class libraries, but it is somehow elegant and minimalistic. It includes mutithreading primitives in the language definition and it tries to achieve good performance while staying immutable!

As I watched this presentation by Rich Hickey I noticed one important thing: he's an architect who was facing design problems in complex real life systems* (somehow like me, although I must admit that he's been exposed to more complexity than me) an tried to find a solution. And he really knows his stuff - all the conventional wisdom of software architecture! And Clojure (and Clojure's features) is an answer to these problems: it takes from LISP what seems suitable and isn't afraid to change it** otherwise.

Contrast this with Scala - sometimes feeling like an academic exercise in trying implementing all the features known to man in a single programming language just to show that it can be done (caveat: totally subjective opinion here). Maybe this pragmatic legacy is one of reasons why I like Clojure much more then Scala.

4. F# (and C#)

C# seems to be evolving very rapidly. From the ugly duckling it was a couple of years ago, it developed a respectable language with a decent amount of innovation. It has good lambda function support and the LINQ concept is rather interesting. I was surprised!

F# in its turn seems to be syntax-wise another ML derived langage like Scala, but it's not so overloaded with features. Sorry, I'm still not done with the free F# book...

Some (for example Scala Lift's creator here) say that it could be the single one of functional languages to success in the mainstream!

5. Go-lang

I didn't like it that much despite of its heritage. The obligatory, one and only true formatting rules, lack of subclassing... But on the other side, there is a CSP implementation on the language level! And as another programmer, who actually tried it out, stated here, it's much like writing in Python, only without its excruciating slowness. So maybe I should switch from Python to Go? Google Application Engine supports it now!

Update: in the meanwhile I managed to have a look at Go!

6. Summary

Warning: You see, it's not the comprehensive comparision I wanted to write back in 2009/10, but before I ditch the project altogether, I'd rather write this stub and extend it when I find some time. But the summary is already complete now, so... ---->

As I said above, I'd use Scala as a "better Java" as to avoid Java verbosity, lack of operator overloading and multiple inheritance. I'd use Closure for the joy of if (pun intended), I'd like some of my clients to want me to use Go, but wouldn't propose it myself, and I probably won't use C# unless I switch to .NET which is rather unlikely. On the other side, I might install F# on my PC - just to play with it a little.

--
* according to InfoQ "...Rich has worked on scheduling systems, broadcast automation, audio analysis and fingerprinting, database design, yield management, exit poll systems, and machine listening."

** e.g. even the venerable parentheses!

Exegi Monumentum...

2011-06-26T13:16:00.000-07:00

"Exegi monumentum aeris perennis" I don't know why but this line is always coming to my mind when I'm thinking about my father, who deceased more than 2 years ago. Just because he was fond of classical studies and Latin?

He also cherished that other lines of poetry, in a language that you can't possibly know, but which, as I can assure you, are work of a genius (I cite form memory):

"Wsrod takich pol przed laty, nad brzegiem ruczaju,
na pagorku niewiekim, we brzozowym gaju,
stal dwor szlachecki, z drewna, lecz podmurowany,
swiecly sie z daleka jego biale sciany,
tym bielsze, ze odbite od ciemnej zieleni,
topoli, co go strzegly od wiatrow jesieni."

You're right, that's a single sentence. And he could recite one after another in the endless flow of the language. Yes, you are right, this is a hommage to my late father, which I'm writing 2 years too late. The first thing I'll always associate with him will be the classical poetry. The another one, linked via the above phrase of Horace, is a story of an interview and things which it revealed to me in retrospect.

It was my school appointment to interview someone and I choose my father. I remeber just a single question I asked: what do you want to be remembered for? The answer was somehow unexpected to me: he cited the line of Horace, and said: I want to be remembered for the things I accomplish. And he did, he was working in local political bodies and was involved in building roads and water lines in our rural neighbourhood.
So when it came to finding a vocation, I wanted to build things! Not maths, physics, literature or music (things I was interested in), but engineering!

As for us in life there are 2 things which are worth of pursuing: for yourself, trying to understand the great scheme of things in that world (for practically minded, if you are more ambitious it's the universe then). But you owe th the others, those before you and those to come, to keep the world running smoothly, and build new smootly running things to replace old ones.

I think, that's my father's heritage. And that's why I cannot be a geek - because I grew up listening to poetry, looking at nature of the rural coutryside and experiencing very practical achievements in very real world.

Agile Rant Update

2011-06-06T06:41:00.000-07:00

After you've read my Agile rant you may have though: "just another old-fashioned hacker who just doesn't get it...". Fair enough, I'm myself not that sure if that what I'm saying isn't just result of lack of experience with Agile? Maybe with even more exposure all my reservations would vanish?

So I'm happy to find out that there are other people thinking roughly along my lines. For example here: Agile Hokum, in DDJ reader mail. He (the reader)says exaclty what I said:

But the "TDD corrupted by writing the tests after coding" is about as extreme as the Taliban. Testing is good. Write them before, during, after, much after, etc., all have their place.

and doubts usefulness pair programming as well. The standard TDD answer of the expert goes than like that:

I agree that tests written at anytime will generally have value. However, TDD was not designed as a way of validating code — plain unit testing will do that. TDD is first and foremost a design practice. And so doing TDD by writing the tests after the code is simply a misunderstanding of what you’re actually doing.

Well, that opinon is already dealt with in my Agile rant. But taking the simplistic road I'd just say that's the way it is handled in agile projects: not as a design technique but rather as a testing one. OK, here seems to be some confusion on the term itself: it's either Test Driven Development or Design...

On the other side the author (i.e. the reader) openly admits that he's not an Agile practitioner, so that opinion doesn't give any earnest support for my criticism?.

But look what an ex Googler says here:

This is not to say that all unit testing is bad--of course not. The dispatch logic in Guice Servlet and Sitebricks benefit from rigorous, modular unit tests. If you have ever used Gmail, Blogger, Google Apps, AdWords or Google Wave (all use Guice Servlet to dispatch requests) you have seen the benefits of this rigor first-hand. But take it from me, we could have achieved the same level of confidence with a few well written unit tests and a battery of integration tests. And we'd have been in a much better position to improve the framework and add features quickly.

That strikes a chord with me, because it's exactly the way I tested my last system - a lot of integration and pre-integration tests, and a couple of unit tests for classes which couldn't be easily tested thorougly that way. Wouldn't that be allowed at your company? Because this isn't Agile?

Don't Decay your Arrays!

2011-05-28T13:23:00.000-07:00

OK, that's nothing new, the technique is old, will be obsolete soon with the new std::tr1::array, and why should we use naked arrays anyhow? But recently I couldn't write it down as easily as I would like to, and as it's useful in the pre C++2011 code, why not discuss it shortly here?

Imagine you have to implement a Netbios/SMB remote file access protocol. The problem we are faced with while doing this is trivial but important - we need to log different packet types, operation codes and command options for debugging. They are all defined as integer type values and C++ doesn't (nor does any language I know) offer much support in that respect. So we define our own mini framework using macros. Don't panic, macros aren't that bad as they are told, and in our use case they are a perfect fit: translating between code (integer value) and strings (its descriptions). It's like eval() but in the other direction :). And it is working like this:

  // trace helper tables
  #define DEF_DESCRIPTION_ROW(a) {a, #a}
  struct CmdNameEntry { int cmdId; string cmdName; };

  const CmdNameEntry g_cmdNameTableNetbios[] = {
      DEF_DESCRIPTION_ROW(SESS_MESSAGE),
      DEF_DESCRIPTION_ROW(SESS_REQUEST),                             
      ...
  };
  const CmdNameEntry g_cmdNameTableSmb[] = { 
      ...
  };
  const CmdNameEntry g_cmdNameTableTrans[] = { 
      ...
  };
  const CmdNameEntry g_cmdNameTableTrans2[] = {                 
      ...
  };

Got it? We use C's stringize operator (#) to convert a constant's name to a string in compile time. You see, we need quite a couple of tables, and each of the tables has many entries (you have to believe me, Netbios & Co are rather a chatty bunch). So why are we using naked arrays? Because the std::vector class didn't have a convenient initializer before C++2011*, a shame! Alternatively you could copy the array's contents to a fresh vector just to pass it to futher processing, but how ugly is that?

Now you'd need to write something like this to translate between integer codes and their descriptions:

  getCmd(g_cmdNameTableSmb, sizeof(g_cmdNameTableSmb)/sizeof(CmdNameEntry)),

because in C/C++ you cannot define a true function on arrays like this:

  getCmd(CmdNameEntry table[N]).

Well, it seems you can after all, but it's of no use anyway because (as you know) inside of the function the array will decay to a pointer and the size information will be lost. I don't know how other people may feel about it, but after some time I got pretty annoyed with that. Couldn't we wrap this in a macro, or even better, use some template trickery? Let's try and write a small utility:

  template<typename T, size_t N>
    string SmbProtocol::getCmdName(int code, T(&table)[N], const char* errorStr)
  {
      for(size_t i = 0; i < N); i++) // <- oops!!! compile error!

      {
          if(table[i].cmdId == code)
              return table[i].cmdName;
      }    
      return errorStr;
  }

Uh-oh, that doesn't compile, but frankly, how should it? N isn't a runtime construct, it is type information, and it lives only at the compile time! What we need is something to translate between type and value, i.e. a bridge between compile and run time. It appears that's not so complicated as it sounds, just write**:

  namespace util
  {
  template<typename T, size_t N>
    size_t array_size(T(&)[N]) { return N; }      
  }

As we are in a template, we can access the type information, and as we are a function, we can return a value. Bridge ready! Now we only need to replace the incorrect line with:

  for(size_t i = 0; i < util::array_size(table); i++)

and now we are ready to go:

  string SmbProtocol::getNetbiosCmdName(int code)
  {
      return getCmdName(code, g_cmdNameTableNetbios, "UNKNOWN_SESS_MSG");
  }

Now we can define new description tables, forward them to functions using the pair of T(&table)[N]and util::array_size() and let our pre C++2011 code look a little more elegant.
--
* You can have a look at this previous installment of this blog for the new C++0x initialization syntax .
** if you wonder about the array parameter syntax look here for an explanation: template-parameter-deduction-from-array-dimensions

Bad Case of Code Reuse

2011-05-26T08:51:00.000-07:00

Bug story!

We all strive to write elegant, concise and clear code. One of the tenants of this pursit is the "DRY" principle - don't repeat yourself! But as usual you shouldn't always follow a principle blindly. Take for example the following bug which caused me some problems at customer's premises.

I just extended the client library of the system I was in charge of at that time to suppport parallel requests, as the legacy server code would would be too risky to be refactored. Why? Well, not everyone has the luxury to have (or even sometimes to be able to define) a comprehensive set of unit tests. In case of the legcy product there were none, and worse, the sheer number of possible inputs (any satellite data, corrupted or not) does not allow the system to be wholy tested!

I decided to supervise long running tasks and send a short running ones in parallel, so the concurrency level would be bounded by 2. At two points in the code I have got to check if the next task is a long-runner or not: when a single new task is submitted for processing, an when I need a second, parallel task (which cannot be a long runner again!). As I (true to the agile) first wrote the new task submission code, I defined the:

isLongrunner(filename)

method, which looked up if the input data file could be a potential long-runner. This method was used to check if a newly arrived request should be run in parallet to the currently running one. Then the library was tested and the first version was finished.

In the next test iteration I added pulling of the requests from the input queue when a short-runner gets ready and the long-runner is still busy. So I had to check if a request sitting in the input queue were a short-runner. But wait, our requests are files, and we already have a method to check for it! DRY baby! Elegant code! I tested it locally, and then it was deployed to our customer on the other side of the globe, namely in Asia. Mission acomplished, no problems expected... You wish! Of course, in Asia the new client library didn't quite work as expected, and the problem that had to be corrected still showed up. Embarassement, client unhappy! After many calls and emails I had at last the logfiles with the error showing clearly up.

What was the problem? In the parallel task submitting code at the end of a short running request I DRY-reused the

isLongrunner(filename)

method, I should rather use:

isLongrunnerSession(sessFilename)

What's the difference? Well, in my client library there is one: the filenames waiting in the input queue will be processed and can be renamed in some specific cases. The isLongrunner() method will check the already processed filenames, as it's passed as paramether to the WorkerThread::process() method*. Unfortunately, in the second case, we are already in the process() method, and are looking directly into the input queue, where the raw file names are to be found! Bugger! And the local test worked, because..., well I can't remember the reason now, probably I didn't test the interface variant (there are two, legacy and modern one - it's a real, living and growing system) the asian client was using. Ooops :(

So what do we learn from this story?

If I wasn't so obsessed with code elegance and DRY, I wouldn't be lured into bad reuse, and then the omission to test the right interface version wouldn't have such unpleasant consequences.
If I'd care about the naming of the methods and its parameters to the point of obsession, I wouldn't settle for the first name which somehow filled the bill. It was sloppy!

So the advice is as with the optimization: first get the code right, that get it DRY! And sometimes the techniques we are eager to use are more of hindrance than of help in a subtle way.

--
* WorkerThread is a part of my portable (as it's Qt-based) worker library, maybe I'll find the time to blog about it. At least the title is already chosen: "Are workers actors?".

A Little Agile nad XP Rant

2010-12-22T04:14:00.000-08:00

The title of this blog is "On Languages ... etc etc ... Systems and Proceses", but utill now I never said anything about processes (or did you think I meant UNIX processes?) so let us write something about the latter now. As lately everybody seems to be happy with Agile/XP (like here and here), let's rant about that a little. So beware, it's (at least partially) a #RANT#, so don't try to kill or evagelize me!

1. The Gripes

The one single thing you encounter today in the process land is the Agile. Well, I have a few gripes with that, which I should have wrote up earlier (and I wanted to, really, for the last 3 years or so), but now the time has come. On the other side, now when Agile is mainstream, maybe my points aren't valid anymore? You know, "the normative force of the factual"* :(. Maybe, but let us begin from the start of all that:

As I read the "Agile Manifesto" and the XP explanations some 10 years ago or so, I was just amazed. It was the liberation! And what I particularly liked were things like pair programming, test driven design, and iterative deleopment. But today I must realize that I was somehow mistaken - maybe did I get it all wrong? Somehow it's all different from what I imagined it'd be, and I don't like it very much. Two examples:

a) the pair programming

In my eyes it was all about two people caring about a piece of code, discussing its design, reviewing main rewrites, just because discussing something with another person can show you another perspective on the problem.

And now? The reality? It is crazy, it's about keyboard swapping, and two people doing coding at the same time. For 15 minutes it's my keyboard, and then it's yours. Come on, are we in Henry Ford's car factory? Or were we supposed to be doing some creative work? #RANT#

But there are people who actually like that (here, here) aren't they? I'd say this could work, but only if you are paired up with someone on your level. Not just another beginner guy with a big ego, thank you very much! Even then I find it to be an overkill, 2 people doing work of 1? Aren't there people able to be doing decent work by themselves anymore? #RANT#

b) test driven development/design (TDD)

I liked this idea very much. OK, I've only seldom written the tests before code myself, but there always have been tests for a new feature: either before I coded it or after that. Because really, that doesn't matter if you know what a feature you are going to implement.

But now? The reality? You should always write tests first. Or you are not agile. And why that? It comes even better - TDD can mean test driven design! A good example of this was to find in a blog posting** of the venerable Uncle Bob where he develops a prime factorization algorithm by TDD, i.e. by writing a series of increasingly complicated tests and reworking the algorithm accordingly. That looks for me like crazy, groping in the dark for the right solution, excuse me?! #RANT#

Why should we waste time and rework our solution on and on? The received answer is: because it results in decoupled code. Come on, aren't there other ways to write decoupled code? Like old and unfashionable thinking about its design a little? #RANT#

2. Some Summary and More Gripes

So meanwhile the XP is for me not the freedom, bur rather the next horsewhip of the pointy haired boss types. It's more about following the rules and their religious observance than about anything else. As I read about XP/Agile first, I thought it was targeted at proficient developers, you know, at the responsible adults, which should be allowed to be adult and just do their job! But now I have more and more the impression that it's rather targeted at beginners. Obviously, they need clear rules and directions. So maybe it's about the overall prevalence (and importance) of beginners in our industry?

Well, mabe (just maybe) I'm opposing the abovementioned XP principles because they introduce a process-like quality into XP, and I feel kind of betrayed now? It was supposed to be about freedom and interacting free individuals? Maybe it was just an illusion, maybe the recipe of Agile Manifesto was too simple?

In this this excellent post about "post agile" development Gwaredd Mountain states that no methodology is pure "agile" or pure "process-oriented". Rather, each one of them contains "freedom" (agile) and "process" elements, as a mix of both is needed for the real world. So even the supposedly totally free XP has to have some emergency brakes. Well, so much for my illusions of a better world :(. Nonetheless, the emergency brakes are rather expensive, the double pair of eyes to watch over "Brownian motion" like TDD development? That's my private opinion, but arguably it's somehow taking things to the extreme.

But there's 3rd problem, and it's not easily dismissed:

d) evangelism

In the Agile world-view there's either agile or waterfall, stone age, and nuclear winter. A typical Twitter post says:
Agile change is change. Change is hard for people
thus, if you can't get enthusiastic about Agile it means that you are unreformable, and just can't see the light! I think the Agile trainer guild is responsible for that***. As their training courses target mainly the beginners, they must give them rules to be followed in each circumstances. But I cannot understand why this must be promulgated to the general public? You know, "Only a Sith deals in absolutes", it's not a sign of wisdom do divide the world in the only right way and all the others! #RANT#

Ok guys, you must earn a living, but doesn't engineering decency mean anything to you? Or can't you think for yourselfs and would rather become a follower? Silver bullet anyone? That irritates me, but at the same timee it's interesting - what could possibly be a result of a realy intensive full-night talk with an Agile trainer? #RANT#

It's a little known fact, but Dr. Royce's defining paper for the waterfall process already contained iterations (see figures 3 and 4)! Agile is not that original, people were able to think and use common sense before that! #RANT#

3. So What Do I Think Myself?

You may ask, how should we develop software then? We're not supposed to go back to "cowboy coding" are we? Steve Yegge' post describes an alternative: the pretty informal Google development practices**** at that time and praises it as the solution. Well I must admit it does sound good, but then even Google wanted to introduce some process not a long time after that (Shh, we are adding a process). In that article they describe how the Scrum process was introduced in the AdSense project (if I'm not mistaken), however not a full embodiment of Scrum. But it seems that somehow for time critical, heavy-money projects you need some kind of a whip in the end... :(

So what do we need a process for? Do we need it at all?

First let me quote Aristotle (via Pragamtic Programmers link, and no, I don't read him in original...) - "We are what we repeatedly do. Excellence, then, is not an act, but a habit." As I understand it, we have to kind of design the things which we repeatedly do, if we want to ge good at something. And that's a start of a process.

The 2nd insight comes from the field of avalanche avoidance (because I'm somehow of an offpiste skier). What the professionals always do, is to have a checklist in the car and to go through it every time they set off. Because people forget, people do make mistakes, and in that profession, when they do, it can cost them their life. That's a kind of process too - you won't believe that you could do a mistake, but in the grand scheme of things, you probably will. So you need some annoying routine, even if you don't think so!

So what would I take? A methodology I liked rather well was the "Tracer Bullet Development" from the abovementioned Pragmatic Programmers book. It's simple, it has a lot of common sense (what I seem to like more then religious fervor) and it goes along the lines of my own understanding of the agile development which I came to during the last 10 or so years.

Because I'm not an enemy of agile. More than taht, I think it's somehow a necessity! In real life there's constant change, and the original design***** of a system cannot be sustained - either it will be dying or it must adapt! So for me, agile has got it right in its basic proposition - you must always iterate. As the ancients used to say "iterare necesse est, vivere non est necesse" :).

And what about my Agile/XP gripes? Here I'm with Steve Yegge - there's "Agile" and "agile", Agile of the trainers and pointy-haired bosses, and agile of the pragmatics. And I'm with the latter.

Waiting for your comments.

---
* or "die normative Kraft des Faktischen", by Immanuel Kant. This means: all that is, is good ,because it is (i.e. exists) or something like that.

** sorry, the original blogpost I read is vanished, this one is the current version reworked as a Kata:
http://butunclebob.com/ArticleS.UncleBob.ThePrimeFactorsKata

*** Read Steve Yegge's post about the so called "Agile" experts: "Good Agile, Bad Agile ", http://steve-yegge.blogspot.com/2006/09/good-agile-bad-agile_27.html, or this one: "Agile Ruined My Life", http://www.whattofix.com/blog/archives/2010/09/agile-ruined-my.php and look out for the Fraud markers!

**** BTW, as I was at Google's HQ in London once, this did just look like any anther SW outfit, maybe even a little like at NEC's HQ in Tokyo (sic!) - small desks (ok, not as small as in Japan), a big open space, no books articles on notes on the desks, just the programmer and the computer. I wasn't that impressed... Sorry, I digress, I know.

***** For example, take this fascinating and/or terryfying case of dying architecture (from this book):

Alas, everything started out so well! The concepts were clear-cut. The technical prototype was a success, the customer was thrilled and Gartner Group decided that the architecture was flawless. Then real life began to take its toll: the number of developers went up, the average qualification dropped, the architect always had other things on his plate – and time pressure increased.

The consequence was that the architecture concepts were executed less and less consistently. Dependencies were in a tangle, performance dropped (too many client/server hops and too many single database queries), and originally small modifications turned into huge catastrophes. To a certain extent the architecture concepts were circumvented on purpose. For example, classes were instantiated via Reflection because the class was not accessible at compile-time.

Some Observations on Software Estimation

2010-12-11T08:36:00.000-08:00

In my prevoius post post in introduced the "real life factor" frw for software estimation, and conjectured it to be as high as 3. I remebered that as I read James Iry's short blogpost. Let me quickly restate its contents. He recollects the course of a software project and the estimates at each point of its history. I summed that up and added the reason for each estimate failure:

             | Project | 
   Estimate  |  Phase  |   Error Reason 
  -----------+---------+-----------------
    2 weeks  |    0%   |  initial guess
             |         |  
    4 weeks  |   50%   |  too optimistic!
             |         |  
    6 weeks  |   90%   |  hidden complexity
             |         |  
    8 weeks  |   99%   |  forgotten the tests,
             |         |  docs and review
             |         |  
    8 weeks  |  100%   |  not really ready, but 
             |         |  has had enough!

So there it is, when estimating we are too optimistic from the start, and then we tend to forget a lot of things .So using this error "model" the real life factor would be frw=4! I'd say rather high. Maybe we need a better model? And what if we want to be really careful?

When speaking about serious software estimation you will usualy end up with the "Demistyfying the Black Art" book (an excellent book indeed, read it!), and will probably learn and use the PERT technique, where you have to give 3 estimates: the best case, the normal case, and the worst case estimates. Then you take a formula and get the expected project duration.

That's all very well and good, but when using this technique, we still tend to be overly optimistic and to forget a lot of things, exactly as in the above example!

The following blogpost hits the nail on the head:

Our experience has taught us that simply asking developers to make two estimates (“low” and “high”) doesn’t really result in much more information than a single point estimate. Developers tend to be optimistic when it comes to individual estimates. So without a little structure, the “low” estimate usually means the absolute smallest amount of time the task could take – an event with a very low probability of coming true. The “high” estimate means what it’s most likely to take to get done.

Given that data, let us do a smal, back of the envelope calculation for the probabilities of the 3 PERT estimates. Thus:

  low estimate => rather improbable => say 20% probability
  high estimate => most likely => i.e. 50% probability

then

  medium estimate => will lie inbetween => ~30% probability

So the real life factor for the medium (i.e. realistic) here is frw=3.3, not that different form the above, unscientific guess! PERT will correct this somehow, by not that much. So should we really always multiply our carefully crafted estimates by 3 (or 2), thus admitting that we cannot do it? That's scary.

But wait, we don't need 100% certainty, that would be sandbagging! Let us say, 80% is enough (see Pareto), but that's still 2,7 times the our "realistic" estimate or 1,6 times the "worst case estimate". I don't want to make any conclusions though, it's only some observations because it makes me curious that the value of 3 seems to resurface rather often!

Yet Another... Web Framework (in C++)

2010-11-23T10:57:00.000-08:00

What? YAWF - yet another web framework? But this is mine(!) ... and it's very small - a Simple Embeddable C++ Webframework for Qt!

1. The need

I have discussed this topic on this blog earlier*, so you shouldn't be surprised to hear about it again: a C++ web framework. You'd think there isn't such thing? Wrong! From my own research and from the comments of friendly internauts I collected a quite a list of possible frameworks:

And recently I found even another one:

CppCMS

Do you see its cool logo? It says that programming in C++ is a good thing, as it isn't hard on the environment, and thus can maybe even stop the global warning ;). I must admit that I never seen C++ from that angle! Interesting point, isn't it? Ok, when you think of all that multicores and figths about cache lines then C++ does quite naturally comes to mind with its very direct memory control, doesn't it? Kind of "lean programming" :).

And don't forget the new kid on the block:

cpp-netlib by Dean Michael Berris

It's rather a wide range of possiblities, so you'd think the world doesn't need another framework, and least at all by yours truly. That's what I thought too, untill I stumbled upon a rather simple problem. Currently I develop a Qt application that runs in the background (call it demon, service, headless, etc.) and would like to have an admin interface for it. Of course I could write a standalone GUI client connecting to the application's command socket but that's so nineties! Only a web interface is fashionable these days. Thus I need a small webapp for the admin console.

So at first then, I thought I'd use some open source code from the above list. But all of them would either not integrate well with Qt events, or require 3rd party includes and dependencies, and gwan would even have to be started as an external server! Well, I didn't use gwan before and it does sound like a border case usage for it, just imagine all that debugging of its servlet compiler in run-time...

Then you may ask: isn't there any HTTP server in the Qt framework? Yes indeed, I found one, but it's an extension of the framework (they call it an option I think...) and it's running as Windows service, not as an additional interface to an existing binary. So we've got the same case as with gwan. BTW, why hasn't C++ a standard HTTP server class? Go has one, Python has one, Javascript has one, but C++ - nope! Well, that's not completely true, CINDER framework has one, and cpp-netlib got one as well, but I wanted to avoid big includes and dependencies like Boost.

But hey, Qt has a couple of HTTP support classes, so I thought: boy, that cannot be that hard! And I just hacked my own mini HTTP server.

2. The design

Of course that gives me an opportunity to design a web framework the way it should be always designed at last :-), because all the others seem to be somehow overcomplex or overcomplicated :-). So let's go down to the brass tacks then!

First we have to decide on the name - let simply try YAWF(4q), and in best software giant tradition I'll use an internal codename, for example "CutieW".

The design will be simple, plain and oldskool, as I need something working and that fast. Besides, never underestimate the power of KISS!

The basic ingredients are the Qt classes for TCP socket handling and HTTP request and response parsing. Then we'll have request handler objects with a process() method. The handler objects (kind of servlets for the Java minded) will have to be derived from a superclass (how uncool, I know) and statically instatiated, so that they can register themself with the server in the startup phase. They register with a name string**, for example "user" or "page", so they can be bound to an URI (see below for the example).

Let's sum it up: there will be a server executable, it will set the URL path configuration in startup phase, each path wil get its own handler. When a request arrives, a fresh instance of the handler will be created, this the handler classes need a clone() method for this.

The server will thus find the matching handler prototype for the URI, clone it, and then invoke its process() method. As you see, there's 1 to 1 relation betweem URI and the handler object in the current implementation, but an extension to "multiple dispatch" isn't difficult, and I'll program it when I have some spare time.

Now to the general usage pattern of YAWF4q. In Clojure you'd write the following:

  (defn display []
     (html [:h1 "hello!"]))

  (defroutes myroutes
     (GET "/" [] (display)))

  (defonce server (run-jetty #'myroutes
       {:join? false
        :port 8080}))

In cpp-netlib this:

  struct hello_world {
    void operator()(server::request const &request,
                    server::response &response) {
                       // your code here ...
                   }
  } handler;

  server server_(argv[1], argv[2], handler);
  server_.run();

But in yawf4q it's all pretty plain, non-clever, and almost boring:

  #include "CuteHttpServer.hpp" // the cutie

  int main(int argc, char* argv[])
  {
    // your application:
    QApplication* qtAppl = new QApplication(argc, argv);

    // the webserver on port 3100
    ibkrj::yawf4q::CuteHttpServer testServer(3100);
    string error;

    if(!testServer.start(error))
        return -1;

    // run Qt
    qtAppl->exec();
  }

The server just starts, finds the config file (using a default, hardcoded name), sets the routes, and waits for requests. If you don't define any handlers, a simple "I'm alive" response is sent back.

3. The example

How to write your handlers? Look, you have to derive them from the TestHandlerBase class. First let us implement the handler of the index page:

  class TestHandlerBase
    : public MiniHttpHandler
  {
  public:
    TestHandlerBase()
        : MiniHttpHandler("TestHandlerBase") {};
    // base overrides
    virtual MiniHttpHandler* clone() { return new TestHandlerBase; }
    virtual void process(AnalyzerRequest& reqData, std::string& respData);
  } g_handler1;

OK that's rather much boilerplate, as we need to supply the name string plus the clone() and process() methods, so maybe I should provide a macro like this:

  // --- OR maybe a MACRO?:
  class TestHandlerBase
    : public MiniHttpHandler
  {
  public:
    BASE_TEST_HANDLER_FUNC(TestHandlerBase)

    ... your functions ...
  } g_xxx;

I don't know, as a matter of fact I hate macros in frameworks! But as an alternative maybe it isn't that bad. Another possibility would be using CRTP, as to automatically add the clone() method, but wouldn't you say this would be somehow too clever when compared with the rest of the design?

And here is the processing method:

  void TestHandlerBase::process(AnalyzerRequest& reqData, std::string& respData)
  {
    respData = " TestHandlerBase alive!!!! ";

    respData =
        "<html><head>"
        "<title>TestHandlerBase is alive!</title>"
        "</head><body>";

    respData.append("<h1> Input your user data: </h1><p></p>");
    respData.append(testform); // HTTP form for UserName, UserMail and Text, action="person"

    respData += "</body></html>";
  }

OK this doesn't look very cool, but the view component isn't yet there, wait a little. What this handler does is to ask for some user data. Now the second handler follows, which will processes the submitted user data:

  class TestHandlerPerson
    : public MiniHttpHandler
  {
  public:
    BASE_TEST_HANDLER_FUNC(TestHandlerPerson); // here process() is already defined

  } g_handler2;

  ///////////// impl:
  void TestHandlerPerson::process(AnalyzerRequest& reqData, std::string& respData)
  {
    respData =
        "<html><head>"
        "<title>TestHandlerBase is alive!</title>"
        "</head><body>";

    respData.append("<h2> Thank you for your data! </h2>");

    // parse the data for display:
    map<string,string>& data = reqData.formularData;
    map<string,string>::iterator iter;

    respData.append(
        "<table border=\"1\"  width=\"70%\"><tr>"
        "<th><b>Field</b></th><th><b>Value</b></th>"
        "</tr>");

    for(iter = data.begin(); iter != data.end(); iter  )
    {
        respData.append("<tr><td>").append(iter->first).append("</td>");
        respData.append("<td>").append(iter->second).append("</td></tr>");
    }

    respData.append("</table>");
    respData.append("<p></p> <a href=\"/index.html\">Back</a><br>");

    respData += "</body></html>";
  }

Rather grotty servlet or .jsp style here, isn't it? Mixing presentation with code?!

Don't despair, you can use a template. Yes, let's be modern! I chose the mustache :{{ template syntax. I decided for mustache, well, because it's pretty new, generates much buzz, and isn't looking too bad. For those who don't know mustache, it is a simple HTML templating syntax with values encoded like this:

  {{value}}

and sections, which can be rendered multiple times, if they receive a list as input data:

  {{#section_X}}
    {{value1}} => {{value2}}
  {{/section_X}}

So there's no need to code anly loops in the template. YAWF contains an implementation of mustache :{{, which seems to render correctly all the examples I found in the online mustache docs. The implementation is based on a simple idea: the template is parsed into a linear sequence of "nodes", and the data items passed to the template take care of rendering and walking the node list. Have a look at the source code*** if you want.

So our response handler would now look like this:

  // using templating:
  #include "CuteMstchData.hpp"
  void TestHandlerPerson::process(CuteSrvRequest& reqData, std::string& respData)
  {
    string templ =
      "<html><head><title>TestHandlerBase is alive!</title>"
      "</head> <body> <h2> Thank you for your data! </h2>"
      "<table border=\"1\"  width=\"70%\">"
      "<tr><th><b>Field</b></th> <th><b>Value</b></th></tr>"
      "{{#datatable}}"
      "<tr><td> {{field}} </td> <td> {{value}} </td></tr>"
      "{{/datatable}}"
      "</table><p></p>"
      "<a href=\"/index.html\">Back</a><br>"
      "</body></html>";

    // convert data to JSON represenation:
    map<string,string>& inpData = reqData.formularData;
    map<string,string>::iterator iter;
    int i;
    ibkrj::yawf4q::CuteMstchValMap templData;

    for(iter = inpData.begin(), i = 0; iter != inpData.end(); iter++, i++)
    {
      templData.addToList("datatable", i, "field", iter->first);
      templData.addToList("datatable", i, "value", iter->second);
    }

    // display page with data (the View of MVC)
    respData = CuteHttpHandler::renderThis(templData, templ);
  }

It maybe doesn't look much better than the plain HTML code, but here we could save the template in file, and read it in at the runtime. And we've got the View part of the MVC pattern. The handler objects stand for "C", but so far don't have support for "M" in YAWF4q.

Now, as last part of the puzzle, we have to configure the server paths, but this is simple:

#
# test configuration
#
index.html$ :: TestHandlerBase
person/.*$ :: TestHandlerPerson

You see, regular expressions are supported. Now let it run baby! The first handler gives us:

Then we willingly give our data:

And see the results:

4. Summing up

At the moment the code*** is only very basic, as I haven't got time to work on it a much as I'd like. I wrote it on my private time, as my customer did decide in favor of a general, grand-scheme, system management solution. So as for today, I wrote only a single test, that one which was described above, and tested it with Visual C++ on Windows. YAWF isn't more than a toy just now as many features are simply missing, for example I'd like to have a built in support for configurable error pages to round the things off. And as next thing, I'd like to add the Model component using my CuteWorkers framework (I hope I'll come to write a post about Qt-workers too).

But there's much more that could done, like:

- HTTPS support in the sever (that shouldn't be very difficult as Qt offers QSslSocket and other classes)
- cookies and session data handling
- dynamic loading and caching of the templates
- dynamic loading of the servelts vial DLL (kind of what ASP is doing)
- a "Clojure example" like dynamic interface

and so on, and so on. Look at my TODO list in the github repository. Nonetheless it's simple (KISS!), it's pluggable, it doesn't have any dependecies other than Qt. Requirements fullfilled!

--
* check out these previous posts: servlets-in-c.html, c-servlets-again.html, c-server-pages.html

** we cannot forget that C++ hasn't much to offer in terms of introspection. But we won't give up automatic registration and detection of handler prototypes!

*** You can find the code at Github: https://github.com/mrkkrj/yawf4q

Engineering Decency

2010-11-13T06:21:00.001-08:00

A couple of days ago I was quite unpleasantly touched by a tweet (a Twitter tweet :-) of a so called "expert" (you know, the people doing only presentations and learning new languages only to write their next book...) ridiculing a developer about a failed project. Somehow I didn't like that - come on, the guys just tried to capitalize on a promise Google (yes, that big engineering powerhouse) made! And they were not alone...

What is that, that the programmers find it so hard to show some engineering decency? I've seen it again and again. Is it so much fun* to flame someone to feel better? The experts should know best that doing stuff mean means making mistakes. Or maybe have they forgotten how it is to be working with real stuff instead of powerpoint metadata?

If you are skiing difficult terrain, you know that the question isn't if you will fall, the question is when you will. That's the same in software - you don't want to believe this now (and me neither) but you will make (sometimes stupid) mistakes! So please, show a little decency. The guys you are going to deride probably just tried to create a working system for a given set of requirements which then turned out to be lagerly false and for a specific platform, which then emerged to be broken. We've all been there.

And because it's difficult and needs to be learnt, I have an example of how to excercize engineering decency (found here). Instead of the usual:

What idiot with even half a handful of moldy cottage cheese for brains would ever create a pile of stinking filth like this and call it an application?! The true feat of engineering here is that this code monster didn’t swallow your whole organization in the black hole of its utter lameness. Where did this person learn programming — NBC?”

Try saying that:

“This will take me longer than we had discussed. In my initial examination of the system, I missed some design decisions in the existing code that conflict directly with what we need to accomplish. I didn’t anticipate the approach adopted by earlier developers, because I would not have designed it that way myself. I’m sure they had their reasons for choosing that direction, but so far those reasons have eluded me.”

That's decency, that's humility. Maybe there's a reason you just don't see now. But even if that's really a load of crap, frankly, who of us can say he never ever wrote bad code (or a bad powerpoint presenation for that reason)?

--
* well, if we'd like to get freudian, we could say that the unconscoius fear of doing some big, stupid mistake reveals itself by attacking others, and that we can maybe reason from that, that the majority of programmers are not well trained for their job. But that would be only a mere speculation to discuss with friends on a lazy evening...

Named function arguments for C++

2010-08-16T14:14:00.000-07:00

For me a high quality code is a readable one. Consider a hypothetical method like this:

  class XXX:
  {
  public:
    void func(int aaa, string bbb, bool optionX = true);
    ...
  };

now try to call it:

  func(1, "test", true);

What irritates me here is, that I don't know what exactly does true mean at the first glance! Thus I found myself quite often writting code like this:

  bool useOptionX = true;
  func(1, "test", useOptionX);

Well, that's better, but what about this:

  bool useOptionX = false;
  func(1, "test", useOptionX); // should I use it or not?

a little bit misleading, isn't it? Not that this would be a big problem, but it's nevertheless a small hitch obstructing us from reading the code effortlesly. OK, next try:

  bool dontUseOptionX = false;
  func(1, "test", dontUseOptionX ); // don't use is false, so I should use it?

A total mess! And what about constructors?

  class XXX:
  {
  public:
    XXXX(bool optionY);
  ...
  };

  class YYY: public XXX
  {
  public:
    YYY() : XXX(true) {}; // true what???
  ...
  };

Call me pedantic, but I don't like that at all! What we really need here are Python's named arguments:

  sameFuncInPython(aaa=1, bbb="pythonistas", optionX=True)

Simple, effective, readable! Is there a way to emulate this in C++? Recently, as I was especially annoyed with the constructor example from above, I devised a following solution:

  class XXX:
  {
  public:
    XXXX(bool optionY);
    ...
    typedef useOptionY bool;
  };

  class YYY: public XXX
  {
  public:
    YYY() : XXX(useOptionY(true)) {};
    ...
  };

and now the other code can be wtritten totally plausibly too:

  typedef bool useOptionX; // local namespace!

  func(1, "test", useOptionX(false));
  func(1, "test", useOptionX(false));

and what would you say to the following?

  waitOnSocket(sec(1), usec(20));

Small thing, but it improves my source code. I like it!

---
PS: After I wrote this, I remembered reading something in this vein about Java. And really, this post http://codemonkeyism.com/never-never-never-use-string-in-java-or-at-least-less-often/ by Codemokeyism proposes generally the same solution:

  new Customer(firstName("Stephan"), name("Schmidt"));

Alas, instead of a benign typedef, you have to write a whole new class named name and firstName etc. Without doubt it's and improvement, but the costs and the clutter... But on the other side Java tends to be verbose nonetheless, so maybe it's OK.

Javascript on the server?

2010-06-14T10:07:00.000-07:00

Recently, I stumbled upon an iteresting presenation about server I/O parallelism and multithreading. Surprisingly it was in a Javascript conference, as I sometimes have a look at the Javascript language developments. I'd never expect to find something like that in this contex though.

The general message of the presentation was that threads suck and asynchronous programing is the king. Come again? I don't quite agree with that (or rather agreed, as you'll see in brief) - wasn't that for the threads to save us from the onerous asynchrounous programming with its context information and callbacks (remember Windows event programming?)! The threads justly delivered a convenient abstraction to group the logically connected data and logic in separate units of execution. So now we are doing a full circle: from asynchronous to threads to asynchronous?

OK, on the surface that may be looking like that, but there's more to that. The problem is that when we apply threads to parallel I/O they are just to slow (you know, context switching, thread stampedes and all that). Thus there's a host of asynchronous I/O methods starting with the venerable select() UNIX call. Ok, agreed, for that special field the threading model isn't very appealing, but for the general usage I'd just hide this inside of an I/O thread...

Now there's this pesentation (Ryan Dahl’s talk on Node.js) and what the author says is exactly that: callbacks and asynchronicity are good! How that? We all know it's a pain in practice? His response (an this is the valuable insight got when watching this) is that we just didn't have a good enough programming language to realise that! When you do in in C (as you'd likely to do when programming performant I/O) you are messed up, but if you take Javascript...

See, that's the crux here: Javascript was designed to work in a callback driven environment, so the language has unique mechanisms which no other language can offer. Thus if we take Javascript and do event oriented programming it's a breeze then! The idea is to take the ultra higspeed C library (libev in that case) and wrap it with event-friendly Javascript hull. Here's an example code from node.js site:

  var net = require('net');
  net.createServer(function (socket) {
    socket.setEncoding("utf8");
    socket.addListener("connect", function () {
      socket.write("Echo server\r\n");
    });
    socket.addListener("data", function (data) {
      socket.write(data);
    });
    socket.addListener("end", function () {
      socket.end();
    });
  }).listen(8124, "127.0.0.1");

Well, maybe it isn't a thing of beauty (if you're not into Javascript) but it's concise, everything is at one place, and you can read it without reading tons of manuals first. So maybe this is the way of doing this? Another thing I liked that node.js never provides a blocking API - even filesystem calls are asynchronous! That's consequence!

Sorry for the short post, but it's summer, I want to go out and watch some worldcup!

---
Resources:
1. the talk to be found here: http://jsconf.eu/2009/video_nodejs_by_ryan_dahl.html
2. and here's the code: https://gist.github.com/a3d0bbbff196af633995
3. if you'd like some more technical details, here's an excellent post by Simon Wilson: http://simonwillison.net/2009/Nov/23/node/
4. node.js critique (yes, there's such a thing too!): http://al3x.net/2010/07/27/node.html

Addendum:
- here is a simple blog engine in less than 200 lines of code using node.js : http://github.com/zefhemel/persistencejs/blob/master/test/node-blog.js. Not bad, I'd say.
- Multi-node: how to implement a concurrent node.js HTTP server: http://www.sitepen.com/blog/2010/07/14/multi-node-concurrent-nodejs-http-server/
- Hummingbird: an impressive application based on node.js - http://mnutt.github.com/hummingbird/

Software estimation and re-reading the classics

2010-06-13T09:36:00.000-07:00

Recently I stumbled upon the "Software Top-Ten List"* again. At first I though, OK, that's old, they did mainframes at that time! But then I read it nonetheless. Maybe because it's so short? It's only 3 pages. Whatever. I read it I was surprised about how refreshing it was. Not mouldy, stale and irrelevent, but interesting and fresh! Surprisingly, because the findings are supposed to be well known.

One interesting thought came after I read the point 8:

Software systems and software products each typically cost 3 times as much per instruction to fully develop as does an individual software program.

I immediately thought about that old estimation rule: estimate how long it will take to be done, and then multiply it by two. I always though it to be unserious or a manifestation of intellectual laziness. What, you aren't able to say how long this effing program will take to be written? Just sum up the parts (as shown below)!

But let's look at the problem from a little more different angle: what if programmers do not take into account the costs of polishing the program, making it user-friendly, plus all the problems and dependencies from other programmers located in other deparments of the company - "L'enfer, c'est les autres"! Normally you do not take this factors into account when estimating, and you even cannot estimate it properly. We use either a brute-force 20% project buffers or a statistical estimate based on the PERT method, but it somehow doesn't work out that good.

So maybe we should take the above citied measurement data, at use it when estimating? Just multiply our technical-only estimate by the "real world" factor? Personally, I still cannot believe that the "real world" factor be so high as 3! Am I too optimistic? But I think that frw=2 should be a good choice!

Although I'm a little afraid here, as I always imprement a given piece of code faster than my own PERT estimate. But recently, when estimating the entire system, I missed the point completely: new requirements, unexpected dependencies on not so great libraries, the internal pressure to please the customer... This all sums up in a greater project! An at that point I should have been to muliply so high as by 4! Welcome in the real world.

--
* Harry W. Boehm, "Industrial Software Metrics: A Top-Ten List", http://csse.usc.edu/csse/TECHRPTS/1985/usccse85-499/usccse85-499.pdf

Bug hunting or how dumb can you get, really?

2010-04-29T09:59:00.000-07:00

As a programmer you sometimes can't help but feel really stupid! Really stupid-stupid! Look at this code snippet for example of a foolish bug that kept me busy for some time. The story behind this is that I added a feature to the product I'm writing for a customer, and was totally happy with it. There was a real customer benefit and it worked like a charm through my tests. However, when deployed to the internal test system, it crashed unexpectedly now and than.

That was a kind of shock, because the product was running stable for more than a year in this environment, and I was totally unsettled as I didn't want to believe that the new feature could have something to do with it. It was tested rock-solid. There must be a mysterios bug somewhere else in the program. And it hid for one year before it disclosed its identity! Bad!

However, after I ran the collected test data on my development machine, it turned out to be a pretty uncomplicated bug:

    for(size_t i = 0; i < a_prInput->frameCount(); i++)
    {
        // header
        size_t start = a_prInput->NettoDataParts[i].uOffset;
        size_t pos = start;

        DnsHeaderData header = parseHeader(a_prInput, pos);

        // interesting data there?
        if(!header.isResponse)
        {
            TRACE("dns: packet isn't a response, ignoring...");
            continue;
        }

        // skip the questions
        for(int i = 0; i < header.questions; i++)
        {
            if(!parseDnsName(a_prInput, start, pos))
            {
                TRACE_ERR("dns: cannot parse a question record");
                m_uGoodByteCnt = pos - a_prInput->NettoDataParts[i].uOffset;

                continue;
            }

You see? The code was referencing the data form the outer loop with the index of the inner one! And it did it only in error case, and crashed only if it was out of luck. Arrgh... There was no problem with fixing this, but it started me thinking: I'm programming for some 20 years, I know C++ in and out, why did I make this bug in the first place? Did I violate some programming rule or best practice? Is it possible to avoid such bugs by some sound practices? Some answers come to mind:

There should be a warning from compiler
That would be the ideal solution. But what warning should that be? And what if I really wanted this that way? I'd need to use an ugly #pragma then? No, better let it be.

I should have used at()
What should I say, er, well, at() is for wimps! And if that wouldn't be the case, we have a choice between at() and the [] operator. So we want to excercise our rights!

Use telling names
Well, of course, I could name each loop variable accordingly, but come on, can you imagine the hassle? Because if this has to be of any good you have to be consequent. So maybe only do this for nested loops? Maybe, but if I start to do this as to avoid this error, and something else to avoid some other error, you guess it, it becomes a pain! We need something more general,
Use for_each(), as we don't do any addressing here
That's more interesting, as logically speaking, the inner loop isn't indexing single elements, but rather wants only to skip over the uninteresting headers. But, on the other side, for_each() is annoying, as it requires start and end of the sequence, which isn't a problem if I'd use the iterator, but gets pointless if I never want to use the iterator directly!

use iterators to go high level
Isn't that the same as the above? No, because it's not an application of a specific technique in a specific situation, but rather a conscious decition for rising of the abstraction level. And quite serendipitously we can avoid the above mentioned error! Or are we? What if both outer and inner loop are using the same name for the iterators? Here we are agan back on the square one and have to use for_each() for the inner loop as to be safe.

Alas, none of them is satisfactory. Well, I know, there's one more solution: code reviews (or pair programming). But first, my customer isn't doing them that often, and second, I doubt this bug would be found in a review/pair. But even assuming it would be found: the price is much to hight! It took me a couple of minutes to fix it, once I had the right data, and my customer expects "results just now".

What is the key point? What does this all mean? Are we doomed? I think that there are two possiblities here: either use iterators or some other high level constructs (which admittedly is somehow of an overkill), or be prepared for occasionally fits of stupidity. In the end we are only humans, and errare humanum est... And who wants to be safeguarded against everything?

--
PS: Sorry, I started this entry long ago, but in the meantime I was so distracted that I couldn't come round to write anything for the blog. Besides, microblogging isn't doing it any easier ;-).

PS2: Other possibility: write shorter functions, use a skipNameRecords() method and you are saved! But really, that function is under 1 page lenght, How long should functions be? Five lines or so? Or seven as adjust to humain brain's cognitive limitations?

Fooling the ODR (and the compiler)

2010-02-28T11:18:00.001-08:00

I just discovered a technique that is prety convenient but rather not self-explaining, nonportable (?), and possibly a maintenance nightmare. So you should never use it, should you? Except, as it's sort of cool, if you'd like to go a little on the geeky side...

1. The need for speed

Imagine you are writing some quick classes in the Java style (i.e. no header files), locate them in some C++ source file, and forget about them. Then you discover that you need to refactor your program (in my case it was a custom test framework) and to move some classes out to a separate file (a.k.a. compilation unit). Now you shun to do the right thing (i.e. to refactor) because you'd have to cut up your classes in two: the .h file and the .cc file, and to duplicate some (non always trivial) amount of code. So yo aren't doing that...

But stop, following trick did it for me: I defined the classes in a separate .h file and kept the .cc file with the extracted classes unchanged. I included the new .h file in the rest of the old .cc file where the classes were removed from. Then I just linked the the whole thing, et voilá, everything worked like a breeze - a refactoring as we like it!

2. Why does it work?

But wait, we spared some editing work, but didn't we violate the One Definition Rule of C++? I'd say we did. So why didn't the compiler complain? Well, as to answer it, we've got to look at the example a little closer.

The extracted classes looked like that:

TestThreads.cpp:

  #include "WorkerThread.hpp"

  // test thread A
  class ThreadA : public WorkerThread
  {
  public:
    ...
  private:
    ...
  } g_workerA;

  // test thread B
  class ThreadB : public WorkerThread
  {
  public:
    ...
  private:
    ...
  } g_workerB;

Note that there's no TestThreads.hpp definition file here, thus we don't have a direct ODR violation for the compiler to complain! The newly created definitions file TestThreads.hpp looked like this:

  #include "WorkerThread.hpp"

  class ThreadA : public WorkerThread
  {
  public:
    virtual void addRequest(Request& r);
    ...
  };

  class ThreadB : public WorkerThread
  {
  public:
    virtual void addRequest(Request& r);
    ...
  };

The WorkerThread.hpp definition file looks like this:

  class WorkerThread
  {
  public:
    virtual void addRequest(Request& r);
    ...
  };

And the rest of the TestMain.cpp file where the TestThread classes were previously placed:

  #include "TestThreads.hpp"

  extern TestThreadA g_workerA;

  extern TestThreadB g_workerB;

  g_workerA.addRequest(req);

Can you see why we could fool the ODR? Well, in this special case we didn't used the constructors directly in the TestMain.cpp file, only the methods of the WorkerThread superclass which then call the appropriate virtuall methods defined in the ThreadA class.
The linker then generates a reference to the virtual ThreadA::addRequest() method. Well, guess what, a method with the very name can be found in TestThreads.cpp, and the linker can resolve it no probs!

Although we clearly have two definitions of the ThreadA class, we get away with it! The ultimate reason for that is that the C++ compiler doesn't have the global knowledge of the whole program - it works only on the compilation unit level, and the relinquishes the responsibility for the complete programm to the linker! I.e. there's no whole programm optimization possible (at least not in the standard model, although Visual C++ compiler can do it).

3. How to extend the "hidden" classes?

Suppose we'd like to call a new method in the TestThread.cpp file from the outside. You cannot call this method directly, as we do not include the real definitions of the TestThread classes, but only the "fake" ones. Alas, there is a price to pay for fooling the compiler and for beeing lazy! The price here is, that we have to modify the base class (TestWorker.hpp) by adding a new public virtual method, if we need a new TestThreadA method to be called in the TestMain.cpp file!

Not pretty, uh? You know, sometimes you can go away with breaking the rules, but in the end you'll have to face the consequences. In the art of software design, your skills are measured at your ability to estimate for how long you will go away with laziness and tricks, and when you have to resort to the known, sound and boring "best practices".

But nonetheless, the price I had to pay was OK for the deeper knowledge of the compiler-linker cooperation and the increased knowlede of my own code.

Do you GoF - the belated anniversary and the ladder

2010-02-14T12:11:00.000-08:00

Last year we had the 15th (or so?) anniversary of the seminal "Gang of Four" book. Is it that old already? So maybe discussing the design patterns is all an old and boring stuff now? Well, shame on me, I wanted to write up my ideas on that matter for such a long time that I run the risk of them beeing irrelevant. But on the other side it's a kind of unfinished business for me, and I have a to kind of close it up.

Some time ago I wondered - why didn't I use the design patterns that much in my programming and design work? I came to the conclusion then, that the design patterns are useful as a common vocabulary as to speak about designs, but then some new ideas surfaced in my mind. By the way, why should I use a pattern vocabulary if I don't use any patterns in my designs?

1. Ideas from chess and math

If you're playing chess (need a Wikipedia link here? ;-) you are accustomed to seeing the board in patterns: check-mate patterns it is! You try to imagine what kind of a check-mate setup could be possible in a given situation, and then work towards that by moving your pieces and trying to chase away your opponent's pieces as to complete the pattern you have in your mind. I must admit that I never had this feeling when writing software. Some pattern in my mind which is the solution for some design problem? Nope.

Read the "Refactoring to Patterns" book by Joshua Kerievsky? When I first heard of the book I though he struck the right idea: only use patterns to refactor code which is a mess. But then when actually reading the book, it was rather a feeling (in most cases) of doing something wrong to the code, well, like overengineering it! So the book didn't reconcile me with patterns too, as it didn't give me the chess-like feeling of a pattern just matching a given situation.

The other problem with patterns is their presentation. This overly formal template with "Forces" (why not just problems?), "Intent", "Consequences", etc, etc... This makes me think about "The mathematicians lament"* - where a matematician reproaches his fellow scientists of over-formalization of mathematics with the effect of its incomprehensibility. What he is saying is basically the following: yes we need the formalism, but only in some very difficult cases where our intuitions don't work. But not in every and one case: "nobody's dying here!"*, you can relax a bit!

It's the same with patterns: the stern formality of presentation is both somehow comical and making the contents rather indigestible, and an dry, uninspiring read. I can only say, relax, we are not deliberating about the meaninig of infinity like Cantor did! We are just describing some not so complicated techniques of object composition! Could you stop to be so pompous?

2. Dreyfus et al.

Do you know the Dreyfus model of knowledge acquisition**? It defines several phases in learning a specific skill: beginner, advanced beginner, competent, proficient and expert. Beginners need rules and recipies, competents can continue learning on their own, proficients and experts see the big picture and fly by the seat of his pants.

Where do the design patterns fit in this scheme? Well, at the beginner level they aren't useful, as they are not "step by step" recipies. On the other side, for the proficients and experts, they are some kind of rules, and experts do not like rules and recipies by definition!

I personally think that patterns appeal rather to the lower ranges of the skill spectrum. Did you see the hopelessly overengineered software full of design patterns written by the beginners? I've seen it only too often!

So maybe they are for the competent? And the usage by competent and proficient users? Let me tell you a little story here. It's not entirely true that I don't use patterns! Lately, I thought that I could warrant the use of the visitor pattern in my current product, as to extract the reporting aspect out of several content managing classes. Well, it has a kind of worked at first, but now, as this whole aspect has to be redesigned, I noticed that this design is rather unintuitive and I cannot simplify the reporting without dumping visitor first. Ok, experiment failed, but my initial problem was that I didn't really search for a solution of my problem, but rather stopped at the point where I found the (as I thought then) matching design pattern! But I didn't think it trough as thoroughly as I should!

3. Summing up

So maybe the thing with patterns is that they are like the "Wittgenstein's Ladder"? You know, what he said in his "Tractatus Logico-Philosophicus" (I cite from memory):

...this book is like a ladder, use it, but then throw it away.

Isn't it the same with patterns? Use them to expand your experience, but don't use them in your software! And as to make the discussion complete, let us state a completely different point of view, a Lisper's critique***:

When I see patterns in my programs, I consider it a sign of trouble. The shape of a program should reflect only the problem it needs to solve. Any other regularity in the code is a sign, to me at least, that I'm using abstractions that aren't powerful enough-- often that I'm generating by hand the expansions of some macro that I need to write.

This is the typical smug Lisp programmer assertion, as they mean that all other languages will in limes converge to Lisp, but newly I tend to think along the lines of the Lispers: the design patterns are a fix for lacking feature in language A, which you know from language B. But as we cannot change the language in most cases, and we are doing OO in the mainstream, just mind the ladder parable!

Update (June 2010): There's another explanation which I didn't think of before. I came across that when reading some programming book (which I cannot come up with now, sorry for the author!). Recall the mid 90-ties: it was the time of the "object craze", everybody wanted to be part of OO. But nobody really understood OO, and as the book went "Stroustrup gave us the C++ but he didn't tell us how to use it"! Her come the patterns, a kind of missing OO manual! This would explain my feelin that half of them was trivial (or not so difficult to figure them up by yourself) and the other half was implemeting language features missing from C++. Very sober diagnose, but at the moment it's my favourite.

---
* Paul Lockhart, "A Mathematician’s Lament", www.maa.org/devlin/LockhartsLament.pdf

** I found it in the book of Andy Hunt, "Pragmatic Thinking and Learning", Pragmatic Bookshelf, Oct. 2008, but here's a blog post link, and here another one

*** Paul Graham, "Revenge of the Nerds", http://paulgraham.com/icad.html

Local language trends 2009

2009-12-21T00:56:00.001-08:00

I just had a look on the statistics pages on our local IT freelancer website (http://www.freelancermap.de/) and found the results pretty interesting. First let's have a look at the languages which are most popular in the freelance projects. The picture below shows the languages, the supply of programmers (the blue bar) and the demand in projects (the red bar).

As it seems, here in Germany there are only two serious languages: C/C++ and Java. Well, ok, Visual Basic and C# are sough after rather strongly (red bars), but there seems to be too little supply. Is that the fear of a vendor lock-in or is it just because Microsoft is considered evil that the programmers don't like .NET? The same case of too little supply for PHP, but here I must mention that the hourly rates for PHP tend to be rather low - PHP still considered a hacker language?

And now look at the last but one language with a longwinded German name I won't retype here (too lazy). In fact it's not a language of its own, but rather denotes "all the other ones in there", which includes everything fancy (or just plain old). You see there is too much supply, the IT managers seem to be more conservative than programmers, no surprise here.

Which leaves us with C/C++ and Java. Java is unmistakeably the 500 pound gorilla here, but it's hopelessly over-supplied (blue bar)! For C/C++ the situation isn't so dramatic, and I'm glad I specialize in C++!

The proposed course of action here: go away from Java (or well, from everything else for that matter) and start on .NET!

Now the overview of technologies in the freelancer projects:

Software development comes first (still a little place for more work - red bar!), then system management (plus "service and support"), however the suply here is much too big (blue bar)! It's just as bad as for SAP programming - in the past the eternal cash cow with ridiculously high hourly wages, now it's a real "problem child" - the market seems to be breaking down. Banking crisis anyone? For Web development and "graphics, content and media" there's much too much demand, just like in the PHP case above. Then the ominous field of "IT consulting", where I don't have any idea what it is, sorry, no comments there :-/. One interesting field is "Executive consulting" with so much demand and so little supply, maybe everyone is just too shy to position theirself in that niche?

The proposed course of action here: either stay with software development or move into "Executive consulting"!

Having said that: Merry Christmas you all!

Even more new C++ features!

2009-12-16T14:07:00.001-08:00

Hi everybody! As I had a look on the new C++ proposal the last time , I found many small interesting things apart form the big, important features. But this was only a first look. Recently I had another look there (and at the C++0x FAQ* by Bjarne Stroustrup as well) and discovered even more interesting crittures. So what did I miss?

1. Some Syntax

First of all the new function definition syntax! Now I can write something like*:

  using PF [](double) -> void;

along the lines of lambda function definition! Well, that's definitely a treat, what a relief from

  typedef void (*PF) (double);

To be honest, I don't know if I've got the last one right, I have always to look thes syntax up or copy and paste from my other code! The [] notation can be generally used to say to the compiler that "the function type will be specified later"! Thusly we can write the following when usign embedded type like List::Link

  [] List::erase(Link* l) -> Link* { ... } ;

We postpone the return type definition utill we have entered the scope of List, and thus no scope specifier is needed!

I liked this notation (it makes the code look a little Groovy-like), but alas, it seems that not everyone does so, as the related "auto arrow" notation proposal was rejected recently**! What is auto-arrow? Look at that***:

  auto f() -> int;
  typedef auto PF() -> int;

the first one is a function from void to int, and the second one is a function type definition for the same signature. Looks good enough, but consider this one:

  auto *pf(auto *p() -> int) -> auto *()->int;

I don't even want to discuss what it means, if you are curious look it up in ***! So after that proposal was justly rejected, I'm a little afraid of the fate of the [] notation - is it the next one to be scrutinized? I hope not!

2. Initalization and initializers

I though I've understood the new initalizer notation, but I was surprized how ubiquitous it became! At first I though it would be only used in assignment like

  vector<int> xs = {1, 2, 3 };

Not so, nowadays you can use it (i.e. the new {} notation) everywhere, even instead of classic parentheses!

  X x{a};
  f({a}); // initializes a temporary
  X* p = new X{a};

This all does look pretty strange to me, and to be honest I don't like it that much, but on the other side you can use it to avoid verbosity with pretty good results:

  map1.insert({a, b}); // sic!

The other improvement with initialization is that we can initialize the string and integer class members directly in the definition of the class!

  class X
  {
    int a = 7;
    string s{"abcd"}; // new syntax!
  };

I can only say AT LAST and amen!!!

Belonging to the "initialization" block of features, we must mention the access to the inherited constructors (in addition to the also new feature of delegating constructors, i.e. constructors which can be invoked of another constructors of the same class):

  class A { public: A(){} };
  class B : public A
  {
    using A::A; // imports all the A's constructors into current scope!
  };

  // now this is possible:
  B b;

At this place I'd like to mention another related feature: disallowing copy and assignment operators. You can write for example:

  X& operator= (const& X) = delete;

to disallow it! Well, if I consider that Google even has a macro exaclty for that reason (DISALLOW_COPY_AND_ASSIGN(), see here) and that I used an UML tool to generate the private version of it for my every class I think that the default-generating them was a bad language design idea! If we want a copy semantics we'd better tell the compiler so! Hindsight is 20/20.

3. Threading etc.

Of course thread, mutex, condition variables and atomics are there as basic support (as expected). On the higher level we have futures and a packaged_task class, which wraps features for simpler usage. Moreover, Bjarne does promise that an async task launcher (logically named async) will be included in C++0x as well. Then we could launch tasks simply like that:

  auto f1{async(funcX(100))};
  auto f2{async(funcX(200))};

  return f1.get() + f2.get(); // waits on both futures

If he's right it would be very nice!

As GC is connected to multithreading in an intimate way (at least in my eyes, sholud blog about it some time soon!), I must state here that the GC is still missing!

But not entirely, we've got some minimal support for third party garbage collectors in there (20.8.13.7 Pointer safety). It defines what a "safely drived" (i.e reacheable) or an "disgused" pointer is (3.7.4.3 Safely-derived pointers) and lets the programmer specify where such pointers can/cannot be found with declare_reachable()/declare_no_pointers() functions.

One thing that struck me however in the threading library classes is the flexibility of interfaces. Take the std::thread's constructor for example. You can use it like that ****:

  void hello_func() { ... }
  // say hello
  std::thread t{hello_func};

or like that, using a functor:

  class Greeting
  {
  public:
    explicit Greeting(std::string const& message) { ... }
    void operator()() const  { ... }
    void greet(std::string const& message) const  { ... }
  };
  // say goodbye
  std::thread t(Greeting("goodbye"));

or currying the thread function with std::bind:

  void greeting_func(std::string const& message) { ... }
  std::thread t(std::bind(greeting_func, "hi!"));

or without std::bind:

  std::thread t(greeting_func, "hi!");

or without std::bind and with more arguments:

  void write_sum(int x, int y){ ... }
  std::thread t(write_sum, 123, 456);

or using the member function pointer

  Greeting x;
  std::thread t(&Greeting::greet, &x, "greet");

  // or by a smart pointer
  std::shared_ptr<Greeting> ptr(new Greeting);
  std::thread t1(&Greeting::greet, ptr, "hello");

  // or by reference
  std::thread t2(std::ref(x));

  // or directly!
  std::thread t2(x);

See, you don't need to worry about the initalization, the new constructs will just take anything you throw at it!

4. Miscellanea

First the variadic templates. What a name for templates with viariable number of arguments! I guess it's an interpolation of the "dyadic" concept which just means "of two arguments" :-). But seriously, we can writhe things like

  template<typename Head, typename... Tail>

doing recursion and typematching like in Clojure or Haskell ;-) Well a kind of, but it looks like that at least. Look at the n-tuple definition using variadic templates in Bjarne's FAQ*.

Further, the new and improved enums (which are class based now) can be now forward defined ! But I'm doing this all the time with C++ Visual Studio 2005! Ouch...

No for the geeky ones: we have user defined literals! You can define new literal types using the following new operator type: operator""() (plus constexpr!), for example

  "abc"s; // s: string literal
  2i;     // i: imaginary literal

For details have a look at Bjarne's FAQ*. NIce feature, but will I use it? It is readable for the maintainer? We'll see.

The next 2 features look to me as if they came straight from Python. First the raw strings:

  string s = R"[ahjh/?=(%/$§%§%\w\\w///]";

Why not just R"..."? Dunno. Next the foreach variant of the old for() loop.Together with initilizers we can now have some Python feeling in C++:

  vector<string> xs = {"aba", "bab", "aab" };
  for(auto x:xs) cout << x; // x:xs looks even a little Haskell-like ;-))

So there are many new things, some of them making C++ look like entirely another language. No I wonder - will the C++ programmers use it, or will they stuck to the old, but known evil (like the old function definition syntax)?

---

*Bjarne's "C++0x - the next ISO C++ standard", http://www2.research.att.com/~bs/C++0xFAQ.html

** Danny Kalev: "The Rejection of the Unified Function Syntax Proposal", http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=460

*** Daveed Vandevoorde: "The Syntax of auto Declarations", http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2007/n2337.pdf

**** for completness' sake, the examples are partially taken from: http://www.justsoftwaresolutions.co.uk/threading/multithreading-in-c++0x-part-1-starting-threads.html

More about Java Closures

2009-11-24T08:23:00.001-08:00

Surprize, surprize! Hear, hear!

One year ago, Mark Reinhold, Principal Engineer at Sun Microsystems, announced At the Devoxx conference in Antwerp, Belgium that the next major release of Java, JDK 7, would not include closures. At the same conference this year, however, Reinhold announced in a surprise turn around the Java would be getting closures after all in JDK 7.*

More specifically, they adopt the BGGA proposal (look here) but without the non-local return (look here) and control invocation statements. This leaves Java without any destructor or C# style using statements support!

What can I say? I can only reinstate what I said in my previous post: comparing to the development of the C# language (look here ** for an interesting inteview about LINQ and its supporting language features) the Java language looks increasingly mouldy (ouch!!!).

But maybe I'm mistaken? maybe there will be some support for modern ressource management in Java but it's decoupled from the closure proposal? Let's hope so.

Update: well, I didn't hope in vain, there will be a feature for this: Automatic Ressource Management (see here)!!! Moreover, we'll got a little bit of type inference for the right hand generic expressions with new (as Google Collections provides it) with a "diamond" operator (?) <>, and strings in the switch statement (as in Groovy). All in all you cannot complain, I'd say, and some bloggers (ehm...) shouldn't be too quick to bash Java 7!

--
* I found the news here: http://www.artima.com/forums/flat.jsp?forum=276&thread=274496
** http://www.infoq.com/interviews/LINQ-Erik-Meijer, BTW I found an interesting quote of general interest there:

Actually, if you look at where I'm going, I'm getting more and more into old school object orientation with interfaces and so on, going away from functional programming. If you are passing a high order function you are passing one thing, one closure. If you're passing an interface or a class, you are passing many because you are passing a whole V table. These can all be recursive and so on. I'm rediscovering object oriented programming and I feel that maybe I wasted some of my time doing functional programming and missed out on objects.

You see, it's not only the functional programminch which is gonna be the Big Next Thing! I think that we cannot just condemn OO and go all for the functional. OO has its merits when structuring code, and a hybrid OO/functional approach seems to be a good thing to me!

Songs in code

2009-11-11T23:36:00.000-08:00

Won't explain what they are, because it's obvious. These two made me laugh:

Bob Marley's (and Groovy's):

  4.times{
  !woman == !cry
  }

and REM's (and SQL's):

  DROP TABLE MyReligion

but normally they are boring, uninspired or plainly wrong, like this one:

  while(young)
  {} // loops forever

Come on, the whole thing is that young is a volatile variable, which will somehow, sometimes be set to false! ;-)

Java's Closures Debate for C++ Eyes

2009-10-30T09:29:00.001-07:00

Years ago Sun described Microsoft's delegate extension proposal for Java as:

Many of the advantages claimed for Visual J++ delegates -- type safety, object orientation, and ease of component interconnection -- are simply consequences of the security or flexibility of the Java object model.
....
Bound method references are simply unnecessary. They are not part of the Java programming language, and are thus not accepted by compliant compilers. Moreover, they detract from the simplicity and unity of the Java language.*

blah, blah, blah, marketing drivel! I don't know what is more annoying here: 1. the object-oriented orthodoxy, or 2. the complacency that "we know it all better! Be that as it may, but now James Gosling (Mr Java himself) says he'd always wanted closures instead of anonymous inner classes in Java, and there is not one, but whole 3 (!!!) proposals how to include functions as first class citiciens of the Java language! So much for the "simply unnessary" argument...

And there is considerable controversy around the very idea of closures, somehow similiar to the C++ debate around Concepts we discuseed recently, deeming it unnecessary and "too complicated". As we are lucky to get lambda functions in C++0x and don't complain the least, the debate in Java community made me curious. Come on, why not to have closures? Even C# has some!

So what is a closure? For the sake of this blog entry it is something like a lambda function in C++ (or Python or Groovy), i.e. an anonymous, freestanding function:

  auto print = [] (int x) { cout << x; }

Now let's do it in Java.

1. Proposals

So let's have a look at the three proposals.**

a) BGGA (Bracha, Gafter, Gosling, Ahé)

This is the most ambitious proposal. It introduces a new type into the language: a function type, and a totally new notation for it. This notation remainds me of Haskell's or of the lambda calculus types.

{T => U} for a function from T to U
{T => U => V} (or is it {T,U => V}?) for a function from T and U to V

I think this notation is one of the purposes why many in the Javaland doesn't like this proposal a bit. The second one will be surely the nonlocal control statements (returns, breaks etc). Come again, non-what? Well, it means that for example the return statement doesn't return from the closure but from the scope which invoked the closure! They don't bind to the closure but to the environment! Why should that be good for? Wait till next section for expanation, dear reader.

An usage example:

  {Integer => void} print = { Integer x => System.out.println(x); }

b) CICE (Concise Instance Creation Expressions)

This is the simplest of the three proposals. Basically it's only a syntactic sugar for defining anonymous classes on the run, nothing more. No first class functions! No obscure functional programming theory!

An usage example:

  public interface IPrint { public void invoke(Integer x);} // exacltly 1 method!
  IPrint print = IPrint(Integer x) { System.out.println(x); }

It's the simplest of the proposals, but for my mind, the most annoying one: you have always to refer back to some interface definition, and it sucks!

c) FCM (First Class Methods)

This is a kind of middle ground between the previous two: it introduces first class functions, but doesen't have the nonlocal binding for closures return statement and the irritatinmg lambda-type notation.

An usage example:

  #(void(Integer)) print = #(Integer x) { System.out.println(x); }

From all the three syntactically I like the BGGA proposal the best, as you can just write your code in a familiar code-block manner, without the FCM's hash or CICE's interface tag.

2. The Debate

To put it mildly, sometimes I find the debate somehow childish. Here the typical arguments:***

pro: If the don't implement closures, I'll never code any line of Java an will go to Scala
con: If they implement the closures, I'll just break down and cry... I cannot change to any other language, but I'll hate you all for that!

Come on people! It's just a language feature! And you don't have to use it at all if you don't like it. Beside this, it's not the developers that choose languages, it's the managers! If they can be convinced that a language offers some substantial benefits, it will be adopted. But not because you like Scala or something else better!

Will the closures make Java too complicated? Would they really

"...complicate the language beyond normal usability: mainstream programmers will turn away from Java, so the argument goes, and move on to simpler programming languages
...
Java will turn into a guru language used only by experts in niche areas"**?

So tell me which the simpler languages for the JVM are! I can see none (nonscripting one) which would be simpler. Guru language? Are you kidding me? The closures are an attempt to make Java less verbose, so everyone should be happy. So why the fear?

One (non PC) answer might be, that the "too complicated" camp is perfectly happy with the language as it is now, doesn't mind the lack of elegance, and don't want learn new mechanisms. And maybe there is some point where there are to much mechanisms (or paradigms) in the language? Me, as a C++ guy, I'm accustomed to the "multiparadigm design", as C++ is a real behemot in that respect. I just try not to think about the mechanisms I'm not using now.

Maybe in Javaland this isn't so simple, as the programmers were lectured for years that there is only one paradigm in the world: the object oriented one. This would explain why some people think that the addition of generics stressed the complexity of language to the limits (the other explanation is the design choices for backward compatibility, in particular the wildcards).
So it seems to be either the the irritation caused the generics or the unusual nonlocal returns in BGGA which provoked the strong feelings, dont you think?

3. Discussion

But let's get more technical: why do we need closures in Java? The answer is probably: because Ruby has got it! The Java community seems still to be in a collective Ruby trauma, and emulating Ruby features is a good enough reason in itself. As they used to say behind the Iron Curtain: to learn form Ruby is to learn how to win! So now, just as in Ruby (or Groovy and C# for that count) we'll be able to write in Java:

  Utils.forEach(names, { Integer x => System.out.println(x); }); // BGGA

So, that's kind what we've always had in C++ with STL and lambda libraries.

But that's not all there is! In each of the three above proposals there is a second part, which allows some kind of finer-grained resource management, i.e. a kind of "destructors for the modern times". I can only say "finally", because Java's resource management is just a big pain in the rear.

BGGA tries to achieve this goal by nonlocal control statements mentioned above, which will then allow to implement the "execute around" pattern as so called "control abstractions"** , but are somehow not very intuitive. In the BGGA proposal the keyword this will be lexically bound and will automatically refer to an instance of the enclosing type in which the closure is defined. For me, Groovy solves the problem of nonlocal bindings somehow clearer - introducing two members for a closure: owner (refering to the enclosing object) and delegate (defining the lexical binding). In tis way wee can always explicitely say what type of lexical binding we need, the default value of delegate being owner!

The other proposals try to address the same question as well: CICE by ARM (Automatic Resource Management Blocks - a special type of block where a dispose method will be automatically applied when leaving it) and FCM by JCA (Java Control Abstractions - this one is more like the "execute around" pattern mentioned above, and requires nonlocal lexical bindings as BGGA does).

At this point let us try to draw some conclusions.

Firstly: with C# having the lambdas in its 3.0 version****, and C++0x having lambdas and the auto specifier, leaving closures out of Java 7 would let Java looking pretty old in comparison!

But don't despair, there is a library solution out there (http://code.google.com/p/lambdaj/). Maybe a library solution will be suffiecient in this case? I don't know, the syntax doesn't look very intuitive, but it's probably the best you can get.

Secondly: what I find the most instructive lesson here is however, that we can see that the old notion of destructor isn't so old-fashioned after all! C# has meanwhile its using statement and Java deserves (and needs) some destructor-like mechanism as well! So much for the (former) "modern languages".

--
* form SUN's rebuttal of delegates - http://java.sun.com/docs/white/delegates.html, where they claim that delegates (i.e. C# equivalent of C++ member function pointers) are bad, because the anonymous classes can do all the same ans are object oriented too! A typical case ob the object-craze of the nineties IMHO

** I will only skim the surface here, as my aim is only the look-and-feel. If you need more information a good place to start is: http://www.javaworld.com/javaworld/jw-06-2008/jw-06-closures.html ("Understanding the closures debate - Does Java need closures? Three proposals compared", by Klaus Kreft and Angelika Langer, JavaWorld.com, 06/17/08)

*** rephrased ;-U from the discussion here - http://infinitescale.blogspot.com/2007/12/bgga-closures-end-of-many-java-careers.html, but seen in that form on other places in the Web

**** Let us digress a little: in C# we'd define the above lambda like that:

  Function<int, void> p = name => Console.WriteLine(name);

or use it directly like that:

  names.FindAll(name => name != "Marek")
       .ForEach(name => Console.WriteLine(name));

Note that in C++ the lambdas aren't templates (they are monomorphic, i.e. we must define the argument types) but in C# they are! To achieve the same in C++ you have to use a template lambda library.

I mean the C# language is looking increasingly cool, perhaps I should learn it, as everybody can program Java these days???

Tony Hoare and the meaning of life (well, almost...)

2009-08-27T03:52:00.001-07:00

You know, I was always wondering about programming - is it an art, a craft, or is it an engineering discipline? Some crazy hackers maintain it be an art, more down-to-earth types oscillate between craft and engineering.

My personal feeling was that it couldn't be engineering - I missed the scientific part of it, it was too "soft". For me software engineering appeared rather to be a set of rules of thumb for organizing our mental work (functions, classes, modules...), something from the realm of cognitive science perhaps :-). For example I couldn't take my multithreading program and prove it not to deadlock in a future (i.e not yet written, but planned) installment of this blog...

1. The presentation

And then, some months ago, I came across that presentation* given on QCon 2009 Conference by Sir C.A.R. Hoare (aka Tony Hoare of the Hoare logic, Quicksort, and CSP fame) about relationships between programming and computer science.

Computer Science is about how programs run and how correct programs can be constructed *

By the way, I used to dislike the all-present video presentations, which seem to be replacing the old-fashioned articles as programmer's information source of choice. Well, to put it frankly, I hated them! Instead of beeing able to quickly scoop the essentials and then read the article if it'd interest me, I'm now forced to hear for hours to some uninspiring presentations and often to some really horrible accents to boot, as to discover that in the end only the title of the video was interesting!

But on the other side, with the video presentations I'm no able to hear people like Linus Thorvald** and Tony Hoare in person, and it proved to be very rewarding to me in both cases. Watching Tony Hoare was a great pleasure - he's a wonderfully gentle and good humored old man, I'd say my idol of sorts. And as he's got his first degree in philosophy, so he's bound to have some interesting insights in the question at hand.

His answer is both simple and compelling (I'm rephrasing it here):

Software Engineering is an engineering discipline because it uses Computer Science, and Computer Science is a science because its results are exploited and confirmed by software products!*

Sounds at first rather convincing, doesn't it? But wait, for a philosophy major, isn't he overlooking something?

2. Circulus Vitiosus?

Well, embarassingly, it's looking suspiciously similiar to a circular argument , right? For SW Engineering gets justified by the CS but the CS is justified by SW Engineering? It certainly appeared like this to me at the first glance So let us have a closer look at it:

science is: "the systematic study of the structure and behaviour of the physical world, especially by watching, measuring and doing experiments, and the development of theories to describe the results of these activities "***

Does this apply here? Well, maybe, if for "physical world" we substitute the human-built things like hardware and binaries. But wait, don't we rather work with mental constructs instead (you know, languages, functions, classes)? Well, yes, but they are models, like math is a model in civil engineering. In the end it's all about how the binary runs!

engineering is: "to design and build something using scientific principles"***

This one seems to fit perfectly. So both parts of the argument are corrct, but it is a circular one? Let it translate it form English into the language of logic:

(CS usedIn SWEng => SWEng is Eng) and (CS usedIn SWEng => CS is Sc)

You see it's not really circular, as it's two separate propositions, and they are just rephrasing the two above definitions (check it!). Because science is defined as a special, rigorous kind of examination of the physical world, and engineering uses science when dealing with this world directly! But wait again, it's not said that we are allowed to use only scince in engineering, or ist it?

3. The Engineer an the Scientist

This leads us to another interesting aspect of that presentation, which was the comparison between science and engineering, and how they relate to each other. Somehow explaining the above realtionship (I must paraphrase again):

so it's first:
scientists are interested in pure truths, i.e. programs which doesn't have errors, while an engineer must compromise, i.e. live with programs that have got some errors,

and second:
scientists are interested in certainity, while an engineer lives in incertanity, must take risks and perform risk management,

and thus:
scientists should be women, engineers should be men (Yes, he really made this little joke:)

There are several other comparisions along these lines in the presentation, and furthermore, even a gradation between the two extremes like applied scientist or scientific engineer are introduced. So there is (almost) a continuum of positions between the two extremes of pure CS scientist an a common hacker.

And that fact harbors some kind of consolation, particularly if I'm in a very pedestrian project and must do the most boring things. Just remember, you can always move up the ladder, and use more science, more absolute truths, and thus (as the ancients believed) come in contact with more beauty. An that's maybe the ultimate "consolatio philosophiae“, or do you think this is an exaggeration on my part?

--
* all citations are not original but rephrased by yours truly, so if they are wrong it's my fault! Original source: Tony Hoare, "The Science of Computing and the Engineering of Software", QCon Conference, Jun 11, 2009 , London: http://www.infoq.com/presentations/tony-hoare-computing-engineering

** for example this presentation by Linus was fun: http://www.youtube.com/watch?v=4XpnKHJAok8&feature=player_embedded

*** the definitions are taken from "Cambridge Advanced Learner's Dictionary", as I was looking for simplicity: http://dictionary.cambridge.org/

Two C++ curiosities with a deeper meaning

2009-07-28T07:16:00.001-07:00

Lately, I was quite surprised by two things in the realm of programming, both of them C++ related. The first one is the new attribute syntax for C++09*, which I curiously failed to notice in the new standard proposal as I first had a look at it.

1. The first one

What are attributes? No, they aren't class data members with a convenient access syntax in this case (like Groovy's attribute syntax - or was it Python?).

I personally only knew attributes as an ugly Microsoft Windows COM-specific hack, by virtue of which you can inject COM related information into C++ code. Look at this example:

  #define _ATL_ATTRIBUTES 1
  #include <atlbase.h>
  #include <atlcom.h>
  #include <string.h>
  #include <comdef.h>

  [module(name="test")];

  [ object, uuid("00000000-0000-0000-0000-000000000001"), library_block ]
  __interface IFace
  {
     [ id(0) ] int int_data;
     [ id(5) ] BSTR bstr_data;
  };

  [ coclass, uuid("00000000-0000-0000-0000-000000000002") ]
  class MyClass : public IFace
  {
  private:
    int m_i;
    BSTR m_bstr;

  public:
    MyClass();
    ~MyClass()

    int get_int_data();
    void put_int_data(int _i);
    BSTR get_bstr_data();
    void put_bstr_data(BSTR bstr);
  };

Doesn't it send shivers down your spine? You might say I'm not consequent, because I just lashed out on excessive XML configuration files (here), but when presented with the alternative, I don't like it either! And now, there are annotations in Java too, doing a similiar thing... What I can say, I was growing up with the classical paradigm of the standalone programm using external functionality through libraries. But nowadays you don't program just for the machine or the OS, you program for same giant framework! Did you hear the phrase "programming by bulldozer"? And to my mind there's no satisfactory model of how to do this! For me both the configuration file madness and mixing code with meta information are both somehow unsatisfactory. Look at this example JBoss Seam action class:

  @Stateless@Name("manager")
  public class ManagerAction implements Manager {
      @In @Out private Person person;
      @Out private List<Person> fans;
   }

Well, what do you think, does it look OK? We have (nearly) POJOs here (i.e. no framework class derivation) which is the Holy Grail in Java programming lately, but the code is littered with annotations. To me they are no better than the old C++ event macros used in some very old Windows frameworks!

Now the question arises: is that a cognitive bias caused by the preceeding simple (and thus elegant) programming model? Remember, even Herb Sutter, when charged with the task of making .NET programming possible with C++, didn't embrace the traditional program + library model! He rather created a language extension, the C++/CLI langauge dialect. Maybe he was constrained to do so by his commision of getting Managed C++ up to speed, but on the other side, in the design paper he argued it's the natural way and compliant with the spirit of C++. Personally I didn't like this language extension a bit, opting for a carefully designed library or framework oriented solution, but maybe it's simply not possible???

Sorry, I digressed. So back to the subject!

The second species of attributes I knew were the Gnu attributes, looking like that:

  void kill(int) __attribute__((__noreturn__)); //GCC specific

but I never used them either, because I thought they were only of any use with som Gnu-specific langauge extensions, which is always avoided like the plague. And do you know how the new standard compliant attributes look like now? Look ("C++09 Working Draft n2798", 10.2008, Chap. 7.6.):

  class C
  {
    virtual void f [[final]] ();
  };

It's a crossbred of Microsoft an Gnu annotations, isn't it? That's why everyone is happy, and why the syntax is really ugly. And do you know what? There are a fistful of annotations defined in the standard (but vendors are allowed to add new ones) and one of them is, you guess it, final! Could someone tell me why should this one be an annotation and not a keyword - is it supposed to be matter in specific environments? Or are we making the way clear for vendor specific language extension? So maybe the whole annotation thing is in there to placate the major vendors: Microsoft and Gnu? Because as I got it (correct me please if I'm wrong!) there isn't a mechanism to plug in an attribute processor as in Microsoft COM-attributes or Java's annotations, so the whole construct is aimed at the compiler writers.

Summing up: first, Java annotation syntax, as seen in the Seam example above, seems to be more pleasing to the eyes, and second, I didn't really grasp the need for standardized annotations, except for vendor's conveniency.

2. The second one

The second curious thing I stumbled upon is the removal of Concepts (or should say of the "Concepts" concept?) from the C++09 standard proposal. As I read**, it was the standard library where the Concepts were first removed from. Come again? As I had a look at the draft, all the std library chapters were strewn with concepts, every one of them in pretty blue (or was is green?) colour.

And wasn't the Concepts supposed to be the best thing since sliced bread? I mean not for me, as I had a look at it one, and didn't want to learn it (in patricular the whole concept_map stuff!!!), but for the compiler writers - they could at last avoid the horrendous error messages for type errors in the template instatiations. Beside of that, it was Bjarne's pet project, and I thought that all Bjarne's PhD and post-doc students were working at it!

And now, out of the blue, the whole feature was cancelled! So maybe there's a limit to the pushing of the type system? C++ is a relentless type-monster already, and sometimes programming it feel like writing a mathematical proof to me. Thus building another level of type definitions on top of the generic meta types is maybe too much of the good thing. But, OK, the idea is not to take out the "duck typing" altogether from the templates, but to strike a delicate balance: add an minimum of concept decalarations and get the best error messages and type security possible. Is that task simply to complicated and too big or is it only the current design of the mechanism which is flawed?

As Bjarne said in the paper which finally did the deathly blow to the Concepts***:

Concepts were meant to make generic programming easier as well as safer. It is part of a whole collection of features aimed at simplifying GP, together with template aliases, auto, decltype, lambdas, etc. However, "concepts" is a complex mechanism and its language-technical complexity seems to be leaking into user code. By "language-technical complexity" I mean complexity arising from the need of compiler/linker technology rather than complexity from the solution to a problem itself (the algorithm).

My particular concern is that in the case of concept maps, in the name of safety we have made templates harder to use. We require programmers to name many entities that would better be left unnamed, cause excess rigidity in code and encourage a mindset in programmers that will lead to either a decrease in generic programming (in favor of less appropriate techniques) or to concepts not being used (where they would be useful). We have overreacted to the problems of structural typing. ***

This means that exaclty what I wasn't willing to learn leaked out into the user space! Bjarne made a series of proposals how to avoid it, but my general feeling after having read the paper was that the concept's design is somehow flawed, and cannot be fixed by some quick workarounds. So it's no wonder that the commitee had to make a tough decision. And that after years of work! And nobody have seen it coming all these years? Don't you think it's unacanny - how little we can do as to design a complicated language feature and to avoid mistakes? Or maybe we see here how badly the standardisation process is working. These problems were there for a much longer time and nobody said a word! Because of lack of time, because of politics, or simply because noone really understood that topic? So it had to be the creato of C++ himself to blow the whistle:

The use of concepts is supposed to help people write and use a wide range of templates. The current definition of concept maps and the philosophy that seems to go with them makes it harder.

Addressing this is important. I suspect that the alternative is widespread disuse of concepts and libraries using concepts. I would consider that a major failure of C++0x. ***

I'd say he killed his pet project out of sense of responsibility.

And where is the promised deeper meaning? As I said previously: there's maybe a limit to the type system. Because for me it felt sometimes like constructing inaccessible cardinals on top if the regular ones (wiki) in the set theory - just very, very subtle. And maybe the horrendous error messages for templates are just the proverbial price of the (type) freedom, and we have better to pay it? When you read Bjarne's article*** you see, that the gratest problems are encountered in sublassing and conversions - so C++ is much more "modern" a language than most of us would like to believe, as it allows us so much of the type freedom using the type system itself to achieve it!

And maybe the design problem relatively simple to solve, but the archaic linker technology we inherited from C is simply too old for that? As it reads:

"... complexity arising from the need of compiler/linker technology rather than complexity from the solution to a problem itself (the algorithm)"***

---
* Danny Kalev "Introducing Attributes", http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=440
** Danny Kalev "The Removal of Concepts From C++0x", http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=441
*** Bjarne Stroustrup "Simplifying the use of concepts", 2009-06-21, http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2009/n2906.pdf

For German Speakers only...

2009-06-27T06:58:00.000-07:00

Hi everyone! As it comes, I stumbled recently upon an interesting C++ website, which I quite enjoyed, and even happened to learn something new in the process! Its address is:

http://www.kharchi.eu/wiki/doku.php?id=cpp:start

I particularly liked the coverage of some more modern topics liker my favourite theme of Web programming in C++, embedded C++ scripting, or the string tutorial covering conversions between different encodings (and not to forget the link to the Pirates Party :)). Don't get me wrong, it's not the uber C++ webpage, but it's a cute, interesting little thing!

Unfortunatlely it's all in German :-((. Good for me (see, learning languages pays off sometimes), bad luck for the rest of you. So if you're speaking German, have a look at it!

Speaking about languages, there's a little story as a goodie: on some project several years ago I had to update an open source tool for copying entire websites. It was thousands and thousands lines of plain, spaghetti-style C code. As this wouldn't be bad enough, all the comments that were there weren't in English, no, they were all in French!!! Hadn't I learnt some French for my holidays, I couldn't possibly have this assignment finished! Without comments the code was absolutely incomprehensible! However it worked fine, even if it wasn't any OO or structured programming thing. Why? For all we know it shouldn't ;-))).

Windows not so bad after all?

2009-05-28T05:31:00.000-07:00

This post is for my distinguished hacker colleague Stefan Z. Do you remember how we were always complaining that you cannot do any serious development on Windows, as the following standard Unix command line isn't posible?

    cat out.txt | grep ERROR

Well, as I'm doing Windows programming now, I recently found out that it's possible after all! Incredible but true! Hovever the syntax is somehow different:

    type out.txt | findstr ERROR

On things like that I see how fast the time is passing! A couple of years before, I'd never believe that Windows would ever catch on. I think, the tide change came with XP.

But wait, it comes even better! Here's a script for checking if a process is running and killing it if so:

   tasklist /FI "IMAGENAME eq myProcess.exe" | findstr myProcess.exe
   if ERRORLEVEL 1 GOTO next

   echo "-- kill running myProcess"
   taskkill /F /IM myProcess.exe

 next:
   echo "-- starting the XXX process"

Even a sleep is possible (athough it's arguably very ugly):

    @ping 127.0.0.1 -n 2 -w 1000 > nul

And here (in the IT-Dojo) you can see that the Windows command promtp has some interesting history mechanisms!

Two Java Collection Utilities

2009-05-19T03:02:00.001-07:00

Lately I stumbled upon two iteresting Java libraries which add search capabilities to Java collections: one adding SQL search and another adding XPath search syntax to an existing collection. I think it's cool - you can take a collection of Java objects and treat it in a high level way! I suppose they are implemented using introspection.

The first one is called JoSQL (http://josql.sourceforge.net/index.html) and can be used like this:

try {
    Query q = new Query ();
    q.parse(
        "SELECT *" +
        "FROM java.lang.String " +
        "WHERE length <= :stringSize " +
        "AND toString LIKE '%e%' " +
        "ORDER BY length DESC");

    q.setVariable ("stringSize", "5");
    List<String> names = new ArrayList<String>();
    String[] n = {
        "alpha", "beta", "gamma", "delta", "epsilon", "zeta", "pi", "chi"
    };
    Collections.addAll(names, n);

    QueryResults qres = q.execute(names);
    List res = qres.getResults();
}
catch(Exception e) ...

Here I got { "beta", "delta", "epsilon" } as result. The only problem I can see as far is that the library doesn't use generics. The other one is the somehow unintuitive way it's treating the "SELECT *" query. Nomally the JoSQL library treats a collection of Java objects as a table which columns are formed by the values returned by the class' methods. However, in case of a "SELECT *" query, we won't get all of the object's attributes, but just the whole object as it is. On the other side, it's the equivalent, isn't it.

The second one is called jXPath (http://commons.apache.org/jxpath/), and is part of the Apache project. You can use it like that:

    JXPathContext context = JXPathContext.newContext(employees);

    Employee emp =
       (Employee)context.getValue("/employees[name='Susan' and age=27]");

Pretty nifty, isn't it? But while SQL query syntax seemed obvious to me, the XPath search expressions seems plain ugly and non-intuitive. I don't know both SQL and X-Path very good, but I find myself always forgetting pretty everything about XPath while SQL gets stuck. Maybe because XPath is too low-level an too much a hack?

You can read in many blogs that the above libraries are pretty the same, but in reality, the XPath query syntax is better for hierarchical data, somehow like that:

    Employee emp =
        (Employee)context.getValue("/departments/employees[name='Johnny']");

I assume you can do the equivalent in JoSQL, but you'd like to use joins, an that isn't that convenient for simple hierarchical structures like that above. So, in the end, maybe the XPath isn't so bad after all?

From DLL- to XML-hell

2009-04-29T13:22:00.000-07:00

This is one of the original articles I wanted to write when I started this blog. Well, it's taken some time, it's a little outdated now, but at last I've kept my promise (or rather one of them) !

1. The Lament

As I switched from C++ to Java in some previous project I thought I plunged directly from the DLL- to the XML-hell! Well, not exacltly (as I didn't programm under Windows in the preceeding project) but a little hyperbole makes a good start! The XML part of the title however, is true. The very first thing I noticed, was the ubiquitous XML - you had to use it for compilation (Ant scripts), you had to use it for for deployment (web.xml), you had to use it for your application (struts2.xml, spring.xml, log4j.xml).

The most annoying part of this was the change from makefiles to Ant scripts: so much superfluous verbiage! Look, compare the Ant script with the equivalent Ant-builder script in Groovy, taken from the "Groovy in Action" book, first the XML file:

<project name="prepareBookDirs" default="copy">
  <property name="target.dir" value="target"/>
  <property name="chapters.dir" value="chapters"/>
  <target name="copy">
    <delete dir="${target.dir}" />
    <copy todir="${target.dir}">
      <fileset dir="${chapters.dir}"
        includes="*.doc"
        excludes="~*" />
    </copy>
  </target>
</project>

Then the builder script:

  TARGET_DIR = 'target'
  CHAPTERS_DIR = 'chapters'

  ant = new AntBuilder()
  ant.delete(dir:TARGET_DIR)
  ant.copy(todir:TARGET_DIR){
      fileset(dir:CHAPTERS_DIR, includes:'*.doc', excludes:'~*')
  }

Well, that's at least readable (and writeable), and it states exactly what you want! When I see such a comparison I think that the XML code should be used only as Ant-internal, assembly-code level data representation! You could maybe find it interesting that even the creator of Ant himself admitted that using XML was "probably an error"*.

The second problem was that, (as another programmer put it):

Developers from other languages often find Java's over-reliance on XML configuration annoying. We use so much configuration outside of the language because configuration in Java is painful and tedious. We do configuration in XML rather than properties because...well, because overuse of XML in Java is a fad.**

I can only confirm this. You can get a feel of how much of XML "code" was needed in a Java web application I was then writing from the following (addmittedly pretty old) table*** :

     Metric               Java +              Ruby +
                     Spring/Hibernate         Rails 

  Time to market     4 months, approx.      4 nights,
                      20 hours/week       5 hours/night
  Lines of code          3,293                1,164
  Lines of config        1,161                113
  Number of classes/     62/549               55/126
  methods

Do you see? The configuration, it's 30% of the code (!!!), and that's 10 times more than was needed in Rails! That's some dependency! By the way, note that Rails needs only 3 times less code that Java, despite the fact that Java libraries can be sometimes pretty low level!

Now, for me the Spring 2 platform was somehow an apogeum of the XML overdependency - the endless configuration files weren't even human readable, you should better use a graphical Eclipse plugin:

to edit the wirings. That's like you need something like UML to tame the complexity of the XML files (and of Spring's configuration of course):

A rhetoric question: isn't it too much of a good thing for something as simple as configuration files? I found the Spring-2 MVC (Spring's web GUI application framework) particularly bad in that respect. Comparing it to the Struts 2 configuration, I had an impression that every obvious fact must be expressed in XML, and that there's no defaults altogether:

<bean name="userListController"
    class="de.ib-krajewski.myApp.UserListController">
    <property name="userListManager">
        <ref bean="userListManager" />
    </property>
    ....
</bean>

<bean id="userListManager"
    class="de.ib-krajewski.myApp.UserListManager" />

and so on, and on, and on.

But wait, it comes even better: some people are so enamoured with XML that they are using it as a programming language****, either doing patterns with Spring and XML configuration files, or even inventing an XML-based programming language and writing its interpterer in XSLT (I'm not joking!).

A word of caution: Java community noticed the problem and is trying to reduce the amount of XML needed. Unfortunately, I haven't got any hard numbers, but each new release of a Java framework seems to claim a "reduced XML config file size" - take for example Spring MVC v.3. However, as the DLL hell is still a reality despite of Microsoft's efforts and the introduction of manifests (really, look here!), equally, for my mind the overdependence on XML won't vanish overnight from the Java world - it's at its core!

2. The Analysis

2.1 XML data

Why are we all using XML? The received standard answer is: because it's a portable, human-readable, standardized data format with a very good tool support!

So why does it all feel so wrong? The first reason I see is that XML is too low level for human consumption. It's claiming to be human-readable while it's only human-tolerable - see the Groovy AntBuilder example above. There are tons and tons of verbiage (although that's something a Java programmer will be rather accustomed to ;-)) which is what machines need, but it's not how human mind is working. It needs high level descriptions because it's only all to easy distracted by unneeded details.

When I recall my own config file implementations of yore, I conclude that I never needed more constructs than single config values ans lists of them. And I never needed hierarchical config data for my systems. So if people are using XML for configuration, they are maybe after something more than pure config settings, maybe they are looking for a scripting solution for the application (as mentioned in*) along the lines of the late Tcl? But with Tcl the model was entirely different: the Tcl scipts glueing together passive modules of code, whereas now, the modules are active and use XML as a database of sorts... So is it that at last? People using XML data as a poor man's, read only database? It's a good, ad-hoc solution, and you can emulate hierarchical, network and OO databases without much of a hassle, can't you? So people are using XML as a kind of qiuck and dirty hack, and hacks aren't normally very pretty ;-).

2.2 XML in Java

The second part of the question is: why are we using so much XML in Java? The answer is probably that we need XML config files to increase modularity an require the loose coupling of our systems. The Spring framework allows us to pull the parts of the system together at the startup or run-time, and that's supposedly a good thing, isn't it.

Coming to this part of the question: I think that the loose coupling thing is overrated. As Nicolai Yossutis said in his book about SOA (we reported ;-)):

"...loose coupling has its price, and it's the complexity."

So firstly, a little bit of coupling isn't that bad if it decreases complexity and increases readability of code! To my mind, it's a classical tradeoff, but as some developers are only too prone to introduce tight coupling all over the place, the received wisdom is to avoid the coupling at every price! But be honest, you can't decouple everything, if the parts are ment to be working together! Secondly, I thing that the "inversion of control" pattern is a bit of overkill too.

Personally, I'd like to glue together the parts of the system directly in code, so that I can see how my software is composed, and that in my source code - it should be the sole point of reference! Thus I like much more the approach Guice or Seam are taking (and nowadays the Spring framework itself) - that of annotations. At last we have the relevant information where it belongs - in code! The problem here is, that the annotations aren't really code, they are metadata, and I don't like the idea of using metadata where you could do things directly by using some API. But hey, that's probably the way we are doing such things in Java...

3. A Summary

That's what I wanted to write approximately 2 years ago. Meanwhile I use XML everywhere in my architectures and designs, mainly because it excludes any discussions about formats! No one has ever said anything against XML!

But programming it (in C++ and Qt) is a real pain - I first tried DOM, than SAX - and it was an even greater mess (well, there's a C++ data binding implementation á la Java's Apache XMLBeans but it's not used at my client's :-((). XPath's selectors are supposed to be better, but Qt doesn't have it by now. The good thing is, everyone will think the system is sooo cool because it uses XML, and I have to make my users happy!

PS: And you know what? The new dm application server by SpringSource uses JSON for configuration files!!! I told you so ;-)

--
* "The creator of Ant excercising hit own daemons", this article was originally stored under http://x180.net/Articles/Java/AntAndXML.html, but this link seems to be dead, so I'll host it on my website for a while (hope it's OK...): http://www.ib-krajewski.de/misc/ant-retrospect.html

** as a matter of fact, he's a Ruby programmer too, so he isn't that unpartial: http://www.brainbell.com/tutorials/java/About_Ruby.htm

***taken from the book "Beyond Java" by Bruce Tate: http://commons.oreilly.com/wiki/index.php/Beyond_Java/Ruby_on_Rails, unfortunately the original link stated there isn't working: Justin Gehtland, Weblogs for Relevance, LLC (April 2005); http://www.relevancellc.com/blogs. "I *heart* rails; Some Numbers at Last."

**** "Wiring The Observer Pattern with Spring": http://www.theserverside.com/tt/articles/content/SpringLoadedObserverPattern/article.html, and "XSL Transformations. A delivery medium for executable content over the Internet" in DDJ from April 05, 2007: http://www.ddj.com/architect/198800555 . I cite from the latter:

XIM is an XML-based programming language with imperative control features, such as assignment and loops, and an interpreter written in XSLT. XIM programs can thus be packaged with an XIM processor (as an XSLT stylesheet) and sent to the client for execution.

See, I wasn't joking...

Some distributed computing news: cuddly elephans, pigs, nymphs and couch computing

2009-04-04T12:09:00.001-07:00

As I did some distributed computing research in my previous life, and then tried to get involved with Google, I'm still somehow interested in the subject. If you are into distributed computing, you'd know that Google is using a distributed C++ system called MapReduce* for indexing of the crawled webpages. It happens that there's an open-source Java implemenation of Google's system (based on published Google's research papers) called Hadoop . Up to now I didn't know about any serious usage of Hadoop in big distributed systems, but as it comes, I learned that Yahoo! is using Hadoop internally** for spam detection!

Yahoo's researchers have also designed a new language called PIG Latin (?!)*** for distributed programming on the Hadoop platform. The idea is rather an interesting one: they pointed out that the existing way of writing programs for the mapper and reducer part of a distributed application is too low-level and not reusable, but somehow familiar to programmers. In contrast, a declarative language like SQL poses some problems, so the paradigm there is completely diffrent:

At the same time, the transformations carried out in each step are fairly high-level, e.g.,filltering, grouping, and aggregation, much like in SQL. The use of such high-level primitives renders low-level manipulations (as required in map-reduce) unnecessary.
...
To experienced system programmers, this method is much more appealing than encoding their task as an SQL query, and then coercing the system to choose the desired plan through optimizer hints.

So it provides a middle ground, where we can avoid SQL and write down the computation steps, but can do that at higher level!

Unsuprisingly, Microsoft already has a proprietary distributed computing platform called Dryad, which tries to do something like this as well, but is using C# and LINQ (i.e. an SQL-like query language):

DryadLINQ generates Dryad computations from the LINQ Language-Integrated Query extensions to C#.

But I don't know how much this is a research project and if it's really used internally. On the other side, Microsoft recently aquired a search company called Powerset, which is using Hadoop at its core, and as the gossip runs, should be used to bring Microsoft's LiveSearch up to speed.

Facebook, in their turn, are using Hadoop, but added a Business Intelligence tool called Hive (now released as open source!) on top of it. It uses an SQL-like query language, as BI-users understand it best. Beneath the surface Hive leverages Hadoop and translates SQL-like imperatives into MapReduce jobs.

Not bad, cuddly elephant! It seems everyone is loving you!

Accidentally, when speaking about SQL and non-SQL: there's a non-SQL database by Apache called Couch DB. It is written in Erlang (we reported already**** ;-)) and implements a distributed document database. It's designed for lock-free concurrency using Erlang's "share-nothing" philosophy. To my mind it resembles Linus' Thorvalds git source configuration system, as it holds all the versions of documents determining the "winner" to be visible in defined views. Hear, hear! Erlang's alive, distributing and kicking!

Another thing worth noticing is that's not only SQL anymore, there are attempts to do things differently, athough others are sticking with the ol 'n reliable. Paradigm change on its way? It's interesting times I think.

PS (6 Okt. 2009): Look, look, now even the Big Blue has jumped the wagon too! They offer the M2 enterprise data analysis platform, based on Hadoop and using PIG. See: http://www.sdtimes.com/link/33808. So it seems Hadoop is definitely mainstream and business analyst thingie now, and as such it has lost much of its appeal, as far as I'm concerned.

---
* Jeffrey Dean, Sanjay Ghemawat: "MapReduce: Simplified Data Processing on Large Clusters", OSDI 2004, http://labs.google.com/papers/mapreduce.html
** unfortunately it's a two-parts video: http://developer.yahoo.net/blogs/hadoop/2009/03/using_hadoop_to_fight_spam_-_part_1.html
*** Christopher Olston et al: "Pig Latin: A Not-So-Foreign Language for Data Processing", SIGMOD 2008, http://www.cs.cmu.edu/~olston/publications/sigmod08.pdf
**** http://ib-krajewski.blogspot.com/2007/08/erlangs-change-of-fortunes.html

One cute microdesign example and some thoughts on language design.

2009-03-30T01:06:00.000-07:00

This time I'll begin with a citation of another blogger*:

I made the switch from C++, I'd wished that Java allowed operator overloading (i.e. custom implementations of operators such as '+', or the array brackets). However, some argue that this feature adds to the complexity of C++ with little advantage, and that there is no need to pass this complexity on to the Java language. Someone who never coded in C++ would more than likely agree with that.

OK, someone who never coded C++ won't understand that, and will take the argument about the undue complexity as received wisdom. So let's show you a little piece of design, a microdesign actually, so you can (maybe) appreciate the feature. If you're a Python/Perl/Groovy (or C++ for that matter!) programmer, I must apologise, but maybe you'll find this small example nonetheless instructive.

1. A piece of microdesign

It's a piece of code from my old project I wrote several years ago. I wanted a class for interthread and interprocess messages (you know, for my favourite pattern: share-nothing, message-passing actors). I wanted to be able to construct a message using parameters of the constructor like this:

    AbstractData notif;
    SocketMsg msg(DispInterface::crReqNotifType, &notif, father);

And then simply send the formatted message as binary data:

    m_connSocket.send(msg, msg.buffsize())

On the other side, I wanted to receive the binary data from socket directly into the message object, which would then parse it and provide high level access to the received data, just like:

    SocketMsg msg; // empty
    if(!m_connSock.receive(msg.writebuff(), msg.buffsize()))
    {
        // if too much errors, better bail out here!
        TRACE_AND_CONT("Cannot read from socket in an FD event handler!",
                       E_ID_Q3A_SOCK_ERR); // try to escalate
        return;
    }

    // look what message received
    if(msg.getType() == cmdOkMsgType)
        handleCmdResp(msg.getAbstrData());

Pretty high level, eh? Just hide away all that low level data parsing and formatting and use the message!

It's design on its lowest level, you cannot possibly make a smaller piece of meaningful design, but I liked that piece! Now a question for the astute reader: what pattern is that? Proxy, bridge or a facade? Or a builder maybe? Perhaps none? Well, I don't think in pattrens, I think in abstractions: what I needed was an abstraction of a self-formatting message. Don't you like it too? Unfortunately, you need some operator overloading magic to do this. Look at the following class:

class SocketMsg
{
public:
    SocketMsg(int type, AbstractData* message, DsetId dsetId = nillDsetId,
              int cmdId = 0, int userCode = 0);
    ... etc

    // conversions
    operator const char*() const { return m_charRepr; };
    inline SocketMsgBufferProxy writebuff(); // see inline section

private:
    // buffer
    static const int c_buffsize = sizeof(SocketMsgStruct) + 1;
    char m_buff[c_buffsize];

    // decoded data: don't remove in case we use different encoding!
    AbstractData* m_data;
    int m_type;
    DsetId m_dsetId;
    int m_cmdId;
    int m_userCode;

    friend class SocketMsgBufferProxy;
};

You can see in the above definition that the class has a buffer for the coded binary message and several fields for the decoded message values. In the char array context, it exports its internal message buffer for reading (through the operator const char*()) so the socket's send function can read the raw data from there. In the other direction, i.e. for receiving data into the internal buffer a different mechanism is used: we are returning a SocketMsgBufferProxy object (this time through the writebuff() function) , which is defined as follows:

class SocketMsgBufferProxy
{
public:
    SocketMsgBufferProxy(SocketMsg* sm) : m_sm(sm) {};
    operator char*();
    ~SocketMsgBufferProxy();
private:
    SocketMsg* m_sm;
}

What it that proxy object doing? It's forwarding the address of the internal raw data buffer and revealing it through the operator const char*() again. As a matter of fact I could offer an even cleaner interface:

    m_connSock.receive(msg, msg.buffsize())

So why didn't I simply expose the internal buffer with the char* conversion operator and introduced a SocketMsgBufferProxy object inbetween? Simply because of the SocketMsgBufferProxy's destructor. When its destructor is invoked, it rescans the raw contents of the receive buffer so after the receive() line of code is finished, the message object is parsed and ready! Of course there is another possibility to achieve this: lazy parsing, but I was somehow more excited with that cool proxy object automatically coming inbetween and doing all the work! I'm only a human too and sometimes looking for the unusual...

2. And now some more general discussion

So as you can see, the operator overloading is a pretty useful little feature. So why didn't the designers of Java include it into the language definition? My opinion is that they overreacted. I can still remember the times of the “object craze” when everybody tried to be object oriented without knowing what that really was, and to use every single feature C++ offered without thinking if it's needed or not for the particular problem at hand. That was somehow similar to the “patterns craze” I can see even today: people asking me “so what pattern could I use here?” - you don't need a pattern here, you need a solution of the programming problem, or do you want just a cool name to stick it on your code? But more about that in future entry.

You wouldn't believe how stupid the people got with the operator overloading, maybe it was something like the ”powder rush” of the skiers, an incredible infatuation with the until-then unheard-of possibilities! You must consider that a fresh-made C++ programmer came from a much smaller language (namely C) and normally didn't have exposure to any higher level language concepts form Smalltalk, Lisp or Simula. So the Java designers simply thought “Hey folks, it's time to cool down!” and banned the feature. To my mind, a fatal case of underestimation of people's capabilities and of patronizing. Because the result of it is that “you cannot do any magic in Java!” (well, actually you can with introspection and bytecode insertion, but only so much, and that's simply not enough). Look at the following citation**:

In fact, I'd say that many of today's current hot trends in programming are a direct result of a backlash *against* everything that Java has come to represent: Lengthy code and slow development being the first and foremost on the list. In Java you generally need hundreds of lines of code to do what modern scripting languages do in dozens (or less).
...
So what's the solution? Well, this is the problem. People have been tinkering with Java for years now, and there's still no hope in sight. There's something about the Java culture which just seems to encourage obtuse solutions over simplicity. As a Java developer, I was always so amazed at how difficult it was to use the standard Java Class Libraries for day-to-day tasks.

And you know what? My pet idea is, that the reason why Java code is so dull and repetitive (and that the Java culture is, ehm..., you've just read it...) is the lack of operator overloading! You think I'm overly generalizing now? Well, the fact is, you simply cannot hide the gory details as conveniently as I was able to do it in the SocketMsg class! The best indication that the people in Javaland are thinking the same, is the Groovy JVM programming language: it has the operator overloading at its core, and is capable of doing things you wouldn't dream possible in Java. Well, that's no virtue in and by itself, but as result it gives you much leaner code! And we all need it: less code!

3. Now something even more general

When am already at it, Java programmers I met wouldn't understand the idea of the h-files (aka include files) too. Why are they needed for Heaven's sake? Isn't it an awful bore? I admit that the origin of the h-files lies in the limitations of the early C compilers, but with C++ it gained another significance: a “table of content” of the code proper. I give the word to another man from Java(FX)land***:

From my experience with C++ and Java, having method bodies in the class declaration clutters it with a mass of implementation details which is detrimental to getting an overview of the actual relationships and operations embodied by the class. It was for this reason that I decided to define the bodies of class operations and functions outside the class declaration.

Exactly what I think! You need an overview for the class, so you don't have to skip every one int and bool declaration until your head gets dizzy. You'll retort that this will be done with Javadoc comments. Well... The Javadocs... It's quite a theme of its own. Let me give only a small example here: if I've already written the function declaration like:

    void sendMessage(SocketMsg& msg, Receiver dest);

why should I repeat all this in a trivial Javadoc header? Is it "don't repeat yourself" or what? Don't get me wrong, I wouldn't check in any uncommented code, but I don't use them in (C++) header files - they only clutter up and make the definitions unreadable! For me they are well suited as implementation descriptions. And seconly, you have to run the javadoc tool first, you cannot just have a quick glance at the code! I must admit, sometimes I'm annoyed about having to edit the h-files, but when I'm reading the code afterwards, I'm happy I did that. Pretty non-progressive, isn't it?

4. Summing up

Maybe you'll say that it's not ethical to lambast a slightly ageing, slightly demodée, mainstream language, and that all is only too well known for several years now, but hey, it's just what I always thought! And like with my other favourite subjects (Agile, XML, Python, patterns), it simply takes time till I write it down, and it's no so insightful then, sorry...

--
* http://www.ddj.com/blog/javablog/archives/2007/08/java_and_potato.html
** http://www.russellbeattie.com/blog/java-needs-an-overhaul
*** Chris Oliver, Creator of JavaFX, in his presentation about JavaFX: http://jazoon.com/jazoon07/en/conference/presentationdetails.html

Fibers, at last!

2009-02-21T07:40:00.001-08:00

As I'm recently doing Windows programming again after some 10 years of a pause, I keep learning about new things. One of them are fibers. Come again? What's that??

A fiber, as the name implies, is a piece of a thread. More precisely, a fiber is a unit of execution within a thread that can be scheduled by the application rather than by the kernel. *

So fibers are just user space threads, aka cooperative multitasking, which I knew since the stone-age DOS times and which weren't supposed to be cool at all!

It were native kernel threads which were supposed to be cool back then! But I must admit that in the past as I was working with cooperative multitasking in a rather big, multithreading system, I was quite happy with it! It gave a nice, predicatble feel to the system; you just new at what times what threads are supposed to run! The sole problem was (of course) that a long running user task couls block all the system. But then we moved on to native threads and I forgot about cooperative multitasking.

But something has changed since then: first, as it was already said, Windows has got it, although allegedly only as a good-enough implementation to support the SQL-Server (fibers have severe restrictions on the kinds of Win32 API’s they can call!). Second, Oracle 10g has got it too, I cite**:

Fiber model support
- An extension of the thread support already in place since Oracle7. Users may now run the database in fiber mode, which employs Oracle-scheduled fibers instead of O/S scheduled threads.
- For CPU intensive apps, this will provide a performance boost and reduce CPU utilization.

Moreover, recently I discovered a fiber library for Windows XP to remedy the unsufficient native Windows fibers support named FiberPool***. And then there are Green Threads (user space level, cooperative multitasking threads) in Java's JVM for some time now! The recent Windows 7 beta has introduced User Mode Scheduled Threads (UMS Threads)**** , which are like Win32 Fibers, as they can be scheduled by hand, but are backed by real bona-fide kernel level threads! Sounds rather interesting!

So at last it's the old, good cooperative mutitasking rediscovered? There's a growing evidence! Will be the ol' good times of DSET maybe back then Stefan?

---
* Johnson M. Hart: Windows System Programming Third Edition, Addison Wesley 2004
** http://download.oracle.com/owsf_2003/40171_colello.ppt
***http://www.fiberpool.de/en/research.html
**** http://blogs.msdn.com/nativeconcurrency/archive/2009/02/04/concurrency-runtime-and-windows-7.aspx

C++ Server Pages

2009-02-18T04:24:00.000-08:00

In my prevoius blogs I wondered if there is/could be a C++ implementation of the Java's Servlet interface(s)*, and recently a friendly fellow blogger** pointed out to me that there already is one, viz the CPPSERV: http://www.total-knowledge.com/progs/cppserv/!

Can you see C++ it its logo?

I cite*:

CPPSERV is a web application server that provides Servlet-like API and JSP-like functionality to C++ programmers.
...
CSP (C++ Server pages), the JSP equivalent for CPPSERV, are currently implemented as a precompiler that converts CSP source to C++ source of a servlet, which, in turn needs to be compiled into CPPSERV servlet.
...
Current CSP implementation supports basic parsing, as well as compile-time taglibs.

Well, that's it! Or perhaps not? What we have is the equivalent of the Java's JSPs, which generally suck, and standing alone are too week a framework for serious work. But hey, it's not that bad, as we have got some choices: the CPPSERV for the oldskool JSP-like programming, the late Bobcat for more of the same, or the Wt framework*** for more modern, GWT-like work!

And with some discipline (I mean the model 1 ;-))) servlets, do you remember???) it's possible to get working application in C++ quickly! And all that Java frameworks out there are only trying to hide the JSP processing model, and they aren't that beautiful themselves. So for smaller things a JSP-like solution will be probably sufficient, and for more modern applications (including AJAX support) I'd choose the Wt framework.

There remains the question of the OO-Relational mapping, an the creators of CPPSERV suggest using their SPTK classes****, but it seems to be only a JDBC-like DB access layer. I suppose, I'd rather take Qt's DB classes for that, but on the other side, they are somehow heavyweight. But the OO-Relational thing for C++ is a different question in itself. But JDBC style access isn't that horrible either, and in a whole I think there are some viable options for C++ web applications out there!

Now I guess, I should try out the 2 frameworks, but ... ehm ... you know, so little time, so many books...

PS (07 Oct. 2009): meanwhile I have detected two another C++ and C "Servlets". Well they aren't really Servlets, but are rather one step back on the evolutionary path, and are somehow JSP-equivalent. The first of them is named KLone and allows you to include C scripts in the HTML page in the exact JSP manner. The other is GWAN - it's free and it allows you to write "servlet" scripts in C, which will be executed by the server and generate the HTML output - so in this case we are almost there, except for the very limited set of functions you can use in the script. Nonetheless, very interesting!

--
* see: http://ib-krajewski.blogspot.com/2008/10/c-servlets-again.html
** Gavino, unfortunately his profile isn't available :-(((, but nonetheless, thanks bloke!
*** The Wt-framework ("witty" - http://www.webtoolkit.eu/wt/)
**** SPTK: mainly a GUI framework, but it has some DB access support as well: http://www.sptk.net/index.php

Reading the new C++ Standard

2009-01-27T23:32:00.000-08:00

Recently I've read on my local German C++ forum* that's there was a new draft document for C++09 Standard** and immediately I rushed out to read it. I mean, something tangible at last! Well, actually it wasn't really a read, but rather a very superficial skimming-over. There are good reviews of C++09 on the Web***, so I didn't actually want to read it in depth, but rather to get an overall impression of the changes: just how it all would feel like in C++0x: x < a.

And guess what? It was last year in November (sic!!!) when I read it and then wrote some things up :-(((. Somehow I'm too busy with my current project! Of course I'd like to read the proposal in depth and analyze it here thoroughly, but alas, it's not possible. So let's start with just few of my very personal impressions:

1. Ranges and initializers

What I perhaps liked best of the new features! Now I can write things like:

    map<type-expr> m = {{}, {}, {}};
    int a[] = {1};
    vector<type-expr> v = { a, b, c };
    SomeClass obj({ a, b, c, d });

and that for any class I want! How is that done? You have to implement the initializer_list constructor for your class! Wow, it feels just (like) Groovy to me :-). A great improvement over the oldskool tricks used to force initialization where it wasn't possible by the language definition!

A realated feature I liked are ranges:

    for(int& x: array)
    {
        x = 0; // for example!
    }

This is accomplished internally by a Range<> class being internally instantiated with the array instance: Range<type-expr>(array). It gives you a nice, scripting language like feeling when iterating; I mean, even Java's got it, and the boost::FOREACH macro was in my personal opinion rather an ugly hack than anything else!

When talking about ranges: actually I was expecting that the ranges will be included for the STL algorithms, so we don't have to write the annoing:

    find(v.begin(), v.end(), not_zero());

each time. I was quite positive that we'll going to have something like:

    find(v, not_zero());

as an overload of the traditional STL signature using the new Range<> class (there is a boost::range library after all!), so I must say I'm a bit disappointed here.

2. Some nice utilities I found

There were suprisingly large number of small interesting things to be discovered, like:

error support: error_code, system_error - well, I can definitely use some of it!
compile time (i.e. template) rational arithmetic - suprise! BTW do we really need it?
clock and time_point classes - useful!
stoi: string to int conversion - a suprise!, so don't we take the road of lexical_cast<> as I was expecting all the time?
alignment control - at last I can say what the aligment is and not the machine! Will then the machines revolt?
unique_ptr - just what we need most often, or auto_ptr fixed!
addressof - even if the actual & operator is overloaded!
reference_closure as Callable - what was that again??? But I wrote it up in my original notes, so it must be something funny!

delegating constructors - we can reuse one constructor in another one, like in this example from the draft document:

  struct C {
      C(int){}     // 1: non-delegating constructor
      C(): C(42){} // 2: delegates to 1

nullptr as a keyword - no more the annoing (XXX*)0 casts!

I liked these little things, and there is probably more like that to be discovered. It's refreshing to see that the smaller problems of day-to-day programming are addressed too!

3. Some expected things

As expected we've got the hash tables at last, as actually the whole TR.1 stuff like regular expressions, type traits for templates, tuples, general binders and function objects, is in there too. Plus the move constructors and rvalue references.

Moreover the auto keyword and the lambda functions are there, so now we can write

    auto iter = m.begin();
    // and not
    map<string,string>::const_iterator iter = m.begin();

at last! Just think how many keystrokes/copy and pastes it will save!

Speaking of lambdas - as I read it, there isn't one example of a lambda function in the draft! And moreover, in "Chap. 14.3.1 Template type arguments" there's something missing: namely the old restictions about the non-global linkage of the template parameters! This means, I can write a functor in local scope and pass it to the STL algorithm like that:

    void XXX::someFunc()
    {
        struct my_not_zero {
            bool operator() (const X& p) { return p != 0; }
        };

        // use the local functor 
        find(v.begin(), v.end(), my_not_zero());
        // OR use a lambda function:
        find(v.begin(), v.end(), [] (const X& p) -> bool { return p != 0 });
    }

I'd say, what was the reason for lambdas again (at least for me)? To be able write a functor which doesn't have to have global linkage! Are lambdas obsolete then? Well, both yes and no. Just look at the code example above, writing a local functor still requires quite a lot of plumbing, and a lambda expression is somehow more terse (we could even spare the return type of bool in our example!). IMHO both of them are not wholly satisfactory. Using a lambda library solution I could write it in a much more terse way:

    // OR using my pocket lambda library:
    find(v.begin(), v.end(), _$1 != 0 });

Another expected (and somehow dreaded) new thing are the Concepts. I must say thery are everywhere in the standard library section, like in this new std::list move constructor:

    requires AllocatableElement<Alloc, T, T&&> list(list&& x);

I mean, now the standard library header files will get even less readable, but the compiler errors will be (hopefully) more so. So I'm in two minds about this feature: isn't it too much of the type system? What about the duck typing in templates? Are we giving it up? For me it was a very nice feature. I guess it won't come that bad, and the templates without Concepts will be working as before but are Concepts only for compiler support or are we supposed to use meta-typing everywhere?

The next expected thing, which however deserves its own section is:

4. Multithreading support

Sorry, I didn't read it yet... It's two whole chapters (29 and 30) ant they aren't exactly short.

But instead it started me thinkig: how was I writing multithreading applications before there was the C++ memory model defined? How on earth was this possible? Well, it was all vendor (i.e. compiler) specific. When a compiler supported the POSIX thread library (pthreads) for example, it would pose some memory fences around the pthread calls****. It would treat some pieces of code in a special way - just like in Java's String magic. The local static initializes would be automatically guarded by locks (Gnu compiler) or maybe not (Sun compiler). The volatile keyword would be given an extra visibility guarantee (Microsoft compiler). And I never initialized global static variables out of the multithreading part of the program, only in the main thread portion. An the best of it is, that all these techniques wasn't guaranteed to work****! But hey, the 1st Java memory model wa buggy too, wasn't it?

Now it's all supposed to be portable.

One thing which I did read up (however on the Web and not in the proposal itself), and what was a littel unexpected for me, is about the volatile keyword. The proposal didn't stengthen the volatile keyword in the way it's been done in Java or C#. I suppose that out of the reasons of backward compatibility with C's volatile remained only a compiler (dis)optimisation directive, while for the visibility guarantees between threads we are supposed to use the atomic<> library!

How will the old programs using volatile for visiblity between threads be running now? Are they broken? I suppose so, they probably must be rewritten using the atomic<> types.

5. Summing things up

My impression so far is, that there are many small things you wouldn't expect, and that they are fun! I was expecting the big new chunks of functionality to be added (and they were added indeed) to be the most impressive ones, but it's rather the small things which make the difference. Look at that code snippet in the new, cool layout style (which I picked up in this Ferbruary's issue of the MSDN Magazine) :

    for_each(values.begin(), values.end(), [](int& value)
    {
        // now we can nest STL at last! (or can we???)
        for_each(value.begin(), value.end, DoSomething);
    });

I'd dare to say that C++ seems to have put away its 90-ties image, as it allows to write programs more in style of the scripting languages than the old and venerable K&R C! I can only add "at last" and "Amen".

---
* Thanks Jens!
** Herb Sutter's announcement and some discussion with the inevitable C++ bashing: http://herbsutter.wordpress.com/2008/10/28/september-2008-iso-c-standards-meeting-the-draft-has-landed-and-a-new-convener/, the Draft document itself is here: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2798.pdf
*** like the article in the Wikipedia, which I found to be surprisingly good: http://en.wikipedia.org/wiki/C%2B%2B0x
**** Hans Boehm: "Threads cannot be implemented as a library" Technical Report HPL-2004-209, HP Laboratories Palo Alto, 2004, http://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf

Do we need destructors?

2008-12-11T00:01:00.001-08:00

This post is for my honoured, old-time, old-skool hacker colleague Stefan Z. As we were discussing the new and (then) hot Java language, we couldn't accept that it didn't have destructors. I cite Stefan:

The constructor/destructor pair is an incredibly powerful concept!

Well, you can't have everything: either we support garbage collection or destructors, isn't it? But it just one more point where Java sucks: the ugly try/finally block and then the explicit close() call like in the following code:

    BufferedReader reader =
            new BufferedReader(new FileReader(aFileName));

    try {
      String line = null;
      while ( (line = reader.readLine()) != null ) {
        // process the line...
      }
    }
    finally {
      reader.close();
    }

Ugly? You bet! And what I really don't like is that you cannot hide all the required handling in a library! In C++ you'd just write:

    ifstream f(aFileName);
    string line;

    while(f)
    {
      getline(f, line);
      // process the line...
    }

That's all, the plumbing is hidden in the destructor* and it is there automatically! It's the reason why Stefan called this concept an "incredibly powerful" one. But that powerful concept can cause problems in a multithreading and garbage-collected environment. As a matter of fact, a recent C++ standard proposal for the multithreading execution model opted for removing destructors from the language (!!!), or at least for not executing the static destructors in a multithreading setting! Of course, it's a shortcut in order to solve a rather complicated problem, but you get the idea, right?

So maybe the destructors are a little bit outdated, what do you think Stefan? All the more was I pleased as I recently stumbled on a Smalltalk pattern called "Execute Around Method" pattern** (to be true, I don't do any Smalltalk and have seen it in some Groovy example code). It's another possiblity to hide the plumbing in the library: you just define a static method doing all the dirty work and accepting your "payload" code - just like in an IP packet: we have the framing and the payload, and the user is only delivering the payload! Well, an example explains it best:

    def static use(closure)
    {
      def r = new Resource()
      try
      {
        r.open()
        closure(r)
      }
      finally
      {
        r.close()
      }
    }

The closure parameter stands for our "payload" code. This code is the hidden in the library. An application of this is the following Groovy code:

    new FileReader(aFileName).withReader
    { reader ->
        reader.readLine(line)
        // process the line...
    }
    // no need to close()!!!

We create a new reader, give it to the static withReader() library method, and provide a "code block" (as you'd call it in Perl) for execution. This code block (called closure in Groovy) gets as the parameter the ressource which will be closed at the end, just like the use() method shown above!

A destructor for the modern times! So the "incredibly powerful" idea can be saved?

---
* this is called a RAII-pattern in C++, see: http://www.hackcraft.net/raii/
** Kent Beck: "Smalltalk Best Practice Patterns", Prentice Hall, Englewood Cliffs, NJ, 1996.

C vs C++ and some celebrity gossiping

2008-11-26T23:26:00.000-08:00

Every time I read a post of Linus "Linux" Torvalds I can't help thinking "what a smug, assumptuous, xxx-yyy-zzz!". Well, I don't know the man personally, but I certainly wouldn't like to have him as my boss in any project, betcha! The first quote is a couple of years old (I cite from memory as I cannot find it anymore) and was a reply to some proposal Linus didn't like :

....go and play in your little world...

It looks innoculous enough here and now, but in the context it was realy ugly. And now, for some time everyone seems to feel obliged to speak about Linus' C++-hating post*, so I had a look at it myself. OK, nothing changed, it goes in the same vein:

*YOU* are full of bullshit. ...... is likely a programmer that I really *would* prefer to piss off, so that he doesn't come and screw up any project I'm involved with. ...... the end result is a horrible and unmaintainable mess. But I'm sure you'd like it more than git.

... etc, etc, etc. OK, maybe it's only his personal creative writing coach who's to be blamed, or perhaps it's the macho Linux kernel developer culture? But, aside of personal dislike, what the man says got a bell ringing with me. Why? Read on:

C++ leads to really really bad design choices. You invariably start using the "nice" library features of the language like STL and Boost and other total and utter crap, that may "help" you program, but causes:
infinite amounts of pain when they don't work (and anybody who tells me that STL and especially Boost are stable and portable is just so full of BS that it's not even funny)
inefficient abstracted programming models where two years down the road you notice that some abstraction wasn't very efficient, but now all your code depends on all the nice object models around it, and you cannot fix it without rewriting your app.
In other words, the only way to do good, efficient, and system-level and portable C++ ends up to limit yourself to all the things that are basically available in C.

Whoa, that man is really hardcore! What he's actually saying is: don't trust any code you didn't write by yourself! And on a higher level: any abstraction we are using is a trap, lulling us in a false sense of security. And more: we can really build big, fast, complex systems without using OO abstractions!

Didn't I feel the same before? That for the efficient, near system level code we can take C, and that all the fancy object thing, where the is better done in Ruby or (even) Java? So no place for C++ here? Take Wireshark protocol analyzer as example?

Gossiping

Well, the story doesn't end here. First, there are some entertaining comments on digg**. My favourites are:

Linus codes a kernel for a living, so its not that surprising that he hates C++.
Linus is an *****, he lives here in Beaverton and the man has a big ego for someone nobody outside of the Linux community cares about.
Ok... Who cares if Linus Torvalds hates C++. I don't really give a damn.
another episode of "I am Linus and hate everything"

funny that linus prefers kde when it is programmed in C++
Personally I've evolved thinking in OO abstractions, so working with C++ is much more natural for me than C. Does that mean I'm a crappy programmer? Only Linux Torvals knows

The STL is the biggest piece of crap I've seen in 40 years of programming. It's a graduate students project to prove one can write a totally orthogonal, yet totally inefficient, impossible to maintain, piece of crud.

Well, as it seems, first: people aren't taking Linus such seriously, and second: STL ist the culprit!

Living in the past?

So, as to begin with something, what's the matter with STL? As we are gossiping in this installment, it's perfectly fine for me to say that the prevalent opinion on the Web (for example ***) is that Linus is referring to a problem from the past (around 2001 or so), when he's speaking abot the non-portability of C++. At that time the support for the C++ standard, and especially tempaltes, was very unconsistet across the compilers, and so the STL implementations could be nonportable between compilers! But nowadays even Visual C++ is quite up to speed here!

Then the inefficiency allegation. I don't even want to discuss it here, because it's so old (back in time to 1998 or so). There's long refutation along classical lines from that time to be found****, if only not very entertaining, and a shorter one*****, from a practitioner's point of view - Steven Dewhurst actually wrote low level code with C++ and templates:

Just to annoy people like Linus, I've also used typelist meta-algorithms to generate exception handlers with identical efficiency to hand-coded C. In a number of recent talks given at the Embedded Systems conferences, I've shown that commonly-criticized C++ language features can significantly outperform the C analogs.

Who's incompetent?

Next comes the critique that C++ tends to attract substandard programmers, and that:

... limiting your project to C means that people don't screw that up, and also means that you get a lot of programmers that do actually understand low-level issues and don't screw things up with any idiotic "object model" crap.

The first thought that comes to mind is Linus' "software Darwinism": in 2000 he lambasted people wanting a debugger in the Linux kernel. His argument was: I don't need any sissy that needs a debugger! I want people who understand the code as a whole! Any higher level abstraction or language (i.e. STL or C++) will make you a wimp and not careful enough!

The fact is, that is *exactly* the kinds of things that C excels at. Not just as a language, but as a required *mentality*. One of the great strengths of C is that it doesn't make you think of your program as anything high-level...

But isn't this just another management whip for the programmers to keep them under pressure, so they are more obedient? A manager's trick? The Linus' software management process? I'm most hardcore of you all, so I'm the overlord ;-). In that light Linus' diatribes are only politics: he's defending the status quo.

There's also a diffrent response to the "substandard programmers" reproach I must mention here. Steven Dewhurst broght in the point, that for a C programmer C++ is so complex because there are alien idioms, methodologies, tricks, and so on*****. You would be tempted to say it's to difficult for the average C coder, but there's something else! C++ isn't just C, it only happens to be backwards compatible! When you switch from C to Common Lisp you won't be an expert instantly, but the C folks assume they can just come and start programming C++. And then they cry that it's too difficult and complex.

Summary

This time there's no summary. I was just gossiping...

---
* Linus original post (admittedly taken out of context!): http://thread.gmane.org/gmane.comp.version-control.git/57643/focus=57918, but I must admit, when he's speaking, he does make a much better impression!
** Digg gossiping: http://digg.com/linux_unix/Linus_Torvalds_hates_C
*** Hacker News discussion: http://news.ycombinator.com/item?id=51451
**** A typical reply: http://warp.povusers.org/OpenLetters/ResponseToTorvalds.html
***** Steven Dewhurst's reply: http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=411

A letter from DLL hell: msvc60.dll and msvcr80.dll

2008-11-19T00:23:00.001-08:00

This is only a short technical note - for those who (like me) first check the Internet for solution to weird programming questions!

The Problem:

Locally everything worked fine: I could install my old (VC++ 6.0) Windows application and start it without a hitch. But at client's site (1000s of miles away) Windows refused to start as it couldn't find the msvcr80.dll library.

What the heck, I link against msvc60.dll but Windows complains about msvcr80.dll which I don't use, don't link, and don't need??? I that black magic? Help!!! That was the problem I was fighting for the best part of one of last weeks. Welcome in the manifest/DLL hell. Well, I tried first to reproduce this locally: it didn't work (i.e. it did work ;-))) on my develpoment Widows XP machine, as well as a freshly installed Windows Server 2003 Std. Edition (SP2), a running Windows Server 2003 Web Edition (SP1), and it always worked! No probs at all when starting the app!

At the client there was the Windows Server 2003 Web Edition (SP2), with "nothing installed" (as asserted by my colleague at the client' s site) except for our product and Oracle! What's going on? There are no dependencies to msvcr80.dll in my code (I checked with the Dependency Walker) and the installation worked for years before I made that last bugfix!

In the last attempt to reproduce the error I found a running installation of Windows Server 2003 Web Edition (SP2) at my client's company local site and could at last see it with my own eyes: "Cannot start application, msvcr80.dll not found".

The Solution:

It's elementary: if there's no static dependency, ther must be a dynamic one! The Process Explorer has shown, that the process has really tried to load the msvcr80.dll! How? Through the Qt plugin mechanism:

"Qt applications automatically know which plugins are available, because plugins are stored in the standard plugin subdirectories. Because of this applications don't require any code to find and load plugins, since Qt handles them automatically.

The default directory for plugins is QTDIR/plugins*, with each type of plugin in a subdirectory for that type, e.g. styles..."

Although I used the Qt version 3, this generic mechanism pulled in all the plugins found, which were OK, except for the styles plugin qtdotnet2.dll. Because of course "nothing else installed" wasn't true: the Base package of my client was there, and it had various plugins installed, including the culprit: qtdotnet2.dll, which was written for Qt version 4 and pulled in the Visual C++ 2005 runtime support!

---

Erlang and Map-Reduce

2008-10-25T13:54:00.000-07:00

I was a fan of Google's Map-Reduce for a quite long time, as I was first doing my PhD research in distributed systems and then was working in some high-availability projects, so such interest emerged somehow naturally.

Their (i.e. Google's) achievements quite impressed me: first the whole infrastructure (lot of C++ coding - they simple wrote their own filesystem and database!!!), but equally the level of abstraction used when working with distributed data. I wrote about the superiority of that model over the SOA model before, although the SOA model is probably the best you can achieve in a heterogenous environment (and I was probably wrong there...).

So every time I see a map-reduce implementation I can't help reading about it: Hadoop is the most known open source implementation, but there's the QtConcurrent::mappedReduced algorithm in the new Qt 4.5* as well. You see, the idea seems to be catching on.

Now to the news: there is a Map-Reduce implementation in Erlang (!!!)** which runs Python scripts (!!!) and it's called Disco***! And if you don't have a massive parallel cluster at home, you can run it in the Amazon's Elastic Computing Cloud! I don't like Nokia very much, but I must admit that this one is rather cool: you simply write scripts to manipulate your data, much in the vein of UNIX shell programming, only infinitely scalable! And we know that scripting languages are much better for data manipulations than Java or C++. According to its homepage, Disco is quite a success too:

This far Disco has been succesfully used, for instance, in parsing and reformatting data, data clustering, probabilistic modelling, data mining, full-text indexing, and log analysis with hundreds of gigabytes of real-world data. ***

Wow! I like te idea of Erlang and Python working unisono!

---
* Qt 4.5 docs: http://doc.trolltech.com/main-snapshot/threads.html#qtconcurrent
** a small itroduction to Erlang: http://ib-krajewski.blogspot.com/2007/08/erlangs-change-of-fortunes.html
*** Disco's homepage: http://discoproject.org/

C++ servlets - again

2008-10-07T04:47:00.001-07:00

It looks like it's getting to be a new hobby of mine: collecting C++ web application frameworks. After the first one* (a simple HTTP server and session classes) and the modern one (Wt aka witty)**, now the "missing link" was found: the classic Java-like Servlet container implementation in C++! This was made possible by a friendly fellow blogger Eduardo Zea.

Eduado was kind enough to give me a link to the DDJ article describing such an implementation by Rogue Wave named Bobcat***. It's quite old (by SW-industry standards) and the link to evaluation downloads doesn't work anymore, so I think, it didn't quite catch on. But it's another one in my collection! So the actual counters are C++=3, Java=googol.

PS: To be more precise, Bobcat functionality is now part of the Hydra Express****, a Rogue Wave's SOA publishing framework. So are we all going SOAP?

---
* see: http://ib-krajewski.blogspot.com/2007/09/servlets-in-c.html
** The Wt-framework: http://www.webtoolkit.eu/wt/
*** John Hinke, Implementing C++ Servlet Containers, April 01, 2002: http://www.ddj.com/184405023
**** http://www.roguewave.com/blog/so-what-is-it-with-rogue-wave-and-xml-soa/

Beautiful code

2008-09-29T03:03:00.001-07:00

What is beatiful code? The shortest answer (which I've read somewhere but can't remember where) is:

we all know what "ugly code" is: code that someone else wrote...

But beautiful code? Isn't it in the eye of the beholder? Well, for me, beautiful equals readable. You have to see on the first sight what the overall idea of the piece of code is. On the other side, the idea itself might be crap (!!!) but then we should ask the next question: what is a beautiful design/architecture?

I, for my side, am thus a proponent of writing aesthetically appealing code. And I'm not alone! Read this:

Whether it is a natural occurrence, a quirk of human languages, or conditioning, most people find while (x==3) significantly simpler to read than while (3==x). Although neither is going to cause confusion, the latter tends to slow people down or interrupt their train of thought. In this book, we have favored readability over safety—but our situation is somewhat different than that of normal development. You will have to decide for yourself which convention suits you and your team better. *

Here we've got it: the eternal problem with the coding guidelines forcing me to write a plug-ugly (3==x)! The question is: should we write ugly code as to be on the safe side? I admit, that I never wrote such an ignominious line of code in my life. You expect trouble? Nope! I've never had any problems at this point! Well, one single time I mistyped it, but found it out in an instant! In C++ you have to develop certain sensitivity for that construct, that's all.

As for practical examples (instead of dull theoreticizing) - some time ago I asked myself: what is the most cool/beatiful piece of code you eve wrote, what would you show to others in order to impress them or to show them how crystal-clear ;-) your style is? As it was even before my lambda library, and only had plain production code on my disposal, I settled on the following piece:

/*------------------------------------------------------------------------*/ /**
* @brief      configDelta
* @descr
*     This function compares two configurations and returns the differences.
*
* @param[in]  other - the other configuration
* @param[in]  scope - what delta requested: all the new entries, all the deleted
*                     entries, or all changes altogether?
* @param[out] delta - the calculated diffrence
*
* @note       Assumption: both configurations must be sorted!!!
*///--------------------------------------------------------------------------->
void SimpleCfgFile::configDelta(const SimpleCfgFile& other, CfgDelta scope,
                                vector<string*>& delta) const
{
    TRACE_FUNC("SimpleCfgFile::configDelta");
    delta.clear();

    switch(scope)
    {
        case addedDelta:
            TRACE_DEBUG("addedDelta");
            // all in this but not in other:
            set_difference(begin(), end(),
                           other.begin(), other.end(),
                           inserter(delta, delta.begin()),
                           less_then_deref<string*>());
            break;

        case removedDelta:
            TRACE_DEBUG("removedDelta");
            // all in other but not in this:
            set_difference(other.begin(), other.end(),
                           begin(), end(),
                           inserter(delta, delta.begin()),
                           less_then_deref<string*>());
            break;

        case completeDelta:
            TRACE_DEBUG("completeDelta");
            // all in this but not in other + in other and not this:
            set_symmetric_difference(begin(), end(),
                                     other.begin(), other.end(),
                                     inserter(delta, delta.begin()),
                                     less_then_deref<string*>());
            break;

        default:
            TRACE_ERR("Unknown scope requested for configuration delta!!!");
    }

    TRACE_VALUE(delta.size());
}

You see, it's not a rocket science. What I liked in this piece of code was it's conciseness, readibility and (though it's of no real importance for workings of the code) its symmetry. And moreover, I was amazed how a judicious usage of the standard library simplified the task which at first seemed to be rather a daunting one!

An lastly, it's an illustration for the fact that you don't have to use a lambda library to obtain a clear code: I just wrote the following trivial functor:

  template <class T> struct less_then_deref : binary_function<T,T,bool>
  {
      // OPEN TODO ---> constraint: isPtrType(T)...
      bool operator() (const T& x, const T& y) const { return *x < *y; }
  }

instead of the lambda expression (*$1 < *$2) and it is still readable. Or even more readable by using a telling name?

---
* taken from the following book: Groovy in Action, Dierk König et al., Manning 2007, page 157-158

Google's technology stack

2008-09-17T05:00:00.000-07:00

Well, I said I wouldn't write any knee-jerk reaction posts on this blog, only well thoght-through, throughly researched, and insightful entries. Certainly, I have some entries I should be rather working on, like mutithreading testing or lock-free synchronization... But I must admit, that was a bit over-optimistic, as you'll see in a second...

Recently, I stumbled across this one:

Google has recently launched the Google App Engine. From an Java enterprise developers point of view it is shamelessly easy to use, deploy, etc. Well, unfortunately it only takes Python apps for now, but it is stated that there will be more languages supported in the future. But it’s Google again putting its finger into the Java EE wound (first GWT with webapps, then Android shaking the Java ME world, and now App Engine showing how runtimes should look like).*

I blogged before about the "Google phone", which came out not as a phone, but as an SDK (BTW: do you want to make your 1st milion? Take part in the Android Developer Challenge, no kidding!). The local german "Java Magazin" published on this ocasion (i.e Android's release) an editorial, accusing Google of attacking Sun, Java, splitting the Javaland and whatever. What the fuss?

I cite Wkipedia**:

Dalvik is often referred to as a Java Virtual Machine, but this is not strictly accurate, as the bytecode on which it operates is not Java bytecode. Instead, a tool named dx, included in the Android SDK, transforms the Java Class files of Java classes compiled by a regular Java compiler into another class file format (the .dex format).

So you are writng Java code, but it's not running on the JVM! Is that forbidden?

"...some have related Dalvik to Microsoft's JVM and the lawsuit that Sun filed against Microsoft, wondering if a similar thing might happen with Google, while others have pointed out that Google is not claiming that Dalvik is a Java implementation, whereas Microsoft was."***

I don't know. But I think it shouldn't be!

Now for the Google App Engine. As I had a look at it some time ago, it didn't ring a bell with me. I rather though about it as of another grid computing offering, like Amazon's Elastic Cloud: just write your app locally and ther throw it on the grid and it will scale automatically with your needs. But when a Java person sees this, it sees Java technology attacked. The same for GWT: it is JSF as it should always has been. But come on, you are still writing your programms in Java, the difference is that the ideas don't come from Sun! I'd rather say Google is giving a second life to Java by providing new ways for using it. I wouldn't have though that 5 years ago, when they were essentially a C++/Pythonn shop!

Additionally, I can't help feeling that the Java poeople are thinking in an "imperialistic" way: boasting about their superiority, but on the other side always suspicious that someone may have try to challenge their (self proclaimed) supremacy. Like the late USSR...

But on the other side, when you look at Google, you could be tempted to think, that they are writing everything new: newly they published an own C++ test framework**** and an own (C++) transfer data encoding****, just as example. So maybe it's not an assault on Java iteself, but just a manifestation of the "Not Invented Here" syndrome? Now, the employees must do something in their 20% project-free time, so they programm every conceivable thing anew (and better?).

---
* http://adminsight.de/2008/05/05/springsource-announces-an-application-plattform/
** http://en.wikipedia.org/wiki/Dalvik_virtual_machine
*** http://www.infoq.com/news/2007/11/dalvik
**** Google test framework: http://code.google.com/p/googletest/wiki/GoogleTestPrimer, Google transfer encoding: http://code.google.com/apis/protocolbuffers/docs/overview.html, and now they even wrote their own browser: Google Chrome (ok, you knew that already...)

The end of dumbing-down of programming?

2008-08-27T13:30:00.000-07:00

Times are changing. Definitely. Think about this little story: do you remember what Java's USP was at the beginning? Yes, the garbage collection, and yes, the JVM portability. But the main thing was it's philosophy of not allowing bad programmers to make bad mistakes. We don't have pointers, we don't have operator overloading, we don't have multiple inheritance, our core classes are final... As one person expressed it at that time: "...they gave me a paper hammer instead of a real one so I can't hit my fingers!" And that's the reason why i din't like it, didn't really want to use it, and consequently missed out on a cash-cow :-(.*

But yesterday, I read this on the InfoQ**:

... And it is true, my experience weaves that out too: you can create environment really restricted just to keep bad developers out of trouble, but these restricted environments harm the productivity of your best programmers. Basically what you do is: you are not speeding up your bad developers and you are slowing down your best developers and that's why our productivity stinks in software right now, but the attitude is changing around.

I've read similar complaints before, but as it seems, after the Ruby-shock (RoR faster than Struts and 10x mote productive) such opinion is somehow fashionable and even almost mainstream today.

What do you say? Isn't that the old C++ philosophy we are returning to? I cite Bjarne***:

Kierkegaard was a strong proponent for the individual against "the crowd" and has some serious discussion of the importance of aesthetics and ethical behavior. I couldn't point to a specific language feature [....] but he is one of the roots of my reluctance to eliminate "expert level" features, to abolish "misuses", and to limit features to support only uses that I know to be useful.

---
* The language was just plain uninteresting to me, and I didn't see its real merits at the time. Maybe I didn't want to see them?
** http://www.infoq.com/interviews/Languages-Platforms-Neal-Ford
*** http://technologyreview.com/Infotech/17831/page3/

PS: BTW, concerning the "Java is the next Cobol" thing, there was a discussion on some German Java forum here, where I argued with 2 arguments, but as I see now, I (and everyone else) missed the most important one. Namely: Java is the new Cobol, as it's mainly used in corprate and business settings (like Cobol was). The solution is simple, isn't it?

C++ pocket lambda library, the last

2008-07-09T14:16:00.000-07:00

So, this will be definitely the last part! I promise! I planned this to be a three part series, and see how it has grown. But let's go down to the business: in the previous installments we achieved the following:

    // part 1: basics
    find_if(vec.begin(), vec.end(),  _$1 <= 10);
    transform(vec.begin(), vec.end(), vec1.begin(), _$1*2);
    sort(vp.begin(), vp.end(), *_$1 <= *_$2);

    // part 2: function applications
    for_each(vec.begin(), vec.end(), bind(sinus, _$1*(pi/180.0)) );

    // part 2a: member access
    find_if(vecx.begin(), vecx.end(),  _$1->*(&XXX::getValue) <= 2);

    // part 3: output
   for_each(vec.begin(), vec.end(), cout << delay("---") << _$1 << delay("\n"));

You can see, we had defined a kind of a custom (if not too counter-intuitive) mini-language for the lambda expessions. As each language has to be learnt, we'd like it to be kept simple! So I have only one more thing to add, namely:

1. Control structures

This is really fun, because it's not difficult at all and it let us define very cute lamdbas indeed. The entire code, as it stands in the library, is like that:

    // if_then
    //  ---
    template <class S, class T> void eval_then_expr(S& e, T& t) { e(); }

    // helper: Assgn needs the iterator for: if_then(_$1==0, _$1=44)
    // --- OPEN, TODO: make general for e.g.: if_then(_$1>=3, cout << _$1)
    // --
    template <class T> void eval_then_expr(Assgn<T>& e, T& t) { e(t); }

    template<class S, class T> struct IfThen  : public lambda_expr {
        S if_expr;
        T then_expr;
        IfThen(S s, T t) : if_expr(s), then_expr(t) {}
        template <class U>
            // OPEN, TODO: check if then_expr needs the val argument!
            bool operator()(U& val) { if(if_expr(val))
                                          eval_then_expr(then_expr, val); }
    };

    template <class S, class T>
        IfThen<S, T> if_then(S s, T t) { return IfThen<S, T>(s, t); }

    // t.b.c....

you see, there are some TODOs, which we'll discuss in the due time, but the technique is simple: first we store the if_expr and the then_expr (as functors), and when the IfThen functor is evaluated, we just evaluate the sub-functors, and make the then_expr dependant on the value of the if_expr. I didn't think myself this would be so simple! As this isn't a ready library, but rather an exploration of some possibilities in code, ther are a lot of open ends here. First, we probably don't need the val parameter in the function call operator. Second, if the then_expr needs the iterator (i.e. _$1), I currently just hacked in only the overload for the assignment. This must be extended with some general forwarder mechamism. But even with this rudimentary support, we can do this now:

    // count occurences
    for_each(vec.begin(), vec.end(), if_then(_$1 == 1, globvar(yyy_ctr)++));

    // replace values
    for_each(vec.begin(), vec.end(), if_then(_$1 == 1, _$1 = 9));

I think it's pretty useful. Other possible control structures? Well, I think they are possible, but not so useful: a loop, a switch? I don't think that it would be good design. What we really would need sometimes, is rather a possiblility to nest STL algorithms than to use a loop functor. But it's not difficult, maybe something along the lines of:

    template <class U>
        bool operator()(U& val) { auto it = val.begin(); // C++Ox
                                  for(it != val->end(); it++)
                                      eval_loop_body(loop_body, it); }

would do, and then we could process a container of containers:

    for_each(cc.begin(), cc.end(), loop_all(cout << _$1));

and perhaps even to extend it like that:

    for_each(cc.begin(), cc.end(), loop_some(_$2 < 1, cout << _$1));
    for_each(cc.begin(), cc.end(), loop_counted(10, cout << _$1));

But do we need it? I think it's more a gimmick that an useful feature, because it's not orthogonal: you have a host of special loop_xxx lambdas instead of a single mechanism. What do you think?

2. Summing this all up

In conclusion? There are 2 conlusions:

1. lambda library is cool, all this stuff is cool, I'm cool.

2. Frankly, isn't that all just appalling? All that effort and what we got is an unnatural syntax! And it's not transparent: for each new combination of operators I've got to write new code in the library (or almost)! Makes you think of Phillip Greenspun's Tenth Rule of Programming: "Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified bug-ridden slow implementation of half of Common Lisp." Please Mr. Stroustrup, why don't we have lambda-expressions as core language feature in C++???

Actually, as the things are, it seems like we are going to have lambda functions in the new C++0x standard*! For me, they look like Groovy lambdas (or are they really Ruby's ???), compare**:

Groovy: myMap.each { key, value -> println "$key => $value" }

C++0x: for_each(a.begin(), a.end(), <> (int x) -> int { return sum += x; } );
       // or, equivalently, but not much Groovy-like:
       for_each(a.begin(), a.end(), <>(int x) { return sum += x; } );

Should we rejoice then? The proposal paper itself lists the problems with lambda expressions:

1. lambda-libraries may render simpler code in basic cases, compare:

    // lambda lib.
    os << _$1
    // lambda expr.
    <> (int i) extern(os) { os << i; }
You see, we need the old, ugly, annoying type specifications again! An we've just started to ejoy the typeless (ehm, generic...) programming in C++! Isn't that what the whole template thing is for! This leads immediately to the second problem:

2. there are no polymorfic lambda functions!!!

The proposal doesn't allow us to write templated, polymorfic lambda function (i.e. ones with implicit type recognition), as it would clash with the concepts feature. More specific, it wouldn't exaclty clash and explode, but rather the concepts couldn't guarantee the correct type checking in this case. So bye bye type freedom? Do we alway have to use the annoying explicite typing? Imagine a lambda function working with an iterator on a list of strings, an then write this type expicitely as input parameter. Ghastly! When you are using a lambda library solution, you'll simply write _$1, and that's it! In such a case, the whole point of the lambda expression, its conciseness, goes down the drain. I can write a standalone functor as well.

Summing up: if have only a simple thing to do, a lambda library solution is simpler. But if there is some more work to do, and we don't have elaborate type specifications for input parameters, the lambda expression solution has clearly an edge, as we don't have to learn a new, special-purpose mini-language, but can just use the standard C++ syntax.

As it seems none of the solutions is optimal: do we need both of them?. And I can say I'm rather not pleased, although the situation here is definitely better than in Python.

3. Personal note

In the past I thought (or rather believed) that you could just do everything in C++ with libraries, operator overloading and template tricks. But I guess I was just dazzled by the template syntax ;-). After this series I think there are limits to the flexibility of C++ which can be only worked around with ugly syntax (or macros).

---
* as in the last "State of C++ Evolution" document they are on the list of the features which are definitely planned to be included, see: "State of C++ Evolution (between Portland and Oxford 2007 Meetings)" of Dec.01.2007 - http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2142.html
** C++0x lambda expressions proposal - http://www.research.att.com/~bs/N1968-lambda-expressions.pdf

The future of C++

2008-06-25T01:35:00.000-07:00

In my recent blog entry* I complained about not exactly knowing where C++ is heading, what features will C++0x contain when it finally appears, and if we'll need to switch to hex as in C++0a ;-). Then I read some interviews with Bjarne Stroustrup** and the things became clearer.

1. The process

The first ambiguity I addressed, was the problem of the very loooong time which C++ needs when acquiring new features, and hinted at the lack of corporate backing. Bjarne on that**:

BS: The progress on standard libraries has not been what I hoped for. ... We will not get ... I had hoped for much more, but the committee has so few resources and absolutely no funding for library development.

and:

BS: There is no shortage of good ideas in the committee or of good libraries in the wider C++ community. There are, however, severe limits to what a group of volunteers working without funding can do. What I expect to miss most will be thread pools and the file system library. However, please note that the work will proceed beyond '09 and that many libraries are already available; for example see what boost.org has to offer.

So that's pretty clear, the industry isn't backing C++ anymore, and the ISO beaurocracy isn't helping here. Java's JSP is definitely more lightweight. Python hasn't a commitee. IBM, Sun (and Google?) are massively backing Java. Previously Google was a stronghold of C++ (MapReduce, BigTable, BigFile), but lately Java seems to take over, in the real "weed language" manner.

2. The features

The second of my questions was the extent of new features which will make it into new standard, beacuse this was a constant source of confusion: thousands of proposals, each in a different state of progress, constantly wandering between approved, considered, considered strongly, considered not so strongly, not considered, demised, etc ;-). Now the new feature scope seems to stabilize at last. Bjarne again**:

BS: I do — based on existing work and votes — expect to get:

Libraries
- Threads
- Regular expressions
- Hash tables
- Smart pointers
- Many improvements for containers
- Quite a bit support for new libraries

Language
- A memory model supporting modern machine architectures
- Thread local storage
- Atomic types
- Rvalue references
- Static assertions
- Template aliases
- Variadic templates
- Strongly typed enums
- constexpr: Generalized constant expressions
- Control of alignment
- Delegating constructors
- Inheriting constructors
- auto: Deducing variable types from initializers
- Control of defaults
- nullptr: A name for the null pointer
- initializer lists and uniform initialization syntax and semantics
- concepts (a type system for template arguments)
- a range-based for loop
- raw string literals
- UTF8 literals
- Lambda functions

The most important feature (IMHO) is**:

BS: The new memory model and a task library was voted into C++0x in Kona. That provides a firm basis for share-memory multiprocessing as is essential for multicores.

and, of course, the auto keyword and lambdas!

Maybe more important is what won't be there**:

BS: The progress on standard libraries has not been what I hoped for. .... We will not get the networking library, the date and time library, or the file system library. These will wait until a second library TR. I had hoped for much more, ...
BS: ... What I expect to miss most will be thread pools and the file system library. However, please note that the work will proceed beyond '09 ...

But an important change of working style will take place:**

... Fortunately, the committee has decided to try for more and smaller increments. For example, C++0x (whether that'll be C++09 or C++10) will have only the preparations for programmer-controlled garbage collection and lightweight concurrency, whereas we hope for the full-blown facilities in C++12 (or C++13).

and we may expect a host of new extensions in the future!

On the other hand, at least when the multithreading is concerned, there are independent libraries available, and they offer some rather high level concepts! Take for example the latest Trolltech's Qt 4.5 framework***, which implements futures, automatic scaling for multicore, has a MapReduce implementation plus concurrent mapping and filtering algorithms. It's just like Java 7's fork-join framework* and the ParallelArray class. Bravo! Other library with high level threading support are of course the Intel's Thread Building Blocks, it's not bad either! At this point we don't need a standard document, as it seems.

3. The prospects

At last, let's pose a more general question: what kind of language wants C++ to be? In the past Bjarne Stroustrup maintained that C++ should be a "general purpose programming language". Contrast this with the statements from the last interviews**:

JB: You are looking at making C++ better for systems programming in C++0x as I understand it, is that correct? ...

BS: Correct. The most direct answer involves features that directly support systems programming, such as thread local storage and atomic types.

JB: C++ is often used in embedded systems, including those where safety and security are top priorities. What are your favorite examples and why do you think C++ is an ideal language for embedded systems especially where safety is a concern, aside from easy low-level machine access?

BS: Yes, and I find many of those applications quite exciting.

For my taste, Bjarne thinks clearly that C++ is an system and embedded programming language: e.g. he expressed his fondness for robotics systems before. That's bad news, because I don't really like embedded programming and automotive :-(((. On the other hand, system programming is quite exciting for me, provided I haven't to fiddle about low level data structures too much.

Allow me a question in this context: for me OO programming is about hiding low level details, seeing the bigger picture. If the strengths of the language lie in the hardware access then maybe it's not so useful for OO programming? We can use C for low level programming (like Wireshark's code does, and it's not a toy system) and do OO in Java or Python? So where's the place for C++ then?

But maybe the future will be totally diffrent? Maybe we won't be programming C++ anymore but rather the Qt platform, which only happens to be written in C++? This would be akin to the Java platform: because C++ doesn't provide a standard ABI, many comapnies are using Qt as to assure the portability of their code between operating systems (among others my current client). Interestingly, not only in GUI applications, abut also in general purpose programming! But what about the (maybe only preconceived) imcompatibility with the standard library, which I bemoaned in one of my previous entries? If I can give faith to Danny Kalev's words****:

And in other news, Nokia completed its acquisition of Trolltech last week.
...
Qt is currently used in Skype, Google Earth, and Adobe Photoshop Elements.
....
After Nokia's acquisition, it seems that Qt will be modified to support Symbian and other mobile environments, or at least POSIX libraries and better support for mainstream C++.

then it seems that not only yours truly is having that impression, and that there will be a remedy soon!

--
* Language trends and waiting blues: http://ib-krajewski.blogspot.com/2008/04/language-trends-and-waiting-blues.html
** An Interview with Bjarne Stroustrup: http://www.ddj.com/cpp/207000124 (but
also http://www.informit.com/articles/article.aspx?p=1192024 , although I don't quote it here)
*** http://labs.trolltech.com/page/Projects/Threads/QtConcurrent
**** C++ Reference Guide: http://www.informit.com/guides/guide.aspx?g=cplusplus (Notes on 25th of June)

C++ pocket lambda library, part III

2008-06-08T11:09:00.001-07:00

As I first wrote my C++ lambda code about a year and a half ago, I didn't know that I'm hitting such a hot topic. I wanted just to reduce the amount of code I had to write, and didn't have any high-church functional programming thingy in mind. But now lamdbads and closures are all the rage: look at all those Groovy articles on developerWorks for example. Even Java 7 will have closures (or won't it?). Definitely, Ruby has made programming languages an interesting topic again!

BTW, do you know how currying and lambdas looks like in Haskell, a popular functional (dynamically and strongly typed) language? If you don't know what it is, we've discussed currying in a previous posting of this mini-series*. In Haskell it is very natural syntactically, you can just write (well, almost, I skipped the T=>T=>T type definition):

    product = a b = a*b
    double = product 2    <-- curried!
    double 2

This is basic, but look at that:

    doubleEach = map (2 *)

i.e. we define a partially evaluated function (as map needs 2 arguments, a function and a list) waiting for application. It's like you'd be using the bind() template of our last posting* in C++. And look at the cute lambda function shortcut: (2 *) is a function (unsurprisingly, as in Haskell everything is a function, contrary to Ruby or Groovy where everything is an object, even the functions ;-)). I like it.

1. Getting exxpresive

Admittedly the code in the 2nd part of this mini series was rather bland*: some hyper technical stuff but not really very entertaining like the 1st part (which was really fun for me to code). I wrote it only for completness' sake, as it is part of my code. I hope this installment will be more fun again.

So let us tackle the last topic we need to implement as to have an usable lambda mini-library: the expression templates. The what? Wy do we need it exacltly? I'm certainly not going to write code like: __expr template <class Expressible> express_anger(Expressible& e); - not in my life!!! Ok, ok, let's introduce the concept gently.

Do you know for example the Boost (e)Xpressive library? It expresses (expressively) regular expressions (Boost Spirit library does in much greater style for grammars) by C++ code constructs. I.e. instead of:

    sregex rex = sregex::compile( "(\\w+) (\\w+)!" );

you can write

    sregex rex = (s1= +_w) >> ' ' >> (s2= +_w) >> '!';

using the domain specific language (buzzword alarm!!!) instead. The string: "\\w+" is replaced by an (Xpressive) expression +_w. You recognize perhaps the usage of placeholders, like our _$1 or _$2, operator overloading and assignment of partial matches to external variables (external to the closure, you'd say in Perl or Groovy). But in this case we don't have a single operation which should create a functor, neither a combination of two different operations. Here we have one operation applied again and again (>> concatenator), and we have to encapsulate it in a single lambda functor!

The same problem emerges in the context of out mini-library.

    for_each(vec.begin(), vec.end(), cout << "-->" << _$1 << "\n");

we have to collect all the items which have to be sent to cout, which can be infinite in number!

Here expression templates come to the rescue. First described by Todd Veldhuizen**, they let us to define recursive templates with operator overloading. And recursion can go infinitely deep down, so we can accomodate our long shift operator sequences with our usual aplomb! What we need is following tree structure:

                  expr
                    ¦
                   op >>
                 /    \
               op >>   \
             /   \      \
           op >>  \      \
         /        \      \
      s1=+_w  ' ' s2=+w_  '!'

True to the "Modern C++ design" book's ubiquitous typelists usage, we can express this runtime structure in compile time with a following monstrous type:

    Op<Op<Op<Char, Expr>, Expr>, Char> rex;

Here, all the structural information has been recorded: just read the type from left to right and compare it with the picture of the parse tree. Now we have to supply the arguments to the constructor of an object of this type and then call a method of the rex object. But we don't do it in classical sense, the construction and the type definition will be done recursively while compiler is parsing the expression. Hence the name of the expression templates #B-D.

2. First real exxpressive code

To convince the compiler to to some sensible work for us we first define the following recursive template:

    // Shift represents a node in the parse tree
    // ---
    template<typename Left, typename Op, typename Right>
        struct Shift  : public lambda_expr
    {
        Left leftNode;
        Right rightNode;
        std::ostream& out;

        Shift(Left t1, Right t2, std::ostream& os)
            : leftNode(t1), rightNode(t2), out(os) { }

        template <class T>
            void operator() (T& t) { Op::print(leftNode, rightNode, t); }
    };

You can see, the structure of the tree node is different form the monstrous type given as example above, well, it's even more complicated. Using this approach we would express the above example tree as:

    Node<Node<Node<Char, Op, Expr>, Op, Expr>, Op, Char> rex;

Ok, why not. If it's supposed to help, I couldn't care less... ;-)

But what has this all with our lambda library? The answer is, we can apply the same concept to the problem of priniting data to cout: supposed we have a following lambda function: cout lt;< "element:" << _$1, we'll can build a type tree like:

                  expr
                    ¦
                  << op
                 /    \
             << op     \
             /   \      \
            /     \      \
         cout "element:" _$1

One interesting thing to note is the ()-operator, which prints the actual node of the parse tree using a mysterious T& t argument: it is the actual parametr of the lambda function, i.e. the uderlying iterator itself!

So if we have the top level tree node (i.e. the expr in the diagram above), we'll just call its ()-operator an the whole tree is printed out, hopefully! Two problem pose themselves here:

1. how to descend to the subnodes of the tree while printing, and
2. how do we get the whole tree structure constructed in the first place?

3. Recursive printing

To answer the first question we need the representation of the recursive printing operation in code:

    struct shiftOp // Represents << operation
    {
        template <class Left, class Right, class T>
            static void print(Left& left, Right& right, T& t)
            {
              print(left.leftNode, left.rightNode, t); // walk down
              left.out << right(t);  // if right needs t? : << _$1*2 ???
            }
 
        template <class Right, class T>
            static void print(std::ostream& left, Right& right, T& t)
            {
              left << right(t);  // at the beginning
            } 

        template <class Left, class T>
            static void print(Left& left, placeholder<1>& right, T& t)
            {
                print(left.leftNode, left.rightNode, t);
                left.out << t; // special case placeholder
            }
    };

First we walk down the tree in the depth first, left to right mode (the first print() function). When we arrive at the lowest left node of the tree we do the first print, then go back and print the corresponding right node. Note that we don't walk a physical tree here, we walk a type expression which is organized like a tree! The print() functions will "match" a part of the type-tree, print the matched part, and match the subtype in a recursive manner. So we are treating types in compile time as we'd treat data in the runtime! That's why this is called template META-programming.

Then we need only 2 specialisations: one for the first invocation of the shift operator, where the left operand is the output stream itself, and the second one to deal with our lambda mini-library's placeholder types. The placeholder will be printed directly to the stream. Our tree expression for the simple example above will be then as follows:

    Shift<Shift<cout, shiftOp, "element"-Expr>, shiftOp, _$1> lambda;

Now just imagine how the print() function will work on it.

4. Growing the tree

To answer the second question we need 2 simple ;-) operator overloading definitions:

    // The overloaded operator which does parsing for expressions of the
    //  form "a<<b<<c<<d" => Shift<Shift<Shift<A, op, B>, op, C>, op, D>()
 
    // for ostream: cannot be taken by value!
    template<class Left, class Right>
        Shift<Left&, shiftOp, Right> operator<< (Left& left, Right right) {
            return Shift<Left&, shiftOp, Right>(left, right, left);
            // left IS an ostream!
        }
 
    // for lambda_expr: must be taken by value!
    template<class Left1, class Left2, class Right>
        Shift<Shift<Left1, shiftOp, Left2>, shiftOp, Right>
            operator<< (Shift<Left1, shiftOp, Left2> left, Right right) {
                return Shift<Shift<Left1, shiftOp, Left2>, shiftOp, Right>(left, right, left.out);
        }

The first one starts the recursive template definition at the "cout <<"-expression, an the second one goes one nesting level deeper and one <<-operator application to the right. It's an elaborate syntax, but conceptually it's not a rocket science! Note how the stream (cout) parameter is handed down the expression tree.

Yeeee-ha! We are done now! Let us try it out:

    for_each(vec.begin(), vec.end(), cout << _$1);  // OK
    for_each(vec.begin(), vec.end(), cout << _$1 << "\n" ); // compile error???
    for_each(vec.begin(), vec.end(), cout << i++ << ":" << _$1 << " " ); // again!!!

Did you see that? We still cannot use our lambda library. What is it this time? Well, we need a last building block, and for discussioon of that we need a separate paragraph.

5. External data in lambda functions

The problem is, that inside of an lambda expression we are working with functors, and not with native C++ data types. This means, we need a function call operator for each element of the lambda expression. What can be easier than that! We can make a trivial functor, which, when evaluated returns our literal value i.e. "\n" or ":"!

    // constant_ref
    //  ---
    template<class T> struct Const : public lambda_expr
    {
        T& val;
        Const(T& t) : val(t) { }
        const T& operator()() const { return val; }
        const T& value() const { return val; }

        template <class S> // for different context: needed for Shift!!!
            const T& operator()(S& s) const { return val; } // ignore input
    };
 
    template<class T>
        Const<T> delay(T& t) { return Const<T>(t); }

This can be described as a delayed evaluation: the compiler first wraps the constant in a functor, and the constant's value is used in runtime instead of compile time. Now a lambda like: auto lambda = (cout << _$1 << "\n"); will work. BTW I wonder if that would compile...

Maybe you don't remebmer, but it the IIa part of this series*** I showed the following code snippet:

    // assign to a external counter
    int aaa;
    for_each(vecx.begin(), vecx.end(), aaa += _$1->*(&XXX::value));

only to say, that it wouldn't work yet. What's the problem? Basically the same one: we are using native C++ type instead of a functor. The solution is of course the delayed evaluation from above, but now must cater for the basic operations:

    // variable_ref
    //  ---
    template<class T> struct Var : public lambda_expr
    {
        T& val;
        Var(T& t) : val(t) { }
        T& operator()() const { return val; }
        T& operator()(T& t) const { return val; } // ignore input
        T& value() const { return val; }

        AssgnTo<T> operator=(T t) { return AssgnTo<T>(val, t); }

        template <class S>  // assign from lambda_expr
            AssgnTo<T> operator=(S s) { return AssgnTo<T>(val, s); }
    };
 
    template<class T> Var<T> globvar(T& t) { return Var<T>(t); }

Here we enabled the assignment to the C++ data type. The addional operators could be implemented like this:

    // OPEN todo: be more modular!!!
    //  --- return lambda_operation<lambda_exp, oper_type>
    template<class T>
        AddAssg<T> operator+=(Var<T> v, const T& t) { return AddAssg<T>(v.value(), t); }
    template<class T>
        AddAssg<T> operator+=(Var<T> v, placeholder<1>) { return AddAssg<T>(v.value()); }
 
    template<class T>
        Incr<T> operator++(Var<T> v, int) { return Incr<T>(v.value()); }

Now we can finally write:

    int xxx = 0;
    for_each(vecx.begin(), vecx.end(), globvar(xxx)++);
    for_each(vecx.begin(), vecx.end(), globvar(xxx) += 5);
    for_each(vecx.begin(), vecx.end(), globvar(xxx) += _$1->*(&XXX::value));

and, for example:

    for_each(vec.begin(), vec.end(), cout << _$1 << delay("\n") );
    for_each(vec.begin(), vec.end(), cout << globvar(i++) << delay(":") << _$1 );

But not, among others: globvar(aaa) = _$1->*(&XXX::value), as it's not a complete, orthogonal, and idustrial strength library. It was only a little pastime of mine...

6. The End?

Ehm, I thought this will be the last part of the description of my simple implementation, but no, I'll need another blog entry as not to bore you to death with this one. Meantime, the writing of this description has taken more time than the actual implementation itself!!! But I have some more points to make, and at the end of a long post, there are rater unlikely to be given much attention. So long then!

---
* Part II: http://ib-krajewski.blogspot.com/2008/01/c-pocket-lambda-library-part-2.html
** Techniques for Scientific C++: http://ubiety.uwaterloo.ca/~tveldhui/papers/techniques/techniques01.html
*** Part IIa: http://ib-krajewski.blogspot.com/2008/04/pocket-c-lambda-library-part-iia.html

God wrote in LISP: programming and genesis.

2008-05-30T00:47:00.001-07:00

Did you ever ask yourself what programming language God used to implement the Universe? I mean, he had a pretty tight deadline - only 6 days - so he must have been using something rather high level. And a rather complex one to boot. And as a programmer, you don't have any doubts thet God must have been using a programming language for the task: without abstraction the task is just too complex ;-).

I came across this song while hearing the OOPSLA podcasts and everything became clear: he used LISP! Hear it here: ( >> )*. I must say, I really like it a lot. It's wonderful: it's like a hymn on a great, dead language. The lyrics come from Bob Kanefsky. All I could trace about him is that he wrote some parody songs, but he must be a programmer himself judging from the quality of lyrics.

And the lyrics are right: Object Oriented languages describe how we humans are thinking about the world, but LISP (or functional languages in general) describe the thoughts of God... so pure... ;-). Let me cite: "Don’t search the disk drive for man.c...". It's almost ontology, I like it :-).

PS: As always there is a monority opinion as well, see http://www.xkcd.com/224.

---
* or here, if embedding works:

.

Why is software engineering not engineering?

2008-05-15T02:26:00.001-07:00

As I had a look over Philippe Kruchten's book "The Rational Unified Process. An Introduction" I noticed the following passage. I mean, the book is rather dry, it just describes some organizational process, but this paragraph was different:

"Software engineering has not reached the level of other engineering discipline .... because the underlying "theories" are week and poorly understood and the heuristics are crude."

Well, nothing new here, just like we all know it to be. Then:

"Software engineering may be misnamed. At various times it more closely resembles a branch of psychology, philosophy or art than engineering. Relatively straightforward laws of physics underlie the design of a bridge, but there's no strict equivalent in software design. Software is "soft" in this respect."

Indeed! In programming we are not working with physical materials but with mental objects (or should I say artifacts?) denoted by an array of characters on a sheet of paper. Sometimes even with just boxes and lines*. So what we really are doing, when we're trying to set up some laws in software, is looking for the "laws of thought", i.e. we are trying to find a good way to organize our ideas! Ideas which will then become flesh when executed on a complicated machine. As we are working with mental objects to a much greater extend than traditional engineering, the methodology cannot be the same. The world of human thought is not so well explored as the physical world. Or maybe the physical world is just much, much simpler?

This discussion would lead us too far**, but one thing is sure: on some level of abstraction, we are no more thinking about the underlying machine, a thing which couldn't happen when we were designing a bridge. Because of that I maintain that in programming we are basicaly working with mental and not physical objects, so it cannot be counted as engineering. That might be the case earlier on, as programmers had all the iron on their hands, setting plugs and connecting cables. It had an engineering-like looks. But today? Just look on an corporate IT department - it's all about organisation, processes, abstractions of every level. For me it has more to do with "management science" and even "social science", only on a different scale.

So it should be best called computer "science" - not in the "scientific" sense, but rather in the "soft-science" sense? This would however mean, that it is somehow an art... Well, it's not what the industry wants to hear! But we don't have to tell them ;-).

---
* MDD (or should I call it MDA?) takes it to the extreme: you don't write ANY code at all, instead you are writing the "computation independent model", of course in UML, which constitutes a description of the functionality of the system. Then you are transforming this model into another model: "platform independent model", which represents the abstract computation model, then you are tranforming this into a "platform specific model" at last. And all of this is done using tools, profiles, configurations, cartriges. Ideally, you should only model the required functionality, choose the target platform, and start the model transformation chain. Cool! But is it still programming? Theoreticall it is.

** i.e. into philosophy. Just consider that civil engineers are working with mental models too, and, on the other side, the matematicians are working with "mental objects", which miraculously can describe the physical worlds with a great precision...

pocket C++ lambda library, part IIa

2008-04-27T23:00:00.000-07:00

Are you wondering about the title of this entry? Well, it really should by part of a previous one*, but after looking at it I decided that the previous entry is pretty long already, and I wasn't willing to blow it up even more. And the theme isn't such an interesting one as to deserve a separate part in this mini-series. What is it we are talking about?

1. Access to the members

In the past I sometimes really wanted to be able to do the following:

    struct XXX { int value; int getValue() { return value; } };
    vecx<XXX*> vec_x;

    find_if(vec_x.begin(), vec_x.end(),  _$1->value == alive);

i.e. to access the members of an object out of the lambda function. Of course I'd like a code like _$1.value the best, but C++ doesn't allow us to overload the dot operator! Why not? This would mean that we could customize the method invocation mechanism itself! As it's the case in Groovy, Perl, Python or even Java**:

// Groovy:
Object invokeMethod(String name, Object args)
{
    log("Just calling me: $name");
    def result = metaClass.invokeMethod(this, name, args);
}

If you have that, you can do things like Rails in Ruby and builders in Groovy. You just intercept calls to the nonexisting methods (i.e. overload the methodMissing()/method_missing() in Groovy/Ruby or __getattr__() in Python) and install the "code block" (a closure, as to be exact) passed as one of the parameters in a custom hash map with the name parameter as key... You've got the message.

This isn't possible in C++, as it would introduce the metaclass notion into the language. At least it would require a common superclass for all C++ objects, and this contradicts the design of C++ classes (AFAIK) as thin wrappers for physical memory segments. On the other side, C++ has a more primitive notion of call intercepting: overloading of the -> operator! Alas, it only works with pointers, so we cannot provide a general solution for value based containers. An that is a bad thing enough.

So let's concentrate on the less ambitious goal: _$1->getValue()! Unfortunately, even this syntax canot be made work in our context! Why? Because for the C++ compiler the expression getValue() doesn't make sense! It doesn't refer to a class' method getValue(), which we could feed to the -> operator, it's just a string which we'd like to transform somehow in a function reference!

So what can we do (if anything)?

2. The Implementation

The best I could produce is the following:

    iterx = find_if(vecx.begin(), vecx.end(), _$1->*(&XXX::value) == 2);
    iterx = find_if(vecx.begin(), vecx.end(), _$1->*(&XXX::getValue) == 1);

Well, what can I say, it's passable. Now the compiler has a meaningful information in form of a member address, and it's not too ugly. How to implement it? With a standard technique from the first part***:

    // ->*
    template<class V, class O> struct ArrowStar : public lambda_expr {
        V O::* mptr;
        ArrowStar(V O::*m): mptr(m) { }
        V operator()(O* o) const { return o->*mptr; }
    };

we are overloading the member call operator for memebr access. For function member calls we need 2 more overloads. First for calls with 1 argument:

    template<class R, class O, class A> struct ArrowStarF : public lambda_expr {
        R(O::*fptr)(A);
        ArrowStarF(R(O::*f)(A)): fptr(f) { }
        R operator()(O* o, A a) const { return o->*fptr(a); }
    };

and for calls without arguments:

    template<class R, class O> struct ArrowStarFv : public lambda_expr {
        R(O::*fptr)();
        ArrowStarFv(R(O::*f)()): fptr(f) { }
        R operator()(O* o) const { return (o->*fptr)(); }
    }

nothing new here as well, just the standard operator overloading technique. For the == operation to work, I extended the EqTo operator from the part 1*** to do a little forwarding. I know, I should use the forwarders (like Le2_forw in part 1), but I was lazy:

    // lambda_expr ==
    template<class S, class T> struct EqTo : public lambda_expr {
        ...
        EqTo(S s, T t) : lexpr(s), val(t) { }
        template <class R>
            bool operator()(R r) { return lexpr(r) == val; }
    };

So let's do something useful at last:

    // shouldn't clash with lambda_expr: *_$1 <= *$2 !!!
    iterx = find_if(vecx.begin(), vecx.end(), _$1->*(&XXX::getVal) <= 2);

    // read field values from vecx
    vector<int> v10_1(10);
    transform(vecx.begin(), vecx.end(), v10_1.begin(), _$1->*(&XXX::getVal));

    // assign to a external counter
    int aaa;
    for_each(vecx.begin(), vecx.end(), aaa += _$1->*(&XXX::value));

Ok, the last one won't be working just now ;-), you must wait for the part 3 of the series! For the first one to work, we must extend the forwarding class from part 1*** a little for the case where only one side must be forwarded:

    template <class S, class T> struct Le2_forw : public lambda_expr
    {
        S e1;
        T e2;
        Le2_forw(S s, T t) : e1(s), e2(t) { }
        .....
        template <class U> // one side is bound! OPEN: assume left side!
            bool operator()(U a) const { return e1(a) <= e2; }
    };

And now everything is buzzing!

3. Discussion

If you think the sytax of the meber function call ist just horrible, there is another possibility to use the member functions: bind them! This you have seen (and frowned upon) in part 2*:

    for_each(vecx.begin(), vecx.end(), cout << bind(&XXX::getVal, _$1));

it's even less readable, is it? Or maybe not? Look, compared with _$1->*(&XXX::value) it's no more SUCH a bad sight! Maybe something like call_func(_$1, &XXX::getVal) would be more readable here? The advantage of this solution is that we could accept value objects in the container, and take it's address internally. I leave the decision to you, the implementation is more or less trivial.

Summing up, I couldn't achieve much progress here, because of inherent C++ design features. So maybe a macro solution? Or another level of indirection? What we really needed here is a hook for compiler errors (method not found), where we could install our own code snippet. Wait a minute! Something like compile time asserts? But how can I get around the string => function coding problem?

Something primitive like this would be possible:

    ...
    ArrowStarStr(string& name): fname(name) { }
    R operator()(O* o) const {
        return (o->func_hashtable.get(name))(); // and don't crash here!
    }

But you cannot use a POCppO anymore, you need some macro gadgetry like in Qt,
for example:

    struct XXX {
        int value;
        int getValue() { return value; }

        // now decorate:
        STORE_LAMDBA_FUNC(getValue);
    };

    // or property based:
    struct XXX {
        DEF_LAMBDA_PROPERTY(value, int);
    };

Not so pretty, not elegant, much to much effort needed. But it is the solution we have to use following the C++ language design. We just don't have introspection and cannot overload the dot. Sorry. Any ideas?

--
* http://ib-krajewski.blogspot.com/2008/01/c-pocket-lambda-library-part-2.html
** with dynamic proxies: see http://gleichmann.wordpress.com/2007/11/22/mimicry-in-action-dynamically-implement-an-interface-using-dynamic-proxy/
***http://ib-krajewski.blogspot.com/2007/12/c-pocket-lambda-library.html

Language trends and waiting blues.

2008-04-06T11:55:00.000-07:00

Hi everbody! I recently skimmed over an interview* about programming language trends in the DDJ. In general, there was nothing new in there, rather a re-statement of the widely known programming trends. But some phrases caught my eye nonetheless:

PJ: C and C++ are definitely losing ground. There is a simple explanation for this. Languages without automated garbage collection are getting out of fashion.
....
Another language that has had its day is Perl. It was once the standard language for every system administrator and build manager, but now everyone has been waiting on a new major release for more than seven years. That is considered far too long.

As I'm mainly a C++ programmer in my bussines life, the news of the proceeding C++'s demise worry me, if only it acknowledges what I see with my own eyes. So this fact alone is not what I want to talk about, but rather about C++ similarity to Perl. Why?

Look, can't you see a parallel to the Perl's fate here? I cite: "everyone has been waiting on a new major release for more than seven years". And there's still no Perl 6! Recall, there were plans for Parrot**, a common VM for Perl and Python (or was it just a joke?). Everyone was excited, but nope, Python won't use Parrot, Parrot is only in its 0.x versions**, so Perl 6 won't come soon, and the situation generally is a mess.

Isn't that somehow similiar to the situation of C++? The last standard (or rather a correction of it) dates back to 1998, i.e. 10 years ago! It's even longer than Perl. So maybe C++'s retreat is due to lack of new language standard like in Perl's case? When I look at Java, I must admit I envy it. Just recall the evolution: while Java 4 was still rather a primitive language without much interesting features (sorry, perhaps with exception of proxies and introspection), already Java 5 brought foreach, generics, annotations, lock-free synchronisation, lock-free data structures, autoboxing and a new the memory model. Admittedly Java 6 wasn't that iteresting language-wise but Java 7 will get things like closures or fork-join support for easy multicore parallelism***! This gets you a wholly new, interesting language.

Contrast that with years-long discussion about what is to be included in the C++0x standard. And I still don't know what exactly is to come! Will a new memory model be included? And lambda functions? Garbage collection? Sometimes we all think that the new Standard won't be named C++0x, because years and years of discussion will be still needed!

So maybe the quote of Bjarne Stroustrup****:

"Java shows that a (partial) break from the past—supported by massive corporate backing — can produce something new. C++ shows that a deliberately evolutionary approach can produce something new — even without significant corporate support."

is false? Mabe only a corporate-backed language has a chance today? Look how quickly Java developed and how the new C++ standard stalls. But maybe it's the "design by committe"-effect on the side of C++? I don't know.

--
* Programming Languages: Everyone Has a Favorite One: ttp://www.ddj.com/cpp/207401593
** Parrot Virtual Machine: http://www.parrotcode.org/
*** Java theory and practice: Stick a fork in it, Part 1: ttp://www.ibm.com/developerworks/java/library/j-jtp11137.html
**** his interview of 2006: http://technologyreview.com/Infotech/17868/page3/

QString conversions, bugs, suprises and design

2008-03-30T11:29:00.001-07:00

1. A trivial bug

It started with a pretty simple piece of code:

    QDomElement e = n.toElement(); // try to convert the node to an element.

    if(!e.isNull())
    {
        TRACE(e.tagName());
        ....

where TRACE() sends an object to cout. The simplest of tasks you'd say, but it didn't compile. What? What year is it now? Are we in the early nineties? Doesn't library writers know about the standard library? It turned out, they know, so I used a slightly modified code:

    TRACE(e.tagName().toStdString());

but it always crashed with Qt 4.3, Windows XP and VisualStudio 2005! Why? No time to check. After some reading of Qt docs, I settled with:

    TRACE(e.tagName().toAscii().data());

It worked, but the code looked extremely ugly! After copy-pasting it for n times (no, n wasn't equal to 3, as it should be according to the Agile gospel, sorry!!!) I longed for something more elegant and explicit, something like:

    TRACE(qstr_cast<const char*>(e.tagName()))

The code was quickly written and worked instantly:

    // helper for QString conversions
    //  -- cute and explicit syntax: qstr_cast<const char*>()!

    template <class T>
    inline T qstr_cast(const QString& s)
    {
        return T(s.toAscii().data());
    }

    inline const char* qstr_cast(const QString& s)
    {
        return s.toAscii().data();
    }

Well, it worked on my machine but not in the target environment, and only in one (different) case. Why? Of course I blamed the Qt runtime support, which already let me down with the toStdString() function. Then I saw the same effect in the debugger on my developemnt machine, and I blamed it on multithreading: there must be something wrong with locking when accessing this particular QString instance. But at last I found time to remove this bug, and looked at the sychronisation, and it was 100% correct. The bug was hidden elsewhere. The new (correct) code is:

    // helper for QString conversions
    //  -- cute and explicit syntax: qstr_cast<const char*>()!

    template <class T>
    inline QByteArray qstr_cast(const QString& s, T* t = 0)
    {
        // the QByteArray's const char* conversion op. will be applied
        // by the compiler in the const char* context!
        return s.toAscii();
    }

As you can see, the old code returned a pointer to the data of a temporary instance of an QByteArray object, and of course it pointed into the void. End of story!

2. Discussion of the trivial bug

Why am I writing this? Isn't it just a banal and stupid bug? I don't think so. I think there are some points to be made.

The first one is that Qt doesn't follow the "Principle of Least Surprise"*. This code should work out of the box: cout << qtStringObj;! Why? Because C++ programmers wrote code like this for centuries! Well, almost. In the "freedom languages" ;-)** you are accustomed to writing just: print someObj; and it always works. Mind I didn't use the word "modern languages" but in modern times we expect it just to be working!

The second one is that Qt doesn't want me to use the standard library! There is no shift operator for std::ostream instead they want me to use their QStream class, which I don't know and have no desire to learn. As for QString class the Qt partisans maintain that it is vastly superior to the std::string class, but what about cout? It is somehow an C++ keyword by now. And besides, why aren't I allowed to make my own choices, even if I choose to use an inferior alternative? Maybe I don't have to be that fast in my prototype implementation? You somehow feel trapped in the proprietary Qt world, and somehow cast back in time. Is that the backwards compatibility with the original 80-ties or 90-ties design? It feels somehow frumpy.

The third one is that once I decided on the syntactic appearance of the function (i.e. qstr_cast<>() and not for example cstr_ptr() or CSTR()), I subconsciously settled on a implementation: just get the internal string data and export the pointer to the char buffer. Alas, in case of the QString this doesn't work that way! Moreover, it's not only the name choice alone, this code would work with all the string classes I knew in my long C++ life. So maybe I chose the name because subconsciously I already knew how to implement it? We pride ourself on being rational beings, but we don't know how much we depend on subconscious shortcuts, which will normally work in a familiar territory, but fail when trying something new. Just like me, as I'm relatively new to Qt.

And what's the moral? It's elementary dear Watson: RTFM first! Or perhaps: don't mix Qt and STL???

---
* for example "Applying the Rule of Least Surprise" from "The Art of Unix Programming" by E.S. Raymond: http://www.faqs.org/docs/artu/ch11s01.html or "Principle of Least Astonishment" at Portland Pattern Repository: http://c2.com/cgi/wiki?PrincipleOfLeastAstonishment

** I thought it was a joke, but not, it's an essay by Kevin Barnes: http://codecraft.info/index.php/archives/20/

Rentree plus Some Comments on Google-Solvers & Co

2008-02-26T23:09:00.001-08:00

Hi everybody! I'm back from my travels and can write some more blog-gobbledegook again (which I'll gladly do). I was away for some serious off-piste skiing in St.Anton, descending some pretty steep, big faces, and taking my first pinwheel tumble on a 40 degrees, Alaska-like, highly exposed, rock-fringed slope. Whow! And then I was back in town, and procuring my next project in just 2 days, taking a head-first plunge here as well ;-).

So let me start my rentree into the blogosphere with something less exciting, i.e. some comments on my previous posts. In fact, I wanted to comment on the Google-Solvers post for quite a long time, so it suits me fine.

Commenting on Java vs. C++

As some of you maybe remember, in a previous post* I compared the perfomance of Python, Perl, Java and C++ (admittedly in a rather ad-hoc manner), and made at first the error of not using the optimization switch of my C++ compiler. Me, a die-hard C++ hacker!!! This resulted in Java and C++ being on par performance-wise. So you can imagine my amazement some time ago when I saw in Uncle Bob's blog these statements**:

You can blame this on the non-optimized gcc compiler I was using (cygwin) but again: Oh boo hoo!. If there are any C++ programmers out there who smirk at the supposed slowness of Java, I think you'd better reconsider.
...
I'll grant you that the JVM can be large. However, if you assume it's already there, then the size of the bytecodes for the application itself need not be very large at all. And the heap can be constrained to a relatively small size to prevent virtual memory thrashing. So, if _engineered_ a java app should give a C++ app a run for it's money in most cases. -UB

Well, this started me thinkig. When you are looking for information concerning Java performance, you can see some typical quotes like this on the web ***:

John Davies: It is just the main reason why there are still diehards that stick with C/C++ because you can guarantee performance, not necessarily faster, but it just guarantees the performance and we will run something in Java, it will run quite frequently faster on C [he means JVM] than it would have done in C/C++, but every now and then it will just pause and that pause can be extremely expensive.

If you'll allow a small comment (hypothesis?, heresy?) from yours truly: what if all those measurements are done without the -O2 switch? It's an all to easy trap to fall into (see my own sufferings*) and an excellent opportunity for the marketing people to use some techniques from the seminal book "How to Lie with Statistics" ;-). Becuse you really can't explain to me that a language which is: 1. interpreted, 2. relies heavily on dynamic memory allocation, and 3. supports garbage collection, can be faster than one which doesn't do it, no matter how much optimization you throw at it! Or can you? If you've read the "Discipline of Programming" book of E. Dijkstra, you know that the first law he establishes there is the "there-are-no-miracles law" :-). So you may guess my position on this.

BTW, maybe I (or someone else) should email Mr. Stroustrup on this one, and see what he's got to say. It would be interesting, I guess, so maybe I'll do it anyway.

And now more comments

Let's continue in the commenting vein: sometime ago I was looking for web-frameworks in C++, and wanted even reimplement Java's Servlet classes in C++ (well, I didn't...). But in the last DDJ I saw an article**** about a C++ web-framework at last! Fortunately, I didn't implement the C++ servlet classes, because it was a really stupid idea! The Wt-framework ("witty" - http://www.webtoolkit.eu/wt/) seems to be rather like modern Java frameworks - GWT, ThinWire or Wicket - you define your application classes in C++, and the HTML rendering comes from the framework! In contrast to the Servlet specification, which is very low level and which nobody uses directly now! Thank God I was lazy! I'll have to try Wt out, perhaps there is an C++ alternative for web-programming anyway.

---
* http://ib-krajewski.blogspot.com/2007/07/google-solvers.html
** here he did basically the same but with Ruby in place of Python: http://butunclebob.com/ArticleS.UncleBob.SpeedOfJavaCppRuby
*** "Improving JVM scalability and performance": http://blog.chinaunix.net/u1/45382/showart_356477.html or http://go.techtarget.com/r/1950475/6108168
**** "Wt: A Web Toolkit": http://www.ddj.com/cpp/206401952

C++ pocket lambda library, part 2

2008-01-31T02:40:00.000-08:00

In the previous post in this series* I promised to tackle some more advanced topics from my pocket lambda library. To start with something, let us gratuitously choose "currying" first. Do you know what currying is? No, it has nothing to do with food, but rather with an american mathematician named, ehm..., Haskell B. Curry. In his formalization of the notion of a computable function (λ-calculus) he used (for simplicity's sake) only functions with one parameter. Come again? What about all the other functions? For this he used a trick: he defined a function which returns a new function where the first argument is fixed, and the second one remains unbound. When used recursively, we can reduce any function of n-arguments to n functions of 1 argument! Mathematics can be as cool as programming sometimes!

I bet you've seen (and used) it before! Yes, STL provides bind1st and bind2nd functions, which are doing exaclty that: currying! In other languages like Python, Perl, Ruby, Haskell, etc, it's possible too, though sometimes through a library extension. Groovy supports IMHO the most explicite (and cool) form:

    def triple_it = {x,y -> x*y }.curry(3);

1. Our task

What we want to do is to extend the STL binder such that they would work the _$1 placeholders. I think of something like the following:

    for_each(vec.begin(), vec.end(), bind(countVector, 'X', 1.222, _$1));
    for_each(vec.begin(), vec.end(), bind(countVector1, _$1, 1.222, 'Y'));
    for_each(vec.begin(), vec.end(), bind(countVector2, _$1));

    for_each(vec.begin(), vec.end(), bind(pr_sinus, _$1*2));
    const float pi = 3.14; // well, not exactly ;-)
    for_each(vec.begin(), vec.end(), -bind(sinus, _$1*(pi/180.0)) );

We assume here that the bind function is pulled in by the appropriate namespace definition, as I want avoid to have to write it like kmx::lambda::bind(sinus, kmx::lambda::_$1). In fact, I never tried out the fully qualified form, shame on me :-(((.

How can we do that? Well there's no need to reinvent the wheel - Boost library already has it, and it's even a part of the C++0x TR.1 standard library extension (tr1::bind). So we can look at the code, or even read about the techniques needed in the implementation**. So if you want, you can spare yourself this post and read ** instead!

2. Basic mechanisms

Well, here is the bind function for the 2 arguments case:

    // 2 args
    template <class F, class A1, class A2>
        binder<F, list2<A1, A2> > bind(F f, A1 a1, A2 a2)
    {
        typedef list2<A1, A2> list2_type;
        list2_type list(a1, a2);
        return binder<F, list2_type>(f, list);
    }

What is it doing? Basically it stores the arguments to be bound in a custom argument list (class list2<>). Then the function and its arguments are used to create the instance of the actual binder class:

    template <class F, class List> class binder : public lambda_expr
    {
        F m_f;
        List m_list;

     public:
        binder(F f, List list): m_f(f), m_list(list) {}

        // OPEN TODO --> only working for void funct!!!
        template <class A1> void operator() (A1 a1)
        {
            list1<A1> list(a1); // store the actual arg
            m_list(m_f, list); // forward to curried funct: m_f+m_list
        }
    };

What is it doing now? It's a functor accepting (in the H.B.Curry's true manner) only one argument. Which one? In the context of an STL algorithm it will be the current value of the iterator. So our task is to sort out to which of the arguments of a function this iterator is bound. Take our previous example: for_each(vec.begin(), vec.end(), bind(countVector1, _$1, 1.222, 'Y')). Here the iterator has to be bound to the first argument of the function - 1.222, and 'Y' to the second and third ones. The binder object stores the actual argument in the custom argument list (class list1<>) and invokes the function m_f in the context of the bound arguments m_list, remembering the current argument in the custom argument list class list1<>. All this is accomplished by heavy use of the listN<> classes, to which we turn our attention now:

    template <class A1, class A2> class list2
    {
        // args: one of them is a placeholder!
        //  -- list1 will detect it!
        A1 m_a1;
        A2 m_a2;
     public:

       list2(A1 a1, A2 a2): m_a1(a1), m_a2(a2) {};

       // apply to arguments and placeholders!
       template <class F, class List1>
           void operator() (F f, List1 list1)
           {
               f(list1[m_a1], list1[m_a2]);
           }
    };

Well, list2<> doesn't do that much, it invokes the curried function, but the detection of bound arguments is redirected to the most important class here: list1<>! Let us have a look at it:

    template <class A1> class list1
    {
        A1 m_a1;

     public:
        explicit list1(A1 a1): m_a1(a1) {}

        // if placeholder:
        A1 operator[] (placeholder<1>) const { return m_a1; }

        // not placeholders: return the ball
        template <class T>
            T operator[] (T val) const { return val; }
    };

Well, this isn't very complicated: complier will detect when a placeholder was placed and use then the current argument (i.e. the iterator). In other cases, the stored argument value will be used, i.e. the value used to curry the function f. As the value is stored in the listN<> class, it is done via a trick: the listN<> class tries to access the placeholder's _$1 value by using the index operator of the list1<> class, but in non-placeholder cases the index value is simply returned to listN<> to be used as n-th argument (a kind of rebound!).

For the sake of completeness, i.e. in case of bind(sinus, _$1) (who'd need something like that?), we need the function call operator in the list1<> class:

    template <class A1> class list1
    {
       ...
       // apply to arguments and placeholders!
       template <class F, class List1>
           void operator() (F f, List1 list1)
           {
               f(list1[m_a1]);
           }
    };

To bind 3-argument functions we need two more templates to be added: the list3<> class and the bind(F f, A1 a1, A2 a2, A3 a3) function, for 4 arguments another two, and so on, and on... But I don't think anyone should need more than 5 arguments in their functions, do you?

3. Extensions

With the above techniques we can write bind(countVector1, _$1, 1.222, 'Y') but not bind(countVector, _$1*2) or bind(sinus, _$1*(pi/180.0)). What can we do about this? Above we wondered about the bind(sinus, _$1) case, but here we can use it to our advantage. We extend the list1<> class like that:

    template <class A1> class list1
    {
       ...
        template <class T>
            T operator[] (Mul<T>& e) const { return e(m_a1); }
   };

So we can see that the list1<> class is indeed a central one! We extended its rebound mechanism to support the multiplication. Oh, uhm, that wasn't pretty! Why couldn't we simply use a general forwading here? The reason is, I left is as a TODO in code (shame on me :-((() but never came around to implement this. I tried the following:

    A1 operator[] (lambda_expr& e) const { return e(); }

but I didn't finish it, and I don't remember if it worked and if it didn't, why. It was almost one year ago and I don't remember all the details, sorry. OK, that could be done better, but (as I said in my previous post*) "it's a simple library". If you need another operation to be supported in the bind context, simply add it to the list1<>. It remains to be clarified if the simplicity of this library is an advantage, but some measurements were reported, which say that the many layers in a fully grown-up lambda library (like Boost) have a detrimental effect on the performance, because the code gets so complex that the optimizers cannot cope with it!*** And the compilation gets longer too. So maybe such a primitive implementation isn't totally in the wrong?

There is another extension I tried. Again we have to extend the central list1<> class:

    template <class A1> class list1
    {
       ...
       // bind a void member function e.g.: int (T::*)(void *)
       template <class T, typename Ret, class List1>
           Ret operator() (Ret(T::*f)(void), List1 list1)
           {
               return ((list1[m_a1])->*f)();
           }
    };

What does it do? It is supposed to be used like that:

    vector<SomeClass> vec(10);
    for_each(vec.begin(), vec.end(), bind(&SomeClass::compute(), _$1);

So for each object in a collection its (void) member function will be called. But again, I don't know if it actually worked, as there is no testcase for that in my code (but I think it should). And not everything that's possible makes sense anyway: here the code is misleading, we'd rather like to express this with _$1->compute()!

4. Last words

Well, that's everything I can say about currying support in my library. In the next post(s) I'll show you how to support lambda expressions like: cout << "---" << _$1 << "\n", and how to access a member or a member function of an object from a collection. What else? Maybe some conditional lambda expressions and some general discussion? Look out.

A general question can be answered here: what about the code of my library? Will it be publicly available? And what about the TODOS? And the missing orthogonality? The anwer is, I don't want to (and haven't got time, for that matter) polish the rough edges and add some more orthogonality. It was a fun writing the library, it took me only a couple of days (about 4, I think) , it is simple, and it has shown me that you don't have to be scared of the template metaprogramming. Of course, I wasn't a novice with templates, but I expected more resistance from C++. Maybe it's because of the library's simplicity?

Whatever. This library is not ready, is not complete, and I leave it as it is now.

---
* http://ib-krajewski.blogspot.com/2007/12/c-pocket-lambda-library.html
** Chris Gibson, "How Do Those Funky Placeholders Work?", Overload 73: http://www.accu.org/index.php/journals/1397
*** C++0x proposal: "Lambda expressions and closures for C++" : http://www.research.att.com/~bs/N1968-lambda-expressions.pdf

SOA in practice: a YAMS?

2008-01-25T14:30:00.001-08:00

In a recent installmet of this blog I already wrote about SOA a little*, but I said that I'm going to read a book of a highly valued author to answer a couple of questions which seemed to stem from my lack of understanding of the subject. Basically my gut feeling was, that SOA (and SOAP) is just a new hype, selling old concepts in new disguise and under new acronyms. Or, in other words: why do we need yet another middleware standard (YAMS)? On the other hand I wondered: if so many highly intelligent people are buying this (i.e. the YAMS), there must be some value in it!

As not to leave my judgements to the mercy of my gut feelings I decided to read some no-nonsense SOA book. This would be close to impossible until recently, as most SOA books were full of marketing drivel and pseudo-scientific, wishy-washy definitions. But recenly a more practical SOA book was published: "SOA in Practice" by N. Josuttis**. As for now, I've read somehow 1/3 of it, which enables me to present some thoughts based based on more solid fundaments.

OK, so was my gut feeling right? Well, both yes and no! SOA is about defining basic services (which can be compared to defining the remote procedures of old), then to combine those simple services (or RPC calls) into more complete, well... services, and then building a business process flow out of those building blocks. And this using SOAP protocol and Web-Services technology. Let me say it: there's nothing new here - nihil novi sub sole! The issues coming up when designing a SOA architecture (i.e. a service-oriented one) are all familiar to me from distributed system design, and the author himself admits it readily. So I was right in my previous post? Technically, yes, with perhaps an exception of BPEL. But???

...but I must admit that SOA indeed has brought in something new, althought in a rather unexpected way! At at the first sight, SOAis only an application of sound distributed system design in corporate settings. And here it is: in corporate settings! As it seems, that wasn't so common before, the organisational and departamental issues were most important then, and now system design issues are on par with them! And that's a huge step forward. Apparently, you need a massive hype vawe to convince corporate people to use some common sense when designing systems!*** And as a Garnter analyst coined the SOA term**** and other Gartner analysts hyped it, the busines people started to hear. And this, and not the usage of SOAP protocol or Web-Services is the real breaktrough of SOA.

PS: And, of course, SOA is SOAP and WS-* in practice, just as I conjectured*, contrary to what you'd read in some high-brow SOA books.

---
* here I tried to answer the question why SOAP and Web-Services aren't simple (which is the first letter of the SOAP acronym!): http://ib-krajewski.blogspot.com/2007/11/web-services-soa-and-rrest.html
** book's homepage: http://www.soa-in-practice.com/
*** or putting it more friendly: there's a huge chasm between business and technical people!
**** SOA-definition by Gartner: http://www.gartner.com/DisplayDocument?ref=g_search&id=302868&subref=browse and http://www.gartner.com/DisplayDocument?ref=g_search&id=302869&subref=browse

Some jokes for the new year

2007-12-31T10:31:00.001-08:00

Happy new year everybody!

What about some (programming) jokes for a smooth beginning of a new year? Let's start with an old one:

How many software developers does it take to change a lightbulb? 10 to discuss the requirements, 10 more to do the analysis, 10 more to do the design, and one to write the code, 12 years later.

Well, that's definitely a "waterfall-model" one! Do you know some XP-jokes? I don't think there are any, as Agility (with the capital A!) is much to cool nowadays to be joked about ;-)! So, as not to let you think I'm a hopeless oldtimer, here is some more modern one, actually a cartoon* :

Well, a discussion could begin about Java, its low-level interfaces, its megatons of frameworks, etc... but not this time.

And here's another one, this time more higher-order and functional (Haskell, if I'm right)**:

Isn't it cute?

And, at last, hava a look at a day in programmers life: http://www.muza.pl/7001.html?cc=0.6113369317218645&mode=image&img=0#gallery. It's really hard sometimes!!!

PS: If you know some XP or Agile jokes, please let me know!

---
* sorry, don't remember the source :-((
** source: http://arcanux.org/lambdacats.html, thanks to Philip Wadler ;-)