Tuesday, 23 November 2010

Yet Another... Web Framework (in C++)

What? YAWF - yet another web framework? But this is mine(!) ... and it's very small - a Simple Embeddable C++ Webframework for Qt!

1. The need


I have discussed this topic on this blog earlier*, so you shouldn't be surprised to hear about it again: a C++ web framework. You'd think there isn't such thing? Wrong! From my own research and from the comments of friendly internauts I collected a quite a list of possible frameworks:
And recently I found even another one:
  • CppCMS


    Do you see its cool logo? It says that programming in C++ is a good thing, as it isn't hard on the environment, and thus can maybe even stop the global warning ;). I must admit that I never seen C++ from that angle! Interesting point, isn't it? Ok, when you think of all that multicores and figths about cache lines then C++ does quite naturally comes to mind with its very direct memory control, doesn't it? Kind of "lean programming" :).
And don't forget the new kid on the block:

It's rather a wide range of possiblities, so you'd think the world doesn't need another framework, and least at all by yours truly. That's what I thought too, untill I stumbled upon a rather simple problem. Currently I develop a Qt application that runs in the background (call it demon, service, headless, etc.) and would like to have an admin interface for it. Of course I could write a standalone GUI client connecting to the application's command socket but that's so nineties! Only a web interface is fashionable these days. Thus I need a small webapp for the admin console.

So at first then, I thought I'd use some open source code from the above list. But all of them would either not integrate well with Qt events, or require 3rd party includes and dependencies, and gwan would even have to be started as an external server! Well, I didn't use gwan before and it does sound like a border case usage for it, just imagine all that debugging of its servlet compiler in run-time...

Then you may ask: isn't there any HTTP server in the Qt framework? Yes indeed, I found one, but it's an extension of the framework (they call it an option I think...) and it's running as Windows service, not as an additional interface to an existing binary. So we've got the same case as with gwan. BTW, why hasn't C++ a standard HTTP server class? Go has one, Python has one, Javascript has one, but C++ - nope! Well, that's not completely true, CINDER framework has one, and cpp-netlib got one as well, but I wanted to avoid big includes and dependencies like Boost.

But hey, Qt has a couple of HTTP support classes, so I thought: boy, that cannot be that hard! And I just hacked my own mini HTTP server.

2. The design

Of course that gives me an opportunity to design a web framework the way it should be always designed at last :-), because all the others seem to be somehow overcomplex or overcomplicated :-). So let's go down to the brass tacks then!

First we have to decide on the name - let simply try YAWF(4q), and in best software giant tradition I'll use an internal codename, for example "CutieW".

The design will be simple, plain and oldskool, as I need something working and that fast. Besides, never underestimate the power of KISS!

The basic ingredients are the Qt classes for TCP socket handling and HTTP request and response parsing. Then we'll have request handler objects with a process() method. The handler objects (kind of servlets for the Java minded) will have to be derived from a superclass (how uncool, I know) and statically instatiated, so that they can register themself with the server in the startup phase. They register with a name string**, for example "user" or "page", so they can be bound to an URI (see below for the example).

Let's sum it up: there will be a server executable, it will set the URL path configuration in startup phase, each path wil get its own handler. When a request arrives, a fresh instance of the handler will be created, this the handler classes need a clone() method for this.

The server will thus find the matching handler prototype for the URI, clone it, and then invoke its process() method. As you see, there's 1 to 1 relation betweem URI and the handler object in the current implementation, but an extension to "multiple dispatch" isn't difficult, and I'll program it when I have some spare time.

Now to the general usage pattern of YAWF4q. In Clojure you'd write the following:

  (defn display []
(html [:h1 "hello!"]))

(defroutes myroutes
(GET "/" [] (display)))

(defonce server (run-jetty #'myroutes
{:join? false
:port 8080}))
In cpp-netlib this:
  struct hello_world {
void operator()(server::request const &request,
server::response &response) {
// your code here ...
}
} handler;
  server server_(argv[1], argv[2], handler);
server_.run();
But in yawf4q it's all pretty plain, non-clever, and almost boring:
  #include "CuteHttpServer.hpp" // the cutie

int main(int argc, char* argv[])
{
// your application:
QApplication* qtAppl = new QApplication(argc, argv);

// the webserver on port 3100
ibkrj::yawf4q::CuteHttpServer testServer(3100);
string error;

if(!testServer.start(error))
return -1;

// run Qt
qtAppl->exec();
}
The server just starts, finds the config file (using a default, hardcoded name), sets the routes, and waits for requests. If you don't define any handlers, a simple "I'm alive" response is sent back.

3. The example


How to write your handlers? Look, you have to derive them from the TestHandlerBase class. First let us implement the handler of the index page:
  class TestHandlerBase
: public MiniHttpHandler
{
public:
TestHandlerBase()
: MiniHttpHandler("TestHandlerBase") {};
// base overrides
virtual MiniHttpHandler* clone() { return new TestHandlerBase; }
virtual void process(AnalyzerRequest& reqData, std::string& respData);
} g_handler1;
OK that's rather much boilerplate, as we need to supply the name string plus the clone() and process() methods, so maybe I should provide a macro like this:
  // --- OR maybe a MACRO?:
class TestHandlerBase
: public MiniHttpHandler
{
public:
BASE_TEST_HANDLER_FUNC(TestHandlerBase)

... your functions ...
} g_xxx;
I don't know, as a matter of fact I hate macros in frameworks! But as an alternative maybe it isn't that bad. Another possibility would be using CRTP, as to automatically add the clone() method, but wouldn't you say this would be somehow too clever when compared with the rest of the design?

And here is the processing method:
  void TestHandlerBase::process(AnalyzerRequest& reqData, std::string& respData)
{
respData = " TestHandlerBase alive!!!! ";

respData =
"<html><head>"
"<title>TestHandlerBase is alive!</title>"
"</head><body>"
;

respData.append("<h1> Input your user data: </h1><p></p>");
respData.append(testform); // HTTP form for UserName, UserMail and Text, action="person"

respData += "</body></html>";
}
OK this doesn't look very cool, but the view component isn't yet there, wait a little. What this handler does is to ask for some user data. Now the second handler follows, which will processes the submitted user data:
  class TestHandlerPerson
: public MiniHttpHandler
{
public:
BASE_TEST_HANDLER_FUNC(TestHandlerPerson); // here process() is already defined

} g_handler2;

///////////// impl:
void TestHandlerPerson::process(AnalyzerRequest& reqData, std::string& respData)
{
respData =
"<html><head>"
"<title>TestHandlerBase is alive!</title>"
"</head><body>"
;

respData.append("<h2> Thank you for your data! </h2>");

// parse the data for display:
map<string,string>& data = reqData.formularData;
map<string,string>::iterator iter;

respData.append(
"<table border=\"1\" width=\"70%\"><tr>"
        "<th><b>Field</b></th><th><b>Value</b></th>"
"</tr>"
);

for(iter = data.begin(); iter != data.end(); iter )
{
respData.append("<tr><td>").append(iter->first).append("</td>");
respData.append("<td>").append(iter->second).append("</td></tr>");
}

respData.append("</table>");
respData.append("<p></p> <a href=\"/index.html\">Back</a><br>");

respData += "</body></html>";
}

Rather grotty servlet or .jsp style here, isn't it? Mixing presentation with code?!

Don't despair, you can use a template. Yes, let's be modern! I chose the mustache :{{ template syntax. I decided for mustache, well, because it's pretty new, generates much buzz, and isn't looking too bad. For those who don't know mustache, it is a simple HTML templating syntax with values encoded like this:

  {{value}}
and sections, which can be rendered multiple times, if they receive a list as input data:
  {{#section_X}}
{{value1}} => {{value2}}
{{/section_X}}
So there's no need to code anly loops in the template. YAWF contains an implementation of mustache :{{, which seems to render correctly all the examples I found in the online mustache docs. The implementation is based on a simple idea: the template is parsed into a linear sequence of "nodes", and the data items passed to the template take care of rendering and walking the node list. Have a look at the source code*** if you want.

So our response handler would now look like this:
  // using templating:
#include "CuteMstchData.hpp"
void TestHandlerPerson::process(CuteSrvRequest& reqData, std::string& respData)
{
string templ =
"<html><head><title>TestHandlerBase is alive!</title>"
"</head> <body> <h2> Thank you for your data! </h2>"
"<table border=\"1\" width=\"70%\">"
"<tr><th><b>Field</b></th> <th><b>Value</b></th></tr>"
"{{#datatable}}"
"<tr><td> {{field}} </td> <td> {{value}} </td></tr>"
"{{/datatable}}"
"</table><p></p>"
"<a href=\"/index.html\">Back</a><br>"
"</body></html>";


// convert data to JSON represenation:
map<string,string>& inpData = reqData.formularData;
map<string,string>::iterator iter;
int i;
ibkrj::yawf4q::CuteMstchValMap templData;

for(iter = inpData.begin(), i = 0; iter != inpData.end(); iter++, i++)
{
templData.addToList("datatable", i, "field", iter->first);
templData.addToList("datatable", i, "value", iter->second);
}

// display page with data (the View of MVC)
respData = CuteHttpHandler::renderThis(templData, templ);
}
It maybe doesn't look much better than the plain HTML code, but here we could save the template in file, and read it in at the runtime. And we've got the View part of the MVC pattern. The handler objects stand for "C", but so far don't have support for "M" in YAWF4q.

Now, as last part of the puzzle, we have to configure the server paths, but this is simple:
#
# test configuration
#

index.html$ :: TestHandlerBase
person/.*$ :: TestHandlerPerson

You see, regular expressions are supported. Now let it run baby! The first handler gives us:


















Then we willingly give our data:



















And see the results:













4. Summing up

At the moment the code*** is only very basic, as I haven't got time to work on it a much as I'd like. I wrote it on my private time, as my customer did decide in favor of a general, grand-scheme, system management solution. So as for today, I wrote only a single test, that one which was described above, and tested it with Visual C++ on Windows. YAWF isn't more than a toy just now as many features are simply missing, for example I'd like to have a built in support for configurable error pages to round the things off. And as next thing, I'd like to add the Model component using my CuteWorkers framework (I hope I'll come to write a post about Qt-workers too).

But there's much more that could done, like:

- HTTPS support in the sever (that shouldn't be very difficult as Qt offers QSslSocket and other classes)
- cookies and session data handling
- dynamic loading and caching of the templates
- dynamic loading of the servelts vial DLL (kind of what ASP is doing)
- a "Clojure example" like dynamic interface

and so on, and so on. Look at my TODO list in the github repository. Nonetheless it's simple (KISS!), it's pluggable, it doesn't have any dependecies other than Qt. Requirements fullfilled!

--
* check out these previous posts: servlets-in-c.html, c-servlets-again.html, c-server-pages.html

** we cannot forget that C++ hasn't much to offer in terms of introspection. But we won't give up automatic registration and detection of handler prototypes!

*** You can find the code at Github: https://github.com/mrkkrj/yawf4q


Saturday, 13 November 2010

Engineering Decency


A couple of days ago I was quite unpleasantly touched by a tweet (a Twitter tweet :-) of a so called "expert" (you know, one the people doing only presentations and learning new languages to write their next book...) ridiculing a developer about a failed project. Somehow I didn't like that - come on, the guys just tried to capitalize on a promise Google (yes, that big engineering powerhouse) made! And they were not alone...

What is that, that the programmers find it so hard to show some engineering decency? I've seen it again and again. Is it so much fun* to flame someone to feel better? The experts should know best that doing stuff mean means making mistakes. Or maybe have they forgotten how it is to be working with real stuff instead of powerpoint metadata?

If you are skiing difficult terrain, you know that the question isn't if you will fall, the question is when you will. That's the same in software - you don't want to believe this now (and me neither) but you will make (sometimes stupid) mistakes! So please, show a little decency. The guys you are going to deride probably just tried to create a working system for a given set of requirements which then turned out to be largely false and for a specific platform, which then emerged to be broken. We've all been there.

And because it's difficult and needs to be learnt, I have an example of how to exercise engineering decency (found here). Instead of the usual:
What idiot with even half a handful of moldy cottage cheese for brains would ever create a pile of stinking filth like this and call it an application?! The true feat of engineering here is that this code monster didn't swallow your whole organization in the black hole of its utter lameness. Where did this person learn programming — NBC?”
Try saying that:
“This will take me longer than we had discussed. In my initial examination of the system, I missed some design decisions in the existing code that conflict directly with what we need to accomplish. I didn't anticipate the approach adopted by earlier developers, because I would not have designed it that way myself. I’m sure they had their reasons for choosing that direction, but so far those reasons have eluded me.”
That's decency, that's humility. Maybe there's a reason you just don't see now. But even if that's really a load of crap, frankly, who of us can say he never ever wrote bad code (or a bad powerpoint presentation for that reason)?

--
* well, if we'd like to get Freudian, we could say that the unconscious fear of doing some big, stupid mistake reveals itself by attacking others, and that we can maybe reason from that, that the majority of programmers are not well trained for their job. But that would be only a mere speculation to discuss with friends on a lazy evening...

Monday, 16 August 2010

Named function arguments for C++

For me a high quality code is a readable one. Consider a hypothetical method like this:
  class XXX:
{
public:
void func(int aaa, string bbb, bool optionX = true);
...
};
now try to call it:
  func(1, "test", true);
What irritates me here is, that I don't know what exactly does true mean at the first glance! Thus I found myself quite often writting code like this:
  bool useOptionX = true;
func(1, "test", useOptionX);
Well, that's better, but what about this:
  bool useOptionX = false;
func(1, "test", useOptionX); // should I use it or not?
a little bit misleading, isn't it? Not that this would be a big problem, but it's nevertheless a small hitch obstructing us from reading the code effortlesly. OK, next try:
  bool dontUseOptionX = false;
func(1, "test", dontUseOptionX ); // don't use is false, so I should use it?
A total mess! And what about constructors?
  class XXX:
{
public:
XXXX(bool optionY);
...
};
  class YYY: public XXX
{
public:
YYY() : XXX(true) {}; // true what???
...
};
Call me pedantic, but I don't like that at all! What we really need here are Python's named arguments:
  sameFuncInPython(aaa=1, bbb="pythonistas", optionX=True)
Simple, effective, readable! Is there a way to emulate this in C++? Recently, as I was especially annoyed with the constructor example from above, I devised a following solution:
  class XXX:
{
public:
XXXX(bool optionY);
...
typedef useOptionY bool;
};
  class YYY: public XXX
{
public:
YYY() : XXX(useOptionY(true)) {};
...
};
and now the other code can be wtritten totally plausibly too:
  typedef bool useOptionX; // local namespace!
  func(1, "test", useOptionX(false));
func(1, "test", useOptionX(false));
and what would you say to the following?
  waitOnSocket(sec(1), usec(20));

Small thing, but it improves my source code. I like it!

---
PS: After I wrote this, I remembered reading something in this vein about Java. And really, this post http://codemonkeyism.com/never-never-never-use-string-in-java-or-at-least-less-often/ by Codemokeyism proposes generally the same solution:

  new Customer(firstName("Stephan"), name("Schmidt")); 
Alas, instead of a benign typedef, you have to write a whole new class named name and firstName etc. Without doubt it's and improvement, but the costs and the clutter... But on the other side Java tends to be verbose nonetheless, so maybe it's OK.

Monday, 14 June 2010

Javascript on the server?

Recently, I stumbled upon an iteresting presenation about server I/O parallelism and multithreading. Surprisingly it was in a Javascript conference, as I sometimes have a look at the Javascript language developments. I'd never expect to find something like that in this contex though.

The general message of the presentation was that threads suck and asynchronous programing is the king. Come again? I don't quite agree with that (or rather agreed, as you'll see in brief) - wasn't that for the threads to save us from the onerous asynchrounous programming with its context information and callbacks (remember Windows event programming?)! The threads justly delivered a convenient abstraction to group the logically connected data and logic in separate units of execution. So now we are doing a full circle: from asynchronous to threads to asynchronous?

OK, on the surface that may be looking like that, but there's more to that. The problem is that when we apply threads to parallel I/O they are just to slow (you know, context switching, thread stampedes and all that). Thus there's a host of asynchronous I/O methods starting with the venerable select() UNIX call. Ok, agreed, for that special field the threading model isn't very appealing, but for the general usage I'd just hide this inside of an I/O thread...

Now there's this pesentation (Ryan Dahl’s talk on Node.js) and what the author says is exactly that: callbacks and asynchronicity are good! How that? We all know it's a pain in practice? His response (an this is the valuable insight got when watching this) is that we just didn't have a good enough programming language to realise that! When you do in in C (as you'd likely to do when programming performant I/O) you are messed up, but if you take Javascript...

See, that's the crux here: Javascript was designed to work in a callback driven environment, so the language has unique mechanisms which no other language can offer. Thus if we take Javascript and do event oriented programming it's a breeze then! The idea is to take the ultra higspeed C library (libev in that case) and wrap it with event-friendly Javascript hull. Here's an example code from node.js site:
  var net = require('net');
net.createServer(function (socket) {
socket.setEncoding("utf8");
socket.addListener("connect", function () {
socket.write("Echo server\r\n");
});
socket.addListener("data", function (data) {
socket.write(data);
});
socket.addListener("end", function () {
socket.end();
});
}).listen(8124, "127.0.0.1");

Well, maybe it isn't a thing of beauty (if you're not into Javascript) but it's concise, everything is at one place, and you can read it without reading tons of manuals first. So maybe this is the way of doing this? Another thing I liked that node.js never provides a blocking API - even filesystem calls are asynchronous! That's consequence!

Sorry for the short post, but it's summer, I want to go out and watch some worldcup!

---
Resources:
1. the talk to be found here: http://jsconf.eu/2009/video_nodejs_by_ryan_dahl.html
2. and here's the code: https://gist.github.com/a3d0bbbff196af633995
3. if you'd like some more technical details, here's an excellent post by Simon Wilson: http://simonwillison.net/2009/Nov/23/node/
4. node.js critique (yes, there's such a thing too!): http://al3x.net/2010/07/27/node.html

Addendum:
- here is a simple blog engine in less than 200 lines of code using node.js : http://github.com/zefhemel/persistencejs/blob/master/test/node-blog.js. Not bad, I'd say.
- Multi-node: how to implement a concurrent node.js HTTP server: http://www.sitepen.com/blog/2010/07/14/multi-node-concurrent-nodejs-http-server/
- Hummingbird: an impressive application based on node.js - http://mnutt.github.com/hummingbird/


Sunday, 13 June 2010

Software estimation and re-reading the classics

Recently I stumbled upon the "Software Top-Ten List"* again. At first I though, OK, that's old, they did mainframes at that time! But then I read it nonetheless. Maybe because it's so short? It's only 3 pages. Whatever. I read it I was surprised about how refreshing it was. Not mouldy, stale and irrelevent, but interesting and fresh! Surprisingly, because the findings are supposed to be well known.

One interesting thought came after I read the point 8:
Software systems and software products each typically cost 3 times as much per instruction to fully develop as does an individual software program.
I immediately thought about that old estimation rule: estimate how long it will take to be done, and then multiply it by two. I always though it to be unserious or a manifestation of intellectual laziness. What, you aren't able to say how long this effing program will take to be written? Just sum up the parts (as shown below)!

But let's look at the problem from a little more different angle: what if programmers do not take into account the costs of polishing the program, making it user-friendly, plus all the problems and dependencies from other programmers located in other deparments of the company - "L'enfer, c'est les autres"! Normally you do not take this factors into account when estimating, and you even cannot estimate it properly. We use either a brute-force 20% project buffers or a statistical estimate based on the PERT method, but it somehow doesn't work out that good.

So maybe we should take the above citied measurement data, at use it when estimating? Just multiply our technical-only estimate by the "real world" factor? Personally, I still cannot believe that the "real world" factor be so high as 3! Am I too optimistic? But I think that frw=2 should be a good choice!

Although I'm a little afraid here, as I always imprement a given piece of code faster than my own PERT estimate. But recently, when estimating the entire system, I missed the point completely: new requirements, unexpected dependencies on not so great libraries, the internal pressure to please the customer... This all sums up in a greater project! An at that point I should have been to muliply so high as by 4! Welcome in the real world.

--
* Harry W. Boehm, "Industrial Software Metrics: A Top-Ten List",
http://csse.usc.edu/csse/TECHRPTS/1985/usccse85-499/usccse85-499.pdf

Thursday, 29 April 2010

Bug hunting or how dumb can you get, really?


As a programmer you sometimes can't help but feel really stupid! Really stupid-stupid! Look at this code snippet for example of a foolish bug that kept me busy for some time. The story behind this is that I added a feature to the product I'm writing for a customer, and was totally happy with it. There was a real customer benefit and it worked like a charm through my tests. However, when deployed to the internal test system, it crashed unexpectedly now and than.

That was a kind of shock, because the product was running stable for more than a year in this environment, and I was totally unsettled as I didn't want to believe that the new feature could have something to do with it. It was tested rock-solid. There must be a mysterious bug somewhere else in the program. And it hid for one year before it disclosed its identity! Bad!

However, after I ran the collected test data on my development machine, it turned out to be a pretty uncomplicated bug:
    for(size_t i = 0; i < a_prInput->frameCount(); i++)
    {
        // header
        size_t start = a_prInput->NettoDataParts[i].uOffset;
        size_t pos = start;
        DnsHeaderData header = parseHeader(a_prInput, pos);
        // interesting data there?
        if(!header.isResponse)
        {
            TRACE("dns: packet isn't a response, ignoring...");
            continue;
        }
        // skip the questions
        for(int i = 0; i < header.questions; i++)
        {
            if(!parseDnsName(a_prInput, start, pos))
            {
                TRACE_ERR("dns: cannot parse a question record");
                m_uGoodByteCnt = pos - a_prInput->NettoDataParts[i].uOffset;

                continue;
            }
You see? The code was referencing the data form the outer loop with the index of the inner one! And it did it only in error case, and crashed only if it was out of luck. Arrgh... There was no problem with fixing this, but it started me thinking: I'm programming for some 20 years, I know C++ in and out, why did I make this bug in the first place? Did I violate some programming rule or best practice? Is it possible to avoid such bugs by some sound practices? Some answers come to mind:
  1. There should be a warning from compiler
    That would be the ideal solution. But what warning should that be? And what if I really wanted this that way? I'd need to use an ugly #pragma then? No, better let it be.

  2. I should have used at()What should I say, er, well, at() is for wimps! And if that wouldn't be the case, we have a choice between at() and the [] operator. So we want to excercise our rights!

  3. Use telling names
    Well, of course, I could name each loop variable accordingly, but come on, can you imagine the hassle? Because if this has to be of any good you have to be consequent. So maybe only do this for nested loops? Maybe, but if I start to do this as to avoid this error, and something else to avoid some other error, you guess it, it becomes a pain! We need something more general,
  4. Use for_each(), as we don't do any addressing here
    That's more interesting, as logically speaking, the inner loop isn't indexing single elements, but rather wants only to skip over the uninteresting headers. But, on the other side, for_each() is annoying, as it requires start and end of the sequence, which isn't a problem if I'd use the iterator, but gets pointless if I never want to use the iterator directly!

  5. use iterators to go high level
    Isn't that the same as the above? No, because it's not an application of a specific technique in a specific situation, but rather a conscious decition for rising of the abstraction level. And quite serendipitously we can avoid the above mentioned error! Or are we? What if both outer and inner loop are using the same name for the iterators? Here we are agan back on the square one and have to use for_each() for the inner loop as to be safe.
Alas, none of them is satisfactory. Well, I know, there's one more solution: code reviews (or pair programming). But first, my customer isn't doing them that often, and second, I doubt this bug would be found in a review/pair. But even assuming it would be found: the price is much to hight! It took me a couple of minutes to fix it, once I had the right data, and my customer expects "results just now".
What is the key point? What does this all mean? Are we doomed? I think that there are two possiblities here: either use iterators or some other high level constructs (which admittedly is somehow of an overkill), or be prepared for occasionally fits of stupidity. In the end we are only humans, and errare humanum est... And who wants to be safeguarded against everything?

--
PS: Sorry, I started this entry long ago, but in the meantime I was so distracted that I couldn't come round to write anything for the blog. Besides, microblogging isn't doing it any easier ;-).
PS2: Other possibility: write shorter functions, use a skipNameRecords() method and you are saved! But really, that function is under 1 page lenght, How long should functions be? Five lines or so? Or seven as adjust to humain brain's cognitive limitations?

Sunday, 28 February 2010

Fooling the ODR (and the compiler)

I just discovered a technique that is prety convenient but rather not self-explaining, nonportable (?), and possibly a maintenance nightmare. So you should never use it, should you? Except, as it's sort of cool, if you'd like to go a little on the geeky side...

1. The need for speed

Imagine you are writing some quick classes in the Java style (i.e. no header files), locate them in some C++ source file, and forget about them. Then you discover that you need to refactor your program (in my case it was a custom test framework) and to move some classes out to a separate file (a.k.a. compilation unit). Now you shun to do the right thing (i.e. to refactor) because you'd have to cut up your classes in two: the .h file and the .cc file, and to duplicate some (non always trivial) amount of code. So yo aren't doing that...

But stop, following trick did it for me: I defined the classes in a separate .h file and kept the .cc file with the extracted classes unchanged. I included the new .h file in the rest of the old .cc file where the classes were removed from. Then I just linked the the whole thing, et voilá, everything worked like a breeze - a refactoring as we like it!

2. Why does it work?

But wait, we spared some editing work, but didn't we violate the One Definition Rule of C++? I'd say we did. So why didn't the compiler complain? Well, as to answer it, we've got to look at the example a little closer.

The extracted classes looked like that:

TestThreads.cpp:

  #include "WorkerThread.hpp"

// test thread A
class ThreadA : public WorkerThread
{
public:
...
private:
...
} g_workerA;
  // test thread B
class ThreadB : public WorkerThread
{
public:
...
private:
...
} g_workerB;
Note that there's no TestThreads.hpp definition file here, thus we don't have a direct ODR violation for the compiler to complain! The newly created definitions file TestThreads.hpp looked like this:
  #include "WorkerThread.hpp"
  class ThreadA : public WorkerThread
{
public:
virtual void addRequest(Request& r);
...
};
  class ThreadB : public WorkerThread
{
public:
virtual void addRequest(Request& r);
...
};
The WorkerThread.hpp definition file looks like this:
  class WorkerThread
{
public:
virtual void addRequest(Request& r);
...
};
And the rest of the TestMain.cpp file where the TestThread classes were previously placed:
  #include "TestThreads.hpp"
  extern TestThreadA g_workerA;

extern TestThreadB g_workerB;
  g_workerA.addRequest(req);
Can you see why we could fool the ODR? Well, in this special case we didn't used the constructors directly in the TestMain.cpp file, only the methods of the WorkerThread superclass which then call the appropriate virtuall methods defined in the ThreadA class.
The linker then generates a reference to the virtual ThreadA::addRequest() method. Well, guess what, a method with the very name can be found in TestThreads.cpp, and the linker can resolve it no probs!

Although we clearly have two definitions of the ThreadA class, we get away with it! The ultimate reason for that is that the C++ compiler doesn't have the global knowledge of the whole program - it works only on the compilation unit level, and the relinquishes the responsibility for the complete programm to the linker! I.e. there's no whole programm optimization possible (at least not in the standard model, although Visual C++ compiler can do it).

3. How to extend the "hidden" classes?

Suppose we'd like to call a new method in the TestThread.cpp file from the outside. You cannot call this method directly, as we do not include the real definitions of the TestThread classes, but only the "fake" ones. Alas, there is a price to pay for fooling the compiler and for beeing lazy! The price here is, that we have to modify the base class (TestWorker.hpp) by adding a new public virtual method, if we need a new TestThreadA method to be called in the TestMain.cpp file!

Not pretty, uh? You know, sometimes you can go away with breaking the rules, but in the end you'll have to face the consequences. In the art of software design, your skills are measured at your ability to estimate for how long you will go away with laziness and tricks, and when you have to resort to the known, sound and boring "best practices".

But nonetheless, the price I had to pay was OK for the deeper knowledge of the compiler-linker cooperation and the increased knowlede of my own code.