C low level programming

Recently I was repeatedly exposed to low level Windows code using C and Win32, and you know what, I didn't hate it completely! Surprise!

Let's go through the usual complaints:

1. Windows coding style --> horrible at the first sight, but you get used to it (see my previous post).

  bool DeleteRegistryValue (HKEY hKeyRoot, LPCTSTR pszSubKey, LPCTSTR pszValue)
  {
    HKEY hKey;
    LONG lRes = RegOpenKeyEx (hKeyRoot, pszSubKey, 0, KEY_SET_VALUE, &hKey);

    if (lRes != ERROR_SUCCESS)
    {
        SetLastError ((DWORD)lRes);
        return false;
    }

    lRes = RegDeleteValue (hKey, pszValue);

    //... etc

You see what I mean? Hungarian notation, uppercase function names, weird data type macros... But as I said, you'll get used to everything :-\.

2. Low level, unsafe C-library APIs --> come on, every programmer should know C (vide Joel*) and after a while you just know it and can work as quickly with lstrcmp() and others as with std::string (Heaven forbid CString's!)

3. Ugly OS API? --> OK, that's a bummer if you, like me, grew up with clean Unix kernel interface**. But what the heck, it's still system programming, low level, dirty, cool. I a word, I like it!

And do you know what I liked best? It's clean! No DLL dependencies, no hassle with VisualStudio distributables, Qt versions, manifests, load paths etc. Just your code and the system libraries. Small and beautiful. If you have a penchant for software aesthetics, you'll appreciate it too!

--
* http://www.joelonsoftware.com/articles/CollegeAdvice.html
** somehow the feelings of a LISPer forced to code in OCaml or even worse, in Java!

SimpleCV install and "You need the python image library to save by filehandle"

You'll maybe complain that this site is degenerating into a mingle-mangle of bug descriptions, but if you've ever had a pesky problem and couldn't find anything about it on the web, you'll understand my urge to document such things for the fellow hackers.

Here the story goes: SimpleCV is a Python framework trying to provide a simple interface to both OpenCV* an PIL**. There's even a small book about it. I first  installed version 1.2 on my computer, but then wanted to upgrade to 1.3 as there were some segmentation algorithms added (graph-cut etc.) which I wanted to use. Unfortunately, as I installed the new SimpleCV 1.3 and then tried to run my script written for version 1.2 I got:

WARNING: You need the python image library to save by filehandle

when I tried to display any image. Deinstallation of both versions an clean install followed, but nothing changed. Googling for solution didn't help either. What should we do? Just fix it by yourself! The solution is simple. Well, just open this file:

C:\Python27\Lib\site-packages\SimpleCV\base.py

and change the following line***:

  from Image.GifImagePlugin import getheader, getdata

into:

  from GifImagePlugin import getheader, getdata

So, that's it! And happy OpenCV hacking!


PS: PIL seems to be generally a difficult library to use on Windows. If you follow their own docs an write something like

  from PIL import Image

it won't work! You have to use the little bit confusing:

  import Image

--
* OpenCV: a C++ library containing image processing algorithms. For Python there are wrappers exposing direct calls to it, but they aren't that easy to be used directly.

** PIL: the "python image library " from the warning message - the best know imaging library for Python, however it isn't self-contained. You'll need several other libraries like sciPy etc.

*** somehow for Windows distribution the GifImagePlugin isn't a subpackage of Image, and the PIL package just isn't there to be used as prefix! Despite the warning, the problem isn't lack of the PIL installation, but failure to find one of it's subpackages.

Disclaimer

I just had a cursory look on a couple of my latest posts and I was horrified! So many spelling and grammatical errors in there! But then again, I remembered that I've already spotted some of them before but didn't have the courage to correct them.

Courage? Did you really mean courage? Well yes. I'm sorry to say that, but Blogger is rather a difficult platform to be working with (to put it very mildly) and I didn't want to risk losing of all my formatting just because I'd like to correct some spelling.

So let's make an agreement: I won't touch the old posts (sorry...) but the new ones will be seriously proofread and not just jotted down and published in the spur of the moment. Promised!

GetWindowText() and IsEmpty() doesn't work on Windows 7 ?!?!

Normally, when someone comes to you and blames some bugs on compiler, or even worse, on the computer, you'd only smile mildly and give him some of the age-old programming wisdom. I said normally. Because today I tried to debug some Windows code ported from VisualStudio 2005 to VisualStudio 2010, and stumbled upon two impossible-to-think-of bugs.  And as googling didn't bring ther right results immediatelly, I will document it here for myself an others. Let's pray Google indexes it* for the sake of all the poor programmers forced to work on Windows!

As a matter of fact, I've seen a similiar problem when initially porting the code, but it was only one reptitive  bug, so I forgot about it. Here a code snippet as illustration:

  // bug VisStudio 2010: "LPCSTR comparison not working in 64 bit OS"
  // - workaround: use CString
  //int iSelSysPath = LBCBFindStringExact(*this, IDC_CB_SYSTEMPATH, 0, TRUE, m_ActSubrange.cSystemPath);
  CString sysPath(m_ActSubrange.cSystemPath);
  int iSelSysPath = LBCBFindStringExact(*this, IDC_CB_SYSTEMPATH, 0, TRUE, sysPath);

Ok, that may happen, I'm not particuralry picky on such errors. But today, there were two more bugs I didn't catch initially. One ratehr simple, so I only cite it here for reference reasons:

  // bug Windows 7!!!
  //m_cbSysPath.GetWindowText(m_pBMMSett->Station[m_iStation].Subrange.cSystemPath, 
  // ARRAY_SIZE (m_pBMMSett->Station[m_iStation].Subrange.cSystemPath));
  //m_strPath = m_pBMMSett->Station[m_iStation].Subrange.cSystemPath; 
  CString tmp;
  ::LB_CB_GetLBText(m_cbSysPath.m_hWnd, LB_CB_GetCurSel(m_cbSysPath.m_hWnd, true), tmp.GetBuffer(), true);
  m_strPath = tmp;

As you can see, you cannot read the contents of a combo box! Its filled, but you continue to get an empty striong over and over again! Fortunately, like the bug above, this is a known issue, and my collague told me that that GetWindowsText() fails when the addres of the target isn't even-valued. Well, somehow creepy, but if that's a known issue, then what the fuss...

The next bug however wasn't that simple to fix. In fact we stared some 15 minutes at a piece of code where the IsEmpty() method of the MFC's (notorious) CString class reported true for an obviously non empty string! How can things get that wrong? The reason was found after some googling (though some deeper and harder debgging would solve it too), here's the relevant post:

I wonder if your getting a referencing problem.  CString attempts to not create new values if it thinks it can use the same one for multiple occurrences.  I've been caught by this before.  I usually force the issue by using the LPCTSTR operator like:

CString cs = (LPCTSTR) csValue;

Might be worth a try anyway.
Tom  

And he (i.e. Tom) was right, because the following change forced a copy and in result fixed the problem:

 
  //bool empty = m_aStrSettPath.IsEmpty(); <-- not working! internal CString cache issues
  CString p = (LPCTSTR)m_aStrSettPath[i];  
  bool empty = p.IsEmpty()  

But now hold on! The basic operation on strings is broken? Can it get any worse? If you can -stay away from MFC, if you cannot - wel,l I hope you'll be abble to google this post.

---
* an that's the reason for the grammatical error in the title of this post

Coding Styles Again, or Getting LISPy?

Although in my previous post I argued that coding styles and formatting are only smoke and mirrors, latetly I found myself increasingly entangled in a dilemma concerning just that question. Namely, I suddenly found pleasure in writing my C++ statements using expressions rather than direct flow control constructs.
Want an expample? Here it is:

  // housekeeping
  (type == QAmmMeasModel::MeasChannelNode) 
    ? removeItemRef(m_measDefEntries, m_measDefFilesSz, defFileName, defFileIndex)
    : removeItemRef(m_rangeDefEntries, m_rangeDefFilesSz, defFileName, defFileIndex);

instead of the more mundane (or, if you want, normal):

  if(type == QAmmMeasModel::MeasChannelNode) 
  {
    //........
  }
  else
  {
    //........
  }

As such, it is rather to be seen as an innocent idiosyncracy of mine, but it didn't stopped there, culminating in the following substituton. Instead of the rather boring:

 // houskeeping
 if(type == QAmmMeasModel::MeasChannelNode) 
 {
   assert(m_measDefEntries.contains(key));
   assert(m_measDefFiles.contains(defFileName));

   m_measDefEntries.remove(key);
   
   if(m_measDefFiles[defFileName] == 1)
     m_measDefFiles.remove(defFileName);
   else 
     m_measDefFiles[defFileName]--;
 }
 else
 {
   assert(m_rangeDefEntries.contains(key));
   assert(m_rangeDefFiles.contains(defFileName));

   m_rangeDefEntries.remove(key);
  
   if(m_rangeDefFiles[defFileName] == 1)
     m_rangeDefFiles.remove(defFileName);
   else 
    m_rangeDefFiles[defFileName]--;
 }

I came to consider using something more funky, like:

 // lispy?
 (type == QAmmMeasModel::MeasChannelNode) ?
  (m_measDefEntries.remove(key),   
     ((m_measDefFiles[defFileName] == 1) ? 
      m_measDefFiles.remove(defFileName) : m_measDefFiles[defFileName]--)) :
  (m_rangeDefEntries.remove(key),   
     ((m_rangeDefFiles[defFileName] == 1) ? 
      m_rangeDefFiles.remove(defFileName) : m_rangeDefFiles[defFileName]--));

You must admit that the second version is more succint, looks better and transmits some indefinite "guru" feeling. Unfortunately I had to let the asserts go, but it didn't occur to me as a great loss. So why did I delete the second version in the end? Simple. It was written for my client, and should be readable to all C++ programmers. In that way coolness was sacrificed on the altar of mediocrity ;) for the filthy lucre's sake ;).

Later in the day I chose the simpler solution and just wrote the removeItemRef() function hiding the ugly if() clause, but if I didn't do it, what do you think, which of the two versions should I had included in the code?

Static vs Dynamic - the LISP experiences

Maybe you happen to remember my recent post about static and dynamic languages with the main assertion that the type information is maybe superfluous but helpful for documentation and refactoring. You might be tempted to retort: docs & refactoring are for sissies, you wimp! But wait a moment, who are the most notorious hackers out there? You are right, the LISP hackers. From them you'd expect the most ruthless hacks and also despising of the programming aids mere mortals are using. At least as far as the programming folklore is concerned.

But if you a little bit look closer... Heres the answer* given to a simple question "What you dislike the most about Lisp?":

I must say that the lack of static typing really gets in the way on larger projects. Being able to confidently change datatypes and function signatures, knowing that the compiler will point out most inconsistencies that you introduce, is something that I've really come to appreciate after working with Rust and Haskell. Test coverage helps, of course, but I'm not a very diligent test writer, and tend to feel that type signatures are easier to maintain than an exhaustive test suite.

Surprise, surprise, basically he's reiterating my simple argument: why should we give up something that can help us, i.e. automatic checking if all the interfaces are used correctly?

And for the seconds, some  "opinion as a result of ... programming in Lisp in a production environment on non-trivial code"**:

Right off, I can say that the biggest, most glaring thing that causes problems is the lack of static typing. It happens that when the compiler is unable to detect these issues, the end user is left to find them. This can be very potentially damaging to companies. And it is more prominent when you have 100k lines of code, not all of which one can be familiar with.

Lisp takes some steps to help solve these issues. SBCL has very good type inference capabilities, and can warn or error at compile time with type issues. Additionally, Common Lisp allows one to declare and assert argument types, but there’s no guarantee this information can be used. 

And what's SBCL? Let's cite from it's homepage: "Steel Bank Common Lisp (SBCL) is a high performance Common Lisp compiler". What? Even more surprises! Compilers, type inference, assertions? Ok, looks like (at least some of) the LISP hackers came to appreciate the static typing. Who would have thought it?

---
* Lisp Hackers: Marijn Haverbeke - http://lisp-univ-etc.blogspot.de/2012/04/lisp-hackers-marijn-haverbeke.html
** The Perils of Lisp in Production - http://symbo1ics.com/blog/?p=1321

C++ Annotations not so pointless? Plus some UMTS history.

Let me start this post with a retrospection: long long time ago I promised to write a blogpost about multithreaded testing, temporal logic and (even) the boundaries of knowledge. Pretty high claims, you'd say, and you'll be of course right, as I didn't write it despite of a few false starts. So though the wholesale treatment of this theme isn't possible now because I plain and simply forgotten much of the details, I'll try to catch the opportunity to pay off my "blogging debt" and use that stuff in a condensed form as intro for this post.

1. Past struggles

So then, long long ago I was writing an multithreading application (or rather a framework* for other programmers to use) under Linux/pthreads against an underspecified multithreading interface of a "General Telco Systems Programming Platform" ;) - let call it GTSPP;). It was underspecified because it was still being developed while we were writing production code to be deployed on backplane UMTS switches.

Well, what can I say, the regular multinational project's madness, alleged synergy, and all that stuff... But, perhaps surprisingly to you, I wanted to have that thing thoroughly tested! And I mean tested! After I found some unexpected behaviour of GTSPP;) and resulting deadlocks, I didn't want to allow it to happen again.

As at that time I was interested in logic, automatic theorem proving and generally in foundational speculations, I had that brilliant (or at least I thought that...) idea - why don't we just describe a multithrerading system in terms of modal logic, like that:

• []T (read: box T) meaning: T alway happens
• <>T (read: diamond T) meaning: T sometimes happens

 and then use automatic derivation do derive some theorems of thatt system? Like "Theorem 1: there will never be a deadlock here".  [In the originally planned post a lot of explanantions about Kripke, possible worlds, Buchi automata... and model checking would follow here... BUT this isn't the planned post, sorry!]

At last some sensible use of maths! Can't believe it, so these whole math classes weren't a total waste of time?

Well, yes and no. As I tried several open source products, none of them was able to read C++ code, annotate it with modal logic, and infer the needed theorems. You call me naive? But that's exactly what SPIN is doing! The only problem was that SPIN doesn't accept C or C++ code, but only it's own description language: Promela. So I had to recode my system in Promela. That was out of the question, as I didn't have so much time. Writing a general translator C++ to Promela translator was beyond question as well - Clang wasn't there yet!

SPIN's author did some interesting work for AT&T on translating C code to SPIN format and even open-sourced it, but as I tried to install it on my Linux machine I failed. Some package dependency was missing, and our sysadmin asserted that there's no no such package for the RedHat distribution... At that point I gave up** thinking: the real boundaries of knowledge are that of human laziness...

2. New hope?

You may remember that I wrote a post "Two C++ curiosities..." where I expressed my inability to grasp the reason for the introduction of attributes in C++ (apart from Microsoft's influence). Recently, the same Microsoft organized a C++ conference  with quite a good video coverage. As I happened to listen to the Clang presentation i noticed some interesting usage of that language feature. It looked roughly like that:

or in gcc's syntax:

Do you see that? Now you can add an attribute to multithreaded code and let the compiler check if it's deadlock and starvation free!

That's what I call a sensible vendor specific extension! Well, it could be, as at the present all this isn't available yet (it's a research project at Microsoft's) but at least now I learned one case where the attributes could do quite a good job!

PS: More information on that topic: the boost::thread and thread safety annotations post or LLVM docs.

--

* I know, programmers hate frameworks, we were hating GTSPP;) too, but we were merging an another telco framework used in the previous project with the hapless GTSPP;) as not to expose the application programmers to too much madness - you know, brains melting, people screaming, teams disintegrating... So it was a Good Thing in the end!

** You don't really think I gave up on this, do you? So what was my solution? Right! I wrote a Python test driver repeating testcases by 1000s and grepping the output for specified error messages. The classic and proven multithreading testing strategy. Maybe a little brute-force and old-fashioned, but you know what? It did the job!