Saturday, 4 April 2020

My answer to a young programmer


You maybe know that I offer C++ mentoring via Twitter (DMs are open!) and, as it happened, someone approached me and wanted to be mentored. So I asked him for his questions and then wrote some responses, which ended up to to be qute long... Then the idea to share what I wrote in this blog - maybe someonw will find it interesing.

Without much ado, to the questions.

1. Questions

I first asked about the goals. Thes answers are probably in line with many young programmers want:
1. I just need to be able to solve programming problems and strong myself in
this part, to be qualified for big tech companies interviews like Google,
Microsoft and Amazon. 
2. No the programming language doesn't matter for me, I know the basics of C, C++ and Python but I just want to be a problem solver and think like a programmer 
3.  I just want to understand the theory behind Algorithms and Data Structure
Then also came a list of resources that one finds on the internet using uncle Google. I won't show them here, as thea are quoted in the response.

2 The Response

Here's my response, in full, without any later editing:
Hi Xyz!

I had a look at your references:

>Courses:

>Algorithms and Data structures by Revi
>https://youtube.com/playlist?list=PLEbnTDJUr_IeHYw_sfBOJ6gk5pie0yP-0

Here we have much complexity theory but little of algos - only basic sorting methods.

>Algorithms by Abdul Bari
>https://youtube.com/playlist?list=PLDN4rrl48XKpZkf03iYFl-O29szjTrs_O

Don't know what to think about it's TOC - maybe too academic (too much about recurrences) but also covers a broad range of themes you'll need. So-so.

>MIT 6.006 Introduction to Algorithms
>https://youtube.com/playlist?list=PLUl4u3cNGP61Oq3tWYp6V_F-5jb5L2iHb
Had a look the TOC and liked it! If you can grasp these themes, you'll have good working knowledge of Comp.Sci.

>Books:

>Data Structures and Program Design in C by Robert L. Kruse
>https://goodreads.com/book/show/27846.Data_Structures_and_Program_Design_in_C
Looks like it's pretty old. Probably outdated, looks like waste of time.

>Introduction to Algorithms By Thomas H. Cormen
>https://mitpress.mit.edu/books/introduction-algorithms-third-edition

This is a classic, maybe a little heavy on the academic side, but a good reference to check what the options are. For an algorithms book the "The Algorithm Design Manual" by Steven Skiena is a much lighter read I'd recommed!

>Think Like a Programmer by V. Anton Spraul
>https://goodreads.com/book/show/13590009-think-like-a-programmer

I don't know this one. Had a look at the TOC and it can be OK or even good. It has a couple of programming problems but the title seems to be kind of a clickbait. Notheless. maybe it's good.

>Practice:

>https://hackerrank.com

Don't know this one (sorry) and couldn't check it out, as it requires a login. Back in the day Google recommended topcoder.com for testing your problem-solving skills in competitions, but I don't know the current status of it.

>https://leetcode.com

Don't know this one (sorry), but it looks somehow inetersting at first glance. You have specific questions for Google, Facebook, etc. Could be worth it.

And responses:

>I just need to be able to solve programming problems and strong myself in this part, to be qualified for big tech companies interviews like Google, Microsoft and Amazon.

A classic book for that is "Cracking the Coding Interview" by Gyle Laakmann McDowell - but it's a real tome! "Programming Interviews Exposed" by J. Morgan et. all was a lighte read. "Algorithms for Intreviews" by A. Aziz & A. Prakesh has many tough problems, but the solutions aren't that well explained - a challenge!

>Q1: No the programming language doesn't matter for me, I know the basics of C, C++ and Python but I just want to be a problem solver and think like a programmer

To be a (real) problem solver you need some deep knowledge in at leas one OO-langauge! Programmers think in patterns, you have to learn them by solving programming problems! On a little higher level, you'll need at least cursory knowledge in design patterns - look at the 4 Amigos book!

>Q2: I just want to understand the theory behind Algorithms and Data Structure

You'll have to undertstand the O-notation at least and to be able to estimate the O-complexity of your algoriths. The Master theorem is not really needed though. You have at least to understand NP-complete complexity class (NP=P?). You'll have to be able to estimate run time of an algorithm - which algos would need thousands of years to complete?

Regards,
Marek
3. Summing Up

Hopefully some young programmer will find it helpful!

Trip Report Qt Word Summit, Berlin, 5-6 November 2019


You've probably noticed that I'm not a hell of a conference goer, didn't you? The last conference I attended was the first "Meeting C++" conference in 2014, at that time still taking place in Neuss, Rhineland* instead of the worldly and hipster Berlin.

I tried to go to the two consecutive Meeting C++ conferences in Berlin, but I was both times unable to do it, so eventually I gave up and called it a day on this idea.

But because recently I got somehow more seriously involved with the Qt framework** I thought, hey, why not just go to this one Qt conference*** in Berlin and visit some of my people there as well? No sooner thought than done. And now, three months something later, another idea struck me - why not write up some impressions? So let us go over things I came to remember most vividly.

Of course you could start moaning that it is much toooo late for a trip report, but better late than sorry, so let us begin!

1. Lars Knoll's talk

Lars Knoll, the CTO of the Qt Company was giving a talk on general directions for the future Qt 6 development and it was rather interesting. You can check it out here (PDF), but basically QML will be reworked to be a typed programming language, Javascript will be only optional and graphics will support not only OpenGL but also newer APIs like Vulcan, Metal or Direct3D 12. All that's very interesting, but what struck me the most was the following slide:


As it seems Qt is predominantly used on Windows! That's kind of surprising, taking into account that all the books and articles you read always assume Linux as a platform! I personally did a lot of Qt work on Windows and couldn't ever understand that.

This slide gave me a nice, fuzzy feeling, because my decision to write my Qt book from a point of view of a Windows developer was a right one!

Other books always describe the tooling you can use on Linux, but this doesn't help me on Windows, because most of that tooling (open source) doesn't exist there. So writing of the Chapter 2 (Profiling and Profilers) was a kind of a discovery journey - how can I make open source tooling run and correctly work on Windows?

2. Keynotes

As much as I don't like keynotes - normally they are rather full of business-y bla bla - but one slide of the Natalie Nahai's presentation really impressed me, I mean this one:
You can see the change of the vibes of a webpage from the left (boring, lots of distractions, unusable!) to right (cool, clear, using 1-click donation buttons). Nice! And it really happened during Obama's campaign.

3. Caliatys hydrogen mobility talk

This one was also a keynote (or I think so...) but the CTO of the company has shown us the slide containing a decision table they used for choosing the Qt framework as a platform for mobile development. The application in question should run on both iPhone and Android device. After the recent switch of Dropbox away from a commonn C++ codebase for both platforms you would maybe expect that C++ isn't good for mobile.

See their reasons for choosing Qt in the pic below:


The cloud connector had to be written separately for each platform, apart from this Qt provided for platform-independence!

4. QML on Microcontrollers

I already said, that the new QML will be a compiled, statically typed language in Qt 6. Already today, we can have a foretaste of the future with Qt Quick Ultralite (Qt QUL) that can be run on MCUs (microcontrollers) - a feat in itself, as up to this day Qt and QML needed a much more beefy processors to be run onto! There were quite a few talks centering around that theme, because it seems to be the future of Qt.

Qt Quick Ultralite is thus basically a subset of QML which can be compiled to C++ and run on bare metal (or an RTOS). This is also a direction in which QML will be going in Qt 6. I was pleasently surprised to learn, that Qt QUL was developed by the Qt Company in cooperation with a polish enterprise that ported it to many MCUs (see the picture below πŸ‘‡).


As it's my native country, I couldn't help but to be very pleased. Good job Poland! 😍

5. People and stories

As it goes, I stumbled upon 2 of my old clients at the conference! I tool the opportunity and asked them about how my old projects fared after I left. We had quite good chats and I wanted to talk to them on the next day also, but - you won't believe it - I couldn't spot them anymore! The conference is a pretty big event with masses of people and finding someone is rather matter of luck.

I also could meed some (i.e. two) Qt-involved people I only knew from Twitter in the real world. As I expected they turned out to be very nice! Plus I also wound up talking to people in the queue to the canteen and at the exhibitor stands - I even could ask Qt's CTO a question face-to-face after his talk.

Now let us proceed to the "stories" part of this section:

One of my customers I met at the venue told me a nice story right form the threnches, They had a beast of a performance problem with the embedded device in my old project (accidentally same project as in the perf. story 6.4 in the book**), namely, after a day being powered on, the device started to slow down up to a total grind - the UI became totally unresponsive and you had to reboot it. What is your guess for the reason of that?

Yes, you are right, a memory leak.

But you'll never guess what couse it! When displaying the time of the day, on every update (i.e. every second) the displayed string was changed and then restyled using CSS styles. As it turned out, there was a bug in the Qt version we used and restyling leaked memory at that place! How could you possibly find that?!?!, I asked, shocked. Well, it was by the old and proven method of pinching off parts of the programm until the problem disppears. A kind of "divide and conquer" of debugging.

Want another story? Here we go - I met 2 guys in the queue in the canteen and chattet with them for quite a long time. At the evening, I went out with my Berlin people to their favourite craft beer bar in the neighbourhood. After a couple of pints I catched a glimple of some guy at the counter and had to ask myself - do I know that guy? No, impossible, only locals frequent this bar... But after a couple of further glimpses I just had to settle that and chatted up the guy and his friend. Guess what - they were the exact people I met at the conference. Then we had one more chat and I even could pitch my book** to them! Moral - everybody can google, there isn't such a thing like a hidden gem or insider-only venue. πŸ˜”

6. Merchandise

Of course I also got some merchandise:
  • Qt sticker set,
  • a T-shirt and a lanyard keychain from Qt πŸ‘‰,
  • sleeping mask (Batman style!) πŸ¦‡, also Qt
  • some small stickers from exhibitor companies 
But not that much I'd expect. There was a raffle where you could win a cool drone (sponsored by one of the exhibitors), but this time I was kinda outta luck.

7. Summing up

You tell me - do you think it was a good conference? The main lecture hall was very impressive in any case. An you could see the Berlin TV-tower right out of the window.

---
* early supporter! πŸ™‹

** see this book: https://www.packtpub.com/application-development/hands-high-performance-programming-qt-5

*** i.e. to the Qt World Summit conference. BTW, here are all QtWS-2019 videos to watch: https://resources.qt.io/qt-world-summit-2019

Saturday, 29 February 2020

Export your Qt Project from VisualStudio to CMake (...or how I stopped worrying and learned to love CMake!)


Yes, I know, this blog was supposed to discuss the high-church themes of languages and system design, and I didn't quite abandoned these goals, but at the momenet I don't have many time to write about deep and interesting things (as you guess, it's time-consuming). To make a long story short, I decided that I'll write about things I'm working on/with just now.

So let us start with CMake and Qt.

1. Intro

You might already have learned that in the next major Qt version (it will be Qt 6) the framework will abandon its old and trusty qmake_build tool and switch to CMake. Why's that?

Although as of lately CMake has acquired a standard tool status in the C++ community, I don't think the desire to be with the cool crowd motivated that change. Rather (AFAIK), qmake's source code developed into an unmaintainable mess, so that nobody wanted to change anything for fear of a total disaster (but feel free to correct me if I'm wrong).

So, as the last alternative to CMake went away, I couldn't advise my client to use qmake for cross platform builds on Windows and Linux with a clear conscience, so in the current project I voted in favour of CMake as our build tool.

You might ask yourself "So what? Just use another tool!". Well, CMake has quite a reputation for being overcomplicated and error-prone. Additionally, it has a text based format, so you will have to become friends with the text editor again.

Before that we could just use convenient graphical IDEs to define the structure of a project, should we lose that comfort now!? If I use Visual Studio with the Qt plugin, I can just drag and drop the Q_OBJECT files into a project and everything just jumps into its place. And what can I do if I already started out with a Visual Studio in first place (as I did in the current project, silly me)!?

2. Convert your Visual Studio Project

So I was loking for some converter of Visual Studion projects and I stumbled upon the the cmake-converter Python tool*. I decised to give it a try and:

1. installed Python 3 first (just googled for the installer and let ir run)

2. Installed the last stable release of cmake-converter:
  pip install cmake-converter
3. Let it run it with:
  cmake-converter -pi -s path/to/file.sln
Where -pi means that we ant use private** includes with CMake and -s specifies a path to the soultion file. You can list all the tool's options using the -h switch. You will see the following output (here I used the verbose output option):


The tool will generate the CMakeLists.txt file as result.

After reading the result file I decided to delete all the generated contents up to the "Source Groups" section and replaced it by my generic Qt-project prologue, i.e.:
  if(UNIX)
   cmake_minimum_required(VERSION 3.10)
  else()
   cmake_minimum_required(VERSION 3.13)
  endif()

  set(CMAKE_CXX_STANDARD 17)
  set(CMAKE_CXX_STANDARD_REQUIRED ON)

  project(GdmRestApiTest VERSION 0.1.0 LANGUAGES CXX)

  ################################################################################
  # Use solution folders feature
  ################################################################################
  set_property(GLOBAL PROPERTY USE_FOLDERS ON)

  ################################################################################
  # Use Qt
  ################################################################################

  set(CMAKE_AUTOMOC ON)
  set(CMAKE_AUTORCC ON)
  set(CMAKE_AUTOUIC ON)

  #set(CMAKE_AUTOUIC_SEARCH_PATHS "src/ui")

  find_package(Qt5 COMPONENTS Core Widgets Gui Network REQUIRED)
Why did I do that? First, the generated CMake file uses some custom Cmake-Tools file which was a little too complicated at the first sight, and secondly, the generated CMake code doesn't take into account that we are using a Qt plugin!

Thus we added the AUTO-MOC/RC/UI lines and the required Qt packages. However, for that to work we have to tell CMake where the Qt framework files are to be found. It can be done by setting an environment variable CMAKE_PREFIX_PATH to the directory wher the Qt CMake integration files can be found, In my case:

  CMAKE_PREFIX_PATH=C:\Qt\Qt5.12.5\5.12.5\msvc2017_64\lib\cmake .

Next we need to manually add the Form_Files and Resource_Files source groups:
  set(Form_Files
      "./GdmRestApiTest.ui"
  )
  source_group("Form Files" FILES ${Form_Files})

  set(Resource_Files
      "./GdmRestApiTest.qrc"
  )
  source_group("Resource Files" FILES ${Resource_Files})
Now the generated source groups can be reused.

Next we need to include the Form_Files and Resource_Files source groups in ALL_FILES so that AUTO-UI/RC can be applied:
  set(ALL_FILES
      ${Form_Files}
      ${Resource_Files}   
      ${Header_Files__generated}
      ${Source_Files}
      ${Source_Files__generated}
  )
Now, all the source files of the project are specified and we can turn to the definitions needed by compiler and linker. Again I deleted the Target and all the consecutive sections (because, you know, too complicated...) and replaced them by my generic Qt definitions:
  ################################################################################
  # Target
  ################################################################################
  if(MSVC)
   # set /SUBSYSTEM:WINDOWS
   add_executable(${PROJECT_NAME} WIN32 ${ALL_FILES})
  else()
   add_executable(${PROJECT_NAME} ${ALL_FILES})
  endif()

  ################################################################################
  # Include directories
  ################################################################################
  target_include_directories(${PROJECT_NAME} PRIVATE
      "${CMAKE_CURRENT_SOURCE_DIR}/../qt5cpp-openapi/client;"
      "${CMAKE_CURRENT_SOURCE_DIR}/GeneratedFiles;"
  )

  ################################################################################
  # Compiler flags
  ################################################################################
  if(MSVC)
     # quiet warnings related to fopen, sscanf, etc.
     target_compile_definitions(${PROJECT_NAME} PRIVATE _CRT_SECURE_NO_WARNINGS)
  endif()

  # force more warnings!
  target_compile_options(${PROJECT_NAME} PRIVATE
    $<$<OR:$<CXX_COMPILER_ID:Clang>,$<CXX_COMPILER_ID:AppleClang>,$<CXX_COMPILER_ID:GNU>>:
      Wall>
   $<$<CXX_COMPILER_ID:MSVC>:
     /W4>)

  ################################################################################
  # Dependencies
  ################################################################################
  set(ADDITIONAL_LIBRARY_DEPENDENCIES
    "Qt5::Core;"
    "Qt5::Gui;"
    "Qt5::Network;"
    "Qt5::Widgets"
  )
  target_link_libraries(${PROJECT_NAME} PUBLIC "${ADDITIONAL_LIBRARY_DEPENDENCIES}")
As we see, they define the executable, additional include directories, the compiler flags we need and the library dependencies for the linker. Ready!

3. Run your CMake project

Now it's time to test it out. As Visual Studio has acquired support for directly working with CMake lately, we can use it to test our CMake file right away!

In my Visual Studio 2019 installation (version 16.3.8, Professional) I do the following:
  • Start Visual Studio
  • Choose "Proceed without file" in the starting screen that appears first
  • In Visual Studio's IDE I click "File" > "Open" > "CMake" menu item and choose the CMakeLists.txt file we created.
  • Now we have to wait, as Visual Studio tries to perform the "Configure" and "Generate" steps for the CMakeLists.txt file.
  • After that, we should be able to build the project - select the CMakeLists.txt file in the project panel on the left and choose "Build" in it's context menu (see the pic below).
  • If you want to start the debugger, select the CMakeLists.txt file in the project panel and choose "Set as Current Project". Then just start the debugger clicking on the green arrow button in the toolbar (see the pic below).

Sometimes the automatic CMake generation might fail, or you'd like to reset the project. In that case you select select the CMakeLists.txt file in the project panel and
  • choose "Delete CMake Cache" to reset the project
  • choose "Generate CMake Cache" to recreate the build files
  • start the build as described above
Ready! To thest my Qt application on Linux I have only to:
  • check out the project on the Linux machine
  • open it in Qt Creator
  • and let it build and run!
4. The Moral

I hope you fear CMake no more now! If we aren't overcomplicating things, CMake might indeed be a pretty nice and handy tool for building projects on both Windows and Linux (and of course also on other platforms).

The technique I currently use is:
  • add all .cpp and .hpp files to the Visual Studio project
  • export the project using the cmake-converter tool
  • replace all generated content beside the file lists with my generic CMake prologue and epilogue
  • slightly adapt prologue/epilogue if needed
This works and won't cost you a lot of time.

--
 * "CMake converter for Visual Studio projects" - https://pypi.org/project/cmake-converter/

 ** private CMake includes (i.e. include directories) aren't exported to other CMake projects,

Wednesday, 18 December 2019

No cursor in your embedded Qt-App on iMX-6?


This one is a continuation of (meanwhile) a series of bughunting/bugfixing posts*.  As I already wrote once:
"You'll maybe complain that this site is degenerating into a mingle-mangle of bug descriptions, but if you've ever had a pesky problem and couldn't find anything about it on the web, you'll understand my urge to document such things for the fellow hackers."**
So I'll just document this one gotcha to help the googling programmer, because on that way I received help from others multiple times. I hope it'll make someone's life easier.

1. The Problem

In the current project for one of my customers I'm writing a Qt UI for an application running on embedded Linux. We are using the open source version of Qt, so we build the system image with yocto using standard Qt5 recipies. We are also using the tried and tested iMX-6 board, that was used in iPhones time and ago.

This board has a GPU and we are using Qt's EGLFS platform plugin which enables standard OpenGL graphics. Everything should hunky-dory, no surprizes, this configuration is well-tested, and it should just work. And it worked, a demo Qt application showed on the attached VGG screen. But then we plugged in a mouse, whisked it to and fro, but nothing happened. No mouse cursor on the screen!

2. The Solution

I tried to google the problem away, but no ready-made solution could be found. But then a small hint set the ball rolling, namely this one:
"... very likely the hardware or the drivers doen't support a hardware-accelerated mouse cursor."
Wait, wait... And what does the "i.MX Graphics User’s Guide" say about this exaclty? Let have a look:
"10.1.5 Cursor
  Hardware IPU does not provide a hardware cursor."
In no uncertain terms! 

Problem solved: Qt will by default try to use the hardware cursor to speed things up, but in such a case it will fail in name of performance. A very honourable end - dulce et decorum est pro performancem mori, as the ancient Romans used to say! 

After we clearified that question the fix "follows trivially" πŸ˜‰ - we only have to turn on Qt's software cursor option. How to do that? As Qt's documentation says:
"The KMS/DRM backend also supports custom configurations via a JSON file."
This is our way. We habe to create a JSON config file and point to it with the QT_QPA_EGLFS_KMS_CONFIG environment variable. The contents should be looking roughly like that:

  {
    "device": "/dev/dri/card0",
    "hwcursor": false,
    "pbuffers": true,
    "outputs": [
      {
        "name": "LVDS0",
        "mode": "800x480"
      }
    ]
  }
with "device" set to your device name and "pbuffers" enabling usage of standard EGL pbuffer surfaces.

3. The Moral

There isn't one this time πŸ˜•. I you can think about one that could fit in here, say it in the comments.

--
* like this, thisthis, this and even this one for Python!
** in one of the above posts here.

Friday, 5 July 2019

VTune Amplifier and profiling MinGW on Windows


You might perhaps know that I recently wrote a book (and blogged about it here). In its second chapter, where we discuss profiling and profilers, I mentioned Intel's VTune Amplifier, but because the book only uses open source tools I just wrote that:
"The commercial Intel performance suite comprises of VTune Amplifier, Intel Advisor, and Intel Inspector. VTune Amplifier is an advanced sampling profiler using hardware and OS counters, providing information on CPU, threading, memory, cache, and storage…"
Admittedly, there was some free version of VTune at the time, but you had to request it and you could use it only for a limited time for evaluation (if my memory serves me right), so I just classified it as a commercial tool.

But this week I learned, that there is a free version of VTune 2019 everybody can use. I didn’t believe that but I went to VTune’s homepage to check it – and indeed, an entirely free versions is available there. The commercial version simply comes with commercial-level support, and there are no other differences AFAICS.

Stress-Test with MinGW

So I downloaded the free version and wanted to try it. I knew that Intel tools integrate tightly with Visual Studio, and that Intel’s compiler is (still is?) binary compatible with Microsoft’s compilers, so I was interested if VTune would be able to understand MinGW’s symbol information. I know for sure, that in the past VTune only could understand Microsoft’s debug info format.

I wanted to test it out, because in my book I am using the standard Qt distribution for Windows (at the time of writing it was Qt 5.9) and this distribution is using MinGW as compiler. This also means that the provided Qt DLLs contained symbol information is stored in the MinGW format (i.e. Dwarf2). 

To test VTune's support for various debug info format I profiled a following application of mine:
  +-----------------+
  |                 |          +-------------+
  |                 |          |             |
  |     QML GUI     +--------->+   QProcess  |
  |                 |          |             |
  |     (MinGW)     |          +---+----^-+--+
  |                 |              |    | |
  +-----------------+        start |    | | cin, cout, cerr
                                   |    | |
                                   |    | |
                                   |    | |
                            +------v----+-v---------+
                            |                       |        +------------------+
                            |                       |        |                  |
                            | "Backend" Executable  +--------+   Some SDK DLLs  |
                            |                       |        |                  |
                            |    (Visual Studio)    |        |      (MinGW)     |
                            |                       |        +------------------+
                            +-----------------------+
As you can see, it’s rather a complicated situation where a MinGW-based GUI is starting a Visual Studio-base executable that is loading some SDK DLLs that export a plain C API, but are internally using MinGW! So let have a try and see how VTune will cope with that mess. 

Surprisingly, when I started VTune and let my example application be started and profiled by it I saw the following picture:



We see that VTune can indeed see and understand symbols from Qt’s QML libraries, as well as from my own C++ code in some QML extension modules, which were also compiled with MinGW. A great hooray! But that’s not enough – on closer inspection I found out, that also the “backend” executable symbols (Microsoft format) were visible in profile data, and then even one step further, i.e. the internal symbols of the SDK DLL it is using (MinGW format again)!

Surprise, surprise, it seems to be working rather neatly. Nice one! It turned out that in a new profiling project the "Analyze child processes" option is turned on by default. I had only to provide the location of my sources, everything else just worked!

Summing Up

As an addendum (or call it correction) to the book, I can say that the VTune Amplifier is another option when profiling Qt applications on Windows!

Besides the traditional profiling to find execution hotspots, VTune has a very nice support for diagnosing multithreaded performance, visualizing lock contentions, it can help with the Intel CPUs by showing cache misses and pipeline stalls, it can diagnose memory access bottlenecks and also profile entire system with multiple applications. You’ll admit it’s quite a mighty* tool!

--
* Want to see ho to use all these features? There is a video guiding you around the GUI here, and here is an article about "Thread, Memory, and Vector Optimizations" using VTune and Intel Advisor, showing how to diagnose and improve usage of the available cores, optimize cache usage and introduce vectorization.

Sunday, 28 April 2019

For last 2 days - my book for $10 only!


There are a couple of days left for you all, if you want to get my book for $10 only!


In the course of the "Spring $10 Campaign" on packtpub.com you can purchase the eBook version of my "Hands-On High Performance Programming with Qt 5" book for $10 only until 30th of April.

So hurry on! here's the link: https://www.packtpub.com/application-development/hands-high-performance-programming-qt-5.

Get it when it's still fresh. It only came out this January!

PS: Sorry for the short notice, but somehow I overslept... Shame on me. 😞 Sorry.

Monday, 25 February 2019

Holy cow, I wrote a book!


I always wanted to say that since I saw Raymond Chen's blog, and now. at last, I can! The title of the book is "Hands-On High Performance Programming with Qt 5" and you can get it from Amazon (here) or Packt Publishing (here). The book's cover looks like that:


I have to say that I rather like the look of it!

TL;DR

A new book About Qt's and C++ performance. Uses Qt 5.9 - 5.12 on Windows. Guides the reader from an intermediate to the (lower) advanced level. 4 of 5 stars πŸ˜‰.

There are also code examples illustrating some of the discussed concetps (to be found at https://github.com/PacktPublishing/Hands-On-High-performance-with-QT).

What this book is about.

This book is about many things. Judging from the title it's about performance optimization of Qt 5
(Qt 5.9 - Qt 5.12) programs. But because Qt's underlying language is C++ it is also about C++ optimizations and performance. Because C++ is a close-to-the-metal language (some would say low-level) we also discuss the hardware architectures. Also because Qt framework covers so many areas we also discuss data structures and algorithms, multithreading, file I/O and parsing, GUI and graphics, networking, and even mobile and embedded platforms.

Quite a mouthful, you'd say? Yes, it's true. But it also makes it interesting, and shows how many facets there are to the performance optimization.

Additionally, I decided to use Windows as the development platform for this book. Most Qt books use Linux, probably because Qt Creator IDE offers a much better tooling there. But because Windows is also a very popular platform I wanted to take the road less traveled and try to find out how to compensate the lack of the standard Linux performance tools on Windows.

We use only open-source or free tools, Qt Creator as development environment and Qt 5.9 LTE version for code examples (as it was the last LTE version at the time I started writing this book).

Why did I write it?

Well, because I always found performance optimization very interesting. And because I have been working with Qt for quite a long time now, nothing to say about C++! So, as the publishers approached me with a proposed book title I knew I could write rather a good book about it!

The second reason was that there wasn't a resource about performance optimization I would be content with. There area a couple of C++ performance books, but I wasn't entirely convinced by them. As of Qt and its specific problems there was nothing! Well, there was a host of information scattered in blogs, Stack Overflow, books, articles and Twitter threads. So I thought that before I forget it all, I'd better put it to paper!

I also wanted to write a book I'd have liked to read when I first started to learn about performance and optimizations, because there wasn't such a thing either. Thus I settled on a intermediate-level, approachable, but not trivial book.

What material is covered

This book was planned as an intermediate level read. If you can write some C++ and have learned to write basic Qt application then it is for you. No further knowledge is requires, as every topic (e.g. data structures, graphics, networking) will be introduced in a understandable and (I hope so) entertaining manner.

My initial plan for the book was:
  • Part I - Basics
  • Intro - basic performance wisdom and techniques, hardware architecture and its impact.
  • Profiling - performance tools and how to use them. As I said, we don't want to make it easy for us and look at the Windows platform!
  • C++ - how performant is C++ really? And what are optimizations the compiler (and linker) can do?
  • Part II - General Techniques
  • Data structures and algorithms - what is the performance of Qt containers and strings? Do we have to use them?
  • Multithreading - Qt's take on threading and how to speed up your program with concurrency.
  • Fails - some more memorable performance problems I encountered in my career
  • Part III - Selected Qt Modules
  • File I/O etc - Qt and file system performance, JSON and XML parsing. Also memory mapped files and caching.
  • Graphic - GUI, widget and QML performance. Probably the most interesting part of the book.
  • Networking - network performance and network support in Qt. Because we live in the ubiquitous connectivity era!
However, the editors wished two additional chapters, so I also added them (unfortunately, this made the three-part structure obsolete):
  • Mobile and embedded - how to use Qt on mobile and embedded (a kind of advanced chapter applying all we have learned before to two specific platforms)
  • Testing - a goodie chapter about testing techniques, GUI testing and performance regression tests. At the end this chapter turned out pretty interesting!

I really think that this book can do a good job to guide the reader from an intermediate to the (lower) advanced level!

Table of Contents

On the publisher's book page there is a TOC (here), but it is very high level, and doesn't really show the real contents of the book. It goes rather monotonously like "Some intro for topic", "Qt classes fro that", "Performance techniques for that", "Summary", "Questions" and "Further reading". I agree with you that one cannot judge the level and quality of the book from that. It could be everything - from horrible to superb.

For that reason I include here a complete TOC as it appears in the book, so you can have a better idea of themes covered. Without much further ado, here it is:

   
OK, I tried, but it's way too long, it follows at the end* of the post.


You can see from that more detailed TOC that there is a wealth of information and techniques in there! I really think that This book can do a good job to guide the reader from an intermediate to the (lower) advanced level!

The "Questions" and "Further Reading" sections

As unseemly as they look in the TOC, these sections were my secret personal favorites. The "Questions" sections contain, as you probably guessed, some questions to test your understanding of themes discussed in the given chapter, but they also will try to deepen your understanding and sometimes even to introduce new and interesting information! I enjoyed writhing them because it was fun trying to find out how could I keep an intelligent reader still interested after the real material was already introduced.

As this is an intermediate-level book, and because I didn't want to write a 500-600 pages tome, it couldn't go deep on every of the introduced themes. Because of that I also included the "Further Reading" sections, where more advanced materials are referenced. Qt covers so many areas, and some themes like networking, graphics, embedded or mobile are so deep, that you will definitely need to consult more books and articles! In hindsight I should have to include even more references, but you know, If I had more time etc...

Summary

If I had more time, this book could (of course) be much better. But even so I'm rather content with it, considering that I wrote it in half of a year only, in parallel with my normal working hours. I hope you will enjoy it!

PS: Maybe I will start some kind of online addendum/errata for this book, as there are quite many things I'm still leaning after I finished writing it.

--
* Here comes the TOC in its whole glory:










Preface

Chapter 1: Understanding Performant Programs 1

  Why performance is important 1

    The price of performance optimization 2

  Traditional wisdom and basic guidelines 2

    Avoiding repeated computation 4

    Avoiding paying the high price 4

    Avoiding copying data around 5

    General performance optimization approach 6

  Modern processor architectures 7

    Caches 7

    Pipelining 8

      Speculative execution and branch prediction 10

      Out-of-order execution 10

    Multicore 11

    Additional instruction sets 12

    Impact on performance 13

      Keeping your caches hot 14

      Don't confuse your branch predictor 15

      Parallelizing your application 16

Summary 16

Questions 17

Further reading 17


Chapter 2: Profiling to Find Bottlenecks 19

  Types of profilers 20

    Instrumenting profilers 20

    Sampling profilers 21

    External counters 22

      Note on Read Time-Stamp Counter 22

  Platform and tools 22

    Development environment 23

    Profiling tools 24

      Just use gprof? 25

      Windows system tools 25

      Program profiling tools 27

      Visualizing performance data 30

    Memory tools 30

  Profiling CPU usage 31

    Poor man's sampling technique 31

    Using Qt Creator's QML profiler 32

    Using standalone CPU profilers 35

      Reiterating on sampling profiling's limitations 39

  Investigating memory usage 39

    Poor man's memory profiling 40

    Using Qt Creator's heob integration 41

  Manual instrumentation and benchmarks 45

    Debug outputs 45

    Benchmarks 46

      Benchmarks in regression testing 46

  Manual instrumentation 47

  Further advanced tools 48

    Event Tracing for Windows (ETW) and xperf 48

      Installation 48

      Recording and visualizing traces 50

      Conclusion 53

    GammaRay 54

      Building GammaRay 54

      When can we use it? 56

    Other tools 57

      Graphic profilers 57

      Commercial Intel tools 58

      Visual Studio tools 58

  Summary 59

  Questions 59


Chapter 3: Deep Dive into C++ and Performance 61

  C++ philosophy and design 61

    Problems with exceptions 62

       Run-time overheads 63

       Non-determinism 63

       RTTI 64

       Conclusion 64

    Virtual functions 64

  Traditional C++ optimizations 65

    Low-hanging fruit 65

      Temporaries 66

      Return values and RVO 66

      Conversions 67

    Memory management 68

      Basic truths 68

      Replacing the global memory manager 69

      Custom memory allocators 70

        Where they do make sense 71

        Stack allocators 71

        Conclusion 71

      Custom STL allocators 72

    Template trickery 73

      Template computations 73

      Expression templates 74

      CRTP for static polymorphism 75
      Removing branches 76

  C++11/14/17 and performance 77

    Move semantics 77

      Passing by value fashionable again 78

    Compile time computations 78

    Other improvements 80

  What your compiler can do for you 81

    Examples of compiler tricks 81

    More on compiler optimizations 85

      Inlining of functions 86

      Loop unrolling and vectorization 86

    What compilers do not like 87

      Aliasing 87

      External functions 88

    How can you help the compiler? 89

      Profile Guided Optimization 90

    When compilers get overzealous 90

  Optimization tools beyond compiler 92

    Link time optimization and link time code generation 93

    Workaround – unity builds 93

    Beyond linkers 94

  Summary 95

  Questions 95

  Further reading 96


Chapter 4: Using Data Structures and Algorithms Efficiently 97

  Algorithms, data structures, and performance 98

    Algorithm classes 98

      Algorithmic complexity warning 100

    Types of data structures 100

      Arrays 100

      Lists 101

      Trees 101

      Hash tables 102

  Using Qt containers 103

    General design 103

       Implicit sharing 103

       Relocatability 105

    Container classes overview 105

       Basic Qt containers 106

       QList 106

       QVarLengthArray 107

       QCache 108

    C++11 features 108

     
Memory management 109

    Should we use Qt containers? 110

  Qt algorithms, iterators, and gotchas 110

    Iterators and iterations 111

    Gotcha - accidental deep copies 111

  Working with strings 113

    Qt string classes 113

      QByteArray 113

      QString 114

      QStringBuilder 114

      Substring classes 115

    More string advice 115

      Interning 115

      Hashing 116

      Searching substrings 117

      Fixing the size 117

  Optimizing with algorithms and data structures 118

    Optimizing with algorithms 118

      Reusing other people's work 120

    Optimizing with data structures 120

      Be cache-friendly 121

      Flatten your data structures 122

      Improve access patterns 122

        Structure of arrays 122

        Polymorphism avoidance 123

        Hot-cold data separation 123

        Use a custom allocator 124

      Fixed size containers 124

      Write your own 125

  Summary 125

  Questions 125

  Further reading 126


Chapter 5: An In-Depth Guide to Concurrency and Multithreading 128

  Concurrency, parallelism, and multithreading 128

    Problems with threads 130

    More problems – false sharing 131

  Threading support classes in Qt 133

    Threads 134

    Mutexes 134

    Condition variables 135

    Atomic variables 136

    Thread local storage 136

    Q_GLOBAL_STATIC 136

  Threads, events, and QObjects 137

    Events and event loop 137

    QThreads and object affinities 138

      Getting rid of the QThread class 142

    Thread safety of Qt objects 142

  Higher level Qt concurrency mechanisms 142

    QThreadPool 143

    QFuture 143

    QFutureInterface 145

      Should we use it? 146

    Map, filter, and reduce 147

    Which concurrency class should I use? 149

  Multithreading and performance 150

    Costs of multithreading 150

      Thread costs 150

      Synchronization costs 151

        QMutex implementation and performance 151

      Atomic operation costs 151

      Memory allocation costs 152

      Qt's signals and slots performance 152

    Speeding up programs with threads 153

      Do not block the GUI thread 153

      Use the correct number of threads 154

      Avoid thread creation and switching cost 154

      Avoid locking costs 154

        Fine-grained locks 155

        Lock coarsening 155

        Duplicate or partition resources 156

        Use concurrent data structures 157

        Know your concurrent access patterns 157

        Do not share any data 157

        Double-checked locking and a note on static objects 158

      Just switch to lock-free and be fine? 159

        Lock-free performance 159

        Progress guarantees 160

      Messing with thread scheduling? 161

      Use a share nothing architecture 162

        Implementing a worker thread 162

        Active object pattern 163

        Command queue pattern 164

  Beyond threading 164

    User-space scheduling 164

    Transactional memory 165

    Continuations 165

    Coroutines 166

  Summary 167

  Questions 168

  Further reading 168


Chapter 6: Performance Failures and How to Overcome Them 170

  Linear search storm 170

    Context 171

    Problem 172

    Solution 172

    Conclusion 173

  Results dialog window opening very slowly 173

    Context 173

    Problem 174

    Solution 174

    Conclusion 174

  Increasing HTTP file transfer times 174

    Context 175

    Problem 175

    Solution 176

    Conclusion 177

  Loading SVGs 177

    Context 177

    Problem 178

    Solution 178

    Conclusion 179

  Quadratic algorithm trap 179

    Context 180

    Problem 180

    Solution 180

    Conclusion 180

  Stalls when displaying widget with QML contents 181

    Context 181

    Problem 181

    Solution 182

    Conclusion 182

  Too many items in view 182

    Context 183

    Problem 183

    Solution 183

    Conclusion 183

  Two program startup stories 184

    Time system calls 184

    Font cache 184

    Conclusion 185

  Hardware shutting down after an error message 185

    Context 185

    Problem 185

    Solution 185

    Conclusion 186

  Overly generic design 186

    Context 186

    Problem 187

    Solution 187

    Conclusion 187

  Other examples 187

  Summary 188

  Questions 189

  Further reading 189


Chapter 7: Understanding I/O Performance and Overcoming Related Problems 190

  Reading and writing files in Qt 191

    Basics of file I/O performance 191

      Buffering and flushing 191

      Tied and synchronized streams 192

      Reading and writing 193

      Seeking 194

      Caching files 194

    Qt's I/O classes 195

      QFile 195

      QTextStream and QDataStream 196

      Other helper I/O classes 198

      QDebug and friends 198

  Parsing XML and JSON at the speed of light 199

    QtXml classes 200

      QDomDocument 200

      QXmlSimpleReader 201

    New stream classes in QtCore 201

    Quick parsing of XML 202

    Reading JSON 203

      QJsonDocument's performance 204

  Connecting databases 204

    Basic example using SQLite 204

    Some performance considerations 205

  More about operating system interactions 206

    Paging, swapping, and the TLB 206

    Reading from disk 207

    Completion ports 208

  Summary 208

  Questions 208

  Further reading 209


Chapter 8: Optimizing Graphical Performance 210

  Introduction to graphics performance 211

    Graphics hardware's inner workings 211

    What is a GPU? 211

    OpenGL pipeline model 213

    Performance of the graphics pipeline 215

      CPU problems 217

      Data transfer optimization 217

      Costly GPU operations 217

    Newer graphics programming APIs 218

  Qt graphics architecture and its history 218

    The graphics API Zoo 219

      Qt Widget 219

      QGraphicalView 220

      QOpenGLWidget 221

      QVulkanWindow 222

      Qt Quick 223

      QtQuick Controls 1 and 2 224

      Extending QML 224

        Canvas 2D 224

        QQuickPaintedItem 225

        QQuickItem 226

        QQuickFrameBufferObject 227

        More APIs 228

      Qt 3D 229

    OpenGL drivers and Qt 231

      Graphic drivers and performance 231

      Setting the OpenGL implementation for QML 233

  Qt Widget's performance 234

     QPainter 234

      Images 234

      Optimized calls 235

    OpenGL rendering with QOpenGLWidget 236

      Images 236

      Threading and context sharing 236

      Usage of QPainter 237

    QGraphicsView 237

    Model/view framework 237

  QML performance 238

    Improvements in 5.9 and beyond 239

    Measuring QML performance 240

    Startup of a QML application 242

    QML rendering 243

      Scene graph optimizations 243

      Scene graph and threading 245

      Scene graph performance gotchas 245

        Batching 245

        Texture atlas 246

        Occlusion, blending, and other costly operations 246

        Antialiasing 246

        Use caching 247

      Which QML custom item should you choose? 247

      JavaScript usage 247

      Qt Quick Controls 248

  Other modules 248

    Qt 3D performance 249

    Hybrid web applications 249

  Summary 249

  Questions 250

  Further reading 251


Chapter 9: Optimizing Network Performance 252

  Introduction to networking 253

    Transport layer 254

      User Datagram Protocol (UDP) 254

      Transmission Control Protocol (TCP) 254

      A better TCP? 256

    Application layer 256

      Domain Name Service (DNS) 256

      HyperText Transfer Protocol (HTTP) 257

      Secure data transfer 258

      A better HTTP? 259

  Qt networking classes 259

    TCP and UDP networking classes 259

      QTcpServer and QTcpSocket 260

      QUdpSocket 261

      QAbstractSocket 262

      QSslSocket 264

      Other socket types 265

    HTTP networking using Qt classes 265

      DNS queries 265

      Basic HTTP 266

      HTTPS and other extensions 267

      Qt WebSocket classes 267

      Miscallaneous classes 268

    Other higher-level communication classes 269

      Qt WebChannel 269

      Qt WebGL streaming 269

      Qt remote objects 269

  Improving network performance 270

    General network performance techniques 270

    Receive buffers and copying 271

    TCP performance 271

    HTTP and HTTPS performance 272

      Connection reuse 273

      Resuming SSL connections 273

      Preconnecting 274

      Pipelining 275

      Caching and compression 276

      Using HTTP/2 and WebSocket 276

  Advanced networking themes 278

  Summary 278

  Questions 279

  Further reading 279


Chapter 10: Qt Performance on Embedded and Mobile Platforms 281

Challenges in embedded and mobile development 282

    Basic performance themes 282

    Run to idle 283

    Some hardware data 283

    Embedded hardware and performance 285

  Qt usage in embedded and mobile worlds 285

    Qt for embedded 286

      Qt usage on embedded Linux 286

      Qt's embedded tooling 287

      Supported hardware 288

      Example usage with Raspberry Pi 288

    Qt for mobile 289

      Android support in Qt Creator 289

      Profiling Android applications 290

      Mobile APIs in Qt 290

  Embedded Linux and Qt performance 291

    Executable size 291

    Minimizing assets 292

    Power consumption 292

    Start-up time 293

      Using the current Qt version 293

      Using loaders 294

      3D asset conditioning 294

      Linux start-up optimizations 294

      Hardware matters! 295

    Graphical performance 295

    Time series chart display 296

      Qt Charts and OpenGL acceleration 296

      Polyline simplifications 297

    Floating-point considerations 298

  Mobile-specific performance concerns 299

    Executable size 299

    Power usage 299

    Mobile networking 300

      Batch and piggyback 301

      Consider a push model 302

      Prefetch data 302

      Reuse connections 303

      Adapting to the current network connection type 303

    Graphic hardware 304

  Summary 304

  Questions 305

  Further reading 305


Chapter 11: Testing and Deploying Qt Applications 307

  Testing of Qt code 307

    Unit testing 308

      Qt Test 308

      Test support in Qt Creator 310

    Automated GUI testing 312

      Squish 312

      Example Squish test 313

    Performance regression testing 316

      Adding a qmlbench benchmark 316

      Using Squish 317

  Deploying Qt applications 318

    Flying parts 318

    Static versus dynamic builds 319

    Deploying on Windows 320

      Windows deployment tool 320

    Installation and paths 320

  Summary and farewell 321

  Questions 322

  Further reading 323



Appendix A: Responses to questions 324

Chapter 1 324

Chapter 2 325

Chapter 3 326

Chapter 4 328

Chapter 5 329

Chapter 6 331

Chapter 7 332

Chapter 8 333

Chapter 9 334

Chapter 10 336

Chapter 11 337