When we decided to improve our coding quality by introduced unit testing to our work process in order to test our C code. The concept was new to many of the engineers who programmed in C, and there was a perception amongst some (developers as well as managers) that it makes development slower with all the associated overhead. However, experience told us that without a solid set of scaffolding around the code to ensure code rot and bugs didn’t emerge, the legacy style code that would otherwise emerge is a lot more expensive to maintain and a lot more fragile and bug-prone than we would like.
So I went about researching and experimenting with how to get unit testing to work with our team. On investigation, I found that there were many unit testing frameworks. It was a very difficult choice, and we didn’t have that much time to make one – we could not spend the time required to evaluate each of them and select the best one. It seemed like a lot of people had independently developed their own unit test frameworks. There is literally too much information out there and recommendations are highly subjective, people promoting their pet unit testing framework is rife, few people using any of them seemed really satisfied.
So with the little I knew at the time, I put together a shortlist.
- Which was the most popular framework?
- A google search showed CPPUnit was one of the most popular.
- Which one had good tutorials?
- CPPUnit seemed to have some helpful documentation
- Which one integrated with our tools?
- CPPUnit had an Eclipse CDT Plugin
- CPPUnit had a Jenkins plugin
- Which one could be easily built?
- A tutorial showed how to build CPPUnit.
- Which one had a low barrier to entry?
- CPPUnit seemed to be one of the best.
- I wanted to have few dependencies.
- Some otherwise very nice unit testing platforms required one to pre-parse the code to generate some of the boilerplate test code.
- Others (e.g. Boost-Test) requires you to install Boost libraries. Getting each developer to build and install this didn’t sound like much fun. It didn’t seem to suit our environment very well.
- CPPUnit just needed a single library to be built.
- I also wanted to be able to test C++ code in case we made a switch.
- CPPUnit seemed to fit the Bill
So as you may have guessed – we went with CPPUnit. All in all, it worked okay, but over time we became dissatisfied as we uncovered minor shortcomings.
- Quite a lot of code is required to get the framework going. The solutions work, but it feels clunky and ugly.
- the main function is rather long and complicated,
- Creating a link to a pre-written mainUnitTest file helps.
- each test fixture required a long list of header files.
- This was solved by using Eclipse Templates, but its a lot of repeated code (ugly).
- A possible solution would have been to encapsulate these repeated header includes in a single header file.
- the main function is rather long and complicated,
- The Eclipse CDT plugin didn’t work to an acceptable level – indeed I battled to get any benefit from it.
- It violated the DRY principle (don’t repeat yourself)
- a lot of repetition of each test declaration is required to get a test going,
- The console output is a bit cryptic and difficult to read and parse.
So, with a lot more experience under my belt, I started to look for some more options. I came to realize there were better, cleaner solutions out there which would fit into our workflow a lot better. The most likely potential solutions I found were as follows:
- A nicely designed unit test plugin for Eclipse CDT. It is called CUTE.
- It uses conventional C++ and code generators to make it easier to build tests
- It also includes a useful code refactoring feature.
- It has its own test runner.
- Eclipse CDT has a nice unit test interface. This is presented as a test runner.
- This shows red bar/green bar test results.
- Any failure can be selected with the mouse and Eclipse CDT takes you straight to the failed test.
- Notably, the test runner supports GoogleTest, BoostTest and QtTest.
So I went about experimenting. Starting with CUTE, I installed the plugin and built some tests. My impressions:
- It is a comparatively well developed, well-rounded test framework.
- It followed solid conventional empirical coding practices.
- The downside of this is that it requires extra code to link the various test fixtures to each other. To some people, this may be a good thing as it is flexible, but I found it to be something that could make testing more difficult.
- It relies quite a lot on automation and code generation from the plugin – something I felt was clever, but not that elegant. Some of the automation also occasionally seemed to misbehave.
- It added some nice refactoring features to Eclipse CDT.
Next, I tried GoogleTest (GTest). Some of my impressions:
- It uses static constructors called during initialization to “trick” C++ into discovering and registering your tests before the main function is called. This is done without requiring any manual specifications or extra code, which while viewed by some as being a bit hackish, is really nice and clean for development and reduces overhead and the risk of tests being accidentally omitted.
- The integration into Eclipse CDT works really nicely.
- Included in the library is the unit test framework fused into a single file.
- This slows initial compilation somewhat but makes setting up tests really easy, and incremental builds are still nice and fast.
- Building a test library is another option if you want quicker builds at all times.
- Integration with Jenkins is still possible through an XML test results file that can be generated that followed the CPPUnit standard.
- The testing features and macros are really extensive with many options, types of tests, and ways of testing.
- No code duplication when building tests (i.e. it doesn’t violate the DRY principle). A lot of macro-driven trickery goes on behind the scenes to accomplish this.
- No extra dependencies. Google Test works nicely with standard C++, with Eclipse CDT, using MinGW.
- As with all C++ unit test frameworks, it tests C code without any issues.
- Although we haven’t used it yet, GTest integrates naturally with Google Mock. As you may have guessed, this is for creating mocks (instead of stubs), but this concept doesn’t work so well with C (even though it seems to be possible albeit in a not so portable way).
- The main function required to get the tests going is really small.
- It is what Google uses – which adds some sort of credibility.
This, I realized was the solution I was looking for. After about a year our team is very happy with GTest, and our older test code is slowly being migrated from CPPUnit over to it. I didn’t use the CUTE unit testing framework, but I still install the plugin, since it includes some fairly usable refactoring capability to Eclipse CDT.
I hope to write some future blogs on unit testing, including things I find work well when used with Eclipse CDT.