Using CMake’s CTest and add_test To Run All Your Tests

Whether you’re using GTest , creating your own test executables, or using shell scripts to run your CMake project’s tests, CTest will help you manage all of them.

CTest is the test framework that comes with CMake, and it can help you manage all of your unit and functional tests in one place. In addition, it also provides ways of filtering test runs, passing arguments to your test executables, parallel execution of test, and many more options. In this post, we look at how to use CMake’s add_test(...) to include your tests to this framework.

Contents

1. Video demonstration of usingctest
2. CMake project setup and context
3. Using CMake’s add_test(...) and enable_testing(...) to create and add tests
   1. Enabling test discovery with enable_testing()
   2. Using add_test(...) to integrate your tests to ctest
   3. Options available to add_test(...)
4. Calling ctest from the build to run all tests
5. Further CMake reading on testing

Video – Using CMake’s add_test And ctest To Manage Your Tests

If you’re a visual learner, the video below demonstrates how to use CTest to manage all the tests you have for your C++ project.

CMake Example Project

In this post, we’ll use the project created for integrating GoogleTest into a CMake / C++ project. For this reason, take a look at that post for more information.

Essentially, the structure below shows how the project is laid out.

├── CMakeLists.txt
├── scripts
│   ├── build.sh
│   └── clean-build.sh
├── src
│   ├── CMakeLists.txt
│   └── multiply
│       ├── CMakeLists.txt
│       ├── include
│       │   └── multiply
│       │       └── multiply.h
│       └── multiply.cpp
└── tests
    ├── CMakeLists.txt
    └── multiply_test.cpp

For the sake of this post, we will only focus on the relevant CMakeLists.txt and the libraries / executables we create.

• ./CmakeLists.txt is the root CMake, and it sets up the project name, C++ configuration (such as the standard required), and includes the subdirectories src and tests with add_subdirectory(...).
• ./src contains a passthrough CMakeLists.txt that just includes the multiply subdirectory.
• ./src/multiply contains a CMakeLists.txt that generates a library called multiply, consisting of the file multiply.cpp, and exporting the headers under ./scr/multiply/include. Basically, this library defines three functions for multiplying two numbers: int, float, and double overloads.
• ./tests contains a CMakeLists.txt that includes GTest with CMake, and defines a test executable called multiply_tests, that will test our multiply library.

For more information, or to clone the project, check out the post on integrating GTest to CMake.

Including Your Tests To ctest With enable_testing() and add_test(...)

In this section, we will look at how you can use enable_testing() and add_test(...) to integrate your tests into ctest, so you can later run all of them with a simple terminal call to ctest.

Making Sure You Enable CMake Test Discovery

Enabling test discovery is actually very easy with CMake. In fact, you only need to add a call to enable_testing() in your root CMakeLists.txt, then all the tests that you will add after this call will be discoverable by ctest!

cmake_minimum_required(VERSION 3.16)
project(MultiplyTest LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

enable_testing()

add_subdirectory(src)
add_subdirectory(tests)

Importantly, if you don’t include the call to enable_testing() before you add your tests, ctest will not be able to find them.

A good practice is to call enable_testing() before any of the test subdirectories are included in your CMakeLists.txt.

Lastly, many projects have an outside build flag to dictate whether or not to build your tests. For this reason, if you want to let a flag indicate whether or not to run tests, you can easily wrap enable_testing() inside a CMake if statement that checks for the defined flag.

Using add_test(...) To Add Individual Tests To CTest

Without further ado, let’s look at adding a test to ctest.

Briefly speaking, calling add_test(...) is the meat and potatoes of testing with CMake. For every test executable or script you want to run, you’re going to add a call to add_test(...) with different parameters. For this reason, I’d highly recommend reading the CMake documentation on add_test(...).

The code block below shows how to use add_test(...) in its simplest form. Specifically, it takes a test name (displayed when running ctest, and it should be unique), and the target executable to run.

include(FetchContent)

FetchContent_Declare(
  googletest
  GIT_REPOSITORY https://github.com/google/googletest.git
  GIT_TAG        release-1.11.0
)
FetchContent_MakeAvailable(googletest)
add_library(GTest::GTest INTERFACE IMPORTED)
target_link_libraries(GTest::GTest INTERFACE gtest_main)

add_executable(multiply_test multiply_test.cpp)

target_link_libraries(multiply_test
 PRIVATE
  GTest::GTest
  multiply)

add_test(multiply_gtest multiply_test)

Adding Tests Contained In Scripts / Commands With add_test

For 90% of the cases, this form of add_test(...) will do. However, add_test(...) can basically run any command followed by as many arguments as you want. In addition, add_test(...) itself has other useful arguments you can pass.

For example, say you have a test script that checks for a file in the disk. If the file exists, the script returns 0, if the file doesn’t exist, it returns 1 indicating a failure. You can also integrate this test script to your build if it’s relevant to your project.

set -eu

FILE="test_file"
if [ -f $FILE ]; then
    echo "File found! Test passed!"
else
    echo "File not found! Test failed!"
    exit 1
fi

exit 0

I do understand that checking for a file isn’t a good example for a script test, but it’s just so you can understand how add_test(...) can be used to plug in your custom test scripts to CMake. In a real-world scenario, you may want to run test scripts in order to functionally test your software in a given context.

Note that ctest interprets 0 returns as pass, and non-zero returns as failures!

Given the test script above, let’s include it in our CMake project. The code block below is a modified version of the CMakeLists.txt for the tests directory.

include(FetchContent)

FetchContent_Declare(
  googletest
  GIT_REPOSITORY https://github.com/google/googletest.git
  GIT_TAG        release-1.11.0
)
FetchContent_MakeAvailable(googletest)
add_library(GTest::GTest INTERFACE IMPORTED)
target_link_libraries(GTest::GTest INTERFACE gtest_main)

add_executable(multiply_test multiply_test.cpp)

target_link_libraries(multiply_test
 PRIVATE
  GTest::GTest
  multiply)

add_test(multiply_gtest multiply_test)

add_test(NAME script_test
  COMMAND "${CMAKE_CURRENT_LIST_DIR}/script_test.sh"
  WORKING_DIRECTORY "${CMAKE_CURRENT_LIST_DIR}")

If you look at the last three lines, you can see a more verbose call to add_test(...) . Specifically, we’re specifying the name, command line to run, and the working directory to run the script from.

The following sections will explain these arguments, but note that ${CMAKE_CURRENT_LIST_DIR} expands to the directory of the current CMakeLists.txt, which is ./tests in our case.

Options To add_test(...)

In the example above, we used the following options in our add_test(...) call.

• NAME specifies the test name, in the same way as the shorthand version.
• COMMAND specifies the command to run for the test. Essentially, this can be any command that would normally run in your terminal. You can also pass the name of an executable target, and CMake will replace it with the full location of the executable.
• WORKING_DIRECTORY is the directory to run the command (or executable) from. Very useful if you have your test data in a particular location.

As previously mentioned, add_test(...) takes in a few possible options. I’d recommend reading the CMake documentation for more information.

Running The Tests Through CTest

Once you’ve added your tests, configured, and built your project, you can run ctest. For this, you will need to go into your build directory and run the following commands.

cd build
ctest

In my case, my build files are located under my project’s build directory.

The ctest command itself can take many arguments. You can find out more information on CMake’s ctest command documentation page, However, I’d recommend running ctest -j N --output-on-failure , which runs all your tests in parallel (N threads), and failing tests will show more logs. ctest also has filtering functionalities, take a look at the -R, -E, and -LE options for filtering tests by name.

Interestingly, this can be integrated into pipeline runs. For example, when your project builds, you can invoke ctest to make sure all your tests are passing, or fail the pipeline if they are not.

Further CMake Reading

As well as reading the CMake documentation on testing, there are many more topics to learn from a CMake and testing perspective.

If you’re developing a serious project with CMake and C++, I’d recommend integrating GTest into your CMake project.

Following that, I’d recommend learning how to integrate coverage with your tests. For example, gcov can show you which lines of your code are being tested, and which bits are not being tested properly.

---

Have I missed anything? Do you have any questions regarding this post? Positive feedback? Feel free to comment below and I’ll respond as soon as I can!