The QEMU build system architecture

This document aims to help developers understand the architecture of the QEMU build system. As with projects using GNU autotools, the QEMU build system has two stages, first the developer runs the “configure” script to determine the local build environment characteristics, then they run “make” to build the project. There is about where the similarities with GNU autotools end, so try to forget what you know about them.

Stage 1: configure

The QEMU configure script is written directly in shell, and should be compatible with any POSIX shell, hence it uses #!/bin/sh. An important implication of this is that it is important to avoid using bash-isms on development platforms where bash is the primary host.

In contrast to autoconf scripts, QEMU’s configure is expected to be silent while it is checking for features. It will only display output when an error occurs, or to show the final feature enablement summary on completion.

Because QEMU uses the Meson build system under the hood, only VPATH builds are supported. There are two general ways to invoke configure & perform a build:

  • VPATH, build artifacts outside of QEMU source tree entirely:

    cd ../
mkdir build
cd build
../qemu/configure
make

  • VPATH, build artifacts in a subdir of QEMU source tree:

    mkdir build
cd build
../configure
make

For now, checks on the compilation environment are found in configure rather than meson.build, though this is expected to change. The command line is parsed in the configure script and, whenever needed, converted into the appropriate options to Meson.

New checks should be added to Meson, which usually comprises the following tasks:

  • Add a Meson build option to meson_options.txt.
  • Add support to the command line arg parser to handle any new –enable-XXX/–disable-XXX flags required by the feature.
  • Add information to the help output message to report on the new feature flag.
  • Add code to perform the actual feature check.
  • Add code to include the feature status in config-host.h
  • Add code to print out the feature status in the configure summary upon completion.

Taking the probe for SDL2_Image as an example, we have the following pieces in configure:

# Initial variable state
sdl_image=auto

..snip..

# Configure flag processing
--disable-sdl-image) sdl_image=disabled
;;
--enable-sdl-image) sdl_image=enabled
;;

..snip..

# Help output feature message
sdl-image         SDL Image support for icons

..snip..

# Meson invocation
-Dsdl_image=$sdl_image

In meson_options.txt:

option('sdl', type : 'feature', value : 'auto',
       description: 'SDL Image support for icons')

In meson.build:

# Detect dependency
sdl_image = dependency('SDL2_image', required: get_option('sdl_image'),
                       method: 'pkg-config',
                       kwargs: static_kwargs)

# Create config-host.h (if applicable)
config_host_data.set('CONFIG_SDL_IMAGE', sdl_image.found())

# Summary
summary_info += {'SDL image support': sdl_image.found()}

Helper functions

The configure script provides a variety of helper functions to assist developers in checking for system features:

do_cc $ARGS…
Attempt to run the system C compiler passing it $ARGS…
do_cxx $ARGS…
Attempt to run the system C++ compiler passing it $ARGS…
compile_object $CFLAGS
Attempt to compile a test program with the system C compiler using $CFLAGS. The test program must have been previously written to a file called $TMPC. The replacement in Meson is the compiler object cc, which has methods such as cc.compiles(), cc.check_header(), cc.has_function().
compile_prog $CFLAGS $LDFLAGS
Attempt to compile a test program with the system C compiler using $CFLAGS and link it with the system linker using $LDFLAGS. The test program must have been previously written to a file called $TMPC. The replacement in Meson is cc.find_library() and cc.links().
has $COMMAND
Determine if $COMMAND exists in the current environment, either as a shell builtin, or executable binary, returning 0 on success. The replacement in Meson is find_program().
check_define $NAME
Determine if the macro $NAME is defined by the system C compiler
check_include $NAME
Determine if the include $NAME file is available to the system C compiler. The replacement in Meson is cc.has_header().
write_c_skeleton
Write a minimal C program main() function to the temporary file indicated by $TMPC
feature_not_found $NAME $REMEDY
Print a message to stderr that the feature $NAME was not available on the system, suggesting the user try $REMEDY to address the problem.
error_exit $MESSAGE $MORE…
Print $MESSAGE to stderr, followed by $MORE… and then exit from the configure script with non-zero status
query_pkg_config $ARGS…
Run pkg-config passing it $ARGS. If QEMU is doing a static build, then –static will be automatically added to $ARGS

Stage 2: Meson

The Meson build system is currently used to describe the build process for:

  1. executables, which include:
    • Tools - qemu-img, qemu-nbd, qga (guest agent), etc
    • System emulators - qemu-system-$ARCH
    • Userspace emulators - qemu-$ARCH
    • Unit tests
  2. documentation
  3. ROMs, which can be either installed as binary blobs or compiled
  4. other data files, such as icons or desktop files

All executables are built by default, except for some contrib/ binaries that are known to fail to build on some platforms (for example 32-bit or big-endian platforms). Tests are also built by default, though that might change in the future.

The source code is highly modularized, split across many files to facilitate building of all of these components with as little duplicated compilation as possible. Using the Meson “sourceset” functionality, meson.build files group the source files in rules that are enabled according to the available system libraries and to various configuration symbols. Sourcesets belong to one of four groups:

Subsystem sourcesets:

Various subsystems that are common to both tools and emulators have their own sourceset, for example block_ss for the block device subsystem, chardev_ss for the character device subsystem, etc. These sourcesets are then turned into static libraries as follows:

libchardev = static_library('chardev', chardev_ss.sources(),
                            name_suffix: 'fa',
                            build_by_default: false)

chardev = declare_dependency(link_whole: libchardev)

As of Meson 0.55.1, the special .fa suffix should be used for everything that is used with link_whole, to ensure that the link flags are placed correctly in the command line.

Target-independent emulator sourcesets:

Various general purpose helper code is compiled only once and the .o files are linked into all output binaries that need it. This includes error handling infrastructure, standard data structures, platform portability wrapper functions, etc.

Target-independent code lives in the common_ss, softmmu_ss and user_ss sourcesets. common_ss is linked into all emulators, softmmu_ss only in system emulators, user_ss only in user-mode emulators.

Target-independent sourcesets must exercise particular care when using if_false rules. The if_false rule will be used correctly when linking emulator binaries; however, when compiling target-independent files into .o files, Meson may need to pick both the if_true and if_false sides to cater for targets that want either side. To achieve that, you can add a special rule using the CONFIG_ALL symbol:

# Some targets have CONFIG_ACPI, some don't, so this is not enough
softmmu_ss.add(when: 'CONFIG_ACPI`, if_true: files('acpi.c'),
                                    if_false: files('acpi-stub.c'))

# This is required as well:
softmmu_ss.add(when: 'CONFIG_ALL`, if_true: files('acpi-stub.c'))
Target-dependent emulator sourcesets:

In the target-dependent set lives CPU emulation, some device emulation and much glue code. This sometimes also has to be compiled multiple times, once for each target being built. Target-dependent files are included in the specific_ss sourceset.

Each emulator also includes sources for files in the hw/ and target/ subdirectories. The subdirectory used for each emulator comes from the target’s definition of TARGET_BASE_ARCH or (if missing) TARGET_ARCH, as found in default-configs/targets/*.mak.

Each subdirectory in hw/ adds one sourceset to the hw_arch dictionary, for example:

arm_ss = ss.source_set()
arm_ss.add(files('boot.c'), fdt)
...
hw_arch += {'arm': arm_ss}

The sourceset is only used for system emulators.

Each subdirectory in target/ instead should add one sourceset to each of the target_arch and target_softmmu_arch, which are used respectively for all emulators and for system emulators only. For example:

arm_ss = ss.source_set()
arm_softmmu_ss = ss.source_set()
...
target_arch += {'arm': arm_ss}
target_softmmu_arch += {'arm': arm_softmmu_ss}
Utility sourcesets:

All binaries link with a static library libqemuutil.a. This library is built from several sourcesets; most of them however host generated code, and the only two of general interest are util_ss and stub_ss.

The separation between these two is purely for documentation purposes. util_ss contains generic utility files. Even though this code is only linked in some binaries, sometimes it requires hooks only in some of these and depend on other functions that are not fully implemented by all QEMU binaries. stub_ss links dummy stubs that will only be linked into the binary if the real implementation is not present. In a way, the stubs can be thought of as a portable implementation of the weak symbols concept.

The following files concur in the definition of which files are linked into each emulator:

default-configs/devices/*.mak

The files under default-configs/devices/ control the boards and devices that are built into each QEMU system emulation targets. They merely contain a list of config variable definitions such as:

include arm-softmmu.mak
CONFIG_XLNX_ZYNQMP_ARM=y
CONFIG_XLNX_VERSAL=y
*/Kconfig
These files are processed together with default-configs/devices/*.mak and describe the dependencies between various features, subsystems and device models. They are described in QEMU and Kconfig
default-configs/targets/*.mak
These files mostly define symbols that appear in the *-config-target.h file for each emulator [1]. However, the TARGET_ARCH and TARGET_BASE_ARCH will also be used to select the hw/ and target/ subdirectories that are compiled into each target.
[1]This header is included by qemu/osdep.h when compiling files from the target-specific sourcesets.

These files rarely need changing unless you are adding a completely new target, or enabling new devices or hardware for a particular system/userspace emulation target

Support scripts

Meson has a special convention for invoking Python scripts: if their first line is #! /usr/bin/env python3 and the file is not executable, find_program() arranges to invoke the script under the same Python interpreter that was used to invoke Meson. This is the most common and preferred way to invoke support scripts from Meson build files, because it automatically uses the value of configure’s –python= option.

In case the script is not written in Python, use a #! /usr/bin/env … line and make the script executable.

Scripts written in Python, where it is desirable to make the script executable (for example for test scripts that developers may want to invoke from the command line, such as tests/qapi-schema/test-qapi.py), should be invoked through the python variable in meson.build. For example:

test('QAPI schema regression tests', python,
     args: files('test-qapi.py'),
     env: test_env, suite: ['qapi-schema', 'qapi-frontend'])

This is needed to obey the –python= option passed to the configure script, which may point to something other than the first python3 binary on the path.

Stage 3: makefiles

The use of GNU make is required with the QEMU build system.

The output of Meson is a build.ninja file, which is used with the Ninja build system. QEMU uses a different approach, where Makefile rules are synthesized from the build.ninja file. The main Makefile includes these rules and wraps them so that e.g. submodules are built before QEMU. The resulting build system is largely non-recursive in nature, in contrast to common practices seen with automake.

Tests are also ran by the Makefile with the traditional make check phony target, while benchmarks are run with make bench. Meson test suites such as unit can be ran with make check-unit too. It is also possible to run tests defined in meson.build with meson test.

Important files for the build system

Statically defined files

The following key files are statically defined in the source tree, with the rules needed to build QEMU. Their behaviour is influenced by a number of dynamically created files listed later.

Makefile
The main entry point used when invoking make to build all the components of QEMU. The default ‘all’ target will naturally result in the build of every component. Makefile takes care of recursively building submodules directly via a non-recursive set of rules.
*/meson.build
The meson.build file in the root directory is the main entry point for the Meson build system, and it coordinates the configuration and build of all executables. Build rules for various subdirectories are included in other meson.build files spread throughout the QEMU source tree.
tests/Makefile.include
Rules for external test harnesses. These include the TCG tests, qemu-iotests and the Avocado-based acceptance tests.
tests/docker/Makefile.include
Rules for Docker tests. Like tests/Makefile, this file is included directly by the top level Makefile, anything defined in this file will influence the entire build system.
tests/vm/Makefile.include
Rules for VM-based tests. Like tests/Makefile, this file is included directly by the top level Makefile, anything defined in this file will influence the entire build system.

Dynamically created files

The following files are generated dynamically by configure in order to control the behaviour of the statically defined makefiles. This avoids the need for QEMU makefiles to go through any pre-processing as seen with autotools, where Makefile.am generates Makefile.in which generates Makefile.

Built by configure:

config-host.mak

When configure has determined the characteristics of the build host it will write a long list of variables to config-host.mak file. This provides the various install directories, compiler / linker flags and a variety of CONFIG_* variables related to optionally enabled features. This is imported by the top level Makefile and meson.build in order to tailor the build output.

config-host.mak is also used as a dependency checking mechanism. If make sees that the modification timestamp on configure is newer than that on config-host.mak, then configure will be re-run.

The variables defined here are those which are applicable to all QEMU build outputs. Variables which are potentially different for each emulator target are defined by the next file…

Built by Meson:

${TARGET-NAME}-config-devices.mak
TARGET-NAME is again the name of a system or userspace emulator. The config-devices.mak file is automatically generated by make using the scripts/make_device_config.sh program, feeding it the default-configs/$TARGET-NAME file as input.
config-host.h, $TARGET-NAME/config-target.h, $TARGET-NAME/config-devices.h
These files are used by source code to determine what features are enabled. They are generated from the contents of the corresponding *.h files using the scripts/create_config program. This extracts relevant variables and formats them as C preprocessor macros.
build.ninja
The build rules.

Built by Makefile:

Makefile.ninja
A Makefile include that bridges to ninja for the actual build. The Makefile is mostly a list of targets that Meson included in build.ninja.
Makefile.mtest
The Makefile definitions that let “make check” run tests defined in meson.build. The rules are produced from Meson’s JSON description of tests (obtained with “meson introspect –tests”) through the script scripts/mtest2make.py.

Useful make targets

help
Print a help message for the most common build targets.
print-VAR
Print the value of the variable VAR. Useful for debugging the build system.