Test Architecture

How libbitcoin is tested: code built for testability, the kinds of test, the platforms each change is tested on, and how coverage is measured. The mechanics of writing a test are in Test practices.

1. Testability#

Code is written to be tested through its own interface. There are no test hooks: no test-only branches, no conditional compilation for tests, and no access granted to tests that other callers do not have. Mocks and fakes are injected to alter and detect behavior:

• Mocks. A test derives a mock from the class under test, or from one it depends on, and overrides what it alters or detects (mock_channel_peer, mock_proxy, mock_settings).
• Fakes. A test substitutes a working implementation of a dependency, such as the in-memory chunk_storage fake for the store's memory-mapped files.
• Settings. Behavior that depends on configuration is driven by the settings object the code already takes.

2. Kinds of test#

The C++ suites run in every CI job, except the performance tests, which are compiled only when HAVE_PERFORMANCE_TESTS is defined. The acceptance suite is customer acceptance: it exercises each interface of a running bs as its clients do, over its own protocols.

3. One thing, named#

A test case tests one thing, and its name states that thing: the method, the condition and the expected result. A failing test is then a diagnosis.

4. Reference vectors#

Tests establish correctness, using reference vectors wherever they exist.

5. Continuous integration#

Every change is built and tested by CI (GitHub Actions) before review. Each library repository runs the same matrix. The gate runs on push and pull request, and the expanded matrix, run on demand, covers every compiler in all combinations of debug or release and static or shared (see Continuous integration and Compilers tested in CI). The current status of each repository is on Build status.

Each CI job runs the same install script published for end-user builds (see Building), so CI verifies the build scripts as well as the build. All build artifacts, for every build system and including the install scripts, are generated from templates and configurable metadata (see Build architecture).

6. Coverage#

Line coverage. A GCC debug job in each repository's CI (GitHub Actions) is built with --coverage, and its lcov output is reported to Coveralls, excluding the paths listed in the repository's coverage element of the build model.

Constant time. A libbitcoin-system job in the expanded matrix runs the secp256k1 constant-time tests under valgrind, which reports any branch or memory access that depends on secret data.

Fuzzing. libbitcoin-system is a module of bitcoinfuzz, a differential fuzzer that runs the same inputs through several Bitcoin implementations and reports where their results differ.

Mutation testing. Coverage shows which code ran, not whether a test would fail if it were wrong. Mutation testing changes the code and checks that a test fails. libbitcoin-system is tested on MutantHub.