Defending software build pipelines from malicious attack

Automated build pipelines, like those used in CI/CD (continuous integration/continuous delivery or deployment), can be a great way to secure the building and deployment of software.

The NCSC's Secure development and deployment guidance describes how automation can give much greater confidence in the consistency and repeatability of security controls, and produce detailed log and audit data. As we discuss in principle 6 of the guidance, it's crucial that the pipeline is well-defended, and that it protects each build from other builds in the pipeline. If an attacker compromises other systems, they may use lateral movement to target your build pipeline. If any builds are compromised, poor isolation between builds may lead to the compromise of more important builds.

It's also important to ensure that you have a clear chain of custody from source code to build artefact, so that you can be confident that all of the pipeline's checks have been applied, with nothing added or changed after the checks.

Software build pipelines are a critical component of the software development lifecycle, as they provide access to lots of sensitive data. If an attacker compromises the pipeline:

• they can add malicious code to the software built and deployed by that pipeline
• they can access any secrets used by the pipeline
• they may be able to gain access to other source code repositories and environments

As a result, the pipeline needs to be defended against attack at least as effectively as the environments it deploys to. This will involve the usual security considerations such as:

• robust authentication with strong password management and multi-factor authentication
• authorisation using the principle of least privilege and keeping software up to date
• applying network security and monitoring for attacks

The NCSC's guidance on securing the build and deployment pipeline covers the other important considerations like avoiding 'self policing', and being cautious of untrusted branches and pull requests.

Even if the pipeline itself is well-defended, a malicious build still poses significant risk.

If a compromised build can interfere with other builds, it may be able to spread the compromise further, steal secrets, or make it harder to remediate the compromise later on. This means that builds need to be isolated from each other using robust compute, network, and storage separation.

For example, performing each build in a single-use virtual machine will make it very hard for one build to attack another using shared hardware (like the CPU), whereas two builds sharing an OS kernel will have many more ways to interfere with each other. Similarly, if a build can connect to other builds over a network (including loopback devices), it may be able to attack them or steal information.

Finally, if a build can access stored information on other builds (such as their source code or build artefacts), then it may be able to steal secrets or modify those builds.

One of the biggest advantages of automated build pipelines is that they perform security checks on the software they build, reliably and repeatably. However, if you can't be confident that the pipeline actually enforces these checks, then they're not worth much.

In other words; it's important to be able to demonstrate that the checks were performed, and that the build isn't modified after those checks.

The first step is to ensure that build information is always protected in transit, using protocols like TLS. This includes when the code is fetched from the code repository, and when the build artefacts are sent to the artefact repository and deployed to the final environment.

Next, cryptographic checksums should be used to record the data processed by the pipeline. This might include a checksum of the source code that was built (such as the 'commit hash'), hashes of the resulting build artefacts, and any additional data like the configuration, tools and environment used to build the artefacts. If the pipeline applies a cryptographic signature over these hashes, then it becomes a lot harder for an attacker to manipulate the build in secret. For example, if an attacker adds malicious code to the build, then the checksum on the code will either match the code that was built, or the code that was fetched, but not both (making it easier to detect a problem).

Building and maintaining a secure build pipeline needs a considerable amount of work, resources, and expertise. Many common pipeline products default to insecure architectures, with poor separation between builds, and few defences protecting the pipeline from a malicious build. A good managed cloud service for build pipelines will be designed to defend against a malicious customer, which will also defend against an attack on a legitimate customer.

The harder it is for you to infect the build pipeline, the harder it is for an attacker. At the same time, the kinds of defences that protect one customer's builds from another customers' will often protect each of your builds from your other builds. It also becomes the service provider's responsibility to keep the pipeline updated and improve its security in response to new threats. This means you're typically getting much better security by default. You should use the NCSC's Cloud Security Guidance (including the 14 Cloud Security Principles) to assess the security of a cloud build service, considering the end-to-end security of the entire process.

Securing your build pipeline can take a lot of resources. But it's worth the effort because the impact of an attacker compromising your build pipeline can be huge.

To summarise:

• The pipeline should be protected using the same security principles as with other information systems, as well as the pipeline-specific considerations like performing code reviews and managing pull requests carefully.
• You should use strong isolation to protect builds from one another, and to protect the build pipeline itself.
• By establishing a strong chain of custody, you can make it much easier to detect suspicious behaviour, while also producing an authoritative audit trail.
• All this can be made much easier by using a good managed service for the pipeline.

Don't forget to use the NCSC's Cloud security guidance to get a better understanding of the service's security. Your build pipeline is one of the foundations of your system security, so give it the attention it needs.

Jamie H
Senior Security Researcher, NCSC