This document covers the installation steps for builds.sr.ht, SourceHut's continuous integration/build service.

builds.sr.ht is comprised of two components: a master server and workers. Typically, deployments have one master and many workers, which are distributed across multiple servers.

#Security model

builds.sr.ht runs arbitrary user code as root by design. It's important to understand its security model in order to deploy it safely.

Build jobs run in a sandbox which has several layers of isolation from the rest of your production setup. Builds run in a KVM-based virtual machine, which is facilitated by a Docker container that includes only the required QEMU components, which runs as a non-root user. The build workers are ideally run on a server devoted to this purpose which does not have any other production data or production access, and is not a part of the production LAN.

We suggest you take similar precautions if your system will run untrusted builds.

#Installation

Familiarize yourself with the general installation documentation before getting started.

• Sample config.ini file
• PostgreSQL database schema

#Daemons

The following daemons are provided by the builds.sr.ht package:

• builds.sr.ht — The web service.
• builds.sr.ht-api — The API service.

The build worker is provided by the builds.sr.ht-worker package and includes the builds.sr.ht-worker OpenRC service. The build image configurations are included in the builds.sr.ht-images package.

The master server is the server which runs the API daemon. The web service and workers may run on the same server or any number of additional servers.

#Redict configuration

A Redict database is required to be shared between the master server and all worker servers, and must be exposed to the network (ideally a LAN or VPN) and secured accordingly (with password authentication and possibly TLS).

The Redict database used to communicate between the master and the workers can be configured via the [builds.sr.ht]redis parameter in config.ini, separately from the master's and web daemon's Redict configuration (via [sr.ht]redis-host), which is used as an application cache. Small deployments which only run trusted builds can use a shared Redict instance, in which case these connection strings will be similar.

#Minimal worker configuration

To improve isolation, the worker servers can be configured with a reduced config.ini file, which for instance will limit their access to production secrets. Build workers require the following to be present in their config.ini file:

• [sr.ht]: all fields
• [webhooks]private-key
• [objects]: all fields, if build artifacts are required
• [mail]: required to send email notifications of build results
• [meta.sr.ht]origin
• [builds.sr.ht]: origin, connection-string, and redis are required. The latter two may use different credentials than the master server.
• [builds.sr.ht::worker]: all fields

The SQL user for the build workers must have the following grants:

• read/write: job, artifact, and task tables
• read-only: user and secrets tables

In the future we will remove the need for build workers to access the database.

#Preparing build images

In order to run builds, each worker must have the build images prepared and the qemu docker container available. The builds.sr.ht-images package includes all of the support code for build images in /var/lib/images. In order to build the Docker container, run the following commands on each build worker:

$ cd /var/lib/images
$ docker build -t qemu -f qemu/Dockerfile .

Bootstrapping build images is somewhat more involved.

The builds.sr.ht-images package includes a genimg script is provided for each image which can be run from a working image of that guest to produce a new image. You need to manually prepare a working guest of each image type (that is, to build the Arch Linux image you need a working Arch Linux installation to bootstrap from). Then you can run the provided genimg to produce the disk image. You should read the genimg script to determine what dependencies need to be installed before it can be run to completion.

The directory structure for bootable images should have the format images/$distro/$release/$arch/, placing the root.img.qcow2 image within the $arch directory. Release names and architecture names follow the conventions of the upstream OS/distro.

#SSH dispatch

For SSH access to running (or failed) build jobs you will need to configure sourcehut-ssh as the system-wide SSH authorization hook. See git.sr.ht's installation page for more information.

#Maintenance

#Worker maintenance

To safely reboot a worker without dropping any running jobs, SIGINT the worker process(es). The workers will stop accepting new work and terminate once all existing jobs are completed.

#Build image maintenance

A build.yml file is also provided for each image to build itself on a suitable builds.sr.ht instance, which you may customize as necessary. The contrib directory of the builds.sr.ht source tree includes the submit_image_build script that we use to automatically build new versions of each build image in production, and the crontab that we schedule those builds with. You may adapt this to your deployment as needed.

Note: You will need nested virtualization enabled in order to build images from a build job. If you run into issues with modprobe kvm_intel within the genimg script, you can fix this by removing the module and then re-inserting it with insmod kvm_intel.ko nested=1 in the directory containing the kernel module.