Rockbuild

About

Rockbuild is a system for running an ad-hoc heterogeneous distributed build farm over the internet (long latency and low bandwidth).

The main purpose of Rockbuild is to give developers quick compile results after commit.

Rockbuild was written for the requirements of the Rockbox project. Rockbox has a huge array of hardware targets, each with a different source code configuration. It is very time consuming for each developer to have to test compile every modification on all those targets before committing it. Therefore the focus of Rockbuild is to, as quickly as possible, give developers feedback to their changes. "Did my commit break any build?"

Build speed is priority one. In the Rockbox build farm of ~50 clients, the 200+ build targets that take nearly an hour for a single machine to build is completed by the farm in a handful minutes.

Features

• It is easy for volunteers to contribute by adding their machines to the farm:
  • Build clients do not have to preregister with the master to join a build farm
  • Build clients are sovereign and do not give the build master login access to their machine
  • Build clients can join and leave whenever they like, even in the middle of a build
  • All network connections are client-initiated, avoiding many firewall and NAT problems
  • Clients that connect with old script versions are automatically upgraded
• The build master gives out jobs matched to each client's past performance. Heavy jobs go to fast clients, lighter jobs to slow ones.
• After completion of a job, clients upload the result in the background while starting on the next job
• The build results are displayed on a web page, stored in a database and optionally echoed to your IRC channel.
• Clients can support subsets of the builds/architectures. No client has to support every build.
• Clients that fail building due to a system error (not a compile error) are temporarily blocked from the farm to avoid interfering with the builds.
• Clients that are consistently malfunctioning can be permanently blocked by manual intervention.

Components

Rockbuild, at its' core, consists of two perl scripts:

rbmaster.pl

The script running on the master server. It tells each client what to build and tells old clients to upgrade.

rbclient.pl

The script running on each build client. It connects to the master server and recieves instructions for what to build.

In addition to this, there are a number of helper scripts and tools for managing the build results, creating the build tables etc.

Source

# git clone git://github.com/zagor/rockbuild.git

https://github.com/zagor/rockbuild

Authors

Rockbuild was written by Björn Stenberg and Daniel Stenberg.

License

Rockbuild is distributed under the GNU General Public License.

How is Rockbuild different from <other system>?

Generally, the goals are different. Many build systems are "Contiuous Integration" tools, focusing heavily on automated testing.

Rockbox, being a device driver-intensive embedded project, has a very wide array of source code configurations. The focus on Rockbuild is therefore to provide compile results for all configurations as soon as possible after each commit.

This might match the needs of your project, or it may not. Luckily, there are many other systems to choose from:

distcc

distcc preprocesses each source file on the master, hands over the file to a remote client to be built and then gets the object file back. Hence to be efficient, it requires (relatively) slow per-file compiles and very fast networks. Rockbuild uses big jobs with little network traffic and is therefore much faster over slow networks.

Buildbot, Hudson, Continuum

These systems all use a fixed list of build clients. Rockbuild allows new clients to join without preregistering with the master. Also, none of them match builds to client speeds. Rockbuild gives heavy jobs to fast clients and lighter jobs to slow ones. This makes a huge difference in build completion time.

Integrity, CruiseControl

These tools run all build jobs on the server. Rockbuild's main focus is to complete the build rounds quickly by distributing the workload.

Samba and PostgreSQL custom build farms

These are traditional rigid constructions where each client sets up cron jobs to run builds regularly. This approach can not produce the immediate commit feedback Rockbuild was designed for.