Buildbot

buildbot is a continuous integration system. The server called buildmaster defines and schedules processes of building and testing of software to be performed by so called builders on machines called buildslaves (or buildbots). One buildslave can have several builders associated (for example a given operating system version buildslave performs testing of several software branches builders). One builder can also be associated with several buildslaves (several, “identical” machines running the same operating system), in which case buildmaster will schedule the given builder to switch periodically between the buildslaves. At the time of writing (2012) buildbot does not offer satisfactory security mechanisms, and due to clear text passwords stored/transferred both on buildmaster and buildslaves external security measures like firewall, proxy, ssh/stunnel should be used.

The sections below describe the configuration of buildmaster and buildslaves.

Configuration of buildmaster

The buildmaster should be running as unprivileged user (usually called buildmaster). The user and group need to be created.

In order to install buildbot on RHEL6 system you need:

yum install python-twisted python-jinja2 python-tempita

Additional dependencies + buildbot itself need to be installed manually:

wget https://downloads.sourceforge.net/project/sqlalchemy/sqlalchemy/0.7.9/SQLAlchemy-0.7.9.tar.gz
wget http://sqlalchemy-migrate.googlecode.com/files/sqlalchemy-migrate-0.7.2.tar.gz

It is sufficient to unpack the files and set PYTHONPATH, and PATH accordingly. Buildbot may need to be patched for twisted compatibility http://www.npcglib.org/~stathis/blog/2012/10/20/bug-buildbot-dies-with-exceptions-importerror-cannot-import-name-noresource/

Proceed with buildbot configuration:

  • create the master:

    buildbot create-master --relocatable python-ase
    
  • reconfigure the master with doc/development/master.cfg:

    cp master.cfg python-ase
    buildbot checkconfig python-ase/master.cfg
    buildbot start python-ase
    
  • consider setting up a crontab which starts buildmaster after the server reboot (this solution is not reliable, deploy rather init scripts - see below):

    # run every 5 minutes
    */5 * * * * sh ~/python-ase-restart.sh
    

    with the following python-ase-restart.sh:

    bot=python-ase
    # no pid file - assume buildbot is not running
    if ! test -f $bot/twistd.pid;
    then
        buildbot restart $bot || buildbot start $bot
    else
        # pid file exists but buildbot is not running
        pid=`cat $bot/twistd.pid`
        if test -z `ps -p $pid -o pid=`;
        then
            buildbot restart $bot
        fi
    fi
    
  • or create a system V init script under /etc/init.d

    #!/bin/sh
    #
    # python-ase:	python-ase buildmaster
    #
    # chkconfig:	345 98 02
    # description:	python-ase buildmaster
    
    # LSB init-info
    ### BEGIN INIT INFO
    # Provides:          python-ase
    # Required-Start:    $network
    # Required-Stop:     $network
    # Default-Start:     2 3 4 5
    # Default-Stop:      0 1 6
    # Short-Description: python-ase buildmaster
    ### END INIT INFO
    
    # Source function library.
    if [ -e /etc/init.d/functions ]; then
    	. /etc/init.d/functions
    fi
    
    # LSB functions
    . /lib/lsb/init-functions
    
    # Check that networking is configured.
    [ "${NETWORKING}" = "no" ] && exit 0
    
    RUN_AS=buildmaster-username
    PYTHON_ASE_HOME=/home/$RUN_AS/python-ase
    test -d $PYTHON_ASE_HOME || exit 5
    
    LOGFILE=$PYTHON_ASE_HOME/../python-ase.log
    PIDFILE=$PYTHON_ASE_HOME/../python-ase.pid
    LOCKFILE=$PYTHON_ASE_HOME/../python-ase.lock
    
    start() {
    	echo -n $"Starting python-ase buildmaster: "
    	dostatus > /dev/null 2>&1
    	if [ $RETVAL -eq 0 ]
    	then
    		echo -n $"python-ase buildmaster already running"
    		log_failure_msg
    		RETVAL=1
    		return
    	fi
    	#su - $RUN_AS -s /bin/sh -c "exec nohup /bin/sh $PYTHON_ASE_HOME/../python-ase-start.sh >> $LOGFILE 2>&1 &"
            # don't produce log
    	su - $RUN_AS -s /bin/sh -c "exec nohup /bin/sh $PYTHON_ASE_HOME/../python-ase-start.sh >> /dev/null 2>&1 &"
    	RETVAL=$?
    	if [ $RETVAL -eq 0 ]
    	then
    		sleep 5
    		su - $RUN_AS -s /bin/sh -c "cat $PYTHON_ASE_HOME/twistd.pid > $PIDFILE"
    		su - $RUN_AS -s /bin/sh -c "touch $LOCKFILE"
    		log_success_msg
    	else
    		log_failure_msg
    	fi
    	return $RETVAL
    }
    
    stop() {
    	echo -n $"Shutting down python-ase buildmaster: "
    	kill $(su - $RUN_AS -s /bin/sh -c "cat $PIDFILE 2>/dev/null") > /dev/null 2>&1
    	RETVAL=$?
    	sleep 5
    	if [ $RETVAL -eq 0 ]
    	then
    		su - $RUN_AS -s /bin/sh -c "rm -f $PIDFILE $LOCKFILE"
    		log_success_msg
    	else
    		log_failure_msg
    	fi
    	return $RETVAL
    }
    
    restart() {
    	stop
    	start
    }
    
    condrestart() {
    	[ -f $LOCKFILE ] && restart || :
    }
    
    dostatus() {
    	kill -0 $(cat $PIDFILE 2>/dev/null) > /dev/null 2>&1
    	RETVAL=$?
    	if [ $RETVAL -eq 0 ]
    	then
    		echo "python-ase buildmaster (pid $(cat $PIDFILE 2>/dev/null)) is running..."
    	else
    		if [ -f $PIDFILE ]
    		then
    			echo "python-ase buildmaster dead but pid file exists"
    			RETVAL=1
    			return
    		fi
    		if [ -f $LOCKFILE ]
    		then
    			echo "python-ase buildmaster dead but subsys locked"
    			RETVAL=2
    			return
    		fi
    		echo "python-ase buildmaster is stopped"
    		RETVAL=3
    	fi
    }
    
    # See how we were called.
    case "$1" in
      start)
    	start
    	;;
      stop)
    	stop
    	;;
      status)
    	dostatus
    	;;
      restart|reload)
    	restart
    	;;
      condrestart)
    	condrestart
    	;;
      *)
    	echo $"Usage: $0 {start|stop|status|restart|reload|condrestart}"
    	exit 1
    esac
    
    exit $RETVAL
    

    The service is added with:

    chkconfig --add python-ase-buildmaster
    

    started with:

    service python-ase-buildmaster start
    

    end enabled for boot time with:

    chkconfig python-ase-buildmaster on
    

    See also an example of systemd script in the section below.

  • consider protecting the buildmaster web interface by, e.g. apache reverse proxy (http://httpd.apache.org/docs/2.4/mod/mod_proxy.html). The basic configuration file may look like

    # EEE.EEE.EEE.EEE is the IP of your External apache server:
    # ase-buildbot.fysik.dtu.dk
    
    <VirtualHost EEE.EEE.EEE.EEE:80>
        ServerName ase-buildbot.fysik.dtu.dk
        ServerAlias ase-buildbot ase-buildbot.fysik.dtu.dk
        Redirect / https://ase-buildbot.fysik.dtu.dk/
    </VirtualHost>
    
    <VirtualHost EEE.EEE.EEE.EEE:443>
        ServerName ase-buildbot.fysik.dtu.dk
        ServerAlias ase-buildbot ase-buildbot.fysik.dtu.dk
    
        # important http://httpd.apache.org/docs/current/mod/mod_proxy.html
        ProxyRequests Off
    
        # III.III.III.III:8080 the IP and port of the Internal buildbot web server
    
        ProxyPass / http://III.III.III.III:8080/
        ProxyPassReverse / http://III.III.III.III:8080/
    
        <Proxy http://III.III.III.III:8080/*>
    	Options -Indexes
    	Order deny,allow
    	Deny from all
    	Allow from 127.0.0.1
    	Allow from ::1
    	#Allow from all
        </Proxy>
    
    SSLProxyEngine on
    
    # provide an authority signed certificate
    SSLCertificateFile /etc/pki/tls/certs/localhost.crt
    SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
    
    </VirtualHost>
    
    # vim: filetype=apache shiftwidth=4 tabstop=4 wrap!
    

Installation and configuration of buildslaves

Some examples are given below. The configuration should be performed under an unprivileged buildslave, or buildbot user account.

Installation

Fedora

Install with:

yum install buildbot-slave

Perform Configuration now.

After configuring buildbot-slave, you can configure systemd service by creating python-ase-fedora-18-x86_64-gcc-2.7.service file under /usr/lib/systemd/system.

[Unit]
Description=buildslave python-ase-fedora+18+x86_64+gcc+2.7 stock
After=network.target

[Service]
Type=forking
User=buildslave-username
Group=buildslave-groupname
ExecStart=/usr/bin/sh /home/buildslave-username/python-ase-fedora+18+x86_64+gcc+2.7-start.sh
Restart=always

[Install]
WantedBy=multi-user.target

python-ase-fedora+18+x86_64+gcc+2.7-start.sh simply exports the necessary environment variables (if needed) and starts buildslave (use the full path), e.g.:

#!/bin/sh
bot=/home/buildslave-username/python-ase-fedora+18+x86_64+gcc+2.7
buildslave start $bot

Choose User and Group under which buildslave will be running. The service is started with:

systemctl start python-ase-fedora-18-x86_64-gcc-2.7.service

In order to force the service to be started at boot time create a link:

cd /etc/systemd/system/multi-user.target.wants
ln -s /usr/lib/systemd/system/python-ase-fedora-18-x86_64-gcc-2.7.service .

OS X

Configure Homebrew, and:

brew install subversion

Configure a virtualenv, and then:

pip install numpy
pip install buildbot-slave

Perform Configuration now.

RHEL5

Install recent python-setuptools in order to get easy_install:

mkdir ~/buildbot-slave-el5
export PATH=$HOME/buildbot-slave-el5:${PATH}
export PYTHONPATH=$HOME/buildbot-slave-el5:${PYTHONPATH}
wget http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c11-py2.4.egg
sh setuptools-0.6c11-py2.4.egg --install-dir=$HOME/buildbot-slave-el5

then:

easy_install --install-dir=$HOME/buildbot-slave-el5 zope.interface==3.6.7
easy_install --install-dir=$HOME/buildbot-slave-el5 twisted==9.0.0  # ignore errors
easy_install --install-dir=$HOME/buildbot-slave-el5 buildbot-slave

Perform Configuration now.

RHEL6

Install build-slave and dependencies:

mkdir ~/buildbot-slave-el6
export PATH=$HOME/buildbot-slave-el6:${PATH}
export PYTHONPATH=$HOME/buildbot-slave-el6:${PYTHONPATH}
easy_install --install-dir=$HOME/buildbot-slave-el6 buildbot-slave

Perform Configuration now.

RHEL7

Install build-slave and dependencies:

mkdir ~/buildbot-slave-el7
export PATH=$HOME/buildbot-slave-el7:${PATH}
export PYTHONPATH=$HOME/buildbot-slave-el7:${PYTHONPATH}
easy_install --install-dir=$HOME/buildbot-slave-el7 buildbot-slave

Perform Configuration now.

Windows

build-slave can be installed and configured to start as a service on Windows http://trac.buildbot.net/wiki/RunningBuildbotOnWindows. This involves several steps:

  1. install Python(x,y) from https://code.google.com/p/pythonxy/wiki/Downloads

  2. configure distutils to use mingw. First enable showing file extensions:

    Open a folder with IE -> Folder and search options-> View -> Folder Options:
    Check: Show hidden files, ...; uncheck: Hide extensions for known file types.
    

    Then, in notepad, create C:\python27\lib\distutils\distutils.cfg, containing:

    [build]
    compiler=mingw32
    
  3. install build-slave on the command line:

    C:\python27\scripts\easy_install.exe buildbot-slave
    
  4. create a local (domain computer-name) user that will run the buildbot service:

    control panel->administrative tools->computer management->local users and groups->users->new user: buildslave-username.
    Click the created user: member of: administrators->check names
    
  5. grant buildslave-username the ability to run the services. Login as computer-name\buildslave-username: Run secpol.msc on the command line as administrator (task bar->cmd->right click: run as administrator):

    • Select the “Local Policies” folder
    • Select the “User Rights Assignment” folder
    • Double click “Log on as a service”
    • Use “Add User or Group...” to add your user here.

    Select the correct “from this location” (may require login as the current domain administrator) and Enter the object names: computer-name\buildslave-username.

  6. on the command line install the buildbot service:

    buildbot_service.py --user computer-name\buildslave-username --password thepassword --startup auto install
    
  7. start the service (for the moment it does not start any buildslave, because they are not configured yet):

    Start->Control Panel> Administrative Tools->Services->Buildbot (Start)
    

    There are additional steps mentioned in the buildbot wiki, but it seems just to work on Windows 7.

  8. run regedit as administrator (type “regedit” on the command line) and add “directories” parameter of the String Value type, containing paths to all your buildslave instances (they will be configured in the Configuration section below):

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\services\Buildbot->paramaters->new (String Value): directories C:\python-ase-windows+7+AMD64+msc+2.7;C:\proj2
    
  9. configure buildslave instance as described in the Configuration section below and start the service again (point 7.). Test that buildslave comes online, and verify that the service starts after reboot.

Configuration

After having installed the buildbot create a name which will identify your buildslave. Obtain the first part of the name for your buildslave by running doc/development/master.cfg:

python master.cfg

This will return a string like redhat+6+x86_64+gcc+2.6 (OS+OSversion+Bitness+Ccompiler+PythonVersion). Append it with something that identifies your buildslave, for example gpu. For a very special system you can use a name like mycluster+x86_64+open64+2.5 gpu. Note that ASE buildbot will perform verification of python version based on the buildslave name so stay consistent!

Generate a strong buildslave password with Creating an encrypted password for SVN access. Don’t reuse any valuable passwords. You don’t need to remember it, buildbot stores it plain text!

Create the buildslave with:

buildslave create-slave python-ase-redhat+6+x86_64+gcc+2.6 buildbot.fysik.dtu.dk:ASE_BB_PORT "redhat+6+x86_64+gcc+2.6 gpu" "password"

ASE_BB_PORT is the port ASE buildbot uses for communication with the buildslave. You will have to tell us the name and password of your buildslave. Please contact ase-developers list Mailing Lists, but don’t send the name and password there!

Edit the python-ase-redhat+6+x86_64+gcc+2.6/info/{admin,info} files: describe your buildslave configuration relevant for the builder process in the info file.

Note that before starting the slave you need to perform an temporary svn checkout of ASE in order to accept the certificate permanently.

Start the buildslave with:

buildslave start python-ase-redhat+6+x86_64+gcc+2.6

Don’t forget to configure a crontab job or a service as described in the previous sections.

By default all slaves run the continuous integration for the trunk. If you prefer your buildslave works also on one of the branches, write this in the email to ase-developers Mailing Lists.