Back to documentaton overview

Integrating buildmanager, cvs, template_product, and upd

Abstract

This document will describe using several tools for building ups packages for multiple platforms. While these tools can be used in isolation, they were designed to be used together to provide the following benefits:

Keeping sources in sync between build areas for multiple platforms

Doing builds for several platforms in parallel

Maintaining revision history and revision control of sources.

Extending the base of products that can be built and tested in a standardized fashion.

Overview

We will be describing these tools and how to use them together by way of building an example product,
demo, and in the process give an introduction to several of the tools involved. In particular, we will examine the following in detail:

creating a prototype of your product file tree.

placing the file tree in a CVS repository

establishing sessions on multiple build machines

creating a working copy for each session

building and testing your software in parallel

making changes to your file trees

distributing your software via upd

If you are logged into one of the nodes on the OSS cluster build machine, and have the appropriate permissions already, you should be able to go through the example in this document step by step and build and release the demo product in order to familiarize yourself with the process.

Creating a prototype of your product file tree

This is really the main role of the template_product package for most users; getting a
skeleton of a product together quickly. While the web page for the template_product has
examples of using it to make products out of scripts or precompiled binaries, here we will discuss using it to make products out of packages that are compiled from source code. This generally involves somewhat more customization than the case where no compilation is required, however it starts off very much the same way. We begin by:

setting up template_product to get access to it
making an empty directory and cd-ing there.
using the CloneTemplate script to make the skeleton
adding subdirectories with Makefiles with source code
adding build commands to the toplevel Makefile to build, install, and test your software

Once this is completed, we will have a prototype product tree that we can put in a Fermilab cvs repository.

This looks like the following, with what a user types in bold face (and [tab] is a Tab on your keyboard):

<build_host> # Setup the product
<build_host> setup template_product
<build_host> # make the directory
<build_host> mkdir demo
<build_host> cd demo
<build_host> # run the script and answer the questions
<build_host> CloneTemplate
Product name? demo
Product version? v0_0
Platform specific product [yN]? y
Dependent products [list as fred:joe:harry]?
installing template product files in .../demo
.../demo/ups
.../demo/ups/Version
.../demo/ups/INSTALL_NOTE.template
.../demo/ups/template_product.table
.../demo/ups/.manifest.template_product
.../demo/Makefile
.../demo/test
.../demo/test/TestScript
<build_host> # make a source directory
<build_host> mkdir src
<build_host> cd src
<build_host> # add a source file and makefile
<build_host> echo '#include <stdio.h>' > hello.c
<build_host> echo 'main(){ printf("hello world\n"); }' >> hello.c
<build_host> echo 'all: hello' > Makefile
<build_host> cd ..
<build_host> # edit the toplevel makefile to build and install "hello"
<build_host> ed Makefile
17373
/^all:/
all: proddir_is_set build_prefix
a
[tab]cd src; make ; cp hello ../bin
.
w
q
<build_host>

Of course, you will probably want to use your favorite editor rather than ed, but it is notoriously
hard to show other editors in use in a document. We now have a reasonable first approximation of the file tree we want to have under version control.

Putting the file tree into a cvs repository

Since we will be using cvs to maintain and track our sources, before doing much else, let us
go ahead and put this file tree into a repository using cvs import. If you are not already familiar with cvs you may want to look at the online documentation , in particular the sections What is CVS?, A sample session, and Starting a new project. In our case we will be using the second approach from the
latter manual section, and importing a whole file tree that we have just created.
Before we can do this, however, we must have a CVS repository into which we can place this source code module for our product. On most Fermilab development systems, we accomplish this by setting up a ups
product like uascvs or cdcvs: setup uascvs Actually adding the module is accomplished with a simple command line:
>cvs import -m "demo product" demo Fermilab username_version which will take the current directory tree and install it into our cvs repository as version username_version.
I reccomend you actually put your username in there, so that several people following these instructions will check in different versions of the demo product sources.

If the above command gives you an error like:
Permission denied.
cvs [import aborted]: end of file from server (consult above messages if any)
you will need to contact the administrative contact for the repository, which should be
the first part of your CVSROOT environment variable -- for example if your CVSROOT
is cvsuser@oss-cvs.fnal.gov:/usr/products/olsshare/cvs you would send mail to
cvsuser@oss-cvs.fnal.gov to get permission to access that repository.

Establishing sessions on build machines

We can now start working on several machines in parallel, in order to speed up the process of getting the software built on several machines. The tool I prefer for his work is a package called buildmanager, which is an expectk script to run login sessions on multiple machines. To run it:

<build_host> # Setup the product
<build_host> setup buildmanager
<build_host> # start buildmanager
<build_host> buildmanager demo username_version

This will pop up a login window:
partial screen image

and an interaction window:
partial screen image

You can see the list of hosts you will be connecting to, and click the checkboxes to
the left of each host to include them or not. Once the host list is to your liking, enter your
login name and password in the password window, and hit enter in the password field,
or press the Dismiss button to start logging into the various systems. More details
on using buildmanager can be found by clicking on the [Help...] button, or on the
buildmanager web page .

Checking out a copy on each platform

To get a working copy for each platform build, you can simply pick cvs checkout from the Commands menu, and buildmanager will change directories to the designated scratch area on each system and issue the appropriate commands. You probably then want to select cvs update -A from the same menu in
order to clear up the "sticky" tags that checking out a specific revision creates (see the bottom of this page in the cvs manual).

Doing a test build

For early stages of development you probably want to do this in several stages, first setting up
the product for build by choosing setup -q build, then doing a compilation by doing a make all,
and run any provided tests with make test, all of which can be chosen from the Commands menu.

Propagating changes with cvs

If you make a change in one working copy of the sources, and want to get the copies all the same, first do a cvs commit -m "some reason" in the window for the host where the change was made, and then do
a cvs update in all the windows (either from the entry box at the top of the buildmanager window, or via the Commands menu.)

Distributing your product

Once your product works to your satisfaction, you can distribute it by doing a make kits from the
Commands menu, which will take turns running that make command in each window to add the
product to a distribution system as defined in the Makefile.