TOC PREV NEXT
Fermilab CD logo Complete Guide and Reference Manual for UPS and UPD

Chapter Contents

Chapter 17: Building UPS Products
  17.1 Basic Steps for Making a UPS Product
    17.1.1 Build the Directory Hierarchy
    17.1.2 Create the Table File
    17.1.3 Declare the Product to your Development UPS Database
    17.1.4 Copy the Product Executable to the bin Directory
    17.1.5 Provide Documentation
    17.1.6 Test the Product
  17.2 Specifics for Different Categories of Products
    17.2.1 Unflavored Scripts
    17.2.2 Pre-built Binaries
    17.2.3 Products Requiring Build (In-House and Third-Party)
    17.2.4 Overlaid Products
  17.3 Sample Auxiliary Files
    17.3.1 README
    17.3.2 INSTALL_NOTE


Chapter 17: Building UPS Products


In this chapter we describe the steps needed to prepare a product for inclusion into the UPS framework and then to prepare it for distribution. We go through the steps for a simple case, then discuss the additional steps that may be required in more complex situations. Some sample auxiliary files are provided at the end.

17.1 Basic Steps for Making a UPS Product

We'll use the standard "Hello world" example, with a product hello, version v1_0, of flavor NULL. The executable, which is a script in this case, consists of the following text:

#!/bin/sh

echo "Hello world"

This is a simple case. You don't need a script on how to build this product, because it doesn't get built. It runs on all flavors of UNIX without modification, so you should declare it with the flavor NULL. It would be nice to have the $HELLO_DIR/bin directory added to your $PATH to use the product, and that's what the setup action will do. The unsetup action will remove $HELLO_DIR/bin from your $PATH. No configuration is needed, nor are any special actions when the product is declared current.

The steps you need to complete are:

  1. Create a directory hierarchy for the product and its related files.
  2. Create a README file.
  3. Create a table file in the location you want it to reside (usually either in the product-specific directory directly underneath your UPS development database or in the ups directory, if your product has one).
  4. Declare the product to your UPS development database with the development chain so that it doesn't interfere with other peoples' work. Although the product itself doesn't exist yet, the declaration can be done and we recommend it at this stage for convenience.
  5. Create the product script in the bin directory (or copy it into there).
  6. Create documentation.
  7. Test the product.

17.1.1 Build the Directory Hierarchy

We will take the product root directory to be hello/v1_0. This product root directory can sit anywhere in the file system. An appropriate, simple directory structure underneath the product root directory is as follows:

bin
contains the executable script hello
test
contains the test script(s)

A README file should go directly under hello/v1_0. We'll put the table file, called hello.table, under the database. Some products would have more subdirectories and files than shown here.

17.1.2 Create the Table File

For our example, we'll create the file hello.table and put it in the product subdirectory of the development database. A simple table file for this product might look like:

FILE=TABLE 
PRODUCT=hello 
VERSION=v1_0 
# 
#--------------------- 
FLAVOR = ANY 
QUALIFIERS = "" 
 
ACTION=SETUP 
   pathPrepend(PATH, ${UPS_PROD_DIR}/bin, :) 
   setupEnv() 

17.1.3 Declare the Product to your Development UPS Database

Refer to section 11.1 Declare an Instance for instructions on declaring the product to your UPS database, or see the reference section 23.5 ups declare. In particular, note two things:

  1. For an unflavored script like this example, declare the flavor specifically as NULL (using either the -f NULL or -0 option).
  2. Declare it with the chain development for your pre-distribution testing (using the -d option).

For example:

% ups declare -0dz /ups_dev_db -r /ups_dev_prod/hello/v1_0 \ 
-m hello.table hello v1_0 

We recommend declaring at this stage for reasons of convenience and organization. It allows you to run setup [-d] on the product to make the $<PRODUCT>_DIR environment variable available.

17.1.4 Copy the Product Executable to the bin Directory

Create the script in the bin directory, or copy or move it to this location.

17.1.5 Provide Documentation

In this simple case, documentation isn't needed, but normally a README file is created and possibly a wiki page on redmine.

17.1.6 Test the Product

Now you can setup and test your product. As an example, for our product we might run:

% setup hello v1_0 
% hello 
Hello world 
% unsetup hello v1_0 
% hello 
sh: hello: command not found 

In many cases, writing a good test script can be rather challenging. Include at least a basic test to ensure that the product works properly. For our example, the test script just needs to run our hello program and verify its output, e.g.,:

#!/bin/sh 
hello | grep "Hello world" > /dev/null 

This will exit with a successful exit code if hello prints Hello world, and fail otherwise.

17.2 Specifics for Different Categories of Products

This section discusses all the steps you need for turning virtually any product into a UPS product. We start with the simpler cases and finish with the more complex ones. For all categories of product, if your product has dependencies, either for building or for execution, you need to have them available on your development system when you build and test the product.

17.2.1 Unflavored Scripts

Unflavored scripts, that is scripts with the flavor NULL, are the simplest form of UPS product. The example in section 17.1 shows how easy it is to create a UPS product from an unflavored script. A product like this does not need to be rebuilt on different architectures, and generally does not need CONFIGURE and UNCONFIGURE actions or scripts. Some, although very few, unflavored scripts require INSTALLASROOT actions in the table file to copy specific files into /usr/local/bin, or to perform similar actions.

17.2.2 Pre-built Binaries

Many third-party products obtained from a vendor or downloaded from the Web are binary images without source code. When you go to a vendor's web site, you will often find separate pre-built binaries for several UNIX operating systems/releases. Note that they may use slightly different terminology than we do to refer to the different flavors.

Generally, to run products that consist of executables (as opposed to libraries, for example), you just need to add the executable directories to your $PATH after downloading. To make a product compatible with UPS, you should provide a table file that modifies the $PATH, a README file and some documentation. If the vendor provides examples and/or any other user files, include them. Most products distributed in this manner include documentation.

Follow this general procedure:

  ACTION=SETUP 
     pathPrepend(PATH, ${UPS_PROD_DIR}/bin, :) 

Now it's time to create areas for each flavor of the product that you plan to install.

17.2.3 Products Requiring Build (In-House and Third-Party)

Most locally developed products, and many vendor-supplied products, are distributed as source code which must be rebuilt for each OS flavor. We are trying to get away from UPS-packaging vendor-supplied products, however, we provide instructions in case you need to do so.

If you are building a product which was obtained from an outside source, you may not have control over the product directory hierarchy. Some outside products include configuration options to specify where the resulting libraries and/or images should reside, but in other cases you must give a hard-coded path to the final output file. In the latter cases, when it is absolutely necessary, you may need to use UPS as a "bookkeeping" wrapper and common point of distribution.

If you are developing the product yourself, you should follow these guidelines:

Preparation for Rebuilding Any Product

For any product, you first need to create the infrastructure. Much of the work needs to be done only once, and is reused for each flavor of the product that is built:

Steps for Rebuilding a Product

Once you have created the product structure along with all of the support files, you will need to get down to the business of actually building the product images. If you're planning on redistributing this product to a wider audience than just your machine, you must be careful in selecting a build node. The build nodes should have appropriate levels of compilers, OS, and other products required for building the given product.

We recommend that you create separate build areas, one for each target flavor, so that the different flavors of binary files do not get mixed up. Once you have completed the preparation described above, complete these steps:

Then for each of the target flavors:

17.2.4 Overlaid Products

An overlaid product gets distributed and maintained in the product root directory of its main product. For example, the overlaid products cern_bin, cern_ups, cern_lib, etc., all reside in the product root directory for the main product cern. A patch is another good example of the use of overlaid products. The set of products overlaid on a main product is collectively referred to as the overlay.

A special keyword, _UPD_OVERLAY, is provided for inclusion in the table file of each overlaid product. _UPD_OVERLAY takes as its value the main product name in double quotes. Its presence indicates that the product is an overlaid product maintained in the root directory of the main product listed as the keyword's value. For example, the table files for the products cern_bin, cern_ups, and cern_lib would contain the following keyword line:

_UPD_OVERLAY = "cern"

UPD would then use cern as the product name when determining the root directory.

In addition to including all the overlaid products as dependencies of the main product, we recommend including the main product as a dependency of each of the overlaid products. This allows separate installation of each of the pieces. Circular dependency lists are allowed in UPS.

17.3 Sample Auxiliary Files

17.3.1 README

Following is the README file for the jobsub_tools product. It has been edited for brevity, but shows the information to include:

Jobsub_tools 

A KITS product containing all the Intensity Frontier grid submission tools.

Usage:
setup jobsub_tools 
(then)
jobsub [options]
minos_jobsub [options]
minerva_jobsub [options]
nova_jobsub [options]
dagNabbit.py [options]

...

$JOBSUB_TOOLS_DIR/docs contains additional documentation
$JOBSUB_TOOLS_DIR/test contains unit tests and usage examples

--------------------------
RELEASE NOTES
--------------------------
v0_1  5/17/2012
===========================
First UPS/UPD version, KITS product is jobsub_tools, i.e.  setup jobsub_tools

Includes unit test, to run:
sh $JOBSUB_TOOLS_DIR/test/Run_Unit_Tests.sh

...

v1_1t 3/14/13
==============================================================

-changes behavior of -d flag so that multiple copies out are wrapped
with one CPN lock, similar to -f behavior as of v1_1s

-fixed unit tests broken by v1_1s code changes

v1_1u 3/19/13
=============================================================
-fix of chmod bug for copying output on v1_1t

17.3.2 INSTALL_NOTE

If there is an INSTALL_NOTE, upd will tell you about it. For example:

% upd install ftt -z /scratch/mengel/p
informational: installed ftt v2_25.
informational: product ftt has an INSTALL_NOTE;
    you should read /scratch/mengel/p/ftt/v2_25/Linux-2-6/ups/INSTALL_NOTE.
upd install succeeded.


TOC PREV NEXT

Last revised May 2014