TOC PREV NEXT
Fermilab CD logo Complete Guide and Reference Manual for UPS and UPD

Chapter Contents

Chapter 2: Overview of UPS and UPD
  2.1 Introduction to UPS and UPD
  2.2 Motivation for the UPS Methodology
  2.3 UPS Products
    2.3.1 Versions
    2.3.2 Flavors
    2.3.3 Qualifiers
    2.3.4 Product Instances
    2.3.5 Chains
    2.3.6 Product Dependencies
    2.3.7 Product Overlays
  2.4 UPS Database Overview
    2.4.1 UPS Database Files
    2.4.2 UPS Database Structure
  2.5 Using UPS Without a Database
  2.6 UPS and UPD Commands
    2.6.1 Syntax
    2.6.2 Defaults
  2.7 The UPS Environment
    2.7.1 Initializing the UPS Environment
    2.7.2 Changes UPS Makes to your Environment
    2.7.3 Changes UPD Makes to your Environment


Chapter 2: Overview of UPS and UPD


UPS and UPD are the utilities provided by Fermilab's Computing Division for managing and standardizing software product development, distribution, support and access. This overview chapter describes these utilities briefly, and discusses the reasons for which this particular product methodology was chosen and developed. The chapter also describes:

2.1 Introduction to UPS and UPD

UPS (UNIX Product Support) is a software support toolkit developed at Fermilab for the management of software products on local systems to facilitate product distribution and configuration management. The three principal user benefits are:

UPD (UNIX Product Distribution) is a companion product to UPS, and provides the functionality for uploading/downloading products between local systems and product distribution servers.

A UPS/UPD installation usually has several parts:

2.2 Motivation for the UPS Methodology

The principal reason for developing and using UPS product support methodology is to provide self-contained products. Each release of a self-contained product can be installed and accessed independently of other releases. There are two main advantages to this, which are especially important for applications such as real-time data acquisition:

  • Multiple versions of a software product can be made available concurrently. This is useful in many situations, especially when some products depend on particular versions of others for compiling or running. Multiple concurrent versions also makes possible the second advantage:
  • A new software release can be backed out completely and assuredly if something goes wrong, and immediately start up a previous tried-and-true release.

The UPS methodology provides tracking. The UPS database tells you what version of a product is running; users don't need to keep track of it elsewhere or risk forgetting. Users can tell if different machines are running the same version of a product.

Additional features:

Further advantages for product installers, system administrators and product providers include:

2.3 UPS Products

Products distributed and managed by UPS on a distribution or user node are called UPS products. Typically, UPS products on a system are declared in a UPS database on that system. UPS products can be maintained on different disks, and/or in different directories. Each product in a UPS database is managed via a set of files that provide product management information to UPS, e.g., the location of each copy of the product, what its status is (e.g., appropriate for general use, for testing, for development), what needs to be done when the product is installed or accessed, and so on.

UPS products are generally self-contained and portable, laid out in a directory structure under a product root directory . The structure of a product's directory tree is not dictated by UPS, but generally includes areas for the executables (bin), for test scripts (test), and for documentation (e.g., man, catman, docs, html).

2.3.1 Versions

UPS supports multiple concurrent versions of software products. Each version of a product is installed and accessed independently of other versions. When a new version of a product becomes available, an existing directory tree is not replaced with another; rather, a branch is added for the new version. (See the diagram in section 2.4.2 UPS Database Structure.)

2.3.2 Flavors

Many programs require separate compilations for each of the different UNIX operating systems. UPS allows a separate directory tree for each compilation (and related files) of the same product. To distinguish between different OS-dependent compilations, the term flavor is used. This allows the same product name and version across different operating systems and OS versions, which is desirable since the same program source files are generally used in the separate compilations.

Several different copies of a product may exist on a given system. The system needs enough information to select the appropriate UPS commands. All UPS commands support allowing the flavor to be specified.

The flavor of a particular product compilation as declared in a UPS distribution database may or may not indicate the version of the OS for which it was compiled. The standard we have adopted for flavors is:

The recommendation is to use the same attributes as where the package came from, and add extra declarations with the same product path, with different attributes as needed. See 3.1 Determining your Machine's Flavor for details.

2.3.3 Qualifiers

The product developer may include information about options used at compilation time (e.g., debug or optimized) or other qualifying information for easy identification of special compilations. This information is declared in the form of qualifiers in a distribution database. When a product is declared to UPS, the installer has the choice of declaring these qualifiers or omitting them in the declaration.

Declaring the qualifiers allows full identification of the product compilation on your system. The drawback is that to access a product compilation declared with one or more qualifiers, a user must specify the qualifier(s) when accessing that compilation.

No standard set of qualifiers has been defined; the naming of qualifiers is at the discretion of the product provider, and thus may vary from product to product.

Some projects (Mu2e, nova, etc) require that users specify the version and qualifiers to avoid the problem of a default qualifier changing in a poorly documented fashion, making it difficult to reproduce results.

2.3.4 Product Instances

Each installed copy of a product that is declared to a UPS database is called an instance of the product. Within a database, a product instance is distinguished by a unique combination of product name, version, flavor, and qualifiers. In the case of multiple databases, the database specification is often also needed to distinguish an instance because identical instances can be maintained in different databases. The concept of chains, discussed next allows users to easily access the appropriate instance of a product according to their needs without having to remember its version number and other details.

2.3.5 Chains

UPS supports multiple versions of software products on a machine. End users do not find it convenient to specify product version numbers each time they set up a product. This is especially true if product setups are needed at login. Most users want to run the latest, tested, approved version of products without having to keep track of the version numbers.

To allow users to specify the version of a product according to its status rather than by its version number, UPS supports chains to product versions. A chain can be thought of as an alias for a particular product instance. It indirectly "attaches" a chain name to a product instance, thereby tagging the product instance according to its status.

Five standard chains, have been defined for use: current, new, test, old, and development.

Chain Names, Options, and Usage

Chain
Option
Usage
current
-c
default instance recommended for general use
new
-n
tested instance that is not yet current
test
-t
instance installed for testing
development
-d
instance under development
old
-o
older instance that was previously current

Additional chains may be defined by developers and installers and other users. The command option -g is provided for this purpose. See section 11.2 Declare a Chain for an example.

2.3.6 Product Dependencies

Many UPS products require that other UPS products be installed, declared and set up for proper functioning or for use of special features. These other products are referred to as dependencies. A dependency is generally an independent product that is maintained in its own individual product root directory. Non-UPS products are sometimes declared as dependencies as well. End users don't normally need to know what dependencies a particular product has, as long as the product runs without problems.

There are two categories of dependencies: those which are required for the main product to function, and optional ones which generally enable nonessential features of the main product.

The coupling of products with their dependencies facilitates product setup (setup is described in section 3.4 Setting up a Product). Users can set up a single UPS product to access any and all of the products listed as dependencies for that product.

Multiple levels of dependencies are possible. A product may have "first level" dependencies which have dependencies of their own, which in turn may have dependencies, and so on.

UPS product dependencies can exist across databases. For this to work, the databases in which they are declared must all be included in $PRODUCTS, or in a database list specified on the command line.

2.3.7 Product Overlays

An additional class of required product is supported by UPD, called an overlaid product. An overlaid product gets distributed and maintained in the product root directory of its main product. For example, the overlaid products cern_bin, cern_ups, cern_lib, etc., all reside in the product root directory for the main product cern. A patch is another example of the use of overlaid products. The set of products overlaid on a main product is collectively referred to as the overlay. An overlaid product is not precluded from being a dependency of other UPS products.

2.4 UPS Database Overview

The information UPS needs for managing products is maintained in a database. A UPS database is a directory tree which contains a subdirectory for each product declared to the database, the subdirectory having the same name as the product. It also usually contains the hidden subdirectories .upsfiles with UPS database configuration information, and .updfiles with UPD configuration information.

Within each product-specific directory under the UPS database directory, a set of ASCII files collectively contains the UPS management information for the product on that system. We call these files UPS database files.

UPS commands refer to the database directory via the UPS environment variable $PRODUCTS described in section 2.7.2, or the -z <databaseList> option. $PRODUCTS can be set to point to one or to several directories, thus allowing support for multiple databases. This allows users to maintain one or more private databases in addition to or instead of the common one(s).

2.4.1 UPS Database Files

There are two types of UPS database files for each product: version and chain files.

In newer versions of UPS, the vx_y.version may be a directory with several files in it, one for each instance. The chain file can also be a directory with several files in it. This provides for overlaying one database over another to add instances, rather than destroying existing ones.

Some users intermingle the database files with the actual product directories, so they can be archived in a tar file all together for distribution.

Another file that strictly speaking is not a database file, but is used in product management is a table file. Table files contain information which is independent of any local installation and which is specific to one or more particular instances of the product. For example, table files tell UPS what needs to be done to configure a product or to make it accessible for use. Table files are provided by the product developer when needed. Not all products have table files. And conversely, some products consist only of a table file. Table files are described in Chapter 36: Table Files.

Whereas version and chain files are constrained to reside under the UPS database, table files can reside anywhere. Table files are usually kept either in the database or within the product instance's root directory structure. Table file paths are indicated in version files, and they are thereby accessible to UPS regardless of location.

2.4.2 UPS Database Structure

2.5 Using UPS Without a Database

This flexibility is provided primarily for off-site users who for one reason or another do not want to maintain a UPS database on their local system. UPS can be installed on a machine to manage products without a UPS database, albeit in a limited way. (Except where noted, this manual is written with the assumption that UPS is used with a database.) In the absence of a database, there are no database files and every UPS/UPD command must include all of the information that normally would have been read from a database. In particular, all commands require specification of the table file name, and usually its location. Functionality requiring or operating on a UPS database is not supported when UPS is implemented in this way.

Product developers may also work in this type of environment, especially if they're using a code management product.

2.6 UPS and UPD Commands

Please see Part VI UPS and UPD Command Reference of this manual for complete information on UPS and UPD commands. This section briefly describes the command syntax and defaults used.

2.6.1 Syntax

Most UPS and UPD commands are of the form ups <command> or upd <command> (the exceptions are setup and unsetup), and take a variety of command line options and arguments. Multiple arguments must be separated by colons (:). The standard syntax is:

% ups <command> [<options>] <product> [<version>] 

For example:

% ups list -f Linux+2 cvs v1_12_13 

The first occurring unflagged argument on the line after the command is generally interpreted as the product name, and the next (if any) is interpreted as the version. With that limitation, the name, version and options can occur anywhere on the command line.

2.6.2 Defaults

If no database is given, UPS uses $PRODUCTS to determine the database. If no instance-identifying options or version are given, UPS/UPD operates on the instance declared as current for the flavor of the machine on which the command is issued (to the highest specification level possible). If there is no instance declared as current or if the current instance has any qualifiers, then the default instance matching will fail.

Projects wishing to ensure that software built at one site will build in exactly the same way on all other sites may not want the notion of current version. For these projects, the version number and qualifiers are fully specified for each command.

For UPD commands, if no product distribution node is specified, UPD uses the central Computing Division product server fnkits.fnal.gov as the default. (The product distribution database on this node is known as KITS.)

2.7 The UPS Environment

2.7.1 Initializing the UPS Environment

Each experiment might have specific setup scripts, so users should consult their experiment's documentation to find out what they are and where they are located. In order to access and use UPS products or manipulate a UPS database, your environment must be initialized to make the UPS commands available.

This is done automatically if something like the following is in your .profile or in your experiment's setup scripts on the system you log into.
# get ups environment, and then setup the login product
#
if [  -f "/afs/fnal.gov/ups/etc/setups.sh" ]
then
    . "/afs/fnal.gov/ups/etc/setups.sh"
    if ups exist login
    then
        setup login
    fi
fi

The UPS initialization does not carry over to any scripts which either create a new login process or invoke a shell from the "other" shell family. If such a script needs to setup and run UPS products from that new process, you need to include a line in the script which sources the appropriate setups.[c]sh file, as described below.

If you need to initialize your UPS environment yourself, the UPS initialization is accomplished by sourcing a UPS-provided script; namely, one of the following (depending on your login shell):

Bourne shell family

setups.sh 

C shell family

setups.csh 

It must be sourced from the directory where the UPS setup files have been installed on your system. Virtually all supported systems on-site provide "courtesy links" in /usr/local/etc so that you don't need to know exactly where the setups.[c]sh files actually reside. If the setups.[c]sh files are not in /usr/local/etc and your system doesn't maintain these courtesy links, you will need to ask your system administrator or UPS database maintainer where to find these files.

You can either source the setups.[c]sh scripts manually (for occasional use), or you can add the following to your .[c]shrc file so that UPS gets initialized every time you log in:

Bourne shell family

. /path/to/setups.sh 

C shell family

source /path/to/setups.csh 

You also need to include this line in any other scripts which setup and invoke UPS products, once per login session and any time a shell from the other shell family is invoked.

2.7.2 Changes UPS Makes to your Environment

The following environment variables get set/modified by setups.[c]sh:

$PRODUCTS
If only one UPS database is defined, this points to it; if two or more are defined, this variable can be set as the colon-separated list of UPS databases. The order of the databases in this list reflects the order of precedence for accessing products.
$UPS_DIR
This points to the top level directory (called the product root directory) of the active instance of UPS.
$UPS_SHELL
Set to sh or csh, depending on shell family in use.
$PATH
Modified to include $UPS_DIR/bin.
$SETUP_UPS
a string containing all the information that the unsetup command needs in order to identify the active instance of UPS (e.g., ups v4_5_2 -f SunOS+5 -z /path/to/db)

2.7.3 Changes UPD Makes to your Environment

The following environment variables get set/modified by the setup upd command.

$FTP_PASSIVE
allows UPD to work behind certain firewalls
$SETUP_UPD
a string containing all the information that the unsetup command needs in order to identify the active instance of UPD
$PATH
modified to include $UPD_DIR/bin.
$UPD_USERCODE_DIR
directory containing UPD configuration
$UPD_USERCODE_DB
database containing UPD configuration



TOC PREV NEXT

This page last revised in May 2014