Chapter 22: Configuration of the fnkits Product Distribution Node

UPS/UPD Doc Home page | Computing Division| Fermilab at Work | Fermilab Home


View/print PDF file

Complete Guide and Reference Manual for UPS, UPD and UPP v4

Chapter Contents

Chapter 22: Configuration of the fnkits Product Distribution Node
  22.1 UPS Configuration for KITS Database
  22.2 UPS Configuration for local Product Database
  22.3 UPD Configuration
    22.3.1 updconfig File Organization
    22.3.2 The Recognized Product Categories
    22.3.3 Matching Product Categories to updconfig Stanzas
    22.3.4 Location and File Name Definitions
    22.3.5 Pre- and Postdeclare ACTIONS
  22.4 fnkits Server Maintenance
    22.4.1 User Accounts and Group Ids
    22.4.2 Database and Configuration File Locations
    22.4.3 Web Server and FTP Log File Information

Chapter 22: Configuration of the fnkits Product Distribution Node

This chapter describes the UPS/UPD configuration on the Computing Division's central product distribution node, fnkits.fnal.gov.1 Information is provided for both the KITS distribution database and the server's local database.

22.1 UPS Configuration for KITS Database

The KITS database on the fnkits.fnal.gov node has a fairly minimal configuration file, typical for distribution databases:

The database is configured to allow all registered nodes to read and use the products in it.
Statistics are not collected for any products.
Locations are defined for product instances, the UPS initialization files and the UPD configuration file.

For reference, we list the contents of the dbconfig file for KITS (minus the comments):
FILE = DBCONFIG 
AUTHORIZED_NODES = * 
PROD_DIR_PREFIX = /ftp/products 
STATISTICS =  
SETUPS_DIR = /fnal/etc 
UPD_USERCODE_DIR = /fnal/ups/db/.updfiles 
22.2 UPS Configuration for local Product Database

The local database on the fnkits node is maintained at /fnal/ups/db. The dbconfig file for this database is typical for databases on user nodes, where products are unwound and available for use. This file happens to contain all the information that the dbconfig file for the KITS distribution database does, except that the product area, defined by PROD_DIR_PREFIX, is different. In addition to this content, there are definitions of target directories for various product information files (e.g., man pages).

For reference, we list the contents of the dbconfig file for the local database (minus the comments):
FILE = DBCONFIG 
AUTHORIZED_NODES = * 
PROD_DIR_PREFIX = /fnal/ups 
STATISTICS =  
MAN_TARGET_DIR = /fnal/ups/man 
CATMAN_TARGET_DIR = /fnal/ups/catman 
INFO_TARGET_DIR = /fnal/ups/Info 
HTML_TARGET_DIR = /fnal/ups/htmldocs 
NEWS_TARGET_DIR = /fnal/ups/news 
SETUPS_DIR = /fnal/etc 
UPD_USERCODE_DIR = /fnal/ups/db/.updfiles 
In particular, notice that UPD_USERCODE_DIR is set to the same value in both files. This indicates that the databases share a UPD configuration.

Soon after the release of this document (mid-2000), the local database on fnkits will point to its own UPD configuration.

22.3 UPD Configuration

22.3.1 updconfig File Organization

The UPD configuration for both the KITS distribution database (/ftp/upsdb) and the database used for locally installed products (/fnal/ups/db) is contained in the same file, ${UPD_USERCODE_DIR}/updconfig.2 ${UPD_USERCODE_DIR} is defined to be /fnal/ups/db/.updfiles in the dbconfig files for both databases.

The dbconfig file includes several stanzas, each of which pertains to a category of product. The product-matching criterion for each stanza is an option which indicates the category. The categories are: default (no option), local, fermitools, proprietary, fnalonly, and usonly. For example, the GROUP: section of the stanza for default products is empty, the one for proprietary contains the options line:
group: 
     options = "proprietary" 
and so on. The contents of the COMMON: sections for each category, namely the location and file name definitions and any actions, are listed in sections 22.3.4 Location and File Name Definitions and 22.3.5 Pre- and Postdeclare ACTIONS.

22.3.2 The Recognized Product Categories

default

The default category is the most commonly used, and is for regular products added to the KITS database (/ftp/products or /ftp/KITS) for distribution to any on-site or registered off-site node3. The products are set to group upd, and group-read-only. No option is associated with the default.

local

(This will be dropped mid-2000.) The local category is for products installed (using upd install) into the local database, ftp/ups/db, for use on the fnkits node itself (as opposed to those added to KITS for distribution). For these products, the -O local option must be included in the upd install command. (Used only by the fnkits system managers.)

fermitools

fermitools products are locally-developed and supported software packages (which are not available elsewhere, generally) that we make available to the public via our FermiTools program 4. These products are installed in the KITS database, are world-readable, and have a symlink hierarchy under /ftp/pub. The /ftp/pub hierarchy has been created with the same structure as /ftp/KITS.

proprietary

The proprietary category includes products for which Fermilab has a limited number of licenses. These products are installed in the KITS database, and made accessible only to special groups.

fnalonly

The fnalonly category is for products accessible only to the fnal.gov domain. They are installed in the KITS database, set to group fnalonly, and are group-read-only.

usonly

US-only (United States only) products are accessible only to U.S. government (.gov) and military (.mil) domains. In general, these are products for which distribution to other countries is illegal. They are installed in the KITS database, set to group usonly, and are group-read-only.

22.3.3 Matching Product Categories to updconfig Stanzas

When adding a product belonging to any of the categories fermitools, proprietary, fnalonly or usonly, do not specify the corresponding option via the -O flag on the upd addproduct command line. Instead, first fill out and submit the Special UPD Product Registration form (at http://fnkits.fnal.gov/specialprod.html) identifying the product and its category. After you receive a confirmation via email, add the product as you would a "default" product (see Chapter 18: Making Products Available For Distribution). The option corresponding to your selected category gets set automatically in order to invoke the proper stanza in the updconfig file.

22.3.4 Location and File Name Definitions

All Product Categories (except local)

The following location and file name definitions are shared by the stanzas for the product categories default, usonly, fnalonly, proprietary, and fermitools:
UPS_THIS_DB = /"ftp/upsdb" 
UNWIND_PROD_DIR="/ftp/products/${UPS_PROD_NAME}/${UPS_PROD_VERSION}/ 
  ${UPS_PROD_FLAVOR}/${UPS_PROD_NAME}_${UPS_PROD_VERSION}_${UPS_PROD_FLAVOR} 
  ${UPS_PROD_QUALIFIERS}" 
UNWIND_UPS_DIR = "${UNWIND_PROD_DIR}/ups" 
UNWIND_TABLE_DIR = "/ftp/products/${UPS_PROD_NAME}/${UPS_PROD_VERSION}/ 
  ${UPS_PROD_ FLAVOR}" 
UNWIND_ARCHIVE_FILE = "${UNWIND_PROD_DIR}.${SUFFIX}" 
UPS_TABLE_FILE="${UPS_PROD_NAME}_${UPS_PROD_VERSION}_${UPS_PROD_FLAVOR} 
  ${UPS_PROD_QUALIFIERS}.table" 
UPS_TABLE_DIR = "${UNWIND_TABLE_DIR}" 
UPS_PROD_DIR = "${UNWIND_PROD_DIR}" 
UPS_UPS_DIR = "ups" 
After the mid-2000 change, we expect a few of these values to change, as follows:
UNWIND_PROD_DIR="/ftp/products/${UPS_PROD_NAME}/${UPS_PROD_VERSION}/ 
  ${UPS_PROD_FLAVOR}/${UPS_PROD_NAME}_${UPS_PROD_VERSION}_${UPS_PROD_FLAVOR} 
  ${UPS_PROD_QUALIFIERS}" 
UNWIND_UPS_DIR = "${UPS_PROD_DIR}/ups" 
UNWIND_TABLE_DIR = "${UPS_TABLE_DIR}" 
UPS_TABLE_DIR = "/ftp/products/${UPS_PROD_NAME}/${UPS_PROD_VERSION}/ 
  ${UPS_PROD_FLAVOR}" 
The UPS_ARCHIVE_FILE definition varies. Different FTP server port numbers with different permissions are used according to the product category. For default, proprietary and fermitools the value is:
UPS_ARCHIVE_FILE = ${UNWIND_ARCHIVE_FILE} 
For fnalonly, it is:
UPS_ARCHIVE_FILE = ":9021${UNWIND_ARCHIVE_FILE}" 
For usonly, it is:
UPS_ARCHIVE_FILE = ":8021${UNWIND_ARCHIVE_FILE}" 
local

The location and file name definitions for local products are (again, this will be in a separate UPD configuration file after mid-2000):
UPS_THIS_DB= "/fnal/ups/db" 
UNWIND_PROD_DIR= "/fnal/ups/${UPS_PROD_NAME}/${UPS_PROD_VERSION}/${UPS_PROD_FLAVOR}${UPS_PROD_QUALIFIERS}" 
UNWIND_UPS_DIR= "${UNWIND_PROD_DIR}/ups" 
UNWIND_TABLE_DIR = "${UPS_THIS_DB}/${UPS_PROD_NAME}" 
UPS_TABLE_FILE= "${UPS_PROD_VERSION}.table" 
UPS_PROD_DIR = "${UNWIND_PROD_DIR}" 
UPS_UPS_DIR = "ups" 
22.3.5 Pre- and Postdeclare ACTIONS

The stanzas for all categories of product except local include a PREDECLARE and a POSTDECLARE action.5

In each case, the PREDECLARE action includes a set of execute statements to chmod/chgrp the files to the right group id and permissions, and another set to symlink files under /ftp/KITS to provide the old-style (UPS/UPD v3) KITS hierarchy6 of KITS/Flavor/product/version. In addition, fermitools includes commands to send notification email.
The POSTDECLARE action makes a convenience tar file of the ups directory for users downloading via ftp.

The execute statements in each stanza are similar, but not identical. We first list them for the default case, and then list the differences for the other product categories relative to the default.

The stanza for local products contains no actions.

PREDECLARE Action for default Products

The PREDECLARE action for the default product category fixes group permissions:
action = predeclare 
Execute("chgrp upd ${UNWIND_TABLE_DIR}/*", NO_UPS_ENV) 
Execute("chmod o-rwx ${UNWIND_TABLE_DIR}/*", NO_UPS_ENV) 
Execute("chmod a+r ${UNWIND_TABLE_DIR}/*.table", NO_UPS_ENV) 
and makes old-KITS compatible hierarchy files:
Execute("test -d  /ftp/KITS/${UPS_BASE_FLAVOR}/${UPS_PROD_NAME}/${UPS_PROD_VERSION} || mkdir -p /ftp/KITS/${UPS_BASE_FLAVOR}/${UPS_PROD_NAME}/${UPS_PROD_VERSION}", NO_UPS_ENV) 
Execute("cd /ftp/KITS/${UPS_BASE_FLAVOR}/${UPS_PROD_NAME}/${UPS_PROD_VERSION}; rm -f ${UPS_PROD_NAME}_${UPS_PROD_VERSION}_${UPS_PROD_FLAVOR}${UPS_PROD_QUALIFIERS}.*", NO_UPS_ENV) 
Execute("cd /ftp/KITS/${UPS_BASE_FLAVOR}/${UPS_PROD_NAME}/${UPS_PROD_VERSION}; /usr/bin/ln -fs ${UNWIND_PROD_DIR}.* . || true",NO_UPS_ENV) 
PREDECLARE Action for FermiTools Products

The PREDECLARE action is the same as for the default products except for the changes noted here.

For fermitools, there is no Execute ("chgrp...") command. The first chmod command is o+rx rather than o-rwx:
Execute("chmod o+rx ${UNWIND_TABLE_DIR}/*", NO_UPS_ENV) 
Also for fermitools, commands are included to send notification e-mail:
Execute("sh -c \"dir=/ftp/pub/${UPS_PROD_NAME}/${UPS_PROD_VERSION}; test -d \\\$dir || mkdir -p \\\$dir\"", NO_UPS_ENV) 
Execute("sh -c \"dir=/ftp/pub/${UPS_PROD_NAME}/${UPS_PROD_VERSION}; cd \\\$dir;  /usr/bin/ln -sf ${UNWIND_PROD_DIR}.* .; /usr/bin/ln -sf ${UNWIND_PROD_DIR}/README .\" || true", NO_UPS_ENV) 
Execute ("echo Fermitools product ${UPS_PROD_NAME} ${UPS_PROD_VERSION} -f ${UPS_PROD_FLAVOR} has been added to fnkits | /bin/mail fermitools_support\@fnal.gov", NO_UPS_ENV) 
PREDECLARE Action for proprietary, fnalonly and usonly Products

The PREDECLARE action is the same as for the default products except for the changes noted here.

For proprietary products, the chgrp command changes to:
Execute("chgrp `echo ${UPS_PROD_NAME} | sed -e 's/^vx.*/vx_dart/' | tr a-z A-Z` ${UNWIND_TABLE_DIR}/*", NO_UPS_ENV) 
For fnalonly and usonly products, the group used in the chgrp command changes to FNALONLY and USONLY, respectively:
Execute("chgrp FNALONLY ${UNWIND_TABLE_DIR}/*", NO_UPS_ENV) 
and
Execute("chgrp USONLY ${UNWIND_TABLE_DIR}/*", NO_UPS_ENV) 
POSTDECLARE Action for All Product Categories (except local)

The POSTDECLARE action makes a ${UNWIND_PROD_DIR}.ups.tar tar file:
action = postdeclare 
Execute("test -d \"${UNWIND_UPS_DIR}\" && cd ${UNWIND_UPS_DIR} && tar cf ${UNWIND_PROD_DIR}.ups.tar . || true", NO_UPS_ENV) 
22.4 fnkits Server Maintenance

22.4.1 User Accounts and Group Ids

The fnkits Web and FTP server configuration files are owned by oss.

The Web server runs as updadmin. updadmin owns all of the FTP-served product files and the KITS UPS database, and therefore also the Web server logs.

Many group ids are used, see /etc/ftpd/ftpgroups on fnkits for the complete list.

22.4.2 Database and Configuration File Locations

The KITS distribution database is /ftp/upsdb.

The local database is under /fnal/ups/db.

The Web server configuration files are maintained under the /fnal/www directory.

The FTP configuration files live under /etc/ftpd, except for usonly and fnalonly products which are kept under /etc/ftpd-usonly and /etc/ftpd-fnalonly, respectively.

"Sanitized" versions of the configuration files (with hostnames and such trimmed) are available at http://ftp.fnal.gov/.

22.4.3 Web Server and FTP Log File Information

Logs are kept for roughly 30 days.

The Web server logs are kept in /fnal/log/www.

The FTP logs are kept under /var/adm/ftpd, except for usonly and fnalonly products which are kept under /var/adm/ftpd-usonly and /var/adm/ftpd-fnalonly, respectively.

The log summaries are maintained under /fnal/www/kits/html/logs, also accessible via http://ftp.fnal.gov/logs. They are prepared using the product ftpweblog.

1
Other names used for this server are: fnkits, kits, kits.fnal.gov, upd, and upd.fnal.gov.

2
This is changing mid-2000; the local database will have a separate UPD configuration file.

3
See the Product Distribution Platform Registration Request form at http://computing.fnal.gov/xms/?pid=1000373.

4
For more information on FermiTools, see http://www.fnal.gov/fermitools/.

5
After mid-2000, some of the functions in the PREDECLARE actions move to a common POSTDECLARE action, namely the functions that make old-KITS compatible hierarchy files.

6
Currently nothing prunes old links or files from this hierarchy.

UPS/UPD Doc Home page | Computing Division| Fermilab at Work | Fermilab Home


View/print PDF file

This page generated on: 10/15/02 14:06:43