Chapter 4: PNFS Namespace

PNFS is a virtual file system package that implements the Enstore namespace. Refer to section 1.2 PNFS Namespace for introductory information on PNFS.

4.1 UNIX Commands You can Use in PNFS Space

Data files do not actually reside in the /pnfs namespace. Errors occur on attempts to read or write the content of the files, or to manipulate the content. Therefore, UNIX commands such as cat, more, less, grep, head, tail, wc, od, file, cp, and so on, fail if you run them on files listed under /pnfs.

However, virtually any non-I/O UNIX command can be used in the /pnfs namespace. For help with these commands, consult a UNIX manual or the man pages. Read and write access is governed by standard UNIX file permissions. Commands that you may find useful include:

The paths: /pnfs/xyz, /pnfs/fs/usr/xyz and /pnfs/fnal.gov/usr/xyz all refer to the same directory. When using Enstore without dCache the first path is most often used. When using dCache (with or without Enstore) the second path is most often used. The third path type is used with SRM transfers.

4.2 PNFS Limitations

• PNFS uses NFS version 2. Newer systems are defaulting to NFS 4. To force the use of NFS 2, include vers=2 mount option.

• On some SGI kernels (confirmed on 6.5.27 and 6.5.28) the NFS implementation is broken. Use the proto=udp mount option as a workaround.

• Recursive commands like ls -lR and find will put a lot of load on PNFS. Their use is strongly discuraged.

• When deleting a directory containing a large number of files do not do rm -[r]f *. Instead use a loop that will sleep after each file is removed. Doing this will take a lot of uncecessary stress off of PNFS.

4.2.1 PNFS Directory Limitations

• It is recommended to keep the number of files in any given PNFS directory under 2000. This is recommended for any NFS-based file system.

• The maximum length of a file in PNFS is 200 characters. However, encp will furthur restrict the filename length with respect to the size of the file.

4.3 About PNFS Tags

Before files can be written to tape, Enstore needs to know where and how to write them. Pnfs uses tag files (usually just called tags) in the /pnfs namespace to specify this type of configuration information, and encp transfers this information to Enstore. Tags are associated with directories in the /pnfs namespace, not with any specific file, and thus apply to all files within a given directory (with the exception noted below). When a new directory in the /pnfs namespace is created, it inherits references to the tags of its parent directory. It is a feature of PNFS that a change to a parent directory will also affect its existing subdirectorys’ tag references.

Manually setting a directory’s tags will destroy references to its parent directory’s tags. This may be what you want to do, but be aware.

A file gets the tag references of its directory as they exist when the file is written to Enstore, and these are what encp uses to access it. Subsequent changes to a directory’s tag references do not affect pre-existing files, therefore it’s possible to have files in a directory to which the current directory tags do not apply.

Allowable characters within tags are: alphanumeric characters, underscore ( _ ), dash ( - ), and slash ( / ).

4.3.1 Tag Listing

The tags include:

file_family This tag determines the file family associated with all files in this directory. See section 2.2.1 File Family for information on file families.

file_family_width This tag determines the file family width associated with all files in this directory. See section 2.2.2 File Family Width for information on file family width.

file_family_wrapper This tag determines the file family wrapper associated with all files in this directory. See section 2.2.3 File Family Wrapper for information on file family wrappers. The default is cpio_odc.

library This tag determines the virtual library (and thus the library manager) associated with all files in this directory. See section 8.3 Library Manager for information about the library.

storage_group This tag determines the storage group associated with all files in this directory, and shows up as your experiment’s top level directory under /pnfs. Typically, one storage group is associated with an entire experiment. A storage group is assigned to each experiment by the Enstore administrators. Users never change this tag.

4.3.2 How to View Tags

Off-site users cannot mount pnfs, and therefore cannot see tags. On-site users: to see the values of the tags for a given directory, first setup encp (with qualifier, see section 6.1 Setup encp) then cd to the /pnfs subdirectory of interest (or enter the directory as an argument to --tags) and enter the command:

% enstore pnfs --tags

.(tag)(file_family) = dcache

.(tag)(file_family_width) = 1

.(tag)(file_family_wrapper) = cpio_odc

.(tag)(library) = eagle

.(tag)(storage_group) = test

-rw-rw-r-- 11 xyz sys 6 Jul 26 10:22 .(tag)(file_family)

-rw-rw-r-- 11 xyz sys 1 May 5 2000 .(tag)(file_family_width)

-rw-rw-r-- 11 xyz sys 8 May 5 2000 .(tag)(file_family_wrapper)

-rw-rw-r-- 11 xyz sys 5 May 5 2000 .(tag)(library)

-rw-r--r-- 11 xyz sys 4 Jul 26 10:20 .(tag)(storage_group)

The output first lists the tags and their values, then the tags again in long format to show the owners and protection modes.

1. Files and directories must not be moved across PNFS database areas. Doing so will destroy Enstore metadata.
2. For ln, hard links must be used to ensure that all the metadata information is linked; symbolic links do not work properly .
3. Stat is not available in all operating systems.