RunII Physics Analysis Software
Support And Maintenance Requirements
Table of contents
Introduction
The main charge of the Physics Analysis Software Support and
Maintenance Requirements Group is to specify a set of questions
to be asked of any candidate software products and tools that
would uncover any support and maintenance issues and bring them
up-front. As part of the process of formulating these questions,
we looked at a number of current candidates, which helped
identify issues. The month spent looking at the candidate
products was not sufficient for a fair comparison, and that was
not the charge. If more issues are documented for a given
candidate, it likely reflects more familiarity with the software
product rather than more issues or defects.
We have formulated the requirements in terms of questions to
ask of candidate software and tried to give some indication of
how to answer the question and provide examples from our
discussions of the candidate products where appropriate. This is
presented as a bulleted list below. The requirements section is
followed by a summary of the support and maintenance issues
uncovered for particular products we looked at. ( Minutes and transparencies from
discussions are available online)We also had a couple of general
issues and concerns which are discussed next.
The issue of how support and maintenance would be coordinated
between Fermilab and CERN for products like Root came up in our
discussions. Even though Root is geared towards High Energy
Physics Analysis so likely will satisfy a good deal of Run II
functional requirements, it is inevitable that there will need to
be development by a local team. Can the development be shared? At
the moment this might be difficult as Root uses CMZ and Fermilab
CVS. If there is independent development, how are the branches
merged? It was felt that we should avoid divergence as in the
case of 4-vectors in CLHEP.
There was general concern that commercial and freeware
packages would not be able to fully satisfy the needs of
physicists in the subset of functionality covered by the package.
For this reason, it was felt that in addition to meeting at least
a subset of the functional requirements, the candidate product
must be looked at carefully in terms of its extensibility in
areas where functionality may need to be added or augmented. As
an example, an attempt to read in and write out ntuple data and
process it in packages like IDL, MATLAB, or OCTAVE could be made
to determine the flexibility of its i/o.
back to top
Support
Maturity and Completeness
- What is the customer base and what is their
experience and opinion? For commercial software
or non-HEP freeware, one should get a list of customers
and references.
- How long has the product been in existence? What
version is the product at? How many major releases have
there been? How often is there a minor release? Several
major releases or regular minor releases with integrated
bug fixes are good signs of a well supported mature
product. Availability of published books on the product
are also a sign of maturity as well an established
customer base.
- How long will the product survive? Are
there any competing products that are likely to win the
market (including freeware)? Who is the product developer
and are they well supported financially (graduate student
or full time staff).
back to top
Who supports users
- Who provides consulting support? Commercial,
other Lab, CERN, Fermilab? Are they responsive?
The user support for Root is reported to be responsive,
can the same be said for commercial? Newsgroups and
dejanews may provide some information on support response
(though these tend to be biased). This is rather
subjective and should be treated as such.
- Who can get support? Particularly for
commercial software, can any user of the product access
the support services or are these limited to a
pre-specified list of local contacts?
- Is the use of the product in the community enough
that there is a pool of people/knowledge to draw from for
support if needed? HEP use should be assessed;
PAW knowledge in the HEP community is widespread and Root
is growing. A dedicated newsgroup would be a plus.
- Is user training needed and available? What is
the cost? Optimally, a product would be familiar
enough or well enough documented that formal training would
be unneccessary. However, in the case of a brand new product,
the availability of low-cost formal training would be a plus.
- Is training required and available for support
staff? What is the cost (time and money)? For
commercial products such as SAS and IDL, support and user
training may be required to optimally use the product,
and that cost should be folded in.
- How much (local) support will be required (is it
complicated and hard to use)? This and the
remaining questions in this section can be determined by
talking to current users or scanning any newsgroups,
mailists or FAQS.
- For commercial or freeware, what kind and quality
of user level support is provided?
- Is the software completely and well documented at
the user level?
- Is a system manager is required in order to
install and/or maintain the package? If so, this
would be significantly complicate matters for some remote
users who do not have ready access to (or a friendly
relationship with) the system managers of their computer.
back to top
Licensing
- What types of licensing are available?Floating
or site licenses are easier to make available than are node-locked
licenses. In particular, site licenses which can be made
available to remote users are very useful and normally cost-effective
- What is the cost? For Universities? Lab?
back to top
Maintenance
Who provides it and how much:
- Who provides maintenance both local and external
to the Lab? What are the fallbacks (if the
maintainer(s) is run over by bus or the company folds)?
- Are the maintainers responsive and are bug fixes
turned around in a reasonable amount of time?A system
whereby we can prioritize bug reports may be useful.
- Does the software maintainer need additional
training (beyond that needed by users)? If so, is it
available and at what cost?
- What are maintenance/licensing costs for
commercial products?
- How much software is there (line count)? How much
needs to be supported locally (how many people required)?
Can/should support be split up into areas of expertise
(e.g. motif/graphics, interpreter, etc.). This
is mainly significant for non-commercial software that
will be maintained locally.
- In the case of commercial software, is source
code available (in escrow)? This would be
required for finding bugs locally or in case the company
folds. This may be an additional cost.
back to top
Maintenance Infrastructure
- What kind of build environment is provided. Is it
robust? This is mostly relevant for
non-commercial software that may need to be
co-maintained. For example, Makefiles or Imakefiles which work
are very useful for Unix applications, as are InstallShield or
NMAKE files for Win32.
- Can the package be built AT ALL on new or
different sub-systems? Root still provides NO
makefiles.
- Is the source repository accessible so that local
support persons can select which changes to accept and
which to reject for local use? Products maintained
by CVS would be easiest for us to deal with. Root still uses
CMZ.
- Will the software have to be maintained and/or
extended locally and externally? If so, can the software
be maintained in a common repository. If separate
repositories, what is the commitment to keep them from
diverging from modifications, extensions and bug fixes. This
excludes locally maintained extensions which use
pre-defined APIs or hooks into the product, which we will
have to maintain ourselves in any case.
- Is the software passed through quality assurance
software such as Purify or Insure++ before being put into
production?
- Are there any restrictions that would prevent the
product from being placed in the run II infrastructure
(i.e. UPS/UPD)? In particular, the ability to
support more than one version of the product on the
system.
- Are release notes and change lists provided with
releases? For example, the commercial product
IDL comes with "what's new" and release notes
lists.
back to top
Maturity and Completeness
- Are there active mailing lists/FAQs/newsgroups
for the product? How do they reflect on the
product? Root has a support list, the commercial product
IDL has FAQs and a newsgroup.
- Are recent releases extensions/enhancements and
not bug fixes?
- Are product releases reasonably paced and useful?
back to top
Modularity:
- If the tool/software need to be upgraded
(additions/replacements) to satisfy Run II functional
requirements, and how difficult will this be? Does
the product provide API/hooks to easily interface locally
written extensions? Would existing support be
able to handle this or would manpower need to be added?
For example adding a command line (e.g. python) to
Histoscope is thought to be difficult. Anticipated
additions and replacements should be identified.
- Is the product modular? Is it broken down
logically and physically into reasonably distinct
sub-systems? Can some of these sub-systems be replaced by
external packages with the same functionality?
For example, in Root can Linear Algebra or Minimization
be done by packages developed specifically to solve these
topics separately from Root? A mini code review should be
performed on the package to determine what would be
involved in replacing such an identified component or
sub-system.
- If new functionality needs to be added, is the
software sufficiently modular such that the code changes
can be localized? For example, it is believed
this is not the case for adding STL support to cint. A
mini code review should be performed to assess whether
extensive and/or destabilizing code changes would be
required to add the functionality.
- Is the software sufficiently modular to be such
that bug fixes are localized? A mini code
review focused on a particular section or component of
the software should provide information on this.
- If a component needs to be replaced in it's
entirety, is the software sufficiently modular such that
a new component can be slotted in with minimal
disruption? For example, Root depends on
functionality in cint other than the interpreter in a
fundamental way.
- What distinct (external) packages or interfaces
are required to build and/or run the package... Motif
(shared libraries), OpenGL, etc. Are there external
software components that are out of the maintainer's
control? LHC++ depends on numerous commercial
packages, Nirvana/Histoscope on motif. All such packages
and interfaces should be identified.
back to top
Portability
- Platform availability: Linux, NT, Solaris, IRIX,
HP, DEC-Unix ... ? If a specific Run II platform
isn't supported, what would it take to get support for it
should be determined.
- Platform priority: In the case of commercial
software, are our platforms a priority with the developers?
In other words, if there is a major upgrade to one of our
supported OS's, which requires a fix to the application
software, will such a fix be provided and released in a
timely fashion?
- Porting of code to new platforms; this applies to
non-commercial software that is currently only supported
on selected systems. The issue to be raised is
the ability of the original developers to accept changes
to be incorporated into the base code so any porting done
here is done once (aside from effects of future OS
upgrades) and does not have to be re-done with each
release of the software.
- How sensitive is the package to minor OS and/or
compiler and/or system header variations? Root
under Linux may be sensitive to which C-libs are in use,
which distribution you are using, which kernel you are
using, which system header patches you have applied
(esp., cint), and so on. PAW is sensitive to OS upgrades.
This can be determined by looking at support history in
newsgroups, other support logs, or talking to the user
community.
- 64 bit considerations: Does the software run on
64 bit platforms/OS (alpha/Unix, SGI/Unix, future, e.g.
merced)? Will it be difficult to port to 64 bit
systems (a la COMIS for PAW).
- Are there Endianship and other heterogeneous
environment considerations?
- Is the software product build dependent on a
specific compiler or is it compiler independent? If
compiles are not needed for the product, are there any
compiler dependencies present in the API used for locally
written extensions? In particular, if the
software needs to be built with Run II compilers, it
should be verified if it can.
back to top
Standards:
- Are standards followed. Compiler standards?
Library standards (e.g. STL, POSIX). Are they fully
supported (e.g. cint and STL)?
- Are there any support and maintenance standards
or procedures? For example, any control over what goes
into releases?
- Are good coding practices (documentation)
followed? Is there good developer's documentation (how
easily can the product be "taken over"?)
- Are good "Computer Science" techniques
and methods used, for example in a language interpreter
(see below)?
- Is there a design methodology applied? Are any
design tools used? (such as a code generator). If
so, do we need to have and/or support these tools?
back to top
Reliability and Security:
- Is there any security maintenance concerns with
the product?
- Is the product likely to crash and if so, how
does it recover? What would be the impact? Do
system managers need to intervene?
- Any government regulation applied to the product?
Export restrictions?
- Are there any Y2K issues?
back to top
Application specific:
- Will an interface/adapter need to be made to fit
the tool in with the rest of the analysis tool framework
(e.g. data import/export). Does the product
provide well defined and documented API/hooks for such an
extension?
- Does the product use a language interpreter. If
so does it support the full language. Is it written
following correct computer science techniques and
algorithms? Is it sensitive against language changes,
etc. (examples are COMIS and cint, which support
only subsets of their language). Or, turning the question
around, is the language computational complete? (Then
forget about the language it was trying to emulate and
treat it as completely new language).
- Is the data processing model correct?
For example, in MATLAB, the model is to read in a file
then process all its data - will this work with Run II
sized files.
- Are there restrictions on the input data (say
size or format)? For example, in Histoscope the
size of ntuples is limited because it is stored and
processed in memory.
- What is the minimal environment to run the
product? How does performance/capability scale
up while the environment scales up?
back to top