Support

Maturity and Completeness

• What is the customer base and what is their experience and opinion? For commercial software or non-HEP freeware, one should get a list of customers and references.

• How long has the product been in existence? What version is the product at? How many major releases have there been? How often is there a minor release? Several major releases or regular minor releases with integrated bug fixes are good signs of a well supported mature product. Availability of published books on the product are also a sign of maturity as well an established customer base.

• How long will the product survive? Are there any competing products that are likely to win the market (including freeware). Who is the product developer and are they well supported financially (graduate student or full time staff).

Who supports users

• Who provides consulting support? Commercial, other Lab, CERN, Fermilab? Are they responsive? Newsgroups and dejanews may provide some information on support response (though these tend to be biased). This is rather subjective and should be treated as such.

• Who can get support? Particularly for commercial software, can any user of the product access the support services or are these limited to a pre-specified list of local contacts.

• Is the use of the product in the community enough that there is a pool of people/knowledge to draw from for support if needed? HEP use should be assessed; PAW knowledge in the HEP community is widespread and Root is growing. A dedicated newsgroup would be a plus.

• Is user training needed and available? What is the cost?

• Is training required and available for support staff? What is the cost (time and money)? For commercial products such as SAS and IDL, support and user training may be required to optimally use the product, and that cost should be folded in.

• How much (local) support will be required (is it complicated and hard to use)? This and the remaining questions in this section can be determined by talking to current users or scanning any newsgroups, mailists or FAQS.

• For commercial or freeware, what kind and quality of user level support is provided?

• Is the software completely and well documented at the user level?

• Is a system manager required in order to install and/or maintain the package? If so, this would be significantly complicate matters for some remote users who do not have ready access to (or a friendly relationship with) the system managers of their computer.

Licensing

• What types of licensing are available?

• What is the cost? For Universities? Lab?

Maintenance

Who provides it and how much:

• Who provides maintenance both local and external to the Lab? What are the fallbacks (if the maintainer(s) is run over by bus or the company folds)?

• Are the maintainers responsive and are bug fixes turned around in a reasonable amount of time?

• Does the software maintainer need additional training (beyond that needed by users). If so, is it available and at what cost.

• What is maintenance/licensing costs for commercial products?

• How much software is there (line count)? How much needs to be supported locally (how many people required)? Can/should support be split up into areas of expertise (e.g. motif/graphics, interpreter, etc.). This is mainly significant for non-commercial software that will be maintained locally.

• In the case of commercial software, is source code available (in escrow)? This would be required for finding bugs locally or in case the company folds. This may be an additional cost.

Maintenance Infrastructure

• What kind of build environment is provided. Is it robust? This is mostly relevant for non-commercial software that may need to be co-maintained.

• Can the package be built AT ALL on new or different sub-systems? Root still provides NO makefiles.

• Is the source repository accessible so that local support persons can select which changes to accept and which to reject for local use? Root still uses CMZ.

• Will the software have to be maintained and/or extended locally and externally? If so, can the software be maintained in a common repository. If separate repositories, what is the commitment to keep them from diverging from modifications, extensions and bug fixes. This excludes locally maintained extensions which use pre-defined APIs or hooks into the product, which we will have to maintain ourselves in any case.

• Is the software passed through quality assurance software such as Purify or Insure++ before being put into production?

• Are there any restrictions that would prevent the product from being placed in the run II infrastructure (i.e. UPS/UPD)? In particular, the ability to support more than one version of the product on the system.

• Are release notes and change lists provided with releases? For example, the commercial product IDL comes with "what's new" and release notes lists.

Maturity and Completeness

• Are there active mailing lists/FAQs/newsgroups for the product? How do they reflect on the product? Root has a support list, the commercial product IDL has FAQs and a newsgroup.

• Are recent releases extensions/enhancements and not bug fixes?

• Are product releases reasonably paced and useful?

Modularity:

• Will the tool/software need to be upgraded (additions/replacements) to satisfy Run II functional requirements, and how difficult will this be? Does the product provide API/hooks to easily interface locally written extensions? Would existing support be able to handle this or would manpower need to be added? For example adding a command line (e.g. python) to Histoscope is thought to be difficult. Anticipated additions and replacements should be identified.

• Is the product modular? Is it broken down logically and physically into reasonably distinct sub-systems? Can some of these sub-systems be replaced by external packages with the same functionality? For example, in Root can Linear Algebra or Minimization be done by packages developed specifically to solve these topics separately from Root? A mini code review should be performed on the package to determine what would be involved in replacing such an identified component or sub-system.

• If new functionality needs to be added, is the software sufficiently modular such that the code changes can be localized? For example, it is believed this is not the case for adding STL support to cint. A mini code review should be performed to assess whether extensive and/or destabilizing code changes would be required to add the functionality.

• Is the software sufficiently modular to be such that bug fixes are localized? A mini code review focused on a particular section or component of the software should provide information on this.

• If a component needs to be replaced in it's entirety, is the software sufficiently modular such that a new component can be slotted in with minimal disruption? For example, Root depends on functionality in cint other than the interpreter in a fundamental way.

• What distinct (external) packages or interfaces are required to build and/or run the package... Motif (shared libraries), OpenGL, etc. Are there external software components that are out of the maintainer's control? LHC++ depends on numerous commercial packages, Nirvana/Histoscope on motif. All such packages and interfaces should be identified.

Portability

• Platform availability: Linux, NT, Solaris, IRIX, HP, DEC-Unix ... ? If a specific Run II platform isn't supported, what would it take to get support for it should be determined.

• Porting of code to new platforms; this applies to non-commercial software that is currently only supported on selected systems. The issue to be raised is the ability of the original developers to accept changes to be incorporated into the base code so any porting done here is done once (aside from effects of future OS upgrades) and does not have to be re-done with each release of the software.

• How sensitive is the package to minor OS and/or compiler and/or system header variations? Root under Linux may be sensitive to which C-libs are in use, which distribution you are using, which kernel you are using, which system header patches you have applied (esp., cint), and so on. PAW is sensitive to OS upgrades. This can be determined by looking at support history in newsgroups, other support logs, or talking to the user community.

• 64 bit considerations: Does the software run on 64 bit platforms/OS (alpha/Unix, SGI/Unix, future, e.g. merced)? Will it be difficult to port to 64 bit systems (a la COMIS for PAW).

• Are there Endianship and other heterogeneous environment considerations?

• Is the software product build dependent on a specific compiler or is it compiler independent? If compiles are not needed for the product, are there any compiler dependencies present in the API used for locally written extensions? In particular, if the software needs to be built with Run II compilers, it should be verified if it can.

Standards:

• Are standards followed. Compiler standards? Library standards (e.g. STL, POSIX). Are they fully supported (e.g. cint and STL)?

• Are there any support and maintenance standards or procedures? For example, any control over what goes into releases?

• Are good coding practices (documentation) followed? Is there good developer's documentation (how easily can the product be "taken over"?)

• Are good "Computer Science" techniques and methods used, for example in a language interpreter (see below)?

• Is there a design methodology applied? Are any design tools used? (such as a code generator). If so, do we need to have and/or support these tools?

• Is the product likely to crash and if so, how does it recover? What would be the impact? Do system managers need to intervene?

• Any government regulation applied to the product? Export restrictions?

• Are there any Y2K issues?

Application specific:

• Will an interface/adapter need to be made to fit the tool in with the rest of the analysis tool framework (e.g. data import/export). Does the product provide well defined and documented API/hooks for such an extension?

• Does the product use a language interpreter. If so does it support the full language. Is it written following correct computer science techniques and algorithms? Is it sensitive against language changes, etc. (examples are COMIS and cint, which support only subsets of their language). Or, turning the question around, is the language computational complete? (Then forget about the language it was trying to emulate and treat it as completely new language).

• Is the data processing model correct? For example, in MATLAB, the model is to read in a file then process all its data - will this work with Run II sized files.

• Are there restrictions on the input data (say size or format)? For example, in Histoscope the size of ntuples is limited because it is stored and processed in memory.

• What is the minimal environment to run the product? How does performance/capability scale up while the environment scales up?