The Fermilab C++ Compiler Test Suite Framework

Marc W. Mengel mengel@fnal.gov

Abstract:

A test framework for C++ compilers has been designed to meet several goals:

• New tests can be easily added.
• Tests can be run automatically.
• Results are reported in HTML for easy publishing.

This framework is in early release, with tests from past C++ compiler testing efforts already put into the framework. This document describes the framework, and how to add tests to it, and is targeted at test developers.

\

Contents

 
• Description
  • Compiler Description Files
  • Test Framework directory layout
  • Test Engine
• Adding tests
  • New compiler flags
• Adding Compiler Definitions
• About this document ...

Description

The Fermilab C++ Compiler Test Suite Framework, which is available in the Fermilab OSS department cvs repository is a combination of:

• description files for various compilers/environments,
• a directory structure for organizing small programs, and
• a test engine which runs the all the tests on a list of compilers, and writes HTML reports of the results.

These parts will now be discussed in detail.

 

Compiler Description Files

Compiler description files are short Bourne shell (/bin/sh) script fragments which define shell variables to hide the differences between compilers and environments. Currently the list of variables is:


Of these, the hardest to discover is usually the definition for VERS.

These variables are used in the Engine script to run the tests, and in the test's Compileflags and Linkflags files.

Test Framework directory layout

Inside the test framework there are several directories:

The test is successful if the Good sources all compile and link, and the Bad sources all fail to compile.

Note that the Compileflags and Linkflags files should always use definitions fromthe Compiler descriptions, rather than actual compiler flags. For example:

$INCLUDE ./incfiles

rather than

-I ./incfiles

Test Engine

The test engine is a Bourne shell script, which takes as command line arguments one or more compiler names; where a compiler name is the name of a Compiler Description File, in the Compilers directory. It performs the following functions for each compiler:

• Reads the Compiler Description File for the compiler
• Runs all the tests in the test directories
• Generates a summary and full report for each compiler
• If possible, puts the reports directly in the web page area, and updates the index of available reports.

Currently, the reports can only be added automatically if the tests are run on a system with the AFS filesystem mounted, and if the user has write permission on the area in question.

Adding tests

This section assumes an understanding of cvs, since that is where the "official copy" of the test suite is kept.

Basically, if you have a test program which you want to add to the suite, you need to:

• Checkout the FCCTSF module from the repository

  $cvs -d cvsuser@oss-cvs:/usr/products/olsshare/cvs checkout FCCTSF
  cvs server: Updating FCCTSF
  U FCCTSF/Engine
  U FCCTSF/mkindex.sh
  ...

• Make a new test directory, and add it to the repository

  $ cd FCCTSF/Tests
  $ mkdir mytest
  Directory /.../FCCTSF/Tests/mytest added to the repository

• Copy your sources into this directory

  $ cp $HOME/breakcompiler.cc atest1.cc
  $ cp $HOME/breakcompiler.h  breakcompiler.h

• Create support files

  $ echo '${EXCEPTIONS} ${RTTI}' > Compileflags
  $ echo '${EXCEPTLIB}' > Linkflags
  $ echo 'It worked.' > ExpectedOutput

• Run just your test on some compiler to make sure it works:

  $ cd ../..
  $ rm -rf Output
  $ sh Engine -j Tests/mytest  gcc
  Test  output for SunOS gcc
  Test Tests/mytest
  $ more Output/*/*/*/*
  ...
  $ rm -rf Output

• And add the test to the CVS repository

  $ cd Tests
  $ cvs add mytest
  $ cd mytest
  $ cvs add Compileflags Linkflags ExpectedOutput atest1.cc breakcompiler.h
  $ cd ../..
  $ cvs commit -m "added mytest"

New compiler flags

If your test needs compiler flags that aren't defined in the compiler definitions, you should add a new variable to *all* the compiler definitions, and to this document, in section 1.1. If you don't know the option for a particular compiler, set the option to the empty string.

Adding Compiler Definitions

This section assumes an understanding of cvs, since that is where the "official copy" of the test suite is kept.

Adding a compiler definition consists of:

• Copying an existing compiler definition

  $ cd Compilers
  $ cp CC newCC

• Editing the definitions as appropriate (see section 1.1 for details).
• Testing the definitions. First you probably want to run one or two specific tests locally for testing,

  $ cd ..
  $ rm -rf Output
  $ sh Engine -j "Tests/test1 Tests/test2"  newCC
  Test  output for SunOS newCC
  Test Tests/test1
  Test Tests/test2
  $ more Output/*/*/*/*
  ...
  $ rm -rf Output

  then run the whole suite.

  $ sh Engine newCC
  Test  output for SunOS newCC
  Test Tests/test1
  ...

About this document ...

The Fermilab C++ Compiler Test Suite Framework

This document was generated using the LaTeX2HTML translator Version 96.1-h (September 30, 1996) Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.

The command line arguments were:
latex2html c++suite.tex.

The translation was initiated by Marc Mengel on Tue Jun 16 11:28:44 CDT 1998