NAME
nnmaster - nn database manager
SYNOPSIS
nnmaster -I [lmit]
nnmaster -w
nnmaster -v
nnmaster -l [ "lock message" ]
nnmaster [ options ] [ groups ]
nnmaster -F [ options ] [ groups ]
DESCRIPTION
nnmaster is the daemon which is responsible for building and
maintaining the database used by the nn(1) news reader.
Normally, nnmaster is started when the system enters multi-
user mode, and runs until system shutdown. To facilitate
this, you should place the following call in /etc/rc (or
similar) to invoke the nnmaster daemon:
$master/nnmaster -l -r -C
where $master is the MASTER_DIRECTORY defined during confi-
guration of nn.
When nnmaster is started as specified above, it will first
unlock the database in case it was locked (-l), perform a
thorough consistency check on the database (-C).
Then, every 10 minutes (-r), it will look at the time-stamp
of the news active file to see whether new articles have
arrived on the system (or whether articles have been
expired). (See -U)
If the active file has been modified, nnmaster will collect
the header information from the new articles and enter them
into the database (or remove the headers of the expired
articles from the database).
If it detects that some articles have been expired, it will
automatically remove the header information of the expired
articles from the database.
ARTICLE COLLECTION OPTIONS
Normally, nnmaster will collect all available news groups
defined in the news active file. The set of collected
groups can be controlled via the argument line. Groups can
be either included or excluded:
A group name, e.g. comp, will cause the group and all its
subgroups to be collected. Individual groups, e.g.
news.software.nn, can also be specified
A group name preceded by an exclamation mark, e.g.
!talk.politics, will cause the group and all its subgroups
to be ignored.
An empty argument, i.e. "", will cause all groups that are
not ignored to be collected. For example, to collect every-
thing but rec and misc, use the following command:
nnmaster -r !rec !misc ""
If the empty argument had been omitted, nothing would be
collected, since the presence of any groups arguments causes
nnmaster to ignore all groups which are not explicitly men-
tioned.
Example 1: The following commands can be executed by cron to
collect different sets of groups at different intervals or
under different conditions:
nnmaster -B -O14 rec misc sci -LBO -u
nnmaster !rec !misc !sci "" -u
Example 2: The group arguments are used in the given
sequence, e.g. to leave out comp.sys, but still collect
comp.sys.ibm.pc, use the command:
nnmaster -r comp.sys.ibm.pc !comp.sys ""
The use of the -u option in the first example is essential,
since each of the commands will update the active file time
stamp which will prevent the other command from detecting
new articles that have arrived.
Using this method to keep specific groups out of the data-
base must be used with great caution; just a single invoca-
tion of nnmaster without any arguments will collect all the
otherwise ignored groups!
COLLECTION OF ARTICLES
The following options control how nnmaster performs the col-
lection of new articles.
-r [ min ]
Daemon mode. The nnmaster will put itself in the back-
ground (unless -f is also specified), and will checks
for arrival of new articles and expired articles every
min minutes (and update the database accordingly). If
min is omitted, the default is to check every 10
minutes.
Without the -r option, the nnmaster will just perform a
single collection of new articles (if any) and then
exit. This can be used to have the nnmaster started by
cron(8) at regular intervals instead of having it as a
daemon which sleeps between checking for new articles.
Since the nnmaster is a bit expensive to start up (it
has to read a few files), it is up to you to decide
which mode is best on your system. (I have also heard
that it works to call nnmaster without -r from
inews(1). I cannot recommend this unless you receive
batched news; invoking nnmaster for every received
article sounds too expensive to me.)
-h [ sec ]
Hold collection of new articles until next scan if new
articles have arrived withing the last sec [60]
seconds. This is useful to prevent nnmaster from com-
peting for disk i/o with an rnews process which is
unbatching incoming news, or a running expire process.
It will have the side effect of limiting the number of
C: entries in the log, since collection of incoming
batches will typically be done in larger thunks.
-f Run nnmaster in foreground in daemon mode (see -r).
Useful if nnmaster is invoked from inittab. (Notice
that if you use a respawn entry in inittab, you will
not be able to stop nnmaster using the -k option, since
init will immediately start another master.)
-C Perform a consistency check on the database on start-
up, and rebuild corrupted database files. This opera-
tion can be quite time-consuming since it reads through
all the database files.
-b Normally, articles without a proper news header (no
Newsgroups: line) are ignored. Specifying the -b
option causes these `bad' articles to be included in
the database (normally with no sender or subject).
-B Remove `bad' articles. Sometimes, articles without a
header ends up in the news spool directory. These
articles have no article id, and therefore, they will
never be expired by expire(8). This option will allow
the nnmaster to silently remove these articles (a `B'
entry will occur in the log file).
-O days
Ignore articles which are older than the given number
of days. This may help keep old 'stray' articles out
of the database. If the -B options is also specified,
the old articles will be removed from the news spool
directories. Old ignored or removed articles will be
reported with an `O' entry in the log file. This
option can be disable for individual groups by the O
flag in the GROUPS file (see below).
-R N Specifies how the auto-recollect operation is performed
on the groups having this option set in the GROUPS file
(see below). Four methods are available (default is
method 1):
1: Run expire on the group when new articles arrive.
2: Run expire on the group on every scan.
3: Recollect all articles when new articles arrive.
4: Recollect all articles on every scan.
-M mode
Normally, nnmaster will send a message via mail to the
news administrator (OWNER) when an error or potential
problems (primarily nntp related) occur. This can be
restricted to only fatal errors (nnmaster terminated)
if mode is 1, and disabled completely if mode is 0.
-Q Normally, nnmaster will print a message on the system
console or in the syslog if a fatal error happens.
This option will prevent this, so only a type 'E' entry
is written to the Log file.
DATABASE EXPIRATION
Since articles does not stay forever in the news system, the
database must be cleaned up regularly to remove the informa-
tion stored for expired articles. Expiration of the data-
base is normally scheduled using the nnadmin(1M) command
executed by cron at a suitable time when expire on the news
articles has completed. The following command will send a
message to the nnmaster and cause it to initiate expire on
all news groups:
nnadmin =EYW
Selective expiration of individual groups can be done from
nnadmin (interactive mode). It can also be done by invoking
nnmaster with the -F option. For example, the following
command will run expire on all groups except the `rec'
groups:
nnmaster -F -k !rec ""
The -k option is required to terminate the currently running
master since two daemons cannot be running at the same time.
Thus to run expire (on all groups) in this way from cron,
the following commands must be used: nnmaster -Fk "" ;
nnmaster -r ...
It is also possible to have nnmaster detect expiration
automatically (see -e). This requires that the min field in
the active file is updated by the news expire (this is not
the default case when Cnews is used). However, this is not
always a safe indication since the first article may not
have been expired, while a lot of other articles have been
deleted.
There are several strategies available in the nnmaster to do
this clean-up, each having their strengths and weaknesses.
Method 1 (default): Rebuilds the database from the existing
database information by comparing the current database con-
tents with the contents of the news group directories, elim-
inating entries whose file no longer exists. This method is
guaranteed to eliminate all expired articles from the data-
base, and it is reasonably fast because it only has to read
the directories, not each article file.
If news is accessed remotely via nntp, the list of exist-
ing articles cannot efficiently be fetched by reading a
local directory. Instead an XHDR request is sent to the
nntp server to get a list of articles.
Method 2: Eliminates only the expired articles before the
first article in the group. This is very fast since only
the active file and the database itself is accessed, but it
will obviously leave some expired articles in the database.
This method requires that the min field in the active file
is updated by expire.
Method 3: Discard the current database information and
recollects all articles. This is obviously very time con-
suming, and it is therefore not recommended, especially not
with nntp.
The options related to database expiration are:
-E N Select expire method N. (If N is omitted, the default
method is used).
-e [N]
Automatically run expire in the database on groups
where the min number in the active file has increased
by N (1 if omitted) articles. This is disabled by
default (since the min field is often unreliable).
-F Run expire once and exit. If a list of groups is
specified on the command line, the matched groups (see
the rules above) will be marked for expiration. If no
groups are specified, only the groups already scheduled
for expire will be expired. Consequently, to expire
all groups, a blank argument "" (matching all groups)
must be specified.
DATABASE LOCKING
The database can be locked centrally, which will normally
disallow all access to the database, and even block nnmaster
from being (accidentally) started. When a lock is set on
the database, all currently running clients will terminate
the next time they try to access the database. Setting a
lock on the database can thus also be used to force running
clients to terminate.
The following options set and clear locks on the database:
-l message
Locks the database with the given message. The message
will be displayed every time a program tries to access
the database.
-l Unlock the database if it was locked.
-i Ignore a possible lock and continue. This can be used
to have nnmaster operate on a database which is blocked
from normal user access.
Since only one nnmaster can operate on the database at any
one time, a running nnmaster daemon must be stopped before a
lock can be set on the database. If neither -f nor -r is
specified with the -l option (in both forms), nnmaster will
terminate after setting or clearing the lock.
DATABASE INITIALIZATION
The following options are used to initialize and update the
central database files:
-I [limit]
Initialize database. This option will erase an exist-
ing database, and create an empty database containing
entries for the currently known groups. nnmaster will
offer you to use an existing GROUPS file when initial-
izing the database.
The optional limit can be used to put a limit on the
number of articles that will be collected in each group
during the first collection of articles following the
database initialization. This is useful on systems
where the 'min' field in the active file is unreliable
or not maintained (Cnews doesn't) to limit the work
done to do the initial collection of news after the
initialization of the database. If news is accessed
remotely from an NNTP server, this is even more impor-
tant! If limit is omitted, or is zero, nnmaster will
trust the min field and collect all articles in the
range min..last.
-G Reread the GROUPS file. This option is used to
instruct nnmaster to parse the GROUPS file after it has
been edited. See the section on the GROUPS file below.
MISCELLANEOUS OPTIONS
The following options controls various details of the
nnmaster's behaviour:
-D [ N ]
equals 1 or 3, this will produce a compact but still
very informative trace of the collection or expire pro-
cess directly on the terminal. This is most useful
when doing the first collection of articles after ini-
tializing the database with -I. If N is 2 or 3, a
trace of the NNTP traffic is written to a file
nnmaster.log in the TMP directory. This option disables
-r.
-H Identifies the host which nnmaster is running on as the
nntp-server for its clients, i.e. that it can access
the news spool directory locally without using NNTP.
Normally, nnmaster will detect this by itself by com-
paring the host name to the contents of the nntp_server
file, so this option should really be superfluous.
-y retries
In some networked environment, opening an article
(shared from another machine via NFS) may fail for no
obvious reason. Using this option, it is possible to
cause nnmaster to perform retries attempts to open an
article before marking the article as non-existing in
the database.
-L types
Exclude the specified entry types from the log file.
This is normally used to exclude the 'C'ollecting and
e'X'pire entries (-LCXO).
-t Trace the collection process. This will place a lot of
information into the log file (T: entries).
-u Normally, nnmaster will just compare the time-stamp on
the active file with a time-stamp saved in the database
to see if new articles have arrived. The -u option
forces the nnmaster to read the active file on start-up
to see if new articles have arrived.
-U Some SVR4 systems (and maybe SunOS) have a useful
"feature". Writing files with mmap() may not update
the last-changed timestamp on the file. Since INN uses
mmap() for writing the active file, this becomes a
problem for nnmaster. The -U option causes nnmaster to
unconditionally read the active file each time the
repeat delay (-r) time expires.
-v Print the release and version identification for nnmas-
ter, and exit.
-w Wakeup the real nnmaster. Send a signal to the nnmas-
ter daemon to have it check for new articles immedi-
ately.
-k Kill the already running nnmaster daemon before
proceeding with the operation specified by the other
options (or terminate if no other operation is
requested).
THE GROUPS FILE
The primary purpose of the GROUPS file is to store the names
of the news groups represented in the database. Each line
in the file corresponds to an entry in the (binary) MASTER
file, and the sequence of the lines in the GROUPS file must
never be changed unless the database is reinitialized after-
wards.
However, the contents of the lines in the GROUPS file can be
edited to control how the nnmaster should handle each indi-
vidual group.
The format of each line is:
news.group.name [ timestamp ] [ flags ]
The news group name is the name of the group, and must not
be changed in any way. If the group is no longer in the
news active file, and consequently the group does no longer
exist, group name can be replaced by a `@' character which
will instruct nnmaster to ignore this entry without having
to rebuild the entire database.
The optional time stamp indicates when the line was added to
the GROUPS file and is used by nn to detect new groups.
When the GROUPS file is built initially from the active
file, the time stamps are omitted which simply means that
they are "old".
One or more of the following flags can be added to the
GROUPS line to control nnmaster's handling of the group:
D Causes nnmaster to treat all articles in the group as
digests, even when they don't initially look like dig-
ests. Articles which are found not to be digests after
all, are still not digested.
N Instructs nnmaster to never digest any articles in the
group.
O Disables the -O option for this group, i.e. all exist-
ing articles will be included in the database (and they
will not be removed if the -B option is specified).
This flag should be set on groups which you never
expire, or have a very long expire time!
R Causes nnmaster to recollect all available articles in
said to be useful is some high-traffic clarinet groups
with many cancelled articles.
>file
Instructs nnmaster to append all new articles to the
specified file. This makes it possible to keep
specific groups out of the way of expire. The archive
file can be access directly from the nn client using
the goto-group command. The file name must be a full
path name to a file in an existing, writeable direc-
tory.
@ Instructs nnmaster to completely ignore this group -
this is equivalent to setting the group name to `@' as
described above.
! or X
Causes nnmaster to ignore the group and not collect the
group's articles in the database.
Comments (starting with `#' and continuing to the end of
line) and empty lines are allow in the GROUPS file, but it
is strongly recommended to keep the changes to the GROUPS
file as small as possible, because of the implicit
correspondence with the binary MASTER file.
It is not recommended to edit the GROUPS file while nnmaster
is running because it may add new groups to the file. After
editing the GROUPS file, the command
nnmaster -G
must be run before restarting the nnmaster to parse and ver-
ify the new GROUPS file.
NNTP SUPPORT
The nnmaster can access the news articles from a local news
spool directory as well as from an NNTP server. When com-
piled with NNTP enabled, nnmaster will compare the name of
the NNTP server and the name of the local host; if they are
identical, nnmaster will bypass NNTP and access the articles
directly.
When it has to access the news articles via NNTP, it cannot
time-stamp the active file, so instead it transfers the
entire active file from the NNTP server and compares it with
a local copy of the last active file fetched from the NNTP
server. This is not very expensive in terms of cpu-cycles,
disk-load, or net-bandwidth, but to stay on friendly terms
with the NNTP server administrator, you should probably not
use shorter update intervals than the standard 10 minutes.
Setting a much higher update interval than the standard 10
minutes is not really recommended either, since an update
normally implies fetching a burst of news articles from the
NNTP server, so setting the interval too long may imply that
the load on the NNTP server will be more un-even.
In expire method 1, the use of XHDR just to get a list of
existing articles in a group is definitely a waste of
resources on the nntp server (but still lower than using
method 3). Before using the XHDR request, nnmaster will
send a non-standard "LISTGROUP" request; if the nntp server
supports this request, it should return an OK_HEAD status
followed by an (unordered) list of article numbers (one per
line) terminated by a `.' line. The nntp servers supporting
this request will be much less loaded during expire.
The -O option does not work with NNTP. The -B option will
only work with NNTP if the nnmaster is running on the NNTP
server.
FILES
The $db, $master, and $news names used below are synonyms
for the DB_DIRECTORY, MASTER_DIRECTORY, and
NEWS_LIB_DIRECTORY defined during configuration.
$db/MASTER Database master index
$db/GROUPS News group names and flags in MASTER
file order
$db/DATA/nnn.[dx] Database files for group number nnn
.../.nn[dx] Database files if located in the group
directories
$master/GATE Message channel from nnadmin to nnmaster
$master/MPID The process id of the nnmaster daemon.
$Log The log file (the location is configura-
tion dependent)
$news/active Existing articles and groups
/usr/lib/nntp_serverContains the name of the NNTP server.
The MASTER file contains a record for each news group,
occurring in the same sequence as the group names in the
GROUPS file. The sequence also defines the group numbers
used to identify the files in the database's DATA directory.
The GATE file will be created by nnadmin when needed, and
removed by nnmaster when it has read it. Therefore, to send
a message to the nnmaster requires that you are allowed to
write in the $master directory.
The contents of the Log file are described in the nnadmin
manual.
SEE ALSO
nn(1), nncheck(1), nngrep(1), nntidy(1)
nnadmin(1M), nnspew(8), nnusage(1M)
AUTHOR
Kim F. Storm, Texas Instruments A/S, Denmark
E-mail: storm@texas.dk