[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Search for runs with suspect RSTDC data using pass2 ntuples



Hi Peter,

On Fri, 11 Jan 2002, Peter Cooper wrote:
> this file is bnlku9:~pcooper/pass2/suspect_rstdc.runs

 It does seem to me that you are probably flagging our runs with bad RS 
TDC data; however, there are a number of runs that I expected to be in 
your list that are not. Do we know why? (is there any data from these 
runs in the ntuple?)

We previously flagged:  

1) 45609,13-14,16-7 whereas you only flagged 45614 [from QOD]
(see From kettell@bnl.gov Sat Jan 12 15:43:04 2002)

2) 45606-7,9,13-6,45648,50 whereas you only flagged 45614 [from Renee]
(see the attachment from Renee)

3) 45607,9,12,14,45648,50 (see the attachment from Erik)

4) 45434-8, 45118, 45189, 45354, 45431; you did not get 45431, 45189, or
45431 (see attachment from me)


  Somehow you missed most of thee runs where we first found the problem...

	Steve

your list
>     run      events     fraction
>   number    all npvrd<15
>   44364.     40.      2.   0.050
>   44416.     22.      2.   0.091
>   44420.     12.      1.   0.083
>   44446.     13.      1.   0.077
>   44454.    660.     34.   0.052
>   44470.    161.      9.   0.056
>   44472.    274.     14.   0.051
>   44578.    336.     18.   0.054
>   44582.   1284.     79.   0.062
>   44614.    288.    278.   0.965
>   44720.   1616.    331.   0.205
>   44942.    494.     55.   0.111
>   44944.    377.     41.   0.109
>   45116.     99.     18.   0.182
>   45116.     19.      3.   0.158
>   45118.    517.     87.   0.168
>   45120.   1308.    233.   0.178
>   45122.    205.     27.   0.132
>   45354.   2988.   1168.   0.391
>   45356.   3063.   1224.   0.400
>   45356.   2291.    942.   0.411
>   45358.   1295.    556.   0.429
>   45360.   6157.   2376.   0.386
>   45362.   2754.   1135.   0.412
>   45380.   5360.    430.   0.080
>   45382.   1629.    191.   0.117
>   45386.   2576.    305.   0.118
>   45386.   2011.    232.   0.115
>   45388.   1372.    147.   0.107
>   45392.     25.      3.   0.120
>   45392.   1108.    128.   0.116
>   45428.    461.    122.   0.265
>   45430.   1491.    324.   0.217
>   45432.   2617.    620.   0.237
>   45432.    820.    189.   0.230
>   45434.    152.     40.   0.263
>   45434.    598.    167.   0.279
>   45436.   2535.    623.   0.246
>   45436.   1051.    246.   0.234
>   45438.     10.      4.   0.400
>   45438.    803.    503.   0.626
>   45450.   1824.    442.   0.242
>   45452.    321.    193.   0.601
>   45454.   2699.   1815.   0.672
>   45456.   2069.   1450.   0.701
>   45536.   2779.    261.   0.094
>   45614.   1764.    304.   0.172
>   46016.     49.     49.   1.000

From kettell@bnl.gov Fri Oct 26 13:39:18 2001
Date: Fri, 26 Oct 2001 13:38:38 -0400
From: Steve Kettell <kettell@bnl.gov>
To: Renee Poutissou <renee@triumf.ca>,
     George Redlinger <george.redlinger@bnl.gov>
Cc: Sasha Kushnirenko <kushnir@fnal.gov>, Bob Tschirhart <tsch@fnal.gov>,
     Peter Cooper <pcooper@fnal.gov>, Erik Ramberg <ramberg@fnal.gov>,
     Steve Kettell <kettell@bnl.gov>
Subject: Re: RS TDC problems

Hi Renee and George,

  I thought I would follow up on Sasha's suggestion and see if there was a
DC2 error before each of our periods of bad RS TDC data. I have been
stymied though trying to understand the time stamps in the
~bnlonl/online_logfiles/ppclog/e787ppc02.log log file. It appears we get a
time stamp only when runs start, is this true? We do not get time stamps
when errors occur?

  Anyway there appear to be 2 errors in the RS TDC system:

1) Lost of synch ,  data in DC2#1, NO data in DC2 #2
2) Error:Ev 4964; DC2 header word is wrong: 0xec100000 ;expecting fffcxxxx


OK, after "grep" consultation with Jingliang, I have generated a list of
DC2 errors (attached). I will include the relevant lines from the last few
days here as well (I have added the CAMAC load times and bad runs):

	LRS3377 files  loaded on Thu Oct 18 08:33:18 EDT 2001 
ddLogTask-e787ppc02: Thu-Oct-18-09:50:44-2001Echo
ddLogTask-e787ppc02: Lost of synch ,  data in DC2#1, NO data in DC2 #2
        LRS3377 files  loaded on Thu Oct 18 09:53:43 EDT 2001 
ddLogTask-e787ppc02: Thu-Oct-18-09:58:55-2001Echo
        LRS3377 files  loaded on Thu Oct 18 11:16:03 EDT 2001 
01      11:59-12:10  run45118   smx_v12j        all_v2_432.cfg  10/18/01
**** Bad RS TDC in QOD *****
        LRS3377 files  loaded on Thu Oct 18 13:08:19 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 13:21:40 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 15:16:41 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 15:35:04 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 15:37:45 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 15:49:07 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 15:50:49 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 15:56:44 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 15:57:32 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 16:01:56 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 16:36:52 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 16:39:40 EDT 2001 
        LRS3377 files  loaded on Thu Oct 18 19:48:11 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 08:53:18 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 09:52:59 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 09:55:23 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 16:20:19 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 16:23:44 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 16:40:11 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 16:44:23 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 16:51:53 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 17:32:09 EDT 2001 
01      18:12-18:27  run45189   smx_v12j        all_v2_432.cfg  10/19/01
**** Bad RS TDC in QOD *****
        LRS3377 files  loaded on Fri Oct 19 18:31:02 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 18:35:31 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 18:36:58 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 19:25:49 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 19:29:10 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 19:49:27 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 21:29:59 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 21:37:03 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 21:46:16 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 22:15:22 EDT 2001 
        LRS3377 files  loaded on Fri Oct 19 23:53:51 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 00:00:41 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 00:06:00 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 03:01:52 EDT 2001 
ddLogTask-e787ppc02: Sat-Oct-20-03:03:27-2001Echo
        LRS3377 files  loaded on Sat Oct 20 03:48:40 EDT 2001 
ddLogTask-e787ppc02: Error:Ev 86949; DC2 header word is wrong: 0xf8100000; expecting fffcxxxx
        LRS3377 files  loaded on Sat Oct 20 04:04:07 EDT 2001 
ddLogTask-e787ppc02: Sat-Oct-20-04:06:02-2001Echo
        LRS3377 files  loaded on Sat Oct 20 04:58:53 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 05:52:12 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 05:54:02 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 14:23:55 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 15:27:14 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 16:42:18 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 17:20:44 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 17:29:12 EDT 2001 
ddLogTask-e787ppc02: Sat-Oct-20-07:23:50-2001Echo
ddLogTask-e787ppc02: Error:Ev 27597; DC2 header word is wrong: 0xf8100000;expecting fffcxxxx
ddLogTask-e787ppc02: Sat-Oct-20-07:48:11-2001Echo
        LRS3377 files  loaded on Sat Oct 20 14:23:55 EDT 2001
        LRS3377 files  loaded on Sat Oct 20 15:27:14 EDT 2001
        LRS3377 files  loaded on Sat Oct 20 16:42:18 EDT 2001
        LRS3377 files  loaded on Sat Oct 20 17:20:44 EDT 2001
ddLogTask-e787ppc02: Sat-Oct-20-17:22:49-2001Echo
ddLogTask-e787ppc02: Error:Ev 1383; DC2 header word is wrong: 0xf0100000; expecting fffcxxxx
        LRS3377 files  loaded on Sat Oct 20 17:29:12 EDT 2001 
ddLogTask-e787ppc02: Sat-Oct-20-17:30:35-2001Echo
ddLogTask-e787ppc02: Sat-Oct-20-18:18:49-2001Echo
ddLogTask-e787ppc02: Error:Ev 3833; DC2 header word is wrong: 0xd0100000;expecting fffcxxxx
        LRS3377 files  loaded on Sat Oct 20 18:34:56 EDT 2001 
ddLogTask-e787ppc02: Sat-Oct-20-18:35:30-2001Echo
ddLogTask-e787ppc02: Error:Ev 10771; DC2 header word is wrong: 0xd0100000;expecting fffcxxxx
        LRS3377 files  loaded on Sat Oct 20 18:43:57 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 18:48:17 EDT 2001 
ddLogTask-e787ppc02: Sat-Oct-20-18:52:05-2001Echo
        LRS3377 files  loaded on Sat Oct 20 19:03:28 EDT 2001 
        LRS3377 files  loaded on Sat Oct 20 19:50:24 EDT 2001 
ddLogTask-e787ppc02: Sun-Oct-21-00:49:03-2001Echo
ddLogTask-e787ppc02: Error:Ev 181021; DC2 header word is wrong: 0xc8100000;expecting fffcxxxx
ddLogTask-e787ppc02: Sun-Oct-21-01:29:36-2001Echo
        LRS3377 files  loaded on Sun Oct 21 01:36:03 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 04:05:38 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 04:18:58 EDT 2001 
ddLogTask-e787ppc02: Sun-Oct-21-04:19:39-2001Echo
ddLogTask-e787ppc02: Error:Ev 9919; DC2 header word is wrong: 0xd0100000;expecting fffcxxxx
        LRS3377 files  loaded on Sun Oct 21 04:33:01 EDT 2001 
ddLogTask-e787ppc02: Sun-Oct-21-04:33:28-2001Echo
        LRS3377 files  loaded on Sun Oct 21 05:01:49 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 05:25:07 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 06:57:22 EDT 2001 
ddLogTask-e787ppc02: Sun-Oct-21-06:58:10-2001Echo
ddLogTask-e787ppc02: Error:Ev 137137; DC2 header word is wrong: 0xc8100000;expecting fffcxxxx
ddLogTask-e787ppc02: Sun-Oct-21-08:10:27-2001Echo
        LRS3377 files  loaded on Sun Oct 21 08:29:11 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 14:01:39 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 15:48:46 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 16:18:08 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 16:28:18 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 16:32:22 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 16:38:59 EDT 2001 
ddLogTask-e787ppc02: Sun-Oct-21-16:39:19-2001Echo
ddLogTask-e787ppc02: Error:Ev 6597; DC2 header word is wrong: 0xd0100000;expecting fffcxxxx
        LRS3377 files  loaded on Sun Oct 21 16:44:20 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 16:50:33 EDT 2001 
ddLogTask-e787ppc02: Sun-Oct-21-16:51:05-2001Echo
        LRS3377 files  loaded on Sun Oct 21 18:53:55 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 20:24:54 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 21:03:04 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 21:20:56 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 21:26:02 EDT 2001 
        LRS3377 files  loaded on Sun Oct 21 23:09:49 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 01:55:22 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 03:16:37 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 03:18:54 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 03:25:52 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-04:01:14-2001Echo
ddLogTask-e787ppc02: Error:Ev 4998; DC2 header word is wrong: 0xe0100000;expecting fffcxxxx
        LRS3377 files  loaded on Mon Oct 22 04:10:27 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-04:10:46-2001Echo
ddLogTask-e787ppc02: Error:Ev 53008; DC2 header word is wrong: 0xf8100000;expecting fffcxxxx
        LRS3377 files  loaded on Mon Oct 22 04:49:36 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-04:49:58-2001Echo
        LRS3377 files  loaded on Mon Oct 22 05:54:50 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 07:09:08 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-07:09:30-2001Echo
ddLogTask-e787ppc02: Error:Ev 98848; DC2 header word is wrong: 0xd0100000;expecting fffcxxxx
ddLogTask-e787ppc02: Mon-Oct-22-08:15:18-2001Echo
        LRS3377 files  loaded on Mon Oct 22 08:26:51 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 08:38:19 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-09:48:09-2001Echo
ddLogTask-e787ppc02: Lost of synch ,  data in DC2#1, NO data in DC2 #2
ddLogTask-e787ppc02: Mon-Oct-22-09:59:17-2001Echo
        LRS3377 files  loaded on Mon Oct 22 10:05:32 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 12:20:37 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 13:08:41 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 13:29:53 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 13:31:06 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 14:02:23 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-14:02:54-2001Echo
ddLogTask-e787ppc02: Error:Ev 40833; DC2 header word is wrong: 0xc0100000;expecting fffcxxxx
        LRS3377 files  loaded on Mon Oct 22 14:39:01 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 14:41:50 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 14:42:29 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 14:43:49 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-14:44:21-2001Echo
        LRS3377 files  loaded on Mon Oct 22 15:39:30 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-15:49:49-2001Echo
ddLogTask-e787ppc02: Error:Ev 23734; DC2 header word is wrong: 0xe8100000;expecting fffcxxxx
        LRS3377 files  loaded on Mon Oct 22 15:58:59 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-15:59:36-2001Echo
        LRS3377 files  loaded on Mon Oct 22 17:05:32 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-17:06:15-2001Echo
ddLogTask-e787ppc02: Error:Ev 42830; DC2 header word is wrong: 0xe8100000 ;expecting fffcxxxx
        LRS3377 files  loaded on Mon Oct 22 17:26:48 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-17:27:24-2001Echo
        LRS3377 files  loaded on Mon Oct 22 17:43:52 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 17:54:12 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 17:55:41 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 19:00:44 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-19:01:08-2001Echo
ddLogTask-e787ppc02: Lost of synch ,  data in DC2#1, NO data in DC2 #2
        LRS3377 files  loaded on Mon Oct 22 19:13:32 EDT 2001 
ddLogTask-e787ppc02: Mon-Oct-22-19:13:57-2001Echo
        LRS3377 files  loaded on Mon Oct 22 19:37:23 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 19:52:00 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 19:57:56 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 20:28:24 EDT 2001 
        LRS3377 files  loaded on Mon Oct 22 23:09:57 EDT 2001 
        LRS3377 files  loaded on Tue Oct 23 00:16:53 EDT 2001 
        LRS3377 files  loaded on Tue Oct 23 01:13:26 EDT 2001 
        LRS3377 files  loaded on Tue Oct 23 01:20:36 EDT 2001 
        LRS3377 files  loaded on Tue Oct 23 01:29:30 EDT 2001 
01      02:47-03:40  run45354   smx_v12j        all_v2_432.cfg  10/23/01
**** Bad RS TDC in QOD *****
        LRS3377 files  loaded on Tue Oct 23 11:04:08 EDT 2001 
        LRS3377 files  loaded on Tue Oct 23 13:41:18 EDT 2001 
ddLogTask-e787ppc02: Tue-Oct-23-15:02:24-2001Echo
ddLogTask-e787ppc02: Error:Ev 3680; DC2 header word is wrong: 0xe0100000;expecting fffcxxxx
ddLogTask-e787ppc02: Tue-Oct-23-15:11:50-2001Echo
        LRS3377 files  loaded on Tue Oct 23 17:07:19 EDT 2001 
        LRS3377 files  loaded on Tue Oct 23 17:53:28 EDT 2001 
        LRS3377 files  loaded on Tue Oct 23 17:58:08 EDT 2001 
        LRS3377 files  loaded on Tue Oct 23 19:29:43 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 12:08:37 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 12:15:38 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 15:18:53 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 15:26:23 EDT 2001 
ddLogTask-e787ppc02: Wed-Oct-24-15:26:48-2001Echo
ddLogTask-e787ppc02: Error:Ev 8821; DC2 header word is wrong: 0xe0100000;expecting fffcxxxx
        LRS3377 files  loaded on Wed Oct 24 15:34:38 EDT 2001 
ddLogTask-e787ppc02: Wed-Oct-24-15:58:29-2001Echo
        LRS3377 files  loaded on Wed Oct 24 16:02:43 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 16:11:26 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 16:16:41 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 16:40:21 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 16:46:17 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 16:54:51 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 17:07:42 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 17:12:00 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 17:39:28 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 18:24:45 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 18:28:59 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 18:34:06 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 18:39:36 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 18:47:56 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 18:52:45 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 19:00:03 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 19:02:54 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 19:08:29 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 19:12:44 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 19:21:13 EDT 2001 
        LRS3377 files  loaded on Wed Oct 24 19:25:52 EDT 2001 
ddLogTask-e787ppc02: Wed-Oct-24-19:50:58-2001Echo
ddLogTask-e787ppc02: Error:Ev 7135; DC2 header word is wrong: 0xe8100000;expecting fffcxxxx
        LRS3377 files  loaded on Wed Oct 24 20:03:06 EDT 2001 
ddLogTask-e787ppc02: Wed-Oct-24-20:06:31-2001Echo
        LRS3377 files  loaded on Wed Oct 24 22:22:55 EDT 2001 
        LRS3377 files  loaded on Thu Oct 25 00:29:15 EDT 2001 
        LRS3377 files  loaded on Thu Oct 25 01:09:09 EDT 2001 
01      04:33-05:09  run45431   smx_v12j        all_v2_432.cfg  10/25/01
**** Bad RS TDC in QOD *****

ddLogTask-e787ppc02: Thu-Oct-25-05:57:48-2001Echo
ddLogTask-e787ppc02: Lost of synch ,  data in DC2#1, NO data in DC2 #2
ddLogTask-e787ppc02: Thu-Oct-25-06:10:05-2001Echo
01      06:25-07:02  run45435   smx_v12j        all_v2_432.cfg  10/25/01
**** Bad RS TDC in QOD *****

01      07:37-07:56  run45438   smx_v12j        all_v2_432.cfg  10/25/01
**** Bad RS TDC in QOD *****

	LRS3377 files  loaded on Thu Oct 25 17:04:16 EDT 2001 
        LRS3377 files  loaded on Thu Oct 25 20:14:01 EDT 2001 
ddLogTask-e787ppc02: Thu-Oct-25-21:06:37-2001Echo
ddLogTask-e787ppc02: Error:Ev 59549; DC2 header word is wrong: 0xf0100000;expecting fffcxxxx
        LRS3377 files  loaded on Thu Oct 25 21:53:41 EDT 2001 
ddLogTask-e787ppc02: Thu-Oct-25-21:54:05-2001Echo
ddLogTask-e787ppc02: Thu-Oct-25-21:06:37-2001Echo
ddLogTask-e787ppc02: Error:Ev 59549; DC2 header word is wrong: 0xf0100000;expecting fffcxxxx
ddLogTask-e787ppc02: Thu-Oct-25-21:54:05-2001Echo


It appears that runs 45434-8 could be blamed on the "Lost of synch" error
without a subsequent CAMAC load.

However, there does not appear to be any such explanation for the bad runs
45118, 45189, 45354, 45431.

	Steve

 On Thu, 25 Oct 2001, Alexander Kushnirenko wrote:
> I came across once of this problem.  IT USUALLY happens when DAQ stops due
> to DC2 header problem.  So I would add a message in the text of this error
> to reload camac.  Whta happens is that DC2 starts to skip one or event 2 DYC
> (out of 6) which correspond to massive loss of RS TDC channels.  
> 
> The only solution I know is to reload CAMAC.  In most cases it is not needed
> as DYC somehow manage to run correctly, but it looks like not in all the
> cases.  Actually one need to reload only RS TDC camac if it is of any
> sppedup.
> 
> Another way to detect it is strange patterrn of Red lights on DYC module,
> but one has to be upstairs to see that.

    [ Part 2, ""  Text/PLAIN (Name: "ppc02_dc2.error")  1,365 lines. ]

    [ Unable to print this part. ]

From ramberg@fnal.gov Sat Jan 12 16:03:27 2002
Date: Wed, 21 Nov 2001 00:48:55 -0600
From: ramberg <ramberg@fnal.gov>
To: Robert_Tschirhart <tsch@fnal.gov>, Peter Cooper <pcooper@fnal.gov>,
     Alexander Kushnirenko <kushnir@fnal.gov>,
     George Redlinger <redlinge@bnlku7>, Renee Poutissou <renee@triumf.ca>,
     Steve Kettell <kettell@bnl.gov>
Subject: Analysis of E949 disk files for RSTDC errors


Greetings -

I've made a few simple additions to the unp3377 routine to check on data
integrity with respect to the RSTDC system.  I check to make sure all
headers are there and that serial event numbers are consistent.  If not, I
find out which TDC's or DYC's are a problem.  I've analyzed a random
sampling of the disk files in the directory:
       /e949dsk/data4/bnlonl/km21/to_be_deleted/
and the results are not the "disaster" that I expected.  They seem to
indicate a problem in DYC chain #5, perhaps in the first few TDC's in that
chain.

The files span the period from Oct. 30 to Nov. 8. I tried to be random in
my sampling, except I am biased in that I included problem files from the
early part of that period, when the problem was noticed.  Maybe I'm lucky
on the later runs, or it is not as prevalent (note that the later runs are
shorter).   Anyway, hope this helps in getting this fixed.

Erik

Run #	Problem		DYC or TDC causing problem
-----	-------		--------------------------
45607	842 TDC errs	   TDC #25 out of synch
45609   8126 TDC errs      TDC's #25,26,27 out of synch
45612   4 calc errs	   DYC and TDC word count don't match
        2 TDC errs	   TDC #8 out of synch
45614   11436 DYC errs     DYC #5 out of synch (contains TDC#25)
        9792 TDC errs      TDC #25 out of synch
45621    no errs
45631    no errs
45639    no errs
45644    no errs
45648   7442 DYC errs      DYC #5 out of synch
45650   284 TDC errs       TDC #25 out of synch
45700    no errs
45750    no errs
45791    no errs
45855    no errs
45900    no errs
45902    no errs
45953    no errs
46009    no errs
46051    no errs
46089    no errs

------------------------------------------------------------
        Erik Ramberg          Email: ramberg@fnal.gov
        Fermilab  MS 231      Phone: +1-630-840-5731
        P.O. Box 500          Fax:   +1-630-840-6311
        Batavia, IL 60510     Home:  +1-630-406-0412



From renee@triumf.ca Sat Jan 12 15:45:48 2002
Date: Thu, 01 Nov 2001 19:05:28 -0800
From: Renee Poutissou <renee@triumf.ca>
To: littenbe@bnl.gov, Steve Kettell <kettell@bnl.gov>,
     ramberg <ramberg@fnal.gov>, Sasha Kushnirenko <kushnir@fnal.gov>,
     Bob Tschirhart <tsch@fnal.gov>, Peter Cooper <pcooper@fnal.gov>,
     George Redlinger <george.redlinger@bnl.gov>
Subject: Re: QOD: RS TDC disaster

For run 45648, where the DYC event mismatch was flagged, 
I looked and the problem is again in the 5th chain.
The DYC event modulo is one BEHIND, the TDC buffer 
numbers are correct !!!!!
It happens at spill #7, ie spill #6 is correct. 

George could you get us also the beginning of that 
run on disk? 

Renee


> 
> Checking runs in  /data4/bnlonl/tdmon/to_be_deleted
> and /data4/bnlonl/tdmon/to_be_backedup
> 
> run              # spills  #spill_error/TDCadd
> 45603  smx_v12j    620         0
> 45604              603         0
> 45605              289         0
> 45606              601       261/24      75/25
> 45607              96         45/24
> 45608   cosmic     147        0
> 45609   smx        262       206/24     206/25      206/26
> 45610   cosmic     515       0
> 45611   smx         18       0
> 45612              582       0
> 45613              17        6/24
> 45614              335       227/24   266- DYC event mismatch !!!!
> 45615              500       252/24  392/25  392/25  140/27  1/11   1/7
> 45616              192       73/24
> 45617              208       0
> 45618              113       0
> 45619              300       0
> ...
> 
> 45648   smx_v12o   322          316      W: DYC event mismatch in CAR2
> 45649              738
> 45650              567       10/24
> 45651              1856      0

-- 
Renee Poutissou                        Phone: (604) 222-7457  
TRIUMF, 4004 Wesbrook Mall	       Fax  : (604) 222-1074  
Vancouver, BC, Canada, V6T 2A3	       Email: renee@triumf.ca