Accelerator Update for February 25, 2009 - February 27, 2009
Wednesday Feb 25
The day shift began with Operations monitoring the Tevatron (TeV) store 6832 (102.37E30) and the Recycler stash (352.1E10), with the Antiproton Source stacking (26.7E10), with the Linac, Booster, and Main Injector (MI) operating normally, with the MiniBooNE and NuMI experiments taking beam, and with the Collider Detector at Fermilab (CDF) and D-Zero (D0) using the colliding beams.
At 8:19 AM, a Linac RF station (LRF3) tripped off. It reset.
Operations terminated store 6832 at 9:09 AM.
Operations put the TeV into shot setup at 9:22 AM.
Operations established store 6833 at 11:10 AM with an initial luminosity of 324.55E30.
Operations disabled the MTest (T971) beam at 3:27 PM. Although the experiment had completed their run earlier, they asked for and received permission to take two more hours of beam. They will be back in April.
Operations reported at 4:01 PM that the Recycler had stashed antiprotons five times during the day shift.
At 5:50 PM, MIRF6 tripped off for the first of many time over the next 44 minutes. It would reset, but would not stay on. Operators finally bypassed the station.
At 9:11 PM, Operations reported that the Linac intensity had dropped by about 2mA. The H- Source output had not changed. Operators investigated.
Operations reported at 9:26 PM that they found a Linac quadrupole’s (L:HQ6Q10) setting had raised one amp. Once this was adjusted, everything returned to normal.
At 11:35 PM, LRF3 tripped off. It reset.
Operations reported at midnight that the Recycler had stashed eight times during the evening shift.
Thursday Feb 26
The midnight shift began with Operations monitoring store 6833 (160.53E30) and the stash (103.7E10), with Pbar stacking (29.1E10), and with NuMI and MiniBooNE taking beam.
At 12:16 AM, a MI power supply (I:V714) caused the NuMI beam permit to trip off. Everything reset.
At 4:27 AM, a Pbar quadrupole power supply (M:Q102) suffered a phase unbalance trip. It would not reset. Operations called in an expert.
Operations terminated store 6833 at 4:33 AM.
Operations put the TeV into shot setup around 4:46 AM.
A power supply expert reported at 5:46 AM that the phase unbalance relay had failed for M:Q102.
Operations established store 6834 at 6 AM with an initial luminosity of 324.3E30.
RF experts began working on MIRF6 at 9 AM.
RF experts brought MIRF6 back online at 11:38 AM.
At 2:18 PM, LRF3 tripped off. It reset.
At 4:10 PM, Operations reported that the 400MeV area tripped off for the first of four times in about an hour due to bad beam positions. It reset.
Operations reported at 4:17 PM that the Recycler had stashed seven times during the day shift.
At 5:04 PM, BRF4 tripped off due to sparking. Operations reported that there was a lot of Booster RF sparking.
At 5:34 PM, BRF3, 4, 15, and 16 all tripped off due to sparking. They reset. A Booster expert investigated.
At 5:49 PM, BRF19 tripped off. It reset.
At 6:41 PM, BRF16 tripped off. It reset.
At 7:05 PM, MIRF7, 8, 9, 10, and 11 tripped off.
At 7:08 PM, the Kautz Road substation tripped off.
At 7:09 PM, store 6834 was lost. There was no quench. Also, all the Recycler readbacks went away along with the stash.
At 7:10 PM, The TeV quenched at sector F3.
Operations reported at 7:18 PM that there was a power glitch from the Kautz Road substation. Several breakers opened. There was a lightening strike at the time this happened. Operations called in experts.
At 7:37 PM, Operations reported that all the MI vacuum pumps tripped off during the lightening strike. Operators restored the tables and reset the pumps.
Operations prepared D0 and CDF for access at 8:23 PM.
Operations reported at 9:18 PM that an off-site ComEd transformer that fed the Kautz Road substation had failed.
At 10:07 PM, Operations reported problems with the MI LCW system. Water experts investigated.
Operations established beam to MiniBooNE at 11:51 PM.
Operations reported at midnight that the Recycler had stashed three times during the evening shift.
Friday Feb 27
The midnight shift began with Operations recovering from the Kautz Road substation power glitch.
At 12:09 AM, Operations reported that the MiniBooNE orbit was bad. They began searching for a power supply problem.
The Operations Crew Chief reported the following status at 12:52 AM.
At 1:52 AM, Operators reported that they could not get Booster intensity up over 6 turns. Operations called in an expert.
Cryo system experts at CHL reported at 2:15 AM that the TeV cryo system was good. The only problems were with the leads at sector A4 and D4.
Operations reported at 2:22 AM that the MI-40 LCW pumps 2 and 3 were back on.
A Pelletron expert reported at 2:56 AM that the Electron Cooling was working. Unfortunately, the crash recovery FSM was not working; they decided to run without it temporarily.
At 3 AM, Operations reported that the Recycler flying wire was not working, the computer associated with the Recycler damper diagnostic scope was down, and the Recycler AC beam monitor was also down. An expert was informed.
Operations established a low intensity beam to MiniBooNE around 3:50 AM. Later, Operations resumed sending normal intensity beam.
A Recycler expert reported at 4:03 AM that the RF was working, but since there were AC problems at MI-60, the amplifiers should be left off.
Operations prepared the TeV for access at 4:34 AM.
The Duty Electrician reported at 5:28 AM that the MI-60 air condition problem was fixed. Operations turned on the Recycler RF system and found a problem. They contacted the RF experts.
At 5:33 AM, LRF3 tripped off. It reset.
At 5:48 AM, Operations reported that the Pbar core transverse cooling pin switches were open and would not close. Operators investigated and discovered a timing problem.
Operations reported at 5:59 AM that MI was ramping.
At 7:04 AM, Operations reported that the MI Center Anode power supply had tripped its breaker. Operators contacted the High Voltage Coordinator.
The Plans for Friday and the Weekend
The plans for today are to continue recovering from the Kautz Road power glitch. The weekend will consist of stacking, stashing, and storing.
Comments and Suggestions
|last modified 02/27/2009 email Fermilab|
|Security, Privacy, Legal|