Operation Summary for September 12, 2011 - September 14, 2011

Monday September 12

The day shift began with Operations monitoring the Recycler stash (339.9E10), with the Antiproton Source (Pbar) stacking (16.2E10), with the Main Injector (MI), Linac, and Booster operating normally, with the NuMI and MiniBooNE experiments taking beam, and with the Collider Detector at Fermilab (CDF) and D-Zero (D0) waiting for the next Tevatron (TEV) store.

During the day shift, operators put the TEV through a wet squeeze at 8:27 AM. Operations put the TEV into shot setup at 9:33 AM and established store 9103 at 10:42 AM with an average initial luminosity (AIL) of 356.88E30. Operators reported short downtimes due to two Booster RF stations (BRF2 and BRF7), a Linac RF station (LRF2), and bad beam positions for NuMI.

During the evening shift, Operations terminated store 9103 at 4:05 PM. Operations established a 450 GeV 3x3 study store at 5:48 PM and terminated the store at 8 PM. Operators put the TEV through a dry squeeze at 8:15 PM. Operations established a 3x3 study store around 9 PM. Operators reported at 9:58 PM that the MI power supply interlock loop tripped several times; after the last trip it showed a ground fault indication. Operators conducted a hipot on the bus. Operators called in MI people for a ground fault hunt.

Tuesday September 13

The midnight shift began with Operations monitoring a 3x3 study store and the stash (180.0E10), with Pbar stacking (93.0E10), and with NuMI and MiniBooNE taking beam.

During the midnight shift, operators aborted the stash and prepared MI for an access at 12:07 AM. At 1:44 AM, operations received an alarm for the TEV sector D1 cryo system building. Operations dumped the TEV ramp, which ended the 3x3 store. The wet engine at D1 had seized; Cryo System personnel reported it would take eight hours to repair. MI personnel reported at 2:29 AM that they found ground water leaking onto the bus; they diverted the water and asked for another hipot. At 4:06 AM, operators hipotted the MI bus and reported that the ground fault was gone. Operators reported at 5:15 AM that when the ramped MI the ground fault returned. Operators prepared MI for access. Operators prepared D0 for an access at 6:41 AM. Neutron Therapy Facility began running beam at 7:25 AM in preparation for treating one patient later in the morning.

During the day shift, D0 came out of access at 9:57 AM. At 10:47 AM, operators, MI, and EE Support personnel began narrowing down where the MI ground fault was located. Operators reported at 1:25 PM that there was still a soft ground fault between MI-30 and MI-50. Booster RF personnel reported at 1:35 PM that they had completed all Booster RF work; Operations established beam to MiniBooNE at 2:01 PM. The Cryo Coordinator reported at 4:06 PM that Cryo System personnel were beginning TEV sector C2 wet engine flywheel repairs. MI reported at 3:17 PM that the MI ground fault was not found, but a way was found to turn on the machine that allowed the machine to run without tripping off; the problem had not been fixed. EE Support will continue to investigate. Personnel also found a quadrupole magnet (Q403) with an internal LCW leak that cannot be repaired; the magnet will have to be replaced eventually, but due to its location, not until it fails outright or there’s a two-week downtime.

During the evening shift, operators reported at 4:55 PM that the TEV was ramping and MI was running at full turns. At 6:06 PM, a MI-50 lower bus power supply tripped off on a ground fault indication; it reset. At 6:12 PM, there was a MI-30 VESDA alarm. Operators prepared MI-30 for an access at 6:22 PM. MI personnel reported at 8:30 PM that they found two spots in MI-30 that showed signs of arcing; Mechanical technicians were called in for repairs.

Wednesday September 14

The midnight shift began with the TEV in a wet squeeze, with MI in access to repair ground faults, and with Operations monitoring the Pbar stack (108.2E10).

During the midnight shift, the MI work and tunnel access ended at 12:32 AM. Operations established a MI ramp at 1:10 AM. Operations had established beam to MiniBooNE, NuMI, and Pbar by 1:23 AM. Two Booster RF stations (BRF7 & 8) tripped off at 1:26 AM for the first of four times in about two hours; they reset. At 3:57 AM, the TEV suffered a four-house quench. At 7:01 AM, operators reported a power glitch that caused problems with the Linac RF, Booster RF, MI kickers, and the TEV ramp. At 7:02 AM, the Booster East Anode Power Supply tripped off; it reset. Operations had all beam restored by 7:07 AM.

The Plans for Wednesday

The plans for today are to recover the TEV and resume stacking, stashing, and storing. The Run Coordinator will allow Booster and MI studies. NuMI personnel reported that they would begin replacing their target tomorrow morning, 9/15/11.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.