Operation Summary for July 27, 2011 - July 29, 2011

Wednesday July 27

The day shift began with Operations monitoring the Tevatron (TEV) store 8930 (95.78E30) and the Antiproton Source (Pbar) stack (95.78E10), with the Recycler off due to a Main Injector (MI) access for NuMI magnet (V108) cabling work, with Booster off in preparation for an access, with the Linac off due to trouble with a Linac RF station (LRF4), with the MiniBooNE and Meson FTBF T-1008 experiments off due to Booster and MI accesses, and with the Collider Detector at Fermilab (CDF) and D-Zero (D0) using the colliding beams.

During the day shift, at 8:52 AM a Recycler vacuum pump (R:MI21VAC) came into alarm; operators and the Recycler Specialist investigated, turned the pump off, and notified a vacuum technician. At 9:27 AM, a TEV sector C feed down sextupole magnet (C:S7BA1) power supply tripped off on a fuse fault indication; operators notified EE Support personnel. Booster came out of access at 9:45 AM. A Mechanical person reported that the tunnel electronics for a Recycler beam valve broke during testing; EE Support personnel began repairs. Stochastic Cooling personnel began working on the Recycler cooling system at 10:04 AM. FESS personnel began cleaning the TEV sector F3 pond water pump strainer; they had it cleaned with in an hour. A Water Works person reported at 10:20 AM that they had applied a temporary patch to the LCW leak on the MI-10 bus. Linac personnel completed their work on LRF4 and Operations established beam to the Linac dump at 10:34 AM. Operations established beam to the Booster dump at 11:04 AM. MTA began taking beam at 11:30 AM. The contract electricians reported at 11:46 AM that they would have the cables in the NuMI stub spliced by noon. The cable splices in MI for the 619 location would be done by 4 PM. EE Support will then test the cables and the connections between the V108 magnet and the power supply and then hook the cables to the power supply. Operations prepared the MI-8 transfer line for access at 1:50 PM. Vacuum personnel reported that they were completing their work in Recycler at 3:45 PM.

During the evening shift, Operations disabled beam to MTA at 5:06 PM. Operators began the search and secure of MI at 5:09 PM; at 6:12 PM they reported finding a LCW leak that was falling onto a magnet (VDPA2) and its BLM. Operators called in Water Works personnel. Operators reported a MI-8 vacuum burst at 8:09 PM. EE Support reported that the V108 hipot was good. At 8:48 PM, store 8930 aborted, Linac went down, as did many Booster power supplies; in all over 200 alarms posted. The Main Accelerators Safety System dropped , which took down all the TEV safety systems. The TEV cryo systems at B1, BA, BD, D1, DA, and DD quenched. All of Linac went down; Operators called in Linac personnel. People from Booster and the TEV came in to help recover their machines. Operators reported at 9:13 PM that the Main Accelerators Safety System had lost power and when they accessed the area , the smell of burnt electronics filled the air and the uninterruptable power supply (UPS) had scorch marks on it. Operators notified many people. Operators reported that interlock personnel replaced the UPS chassis at 10:21 PM and got the Main Accelerator Safety System back online. The Cryo Coordinator reported at 10:56 PM that he was bringing in cryo system technicians to bring the TEV sector A0 compressors back online. FESS personnel will turn on the A0 pond water pump when the cryo techs are ready and then remove the fire hose and barricades. Pbar turned back on. Operators brought the 400 MeV line and the MI-8 line on. MI began ramping at 11:35 PM. Operations began sending beam to MiniBooNE at 11:36 PM. Operations established beam to Pbar at 11:57 PM.

Thursday July 28

The midnight shift began with most of the Accelerator Complex in recovery, with Pbar stacking, with Linac and Booster operating normally , and with MiniBooNE taking beam .

During the midnight shift, at 12:06 AM, a lightning strike caused some Booster quadrupole power supplies to trip off as well as RF stations and a Recycler kicker. BRF17 would not reset due to indications of a power amplifier LCW leak and high vacuum pressure. Neutrino Lab A lost power. LRF2 and 5 tripped off. Operators called in a Booster person to investigate a PLC problem. Then, at 12:21 AM , all the lights and power all over the lab went out, except for the Village

Operations reported at 3:42 AM that a little after midnight, the site had lost all power from the Master Substation and the Kautz Road Substation. The Kautz Road Substation came back on after a minute or so. Operators notified many more people. The High Voltage Coordinator set up to back feed the Master Substation from Kautz Road Substation and restored power to the rest of the site (with some load restrictions). Since power was not restored to the Main Control Room until 2 AM, there was no controls system. During this time, operators visited all the buildings and manually restarted vacuum pumps and many other devices in the Accelerator Complex. At 5 AM, operators began preparing the MI, TEV Transfer Hall, and TEV A-E tunnels for access. Operators reported that the interlocks dropped for many areas and would have to be searched and secured.

The Operations Crew Chief reported the current status of MI, Recycler, and the Pelletron:

• MI ramp was off and Kautz Road Substation (KRSS) feeders were locked out until further notice since the Master Substation (MSS) was now back-fed from KRSS. CHL was running a single big compressor and there were 7 Mycom compressors on ring-wide
• MI high level RF was on
• All MI LCW systems were on, but the MI -30 #1 LCW pump would not start
• All MI and RR vacuum pumps were on and vacuum looked reasonable in both machines. Presently all pneumatic beam valves were closed in both rings
• The Electron Cooling (ECool) vacuum pumps were on and the (SF 6) chiller and blower were running in the MI -31 pit. The Pelletron high voltage remained off
• MI, RR, and ECool controls seemed to be working more or less normally except for a few systems Front Ends (FE):
• MISBD – MI sampled bunch display FE
• MIFW (MI flying wire FE)
• MIFBI (MI fast bunch integrator)
• IPMM1H (MI IPM mark-1)
• ECBPM (ECool BPM vxi)
• ADRR (Recycler damper diagnostic scope)
• MI Low Level RF (LLRF) and MRF were rebooted earlier followed by the MI dampers
• MI BPMs: IBPMA1, IBPMP2, IBPM8A
• RBMP40 (Recycler 40 BPM FE)
• MECAR was rebooted at Controls personnel request since it was not broadcasting MDAT frames. The FE was responding to a node poll.

Operators reported at 6:45 AM that the MI RF system was back online as well as the coalescing system. The Recycler RF also looked good. Operators reset the TEV QPMs that would reset and reported the ones that wouldn’t reset. Operators reported at 7:02 AM that the Pbar vacuum pumps were on and the LCW system flowing, with CAMAC links restored, but with the power supplies off. Operators prepared D0 and CDF for accesses. Operators also restarted the Neutrino and Meson area LCW and vacuum systems.

During the day shift, Instrumentation personnel helped bring up TEV BPMs and BLMs. Linac personnel worked on their low energy RF systems. Booster personnel worked on the 400 MeV area elements. Operators opened the Recycler stochastic cooling light link valve. At 10:20 AM, operators reported strange actions by the MI-50 corrector power supplies; A Controls person rebooted the CAMAC crate. A MiniBooNE person reported a LCW leak in MI-12B on a quadrupole magnet (Q847). A TEV person reported a LCW leak in TEV sector A11. Operations reported that the TEV was going be down for three to six days due to cryo system and vacuum work. The High Voltage Coordinator reported that ComEd restored power to the Master Substation at 2:25 PM; he said they reported losing three 345kV lines last night during the storm. There will be a 30-minute power outage on Friday morning at 7 AM to switch back to MSS power.

During the evening shift, power outage recovery continued. Linac personnel continued work on LRF4 and other equipment and systems. Operations, NuMI, and EE Support personnel turned on and ramped V108 magnet and power supply.

Friday July 29

The midnight shift began with the Accelerator Complex recovering from the power outage.

During the midnight shift, Operators turned off V108 at 5:48 AM. Operations reported at 6:30 AM that all areas were ready for the power outage scheduled for 7 AM. The High Voltage Coordinator reported that in terms of power, everything was back to normal by 7:30 AM. CUB personnel reported at 7:51 AM that they were having a problem with their transfer switch; this was eventually resolved.

The Plans for Friday and the Weekend

The plans for today and the weekend are to continue recovering from the power outage. The TEV cryo and vacuum systems will take anywhere from three to six days to recover.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.