Accelerator Update for February 25, 2009 - February 27, 2009

Wednesday Feb 25

The day shift began with Operations monitoring the Tevatron (TeV) store 6832 (102.37E30) and the Recycler stash (352.1E10), with the Antiproton Source stacking (26.7E10), with the Linac, Booster, and Main Injector (MI) operating normally, with the MiniBooNE and NuMI experiments taking beam, and with the Collider Detector at Fermilab (CDF) and D-Zero (D0) using the colliding beams.

At 8:19 AM, a Linac RF station (LRF3) tripped off. It reset.

Operations terminated store 6832 at 9:09 AM.

Operations put the TeV into shot setup at 9:22 AM.

At 9:44 AM, Operations reported a problem with the Switchyard (SY) and Meson vacuum. Experts investigated.

Operations established store 6833 at 11:10 AM with an initial luminosity of 324.55E30.

Operators search and secured the MTest area at 1:06 PM.

A SY expert established beam to the SY dump at 1:26 PM. He then began tuning beam to MTest.

Operations disabled the MTest (T971) beam at 3:27 PM. Although the experiment had completed their run earlier, they asked for and received permission to take two more hours of beam. They will be back in April.

Operations reported at 4:01 PM that the Recycler had stashed antiprotons five times during the day shift.

At 5:50 PM, MIRF6 tripped off for the first of many time over the next 44 minutes. It would reset, but would not stay on. Operators finally bypassed the station.

At 9:11 PM, Operations reported that the Linac intensity had dropped by about 2mA. The H- Source output had not changed. Operators investigated.

Operations reported at 9:26 PM that they found a Linac quadrupole’s (L:HQ6Q10) setting had raised one amp. Once this was adjusted, everything returned to normal.

At 11:35 PM, LRF3 tripped off. It reset.

Operations reported at midnight that the Recycler had stashed eight times during the evening shift.

Thursday Feb 26

The midnight shift began with Operations monitoring store 6833 (160.53E30) and the stash (103.7E10), with Pbar stacking (29.1E10), and with NuMI and MiniBooNE taking beam.

At 12:16 AM, a MI power supply (I:V714) caused the NuMI beam permit to trip off. Everything reset.

At 4:27 AM, a Pbar quadrupole power supply (M:Q102) suffered a phase unbalance trip. It would not reset. Operations called in an expert.

Operations terminated store 6833 at 4:33 AM.

Operations put the TeV into shot setup around 4:46 AM.

A power supply expert reported at 5:46 AM that the phase unbalance relay had failed for M:Q102.

Operations established store 6834 at 6 AM with an initial luminosity of 324.3E30.

RF experts began working on MIRF6 at 9 AM.

Operations prepared the Meson M01-M05 enclosures for access and safety system tests at 9:57 AM.

RF experts brought MIRF6 back online at 11:38 AM.

At 2:18 PM, LRF3 tripped off. It reset.

At 4:10 PM, Operations reported that the 400MeV area tripped off for the first of four times in about an hour due to bad beam positions. It reset.

Operations reported at 4:17 PM that the Recycler had stashed seven times during the day shift.

At 5:04 PM, BRF4 tripped off due to sparking. Operations reported that there was a lot of Booster RF sparking.

At 5:34 PM, BRF3, 4, 15, and 16 all tripped off due to sparking. They reset. A Booster expert investigated.

At 5:49 PM, BRF19 tripped off. It reset.

At 6:41 PM, BRF16 tripped off. It reset.

At 6:58 PM, the MI ramp tripped off on a ground fault indication. It would not reset.

At 7:05 PM, MIRF7, 8, 9, 10, and 11 tripped off.

At 7:08 PM, the Kautz Road substation tripped off.

At 7:09 PM, store 6834 was lost. There was no quench. Also, all the Recycler readbacks went away along with the stash.

At 7:10 PM, The TeV quenched at sector F3.

Operations reported at 7:18 PM that there was a power glitch from the Kautz Road substation. Several breakers opened. There was a lightening strike at the time this happened. Operations called in experts.

At 7:37 PM, Operations reported that all the MI vacuum pumps tripped off during the lightening strike. Operators restored the tables and reset the pumps.

Operations prepared D0 and CDF for access at 8:23 PM.

Operations reported at 9:18 PM that an off-site ComEd transformer that fed the Kautz Road substation had failed.

At 10:07 PM, Operations reported problems with the MI LCW system. Water experts investigated.

Operations established beam to MiniBooNE at 11:51 PM.

Operations reported at midnight that the Recycler had stashed three times during the evening shift.

Friday Feb 27

The midnight shift began with Operations recovering from the Kautz Road substation power glitch.

At 12:09 AM, Operations reported that the MiniBooNE orbit was bad. They began searching for a power supply problem.

The Operations Crew Chief reported the following status at 12:52 AM.

• The MI-40 LCW pumps 2 & 3 were not working. Each respective motor controller had a failed 480-V contactor. FESS electricians were investigating.
• All kickers in the MI and RR were not working. Experts were investigating.
• Operations was trying to establish beam to the Booster dump. Once established, Operations will send beam down the MI-8 line and eventually to MiniBooNE.
• The MI-60 air conditioning unit was not working and the building was getting hot, which prompted the RF engineers to turn off all the MI/RR RF. Operations contacted experts.
• The Booster pond was flooding over into the road. The mechanic reported that all pumps were running, but were overwhelmed by the volume of water. Since the rain had stopped the pumps ought to catch up sooner or later. Operations will monitor the pond level and respond accordingly.
• The TeV cryo systems were recovering and being monitored by CHL. TeV sectors D3, E3, and all of F-sector were in various states of recovery. CHL expected it would be at least a few more hours before ramp beam permits came back.
• Experts were at MI-31 recovering the Pelletron.
• Linac was working.
• Booster was working.
• The Accumulator had maintained a stack of 24 mA and was ready for more.
• NuMI was ready for beam.
• MiniBooNE was ready for beam.

At 1:52 AM, Operators reported that they could not get Booster intensity up over 6 turns. Operations called in an expert.

Cryo system experts at CHL reported at 2:15 AM that the TeV cryo system was good. The only problems were with the leads at sector A4 and D4.

Operations reported at 2:22 AM that the MI-40 LCW pumps 2 and 3 were back on.

A Pelletron expert reported at 2:56 AM that the Electron Cooling was working. Unfortunately, the crash recovery FSM was not working; they decided to run without it temporarily.

At 3 AM, Operations reported that the Recycler flying wire was not working, the computer associated with the Recycler damper diagnostic scope was down, and the Recycler AC beam monitor was also down. An expert was informed.

Operations established a low intensity beam to MiniBooNE around 3:50 AM. Later, Operations resumed sending normal intensity beam.

A Recycler expert reported at 4:03 AM that the RF was working, but since there were AC problems at MI-60, the amplifiers should be left off.

Operations prepared the TeV for access at 4:34 AM.

The Duty Electrician reported at 5:28 AM that the MI-60 air condition problem was fixed. Operations turned on the Recycler RF system and found a problem. They contacted the RF experts.

At 5:33 AM, LRF3 tripped off. It reset.

At 5:48 AM, Operations reported that the Pbar core transverse cooling pin switches were open and would not close. Operators investigated and discovered a timing problem.

Operations reported at 5:59 AM that MI was ramping.

Operations loaded an energy conservation time line into the Time Line Generator at 6:59 AM.

At 7:04 AM, Operations reported that the MI Center Anode power supply had tripped its breaker. Operators contacted the High Voltage Coordinator.

The Plans for Friday and the Weekend

The plans for today are to continue recovering from the Kautz Road power glitch. The weekend will consist of stacking, stashing, and storing.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.