Accelerator Update for February 9, 2009 - February 11, 2009

Monday Feb 9

The day shift began with Operations monitoring the Tevatron (TeV) store 6787 (154.75E30) and the Recycler stash (157.5E10), with the Antiproton Source stacking (29.0E10), with the Linac, Booster, and Main Injector (MI) operating normally, with the MiniBooNE and NuMI experiments taking beam, and with the Collider Detector at Fermilab (CDF) and D-Zero (D0) experiments using the colliding beams.

Pbar experts began a 10-minute target scan study around 9:55 AM.

A MI expert began a longitudinal bunch measurement study at 10:19 AM. Some problems delayed this study for about 45 minutes.

At 10:42 AM, the TeV sector C4 air compressor tripped off. Operators responded and managed to get it running before the store was lost.

At 2:50 PM, the P1 transfer line beam permit tripped off when the output of a power supply (I:V714) went too high. Everything reset.

At 3:10 PM, the TeV sector C4 air compressor tripped off again. Operators managed to restart it before the store was lost. Experts were searching for the cause of the trips. They added a spare compressor.

Operations reported at 4 PM that the Recycler had stashed antiprotons eight times during the day shift.

Operations terminated store 6787 at 4:05 PM.

Operations prepared D0 for a one-hour access at 4:24 PM.

Operations put the TeV through a wet squeeze at 5:57 PM.

Operations put the TeV into shot setup at 6:50 PM.

At 7:41 PM, Operations reported a LCW leak.

Operators checked the service buildings; around 7:53 PM they discovered an Accumulator power supply (A:LQ) had a leaking hose. Operations called in an expert.

Operations established store 6789 at 8:15 PM with an initial luminosity of 320.88E30.

Operations reported at 9:47 PM that the A:LQ power supply was repaired.

Pbar resumed stacking at 10 PM.

At 11:23 PM, A:LQ failed to return to its nominal position. Operators recycled the busses.

Pbar resumed stacking at 11:32 PM.

Operations reported at midnight that the Recycler had stashed four times during the evening shift.

Tuesday Feb 10

The midnight shift began with Operations monitoring store 6789 (172.18E30) and the stash (50.9E10), with Pbar stacking (18.1E10), and with NuMI and MiniBooNE taking beam.

At 4:37 AM, A:LQ tripped off due to another LCW leak. Operations called in an expert.

Operations put Pbar into standby mode at 4:39 AM.

A Pbar power supply expert reported at 6:50 AM that he had A:LQ back online.

Pbar resumed stacking at 7:09 AM.

Around 8 AM, the Tevatron Electron Lens (TEL1) tripped off when the Linac LCW supply and return pressures began oscillating. Operations contacted a CUB expert.

Operations reported at 8:04 AM that the Recycler had stashed four times during the midnight shift.

At 8:57 AM, Booster RF stations BRF6, 7, and 13 all tripped off. They were reset.

A MI expert began a 90-minute study at 9:49 AM.

At 10:58 AM, TEL1 tripped off on a vacuum indication. It reset.

Mechanical group experts reported at 11:18 AM that they had replaced the overload heaters for the TeV sector C4 air compressors.

At 11:49 AM, BRF7 tripped off. An RF expert investigated and said something “funny” was happening in the East Gallery. Operations lowered Booster intensities.

A Booster expert began tuning around 12:05 PM.

At 12:31 PM, BRF5 began tripping off due to sparking.

Operations terminated store 6789 at 2:39 PM.

Operations prepared the TeV for a proton only study at 2:59 PM.

Operations reported at 3.54 PM that the Recycler had stashed seven times during the day shift.

At 4:57 PM, BRF5 tripped off two times. It reset.

Operations put the TeV through a wet squeeze at 5:45 PM.

Operations put the TeV into shot setup at 6:28 PM.

At 6:37 PM, the P2 transfer line beam permit tripped off due to a power supply (HTF13) going out of tolerance. Everything reset.

At 6:44 PM, the TeV sector C4 air compressor tripped off. Operators reset it.

Operations established store 6792 at 7:44 PM with an initial luminosity or 314.37E30.

At 9:28 PM, the P1 transfer line beam permit tripped off due to V714 going out of tolerance. Everything reset.

At 9:57 PM, a Pbar Debuncher septum (D:ESEPV) tripped off. It reset.

At 10:42 PM, Operations reported that that a Pbar Debuncher kicker (D:EKIK) had stopped firing. Operators investigated and found its ramp was missing.

Around 11 PM, Operations reported the transformer that powered the Cross Gallery, MCR, the Computer Room, and the MAC room had failed – no power to these areas. The TeV quenched at six houses: A4, B1, C4, D1, B3, and C2. Store 6792 was lost. The stash was lost. Pbar managed to hold on to a small stack.

At 11:18 PM, smoke was observed in the TeV sector B3 area. The Fire Department and operators investigated. Later, it was discovered that the dump switches for the sectors B3 and C2 service buildings had failed.

At 11:45 PM, the UPS in the Computer Room started to fail. The MCR lost controls and readbacks for the complex.

Wednesday Feb 11

The midnight shift began with the complex down due to a power failure. Investigation and recovery efforts began. The electronic logbook was down. The following is a report on machine statuses near the end of the shift, ~8 AM.

• Linac
• Linac experts were on site all night to assist with Linac systems.
• Linac never lost power, systems only affected by TCLK going away when UPS died.
• Attempted switch to LCLK that failed due to missing systems in Computer room and MAC room.
• Pumps in CUB utility tunnel off due to power outage, tripped Klystron cooling skids and Klystron RF stations.
• CUB had power and was providing 55 LCW, but stations turned down and quads breakered off as a precaution.
• Booster
• A Booster expert was here all night to assist with systems.
• Booster RF, GMPS, ORBMP and other systems dependant on TCLK failed when TCLK went away.
• Vacuum not affected by outage.
• Systems ready to come back up when controls system stabilizes.
• MI/RR
• MI ramp dumped when power was lost.
• RR stash lost.
• Other MI systems remained unaffected by power outage.
• Pbar
• D:EKIK problem was still under investigation when power outage occurred.
• Pbar actually held 12mA stack during outage.
• Tevatron
• B3, C2 dump switches failed during ramp dump due to outage.
• Houses near low betas as well as B3, C2 all quenched.
• Frig FSMs handled cooldown effectively blind.
• The Cryo Coordinator was here most of the night to monitor and aid with cryo systems.
• B0 low beta helium lead flows capped off to prevent freezing.
• D0 low beta HE leads valved out to prevent freezing.
• House power to B3, C2 cut during fire dept investigation of Code 1s at those buildings.
• Power restored to these buildings a couple hours later.
• Controls
• A Controls expert came in to shut computers down as power was lost by the UPSs.
• Another Controls expert came in to aid in recovering networks.

• Two other Controls experts came in to recover controls.

The Plans for Wednesday

Power was restored to the Cross Gallery and other areas by a large generator. There is a spare transformer in the Proton experimental area, but it must be checked out before it’s moved. The plans for today are to investigate, recover what can be recovered, and repair damage.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.