Accelerator Update for February 11, 2009 - February 13, 2009

Wednesday Feb 11

The day shift began with the accelerator complex in recovery from a power outage caused by a transformer failure at 11 PM on 2/10/09.

Operations reported at 9:34 AM that TCLK was working.

Operations conducted a TeV hipot at 10:13 AM. Operators found a soft ground fault in between sectors C2 and D2.

Operations prepared the Booster for an access at 10:20 AM.

FESS personnel reported at 10:26 AM that they were cleaning basket strainers in MI.

The Cryo System Coordinator reported at 10:42 AM that cryo technicians were warming up TeV sector C4 to repair a Kautzky valve.

TeV experts reported at 10:52 AM the sector B3 electronic dump switch would be repaired and it would remain an electronic dump switch; the sector C2 dump switch would be converted back to a mechanical dump switch; and that they were investigating a problem with the sector F3 electronic dump switch.

Operations logged pictures of the damaged TeV sector B3 dump switch at 11:37 AM.

Operations prepared MI for access at 11:48 AM.

Operations prepared TeV F sector for access at 11:49 AM.

Operations reported at 11:50 AM that a new gas powered generator was installed and running the Cross Gallery and other areas covered by the failed transformer. The spare transformer out in the Proton Experimental Area failed its hipot. Another transformer was found at the Rail Head.

A TeV expert reported at 11:57 AM that he had put all the sector A0 abort kickers in conditioning mode.

Operations logged the following pictures of the bad transformer being removed at 2:11 PM.

Interlock experts reported at 2:38 PM that they had completed testing on the MI-8 safety system.

At 2:48 PM, Operations reported sending a pulse of beam to Booster, but it fell out at low field.

The Neutron Therapy Facility (NTF) reported at 3:09 PM that they had completed treating its last patient for the day.

Operations reported at 3:22 PM that the transformer at the Rail Head had passed its hipot. It will be placed on its pad behind the Cross Gallery tonight, but not hooked up due to the rain.

Operations prepared CDF for an access at 4:13 PM.

D0 came out of access at 4:18 PM. They began a cosmic ray study.

Operations reported at 4:40 PM that they had prepared MI and TeV sector F for interlock testing.

CDF came out of access at 4:40 PM. They began a cosmic ray study.

Operations reported at 5:10 PM that Booster was circulating beam.

At 5:46 PM, a Pbar Debuncher Bendbus power supply (D:IB) tripped off on a ground fault indication. Operators investigated, found nothing unusual, and reset the supply. It tripped off on the same indication. It reset.

Operations turned on MI and established a ramp at 6:26 PM.

At 6:40 PM, D:IB tripped off on a ground fault indication. Operations called in a power supply expert.

Operations had beam established to MiniBooNE and NuMI by 7:13 PM.

The power supply experts confirmed at 9:05 PM that D:IB had a cable ground fault located somewhere between the supply and the yard transformer. Operations informed the Pbar experts. They will decide what to do on Thursday morning. Pbar will not be able to stack.

At 11:06 PM, MIRF13 tripped off on a ground fault indication and would not reset. Operators bypassed the station.

Thursday Feb 12

The midnight shift found the accelerator complex still recovering from the power outage, with Recycler experts conducting studies, and with Pbar down due to a D:IB cable ground fault.

Vacuum experts accessed the TeV sectors A-E and the transfer hall at 5:35 AM.

At 6:29 AM, a Linac Klystron RF station (KRF4) tripped off due to a rise in the cavity temperature. Operators made adjustments and had it back online 20-minutes later.

At 6:39 AM, BRF14 tripped off for the first of six times in 30 minutes. It reset.

An RF expert brought MIRF13 back online at 7:51 AM.

A MiniBooNE expert began a study at 8:01 AM.

CDF accidentally dropped their interlocks at 9:06 AM.

At 9:26 AM, a Linac quadrupole power supply (QPS501) tripped off for the first of two times in 8 minutes. It reset. Operations informed Linac experts.

At 9:50 AM, an Accumulator power supply (A:LQ) tripped off on a ground fault. An expert investigated and reported that the supply had developed another LCW leak.

A Recycler expert began a beam measurement study at 10:23 AM.

Operations established full intensity beam to NuMI and MiniBooNE at 10:24 AM.

At 10:55 AM, a Recycler kicker (R:KPS2A) tripped off on a fail to fire indication for the first of three times six minutes. Operators reported that the trips were occurring every time protons were injected into the Recycler. It reset.

Cryo system technicians reported at 1:11 PM that they were beginning work on the TeV sector A3 wet engine.

A power supply expert reported at 1:58 PM that the Pbar power supplies work was finished.

Pbar began stacking at 2:19 PM.

Operators searched and secured the CDF collision hall at 3:08 PM.

NTF reported at 6:20 PM that they had completed treating patients for the day.

The Recycler proton study ended at 6:26 PM.

Operations established a QBS ramp on the TeV and completed two dump tests at 7:36 PM.

Operations reported at 7:58 PM that the TeV sector B3 dump switch was back on line and successfully tested.

At 10:23 PM, BRF7 tripped off for the first of two times in one minute. It reset.

Operations reported at 10:26 PM that they had the TeV turned on to 90 GeV and would leave it on all night.

At 10:40 PM, LRF3 tripped off and would not reset. Operations contacted experts.

Friday Feb 13

The midnight shift began with Operations monitoring a Pbar stack (141.7E10), with Linac experts working on the LRF3 driver tube, and with the TeV at 90 GeV.

A Linac expert reported at 12:08 AM that the LRF3 driver tube had a water leak. He was calling in experts to help make repairs.

Operations reported at 4:30 AM that they were beginning to prepare areas for the up coming power outage to switch to the new Cross Gallery transformer.

Operations prepared the TeV, MI, and Booster for access at 6:50 AM.

Controls experts reported at 7:32 AM that they were taking down the controls system in preparation for the power outage.

The Plans for Friday and the Weekend

The plan for today is to switch off power at 8:30 AM to connect the new Cross Gallery transformer. Power should be back by 10:30 AM and controls should be back within an hour or two after that. The plans for the weekend are to check out the TeV and all other machines and then begin stacking, stashing, and storing.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.