Accelerator Update for March 13, 2009 - March 16, 2009

The Integrated Luminosity from 3/9/09 to 3/16/09 was 54.36 inverse picobarns.

Friday March 13

The day shift began with Operations monitoring the Tevatron (TeV) store 6891 (126.73E30) and the Recycler stash (207.9E10), with the Antiproton Source stacking (17.7E10), with the Linac, Booster, and Main Injector (MI) operating normally, with the MiniBooNE and NuMI experiments taking beam, and with the Collider Detector at Fermilab (CDF) and D-Zero (D0) experiments using the colliding beams.

At 8:20 AM, Operations reported a power glitch at D0. Store 6891 was lost. The TeV quenched at sector C4. Operators and experts investigated. D0 transformer had tripped off.

At 8:21 AM, Operations reported that there was no power at D0.

At 8:28 AM, Operations reported that the TeV RF system was off and that there was no power at the D0 service building.

At 8:31 AM, Operations reported that two quadrupole power supplies (L:Q42 and L:Q43) were off and an operator reported the smell of something burning. The Fermi Fire Department responded. Later, Operators reported that the L:Q4BULK power supply transformer had overheated.

Operations put Pbar into standby at 8:33 AM.

Operations reported at 9:02 AM that the TeV RF system was back online.

Operations reported at 9:12 AM that they had turned off all the TeV power leads and opened the TeV power supply VCBs. Operators turned off all TeV devices in preparation for the D0 transformer repair.

Operations reported at 9:28 AM that they were preparing to turn off the TeV LCW system.

Operations prepared D0 and CDF for access at 10:08 AM.

Operations began preparing the TeV A-E sectors, Transfer hall, and F sector for access at 10:35 AM.

The Neutron Therapy Facility reported at 10:41 AM that they were beginning to treat patients.

Operations reported at 11 AM that the TeV LCW system was brought back online.

Operations reported the following status at 11:09 AM:

• At 8:20 AM or so, a fault took out power at the D0 service building
• This fault caused a glitch that took down the Linac and Booster accelerators
• Shortly after the glitch, Linac Q4BULK power supply transformer overheated and caused a Linac gallery fire alarm. The Fire Department and Linac personnel responded and dealt with the situation
• Troubleshooting the D0 service building fault required that power be dumped for all of D sector. This meant that the TeV LCW system had to be shut down. All affected groups were consulted and made their systems ready for this. D sector power was dumped, and the D0 service building transformer was isolated
• D1 through D4 house power was back fed by 10:50 AM
• Once D sector power was restored, the TeV LCW system was turned back on
• Linac Q4BULK power supply repairs were complete by 11 AM. Linac experts were bringing the supplies back up slowly
• CDF and D0 were in access
• The entire TeV ring was made ready for access
• Water experts accessed into the TeV to check water flow on the power leads
• The sector E4 LCW pump was replaced
• Controls experts added a status bit readback for the C4 house air compressor
• Pbar Target experts were checking on the vault water leak
• TeV vacuum experts were recovering and monitoring their systems
• Cryo system experts were recovering and monitoring their systems

Linac resumed sending beam at 12:05 PM.

Booster resumed sending beam at 12:15 PM.

Operations reported at 1:16 PM that high voltage experts had brought the D0 power transformer back online. Controls experts began resetting their equipment.

At 1:22 PM, a Pbar expert reported that the water leak on the lithium lens was worse than they had expected.

At 2:05 PM, FESS electricians reported that when the D0 power came on, the tunnel power did not return.

At 2:10 PM, FESS electricians reported that they found two breakers tripped at D0. They need to power down the transformer before they can reset the breakers.

At 2:40 PM, the FESS electricians reported that they found evidence of a flash over in a 480 volt AC panel at D0. They investigated.

Operations began sending beam to MiniBooNE at 2:46 PM.

At 2:50 PM, Pbar experts reported that they would have to pull out the old lithium lens and install a new one.

A TeV expert reported at 3:10 PM that he had placed the TeV sector A0 kickers into conditioning mode.

Operations established beam to NuMI at 3:12 PM.

At 3:36 PM, vacuum experts reported that they needed 2-3 hours in the TeV to turn on pumps and conduct maintenance.

Operations reported that FESS electricians restored power to D0 at 3:44 PM. They reported that they had found some loose connections in a power panel. They tightened and cleaned everything.

Operations established beam to MTest at 4:11 PM.

At 4:25 PM, a Linac RF station (LRF3) tripped off. It reset.

At 4:41 PM, L:Q7BULK power supply tripped off. An expert reset it.

Operations reported at 4:50 PM that all of the TeV front ends were rebooted.

A TeV expert brought the A0 kickers out of conditioning mode at 5:35 PM.

At 6:14 PM, LRF3 tripped off. It reset.

CHL cryo system experts reported at 6:29 PM that all the TeV cryo systems were cold and ready to go.

Operations disabled the Meson MTest beam at 8:22 PM.

Pbar experts reported at 9:37 PM that they were pulsing the new lens.

Pbar began stacking at 11:06 PM.

TeV vacuum experts came out of access at 11:36 PM.

Saturday March 14

The midnight shift began with Operations monitoring the stash (225.9E10), with Pbar stacking (22.4E10), and with NuMI and MiniBooNE taking beam.

At 12:25 AM, LRF3 tripped off. It reset.

At 12:59 AM, Operations reported a ground fault on a TeV power supply (C:B0Q5). Operators investigated and then called in an expert.

At 1:31 AM, Operations reported a TeV power supply (T:L2C4) had tripped off on a quench and safety indication, as well as a ground fault. Operations reseated the modules and reset the supply.

A TeV power supply expert reported at 2:54 AM that C:B0Q5 was okay. He reset the supply and turned it on.

Operations put the TeV through a dry squeeze at 3:27 AM.

Operations reported at 5:31 AM that they had completed two TeV orbit smoothes.

Operations put the TeV into shot setup at 5:55 AM.

At 5:59 AM, LRF3 tripped off. It reset.

Operations established store 6894 at 7:15 AM with an initial luminosity of 273.25E30.

Operations reported at 8:04 AM that the Recycler had stashed antiprotons five times during the midnight shift.

Operations reported at 3 PM that they had completed some Booster injection tuning.

At 3:20 PM, Operations reported that the I- Source’s beam intensity had started to drop a little this morning. Operators adjusted the extractor voltage to compensate.

Operations terminated store 6894 at 6:32 PM.

At 6:45 PM, a quadrupole power supply (I:QF11B) tripped off and caused the P1 beam permit to also trip off. Everything reset.

At 7:02 PM, LRF3 tripped off. It reset.

Operations put the TeV through a wet squeeze at 7:16 PM.

Operations reported at 7:56 PM that the Recycler had stashed 10 times during the last 12 hours.

Operations prepared CDF for a one-hour access at 8:15 PM.

Operations put the TeV into shot setup at 11:27 PM.

At 11:33 PM, a TeV sector C kicker (C:AAK1) tripped off. It reset.

Sunday March 15

The midnight shift began with the TeV in shot setup, with Operations monitoring the stash, with Pbar stacking, and with NuMI and MiniBooNE taking beam.

Operations established store 6899 at 12:50 AM with an initial luminosity of 307.08E30.

At 1:39 AM, a Linac Klystron RF station (KRF5) tripped off. It reset.

Operations reported at 8 AM that the Recycler had stashed eight times during the last 12 hours.

At 10:36 AM, Operations reported that two Booster vacuum pumps (IP61 and IP62) tripped off. They reset.

Operations terminated store 6899 at 1:15 PM and put the TeV into shot setup.

Operations established store 6900 at 2:52 PM with an initial luminosity of 290.2E30.

At 3:41 PM, Operations reported that store 6900’s luminosity was low due to higher vertical and horizontal proton emittances and higher longitudinal antiproton emittances.

At 4:02 PM, Operations reported a couple of small dips in luminosity. Operators investigated and found that a TeV RF station (TRF6) had sparked.

At 4:51 PM, KRF1 tripped off. It reset.

At 5:26 PM, KRF5 tripped off. It reset.

Operations reported at 7:55 PM that the Recycler had stashed 10 times during the last 12 hours.

Monday March 16

The midnight shift began with Operations monitoring store 6900 and the stash, with Pbar stacking, and with NuMI and MiniBooNE taking beam.

At 12:17 AM, LRF2 tripped off.

Operations terminated store 6900 at 4:48 AM.

Operations put the TeV into shot setup at 5:01 AM.

Operations established store 6901 at 6:41 AM with an initial luminosity of 296.42E30.

At 7:30 AM, the 400 MeV area beam permit tripped off due to bad beam positions for the first of three times in 19 minutes.

Operations reported at 8 AM that the Recycler had stashed 10 times during the last 12 hours.

The Plans for Monday

The plans for today are to stack, stash, and store. The Run Coordinator will allow TeV experts to conduct an end-of-store crystal collimator study.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.