Accelerator Update for July 31, 2006 - August 2, 2006

Monday July 31

The day shift began with Operations monitoring store 4862 and a Recycler stash, with the Collider Detector at Fermilab (CDF) and D-Zero (D0) using the colliding beam, with the Antiproton Source (Pbar) off due to problems with A:IB, with the Linac, Booster, Main Injector (MI), and Recycler operating normally, and with NuMI and MiniBooNE taking beam.

At 8:23 AM, a Linac quadrupole power supply (L:QPS105) tripped off twice in eight minutes. It reset both times.

At 9:05 AM, Linac technicians swapped the control cards between QPS103 and QPS105. QPS103 went out of tolerance and had to be adjusted. At 9:11 AM, the cards were swapped back and QPS105 went out of tolerance. A new card was installed in QPS105 at 9:44 AM.

The power supply expert working on the Accumulator power supply A:IB reported that a cable from the transformer was bad. He had A:IB on by 10:29 AM.

Pbar tried to resume stacking at 11:10 AM, but an RF station (ARF1) wasn’t ramping. At 11:24 AM, problems with BPMs arose. At 11:38 AM, Operations reported that Pbar was stacking, but at a poor rate due to a problem with the control program (Overthruster).

Experts began tuning Booster and MI for higher intensities at 12:09 AM.

FESS reported at 1 PM that divers resumed cleaning the A1 pond strainer. They had the job finished thirty minutes later.

At 1:12 PM, all the Booster RF station began sparking due to the tuning and BRF3 & 10 tripped off. The expert put back his most recent changes.

At 2:11 PM, cryo system techs at CHL reported that the ICW pressure was low. They contacted FESS.

At 2:24 PM, Operations reported that a B0 compressor had tripped off. CHL was aware of the problem. They have 26 compressors running.

At 2:40 PM, Operations reported a problem with a MI-10 CAMAC crate; corrector magnet power supplies were not ramping. This caused problems with the MI orbit and with NuMI beam. Controls experts replaced a card at 2:54 PM.

At 2:55 PM, Operations reported that the Recycler lost part of its stash due to over cooling during a study.

At 4:55 PM, BRF13 & 14 tripped off due to a vacuum burst. Everything reset.

At 5:21 PM, Operations reported that the Tevatron (TeV) sector B0 electronics room temperature was rising. FESS personnel placed a portable cooler in the room and the temperature began to drop.

Operations began preparing the Recycler at 6:17 PM to stash antiprotons.

At 6:53 PM, CDF reported that they had to turn off their detector due to the heat. (Much later, investigators found that a valve had been opened to divert pond water back to the pond. This valve was closed and cooling improved.)

At 7:02 PM, Operations reported that not all the beam was getting extracted from MI to the Recycler. Operators contacted a MI expert who said there was a large phase offset between Pbar and MI. The stash ended.

A Pbar expert came in at 9:44 PM to prepare for the next stash.

The Recycler began stashing at 11:01 PM. The transfer looked much better. The stash ended and Pbar resumed stacking at 11:18 PM.

Tuesday August 1

The midnight shift began with Operations monitoring store 4862 and the stash, with Pbar stacking, and with NuMI and MiniBooNE taking beam.

At 12:54 AM, many of the Pbar devices stopped reporting. Operations rebooted AP1001 (a network switch) and this brought back A:IB and a couple more devices, but there were still problems and many alarms. Operations contacted a controls expert.

At 2 AM, Operations reported that all of the Main Control Room consoles seemed to be failing.

The controls expert reported at 2:02 AM that he traced the Pbar problems to a switch in AP-10. An operator power cycled the switch. Everything came back except for the Pbar flying wire, which will get fixed during the day shift.

The Recycler began preparing to stash at 3:26 AM. Problems with GPIBs, a Pbar beam permit trip, and with the reverse protons not making it into the Accumulator caused Operations to halt the stash and resume stacking.

Operations attempted at 4:26 AM to prepare the Recycler to stash. The Pbar beam permit tripped again and the reverse protons never made it. Operators contacted an expert.

Operations terminated store 4862 at 5:02 AM.

CDF and D0 were ready for access by 5:19 AM.

At 5:28 AM, a Recycler expert noticed that two power settings were wrong. Operators set them back to a previously used setting and the transfer went from Pbar to MI, but then fell out of MI. The investigation continued.

At 5:28 AM, the Accumulator IKIK injector shutter was off, which halted stacking.

Pbar resumed stacking at 5:41 AM, but poorly.

A Pbar expert suggested at 6:20 AM that Operations run the stacking aggregates again. This worked, but then they restored a file and the extraction and injection shutters alarmed. Operators restored a different file at 6:31 AM and Pbar resumed stacking.

D0 came out of access at 6:54 AM.

Earlier, a TeV expert came in to look at a kicker (F17K2) problem. At 7:02 AM, he reported that the problem was in the tunnel. Operators began preparing the TeV at 7:10 AM for access.

CDF came out of access at 7:59 AM.

A Pbar expert reported at 8:11 AM that he suspected H926 was the cause of the shot problems. Another expert will investigate further.

D0 accessed their collision hall at 8:20 AM.

The TeV experts came out of F sector at 9:44 AM.

MiniBooNE resumed taking beam at 10:11 AM.

D0 came out of access at 10:16 AM.

NuMI resumed taking beam and Pbar resumed stacking at 10:26 AM.

At 10:30 AM, LAM42 was holding off the MI beam permit.

A power supply expert investigating the MI Lambertson problem (I:LAM42) reported at 12:25 PM that he found a bad magnet Klixon. He bypassed the Klixon and turned on the power supply.

NuMI resumed taking beam and Pbar resumed stacking at 12:28 PM.

A power supply expert reenabled the MI I:LAM42 Klixon at 2:01 PM.

TeV experts began a study period shortly after 3:19 PM.

At 5:03 PM, Operations reported a small glitch; all the Booster RF stations tripped off, including Lam 60 & 61, and ARF1. Everything reset.

The Recycler began preparing to stash at 5:12 PM. The stash ended and Pbar resumed stacking at 5:54 PM.

Operation put the TeV into shot setup at 7:27 PM.

Operations established store 4868 at 9:08 PM with an initial luminosity of 120.16E30.

The Recycler began preparing to stash at 9:52 PM. The stash ended and Pbar resumed stacking at 10:24 PM.

Wednesday August 2

The midnight shift began with Operations monitoring store 4868 and the stash, with Pbar stacking, and with NuMI and MiniBooNE taking beam.

At 2:06 AM, the H- Source tripped off due to vacuum pumps tripping off. Operations reported at 2:25 AM that the vacuum was not recovering. Operations called in an expert.

Operations reported at 2:54 AM that a stash, which had started earlier, ended and stacking had resumed.

At 3:06 AM, the Klystron Debuncher tripped off and would not reset. Operators called in experts.

Linac experts repaired the Klystron Debuncher and got the H- Source running, though at a limited capacity. Beam resumed at 7:26 AM.

Stacking resumed at 7:30 AM.

The Plans for Wednesday

The plans for today are to stack, stash, and store.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.