After nearly a month with repair crews working 20-hour days, improvising strategies like a line-up of off-the-shelf industrial-strength shop-vacs to remove water and clear the cooling nozzles, the NuMI focusing horn is back in operation and delivering neutrinos to the MINOS experiment when proton beam from the Main Injector is available. "We believe it's now essentially clear, and it appears to be running fine," said Accelerator Division engineer Kris Anderson, who directed the effort. "It takes well over a year to build one, and I'm glad we were able to fix it without breaking it."
The NuMI horn focuses short-lived particles called pions, which are generated by protons hitting a target and decay to neutrinos (among other particles). The focusing is achieved using intense magnetic fields generated by 200,000-ampere pulses of electric current. In addition to neutrinos, the particles and electric current deliver enormous amounts of heat. Without 36 gallons per minute of low-conductivity (or de-ionized) water for cooling, the horn would burn itself up. At 4:40 p.m. on Friday, June 30, with one foot out the door before the extended July 4 holiday weekend, Anderson heard the last thing he wanted to hear just then: voices on the phone telling him there was no water flowing to the horn's cooling nozzles. Since the voices belonged to experts, Jim Hylen and Tom Kobilarcik, Anderson knew there was no room for doubt.
Tiny resin beads, about 20 thousandths of an inch (half a millimeter) thick, were clogging the nozzles. The beads, which are sensational at plugging leaks, had moved the wrong way through the de-ionized water system loop due to a failed check valve during maintenance on June 30. About a gallon of the beads had slipped through to the horn cooling nozzle circuit, congregating at the water nozzles. When the resin beads are wet, they get very sticky. At the nozzles, the wet and sticky beads did what they do best: they stopped up the holes and kept the water from going through.
After determining the problem, by viewing beads in the flow meters where there never should be beads, Anderson, Hylen, Kobilarcik and several repair teams met on Saturday morning to plan strategy. It was the beginning of more than three weeks of 20-hour days for two crews of technicians, plus intensive time for other AD-Mechanical Support and Radiation Safety personnel. Using prototype horn components in a test setup, they experimented by packing in fresh resin beads, then trying to flush them out with reverse-flow water pressure. It seemed to work, but there were concerns about the real horn standing up to the pressure.
"The problem is that the horn inner conductor is only 2 mm thick, or 80 thousandths of an inch," said Anderson, whose connection to the horn extends back to R&D days. "So if one wants to use reverse flow, you have to fill the entire horn volume with water to create a pressure drop across the nozzles. If the pressure is too high or excessive pressure waves are generated, you could crush the inner conductor, like crushing a beer can. We did not want to do that with a million-dollar horn."
Next experiment: vacuum pressure. While sticky when wet, the beads blow away like grains of sand when completely dry. Using two vacuum cleaners in another prototype test setup, crews dried out the water header cavity and then were able to suction out the beads. Anderson had some crew members go out to buy industrial-size vacuum cleaners. Some vacuum cleaners were used to blow air into horn, others to suction out the line; the cumulative effect was pressure of about 8 psi across the cooling nozzles. With four vacuum cleaners used to pressurize the horn volume, and two hooked in series to suction out the nozzle header, the system worked well.
Stress limits were re-calculated, to make sure the horn would withstand the pressure. Then, for the next three-and-a-half weeks, the crews worked steadily with cycles of forward and reverse pressures, valving out the horn, and vacuuming out With 50 nozzles cooling the horn's inner conductor, and 19 in the outer conductor, 69 nozzles had to be cleared. After the first couple vacuuming attempts, not all the water had been removed, and some nozzles were still clogged by sticky beads, so they had to be cleared again. Finally, a desiccant dehumidifier was added to the system; with the input air at less than 1% relative humidity, the rate of drying out water was improved. The horn was finally "buttoned up" on July 26, some 27 straight days of work including the July 4 holiday and weekends.
"The guys were amazing," Anderson said. "Hiep Le's technicians basically worked 80-hour weeks. Bob Slazyk's crew worked on repairs to the the water system skids. A lot of guys pitched in. Radiation Safety people were in the cavern for many hours. Vladimir Sidorov, another Mechanical Support Department engineer, came up with different ways to pressurize and scavenge that were very ingenious."
Even the vacuum cleaners rate special mention.
"They came right off the shelf, and they weren't expensive," Anderson said. "We ran those things 20 hours a day for three weeks, they did the job and none of them burned out. Whoever makes those things has a pretty good product."
--Mike Perricone
