Wednesday, July 15, 2015
Top Links

Labwide calendar

Fermilab at Work

Wilson Hall Cafe menu

Chez Leon menu

Weather at Fermilab


Commercializing innovation: office hours at IARC - today

English country dancing at Kuhn Barn - July 26

Deadline for the University of Chicago tuition remission program - Aug. 18

Call for proposals: URA Visiting Scholars Program - deadline is Aug. 31

Fermilab prairie plant survey

Users Center entrance repair on Sauk Blvd in the Village

Pool memberships on sale

Fermilab Board Game Guild

Fermilab Softball League

Outdoor soccer

Bristol Renaissance Faire employee discount

Raging Waves Waterpark employee discount


Fermilab Today

Director's Corner

Frontier Science Result

Physics in a Nutshell

Tip of the Week

User University Profiles

Related content


Fermilab Today
is online at:

Send comments and suggestions to:

Visit the Fermilab
home page

Unsubscribe from Fermilab Today


Mu2e's opportunistic run on the Open Science Grid

To conduct full event simulations, Mu2e requires time on more than one computing grid. This graphic shows, by the size of each area, the fraction of the recent Mu2e simulation production through Fermigrid, the University of Nebraska, CMS computing at Caltech, MIT and Fermilab, the ATLAS Midwest and Michigan Tier-2s (MWT2 and AGLT2), Syracuse University (SU-OSG). and other sites — all accessed through the Open Science Grid.

Scientists in Fermilab's Mu2e collaboration are facing a challenging task: In order to get DOE approval to build their experiment and take data, they must scale up the simulations used to design their detector.

Their aim is to complete this simulation campaign, as they call it, in time for the next DOE critical-decision review, which Mu2e hopes will give the green light to proceed with experiment construction and data taking. The team estimated that they would need the computing capacity of about 4,000 CPUs for four to five months (followed by a much smaller need for the rest of the year). Because of the large size of the campaign and the limited computing resources at Fermilab, which are shared among all the lab's experiments, the Mu2e team adapted their workflow and data management systems to run a majority of the simulations at sites other than Fermilab. They then ran simulations across the Open Science Grid using distributed high-throughput computer facilities.

Mu2e scientist Andrei Gaponenko explained that last year, Mu2e used more than their allocation of computing by using any and all available CPU cycles not used by other experiments locally on FermiGrid. The experiment decided to continue this concept on the Open Science Grid, or OSG, by running "opportunistically" on as many available remote computing resources as possible.

"There were some technical hurdles to overcome," Gaponenko said. Not only did the scripts have to be able to see the Mu2e software, but also all of the remote sites — more than 25 — had to be able to run this software, which was originally installed at Fermilab. Further, the local operating system software needed to be compatible.

"A lot of people worked very hard to make this possible," he said. Members of the OSG Production Support team helped support the endeavor — getting Mu2e authorized to run at the remote sites and helping debug problems with the job processing and data handling. Members of the Scientific Computing Division supported the experiment's underlying scripts, software and data management tools

The move to use OSG proved valuable, even with the inevitable hurdles of starting something new.

"As Mu2e experimenters, we are pilot users on OSG, and we are grabbing cycles opportunistically whenever we can. We had issues, but we solved them," said Rob Kutschke, Mu2e analysis coordinator. "While we did not expect things to work perfectly the first time, very quickly we were able to get many hundreds of thousands of CPU hours per day."

Ray Culbertson, Mu2e production coordinator, agreed.

"We exceeded our baseline goals, met the stretch goals and will continue to maintain schedule," Culbertson said.

Ken Herner, a member of the support team in the Scientific Computing Division that helped the experimenters port their applications to OSG, hopes that Mu2e will serve as an example for more experiments that currently conduct their event processing computing locally at Fermilab.

"The important thing is demonstrating to other experiments here that it can work and it can work really well," Herner said. "Ideally, this sort of running should become the norm. What you really want is to just submit the job, and if it runs on site, great. And if it runs off site, great — just give me as many resources as possible."

Hanah Chang

In the News

Record numbers at Neutrino Day

From Sanford Underground Research Facility's Deep Thoughts, July 13, 2015

As the temperature soared Saturday, so, too, did the number of visitors attending Neutrino Day activities in Lead. As Sanford Lab celebrated 50 years of science, more than 1,800 people attended activities and presentations, shattering the record of 1,100 set two years ago.

Many started the day at the lab with a hoistroom tour, one of the most popular activities of the day. Visitors also saw "wild science" experiments with South Dakota Public Broadcasting's Steve Rokusek, watched demonstrations of a solar powered water pump, learned about water treatment, looked at sunspots using a solar telescope, and got a taste of the Majorana experiment through the "glove box" demonstration.

Read more

From the Technical Division

A lesson in tolerance

David Harding

David Harding, interim head of the Technical Division, wrote this column.

How good is good enough? Suppose you are baking a cake from a prize-winning recipe that calls for a cup of sugar. Will the cake still be superb if you put in an extra teaspoon of sugar or leave out a teaspoon? What if you are off in your measuring by a tablespoon? How about a quarter cup? Perhaps the recipe calls for two large eggs. Surely there will be a difference if you use one egg or three eggs, but what happens if you use medium eggs or extra-large eggs?

In technical jargon, we call the allowable variation from the planned quantity the tolerance.

If you want one part of a magnet to fit into another nicely, not too tight and not too loose, their sizes must agree. It is important to specify how far from the drawing dimension the parts can vary while still fitting acceptably. If you specify your tolerances too loosely, you run the risk of not being able to put the magnet together. If you specify the tolerances too tightly, you run the risk of paying far more than you need to because it generally takes more time and is more expensive to get the parts closer to the drawing dimensions. We use a system called geometric dimensioning and tolerancing in our drawings to concisely convey our requirements to the machine shops that make our parts.

The quality of the magnetic field in a conventional magnet depends predominantly on the precision with which the iron is shaped, forming a "gap" through which the particle beam travels. When we in Technical Division build a magnet with a 2-inch gap, we work with Accelerator Division scientists to decide how close we need to come to 2 inches to meet the magnetic field needs of the accelerator. Typically AD tells us that the accelerator will work well only if we can keep the variation in the gap to under two ten-thousandths of an inch, or about 1/20 the thickness of a piece of paper. That's in a magnet that may weigh 20 tons or more. In your cake you would be asked to measure the sugar to 1/208 of a teaspoon.

We consider our tolerances carefully to achieve the accelerator goals while minimizing the costs. We balance the desire for perfection in the magnets with the reality of the imperfect beams.

We have also learned the value of checking parts as they come in to ensure that they do, indeed, meet our tolerances. Trying to save time by skipping that incoming inspection leads to greater delays when out-of-spec parts do not fit together during magnet assembly.

Tolerance is also an attitude that must be practiced at an appropriate level so that the pieces of the laboratory fit together. Setting limits on behavior can be harder than setting limits on sugar measurements or magnet steel dimensions. We tolerate differences of opinion but not offensive behavior. For a healthy lab we must maintain respect for each other.

Photo of the Day

Under the capacitor tree

You don't often see this side of the capacitor tree at Master Substation. Photo: Mark Kaletka, CCD

In memoriam: Joyce Ludwig Rossi

Fermilab retiree Joyce Ludwig Rossi, 81, of Overland Park, Kansas, passed away on June 30. She was an employee in the Particle Physics Division Support Services Department from 1992 to 1997, when she retired.

A memorial mass will be held Friday, July 17, at 10 a.m. at St. Margaret Mary Church, 1450 Green Trails Drive, Naperville, Illinois. A private burial will be held at Saints Peter and Paul Cemetery.

Donations can be made to Sisters, Servants of Mary, 800 N 18th Street, Kansas City KS 66102 or to Kansas City Hospice House, 12000 Wornall Road, Kansas City MO 64145.

Read Rossi's obituary.

Safety Update

ESH&Q weekly report, July 14

This week's safety report, compiled by the Fermilab ESH&Q Section, contains two incidents.

A visitor saw many insects while cleaning at SciBooNE. That evening he noticed swelling on his left forearm that worsened through the evening.

An employee woke in the morning with knee pain after using a floor jack to maneuver a loaded pallet the day prior. This will likely be a DART case.

See the full report.