How to run a million jobs
At SC08, several experts organized an informal session to share information on up-and-coming solutions for expressing, managing, and executing "megajobs." They also discussed ways of repackaging work to avoid megajobs altogether.
Here iSGTW shares the latest ideas and developments about megajobs with its readers, and plans to follow up with articles on various mentioned technologies and trends in the coming months.
Biting off a megajob-it's a lot to chew
As large systems surpass 200,000 processors, more scientists are running "megajobs", thousands to millions of identical or very similar, but independent, jobs executed on separate processors. From biology, physics, chemistry and mathematics to genetics, mechanical engineering, economics and computational finance, researchers want an easy way to specify and manage many jobs, arrange inputs, and aggregate outputs. They want to readily identify successful and failed jobs, repair failures, and get on with the business of research. System administrators need effective ways to process large numbers of jobs for multiple users.