ASCR Monthly Computing News Report - May 2010



DOE Applied Math Research Meeting Draws 225 Attendees
The 2010 meeting of the Applied Math Research program, held May 3-5 in Berkeley, CA, drew 225 attendees from national labs and other research organizations, including 81 researchers from universities. The conference program included nine plenary talks, 48 contributed talks and 83 posters presented in two afternoon sessions. The previous meeting, held in October 2008 at Argonne National Laboratory, drew 147 attendees. The growth in the size of the conference reflects the overall growth in the Applied Math Research program, according to Program Manager Sandy Landsberg. The meeting was chaired by Esmond Ng, leader of the Scientific Computing Group at Lawrence Berkeley National Laboratory.
Argonne Simulation of Gulf Oil Spill Featured on NBC Nightly News
A simulation produced by senior computational scientist Paul Fischer of Argonne National Laboratory and his collaborator Tamay Ozgokmen at the University of Miami dramatically depicts the plumes of the miles-long BP oil spill. Scientists are using such 3D simulations to gain insight into how ocean turbulence may affect the movement of the spill.
The simulation was shown on NBC Nightly News May 17 and can be seen on the web. The dark blue shows the 3D oil plume, and green-red colors represent local temperatures indicative of the oceanic surface movements. Researchers are carefully tracking the depth, direction, and speed of the spill, as concerns grow that it will join with a warm loop current that could bring it up the U.S. east coast. The model may help answer questions regarding the expected lifetime of the plume, once the oil flow is shut off. The simulation considers physical phenomena such as buoyancy of the oil, rotation of the Earth, and ocean currents.
The simulation was based on computations performed on Argonne’s Blue Gene/P using the 3D nonhydrostatic spectral element code Nek5000. This powerful code combines the geometrical flexibility of the finite-element method with the numerical accuracy of the spectral method. Fischer and Aleks Obabko, an assistant computational scientist at Argonne, have already started higher resolution runs (n = 8192 x 113 gridpoints, vs the original n = 8192 x 73) and will be doing successively higher runs over the week—all on BG/P. With the help of Hank Childs at Lawrence Berkeley National Laboratory (LBNL), they are also rendering new visualizations on Argonne’s Eureka system using VisIt, a parallel visualization and analysis tool that was developed at Lawrence Livermore National Laboratory (LLNL).
The simulation of the oil spill is part of a long-term National Science Foundation (NSF)-funded collaboration involving Argonne, the University of Miami, Illinois Institute of Technology, and Virginia Tech to study ocean currents. According to Fischer, developer of Nek5000, DOE’s role in the collaborative project has been in the agency’s continued support of Nek5000 development on large-scale computing platforms.
Simulations Run at NERSC Reveal That Earth’s Silica Is Predominantly Superficial
Silica is one of the most common minerals on Earth. Not only does it make up two-thirds of our planet’s crust, it is also used to create a variety of materials from glass to ceramics, computer chips and fiber optic cables. Yet new quantum mechanics results generated by a team of physicists from Ohio State University (OSU) show that this mineral only populates our planet superficially—in other words, silica is relatively uncommon deep within the Earth.
Using several of the largest supercomputers in the nation, including the National Energy Research Scientific Computing Center’s (NERSC) Cray XT4 “Franklin” system, the team simulated the behavior of silica in high-temperature, high-pressure environments that are particularly difficult to study in a lab. These details may one day help scientists predict complex geological processes like earthquakes and volcanic eruptions. Their results were published in the May 10 online early edition of the Proceedings of the National Academy of Sciences
LBNL Researchers to Present Three Papers at International Combustion Symposium
Researchers in Berkeley Lab’s Center for Computational Sciences and Engineering will present three papers and co-authored a fourth at the 33rd International Combustion Symposium to be held Aug. 1-6 at Tsinghua University in Beijing, China. Sponsored by the Combustion Institute, the International Symposium on Combustion is held every two years and attracts members of the Combustion Institute as well as others interested in combustion from around the world. The synergism produced at these symposia makes them the principal forum for presenting and integrating combustion research results. LBNL papers are:
  • “Characterization of Low Lewis Number Flames” by Andrew Aspden, Marcus Day and John Bell
  • “Lewis Number Effects in Distributed Flames” by Andrew Aspden, Marcus Day and John Bell
  • “Numerical Simulation of Nitrogen Oxide Formation in Lean Premixed Turbulent Flames” by Marcus Day, John Bell, and Xinfeng Gao, with Peter Glarbor of the Technical University of Denmark.
PNNL Scaling Goes eXtreme, Reaches 34,000 Cores
Following PNNL and ORNL’s petascale demonstration using Global Arrays that performed at 1.3 petaflops and used more than 200,000 processors, researchers at Pacific Northwest National Laboratory (PNNL) have demonstrated the scalability of high-level coupled-cluster approaches for excited states, and have reached 34,000 processor cores for the first time. The excited-state coupled cluster implementation demonstration was also based on the Global Arrays programming model; however this research is targeting software that is capable of describing the behavior of molecules in excited states, as well as simulating their dynamics. Additionally, the research is focusing on integrated multiscale approaches that can be used to model chemical processes in realistic settings and that can capitalize on having highly scalable implementations of electronic structure methodologies. This will allow high-level description of large molecular systems in realistic settings defined by finite temperatures and pressures.
Progress in this field will significantly enhance the systems-size limit that can be managed by computer simulations and will set new standards for accuracies attainable in molecular simulations. In particular, researchers will use new, highly scalable codes to describe the energy conversion in light harvesting (photosynthesis) molecular systems and to simulate the structure, dynamics, and reactions at the mineral (Fe2O3)/solution interface as a function of pressure and temperature. At the completion of the project, researchers expect to have a suite of massively parallel tools to perform excited-state calculations for molecular systems composed of hundreds of atoms and new algorithms to perform dynamic simulations for much longer propagation times. This research is supported through PNNL’s eXtreme Scale Computing Initiative.
Contact: Mary Anne Wuennecke,
Sandia and LANL Researchers Explore Unconventional Uses of Optimization
An ongoing collaboration between P. Bochev, D. Ridzal, and G. Scovazzi of Sandia National Laboratories and M. Shashkov of Los Alamos National Laboratory (LANL) explores novel uses of optimization and control ideas in computational modeling. The main thrust of this research is to restore physical properties such as monotonicity and positivity that may be lost during the discretization, by reformulation of the problem into an optimization problem. This approach has been applied to develop a new mathematical framework for remap, based on ideas from constrained optimization. Compared to the state of the art, the new optimization-based remap (OBR) can generate significantly more accurate solutions at or below the cost of the best currently used remapping techniques. The OBR was presented at the DOE ASCR Applied Mathematics Research PI Meeting (Berkeley, May 2010), and the NNSA BER Meeting on Atmospheric Transport (Sandia NM, March 2010). A technical report (SAND2010-3021) documenting the new approach is available upon request. The research is partially funded by Bochev and Shashkov’s ASCR projects on advanced discretizations and mimetic methods, respectively.
Contact: Pavel Bochev,
LBNL’s Work on Digital Watershed Highlighted in Video by Microsoft Research
Every year, Microsoft Research organizes the Silicon Valley TechFair to highlight the company’s work with the research community. At this year’s event on May 6, the Berkeley Water Center was highlighted both in the opening address and in a video produced for the event. The video focuses on the “digital watershed,” a project between Microsoft Research, Berkeley Lab’s Advanced Computing for Science (ACS) Department, and UC Berkeley to develop better tools for storing and accessing data on water resources. Watch the video, which features ACS Department Head Deb Agarwal.


Four NERSC Users Elected to the American Academy of Arts and Sciences
NERSC users Andrea Bertozzi (UCLA), Adam Burrows (Princeton), Gary Glatzmaier (UC Santa Cruz), and John Wilkins (Ohio State University) are among the 229 leaders in the sciences, social sciences, humanities, arts, business and public affairs who have been elected members of the American Academy of Arts and Sciences. Founded in 1780 by John Adams, James Bowdoin, John Hancock, and other scholar-patriots, the Academy is an independent policy research center that conducts multidisciplinary studies of complex and emerging problems.
Click here for a list of the 2010 fellows.
Victor Markowitz Appointed JGI CIO and Associate Director
Berkeley Lab’s Victor Markowitz is the new Chief Informatics Officer and Associate Director at DOE’s Joint Genome Institute (JGI). In this new role, Markowitz will provide vision and strategic leadership to establish and maintain leading-edge information technology and informatics capabilities for the JGI and DOE partners. He will serve as a member of the JGI senior management team and play a key role in advising the facility’s director on the appropriate direction and formulation of the JGI’s technology infrastructure. He will develop, direct, and evaluate technology strategies including designing and recommending appropriate technological solutions to support the Institute as well as provide a vision and direction for information technology and its linkage to external activities and the scientific community. In addition to his new roles, Markowitz will continue to head the Berkeley Lab Computational Research Division’s Biological Data Management and Technology Center (BDMTC), which has been engaged in partnership with JGI scientists in the development of microbial genome and metagenome data management and analysis systems.
Contact: Linda Vu,
Argonne Researcher Sven Leyffer Named SIAM Vice President for Programs
Sven Leyffer, a computational mathematician in Argonne’s Mathematics and Computer Science Division, has been named Vice President for Programs for the Society for Industrial and Applied Mathematics (SIAM). The organization includes more than 12,000 members in 85 countries who are working in industry, government, and academia.
In this three-year position, Leyffer will chair SIAM’s program committee, with responsibility for the overall program direction and content of all SIAM conferences, including soliciting and reviewing topics for conferences and initiating special workshops in new areas. He will also be responsible for reporting to the SIAM Board of Trustees and SIAM Council about the status of meetings and their budgets.
Leyffer was named a Fellow of SIAM in 2009. He is developing and applying nonlinear optimization methodologies to emerging areas such as mixed-integer nonlinear optimization and optimization problems with complementary constraints. He is also a Fellow of the Argonne/University of Chicago Computation Institute.
PNNL Strengthens Computational Science Expertise with New Staff
Computational sciences and mathematics play an integral role in virtually every type of scientific research being conducted today. PNNL has added three new staff members who will strengthen the organization’s capabilities in data intensive and high performance computing. Adolfy Hoisie, Darren Kerbyson and Kevin Barker joined PNNL’s Computational Sciences and Mathematics Division, which focuses on advancing the frontiers of computational sciences and merging high-performance computing with data-centric analysis capabilities to solve significant problems in energy, the environment, and national security. Hoisie and Kerbyson were appointed Laboratory Fellows, and Hoisie is also the Associate Division Director for High Performance Computing as well as the director of the Institute for Exascale and Data Analytics Computing. Kerbyson is the chief scientist of PNNL’s Extreme Scale Computing Initiative and is leading fundamental research in the new Center. Barker is a senior research scientist and will conduct research for the Center.
Contact: Mary Anne Wuennecke,
LBNL’s Michael Wehner Named Lead Author of IPCC Climate Change Report Chapter
Michael Wehner, a climate researcher in Berkeley Lab’s Computational Research Division who specializes in extreme weather changes, has been selected as a lead author of Chapter 12, “Long-Term Climate Change: Projections, Commitments and Irreversibility,” in the Working Group I contribution to the Fifth Assessment Report (AR5) of the Intergovernmental Panel on Climate Change (IPCC). Wehner’s selection was approved during the May 19-20 meeting of the Bureau of the Intergovernmental Panel on Climate Change in Geneva. Wehner was a contributing author of the IPCC AR4 report, which was honored with the 2007 Nobel Peace Prize.
Contact: Jon Bashor,


Fourth IESP Workshop Builds on Exascale Software Roadmap
The fourth International Exascale Software Project (IESP) Workshop, held April 13-14 at the University of Oxford, UK, extended the exascale software technology roadmap that will lay the strategy for developing the next-generation, open source software for scientific high performance computing (HPC). The conference brought together members from industry, academia, and government with expertise in a range of essential scientific domains to discuss the future of software needed to calculate at exaflops speed —or a million trillion calculations per second.
The IESP aims to advance the world’s simulation and modeling capability by improving the coordination and development of the HPC software environment. Previously, three IESP workshops were held in Santa Fe, New Mexico; Paris, France; and Tsukuba, Japan, in April, June, and October 2009, respectively.
Goals for the April workshop included the following:

  • Develop a governance structure and management model for IESP;
  • Explore how laboratories, universities, and vendors can work together on coordinated HPC software;
  • Refine the roadmap for software and algorithms on extreme-scale systems;
  • Develop a prioritized list of software components for exascale computing as outlined in the roadmap;
  • Assess the short-term, medium-term, and long-term software and algorithm needs of applications for peta/exascale systems.
At the workshop, Argonne Leadership Computing Facility Director Pete Beckman and Jack Dongarra, University Distinguished Professor of Computer Science, University of Tennessee and Distinguished Research Staff member, Oak Ridge National Laboratory, gave an introduction about IESP and outlined the meeting’s objectives. Rick Stevens, Associate Laboratory Director, Computing, Environment, and Life Sciences, Argonne National Laboratory, presented the Department of Energy’s (DOE) Laboratory plan for providing exascale applications and technologies for critical DOE mission needs. He also discussed the research needed to achieve exascale performance.
Four breakout groups — led by Satoshi Matsuoka (Titech) and Michael Heroux (Sandia), Peg Williams (Cray) and Bill Kramer (NCSA), Ed Seidel (NSF) and Konstantinos Glinos (European Commission), David Keyes (KAUST) and Jean-Yves Berthou (EDF) — focused on technical, computational, economic, and management challenges and needs presented by the IESP and the role of national and international funding agencies. ORNL staff members in attendance were Deputy Director of Science and Technology Thomas Zacharia, Associate Laboratory Director for the Center for Computational Sciences Jeffrey Nichols, Computer Science and Mathematics Division Director Arthur Bernard MacCabe, and Computational Chemistry Group Leader Robert Harrison. David Skinner, head of the SciDAC Outreach Center, represented Berkeley Lab and NERSC.
Berkeley Lab Scientists Build Software Framework for ATLAS Collaboration
Three thousand researchers in 37 countries are searching for the origins of mass, new dimensions of space and undiscovered forces of physics in the head-on collisions of high-energy protons at the Large Hadron Collider’s ATLAS experiment. When ATLAS is turned-on, its detectors record about 400 collision events per second from a variety of perspectives, a rate equivalent to filling 27 compact disks per minute.
In order to sift out signs of new physics in this torrent of data, thousands of researchers must be able to process this information and collaborate on results in real time. To facilitate this distributed workflow, they are relying on a software framework called Athena, which was developed by an international team of scientists led by Paolo Calafiura of the Advanced Computing for Science Department in the Lawrence Berkeley National Laboratory’s (Berkeley Lab) Computational Research Division.


ALCF and OLCF Collaborate on Webinar to Help Prospective INCITE Researchers
To help researchers submit their best proposals, an INCITE 2011 Proposal Writing Webinar was offered on May 17 at Argonne National Laboratory. A total of 53 attended the webinar: 38 via the web application and 15 in person. James Osborn, Computational Scientist at the Argonne Leadership Computing Facility (ALCF), and Bronson Messer, computational astrophysicist in the Scientific Computing Group at the Oak Ridge Leadership Computing Facility (OLCF) at Oak Ridge National Laboratory, provided a wealth of tips to improve the quality of INCITE proposal submissions. They discussed critical questions to address when preparing an INCITE proposal, outlined the proposal form, explained the proposal review process, and cited PI obligations, among other key topics.
One attendee remarked, “I am starting to prepare the INCITE proposal to be submitted next month. This webinar was very helpful and will save me time in preparing the proposal.”
For those who did not attend the webinar but would like to obtain the information, the slides from the webinar can be downloaded from the ALCF website at the following link: Over one billion processing hours are available through the INCITE program for 2011. The webinar proved timely for researchers as proposals are due by June 30.
For more information about the INCITE program, see or contact the INCITE manager at
ALCF Leap to Petascale Workshop Provides Hands-on Help
Abundant hands-on help was available to 17 attendees representing 14 different projects at the Leap to Petascale Workshop on May 18-20 at Argonne National Laboratory. Sponsored by the Argonne Leadership Computing Facility (ALCF), the workshop was geared to current INCITE and Discretionary project teams that have scaled to at least two racks on Intrepid, the ALCF’s Blue Gene/P supercomputer. Primarily, the workshop was devoted to tuning applications to scale up to 40 racks on Intrepid. ALCF performance engineers and computational scientists, along with tool and debugger developers, aided the attendees in their applications.
Contact: Chel Lancaster,
OLCF Hosts Spring Hex-Core Workshop, User’s Meeting
A workshop was held at ORNL May 10–12 to help new and returning users acclimate to Jaguar, OLCF’s Cray XT5. The first day of the workshop was broken into two tracks—Track I for new XT5 users, and Track II for more experienced users. The workshop included lectures by staff members of OLCF, Cray, and PGI (provider of compilers and tools for high-performance computers) on such topics as XT5 architecture and programming, AMD six-core processors, and parallel debugging tools. Daily hands-on sessions allowed participants to access Jaguar using their own codes while working one-on-one with OLCF staff members to resolve any issues.
The OLCF Users’ Meeting was held on the final day of the hex-core workshop specifically for research groups that have been granted allocations on Jaguar through the INCITE program. OLCF Users’ Council also convened on this day to elect Balint Joo, from Jefferson Lab, as chair. Joo will function as the liaison between users and the OLCF management team.
Contact: Jayson Hines,
NERSC Staff Share Expertise in Talks at Cray Users’ Meeting
NERSC staff members presented five talks at the Cray Users Group (CUG) meeting held May 24-27 in Edinburgh, Scotland. Additionally, NERSC’s Nick Cardo, who is also CUG vice president, served as program chair for the meeting. Jim Craw, head of NERSC’s Computational Systems Group, chaired the XTreme special interest group sessions at the meeting. Here are the talks by NERSC staff:
  • “Application Acceleration on Current and Future Cray Platforms,” Alice Koniges with Aaron Fisher, Nathan Masters, and Mlaker Velimir, LLNL
  • “Franklin Job Completion Analysis,” Yun (Helen) He, Hwa-Chun (Wendy) Lin, and Woo-Sun Yang
  • “File System Monitoring as a Window Into User I/O Requirements,” Andrew Uselton with Katie Antypas, NERSC; Daniela Ushizima, LBNL; and Jefferey Sukharev, University of California, Davis
  • “External Services on the Cray XT5 System Hopper,” Jonathan Carter with Katie Antypas and Tina Butler
  • “Analyzing the Effect of Different Programming Models upon Performance and Memory Usage on Cray XT5 Platforms,” Alice Koniges with John Shalf, Hongzhang Shan, and Nick Wright, NERSC; Haoqiang Jin, NAS Systems Division; and Seung-Jai Min, LBNL
OLCF Hosts Visualization Software Course
On May 13, the OLCF sponsored a day-long course on VisIt, a software application for analyzing and visualizing very large simulation datasets. “VisIt is unique in that it can handle the largest datasets that users throw at it,” said Sean Ahern, OLCF visualization task leader. “Everything researchers do on a small scale can also be done on huge petascale data sets, and that’s a very important [software application] for us to be supporting, maintaining, and training users on.”
The course included a general overview of how to use VisIt to access data and create visualizations, detailed discussions about how and when to use VisIt’s various plots and operators, how to visualize a subset of a database, and how to create derived variables using expressions. The day concluded with lectures about producing presentation-quality graphics and movies through mastery of session files, color tables, annotations, lighting, and scripting. The course alternated between lecture discussions and hands-on exercises and provided an interactive setting in which to explore the software.
Contact: Jayson Hines,
OLCF Hosts Students, Teachers for National Lab Day
On May 6, ORNL hosted students and science teachers from six area middle schools as they explored real-world science as a part of National Lab Day. Approximately 120 students from middle schools across the area visited the Spallation Neutron Source—the most intense pulsed neutron beams in the world—as well as the OLCF. At the OLCF, students were treated to a presentation of visualizations on the EVEREST PowerWall, a large-scale venue for data exploration and analysis. Visualizations included images from global population distributions, climate science, and astrophysics projects run on Jaguar, the world’s fastest supercomputer. Students also learned about superhydrophobics, forensic anthropology, and electric cars.
“We wanted to show students that science is not abstract, describe to them that there are people here using computers to answers scientific questions, and that’s just one means of exploration,” said OLCF visualization expert Ross Toedte. “There are experiments in labs, people taking measurements in the field, and people developing theories about physics and chemistry. These are all ways of exploring science, and they all synergistically support each other.”
Contact: Jayson Hines,