ASCR Monthly Computing News Report - March 2010



NERSC Helps Locate Jupiter’s Missing Neon
It’s raining helium on Jupiter—and as these droplets fall towards the planet’s deeper interior, they are bringing neon down with them. This new result, published in the March 26 issue of Physical Review Letters, solves a 15-year-old mystery that was initiated on December 7, 1995, when NASA’s Galileo probe plunged into Jupiter’s atmosphere and found only one-ninth the amount of neon that should have been there based on measurements of the Sun’s composition. The authors, Burkhard Militzer and Hugh Wilson of the University of California, Berkeley, were able to answer this decade-old question with supercomputing help from the National Energy Research Scientific Computing Center (NERSC).
Since 1995, astronomers have suspected that neon in Jupiter’s interior was dissolving into droplets of condensed helium. However, Wilson says that the UC Berkeley team was the first to actually test the hypothesis. Jupiter’s atmosphere consists of approximately 74 percent hydrogen, 23 percent helium, and less than 3 percent of other elements like carbon, nitrogen, oxygen, neon, etc. In much of the atmosphere, all of these elements are pretty evenly mixed together. However, 10,000 km into the planet’s interior, where temperatures are about 5,000 degrees Celsius and the pressure is roughly 2 million times the atmospheric pressure on Earth, hydrogen turns into a conductive metal and helium condenses into drops. The new calculations show that neon atoms have a strong preference for dissolving into the forming helium droplets.
NERSC, RHIC Help Shed Light on the Nature of Antimatter
Using the NERSC’s Parallel Distributed Systems Facility (PDSF) and the Brookhaven National Laboratory’s Relativistic Heavy Ion Collider (RHIC), physicists have detected and confirmed the first-ever antimatter hypernucleus, called “antihypertriton.” Translated, the newly detected “antihypertriton” means a nucleus of antihydrogen containing one antiproton and one antineutron—plus one heavy relative of the antineutron, an antilambda hyperon.
“STAR is the only experiment that could have found an antimatter hypernucleus,” says Nu Xu of the Lawrence Berkeley National Laboratory’s Nuclear Science Division, the spokesperson for the STAR experiment. “We’ve been looking for them ever since RHIC began operations. The discovery opens the door on new dimensions of antimatter, which will help astrophysicists trace back the story of matter to the very first millionths of a second after the Big Bang”.
ORNL’s Memphis Improves Performance of Major DOE Applications Over 23 Percent
Collin McCurdy and Jeffrey Vetter of Oak Ridge National Laboratory’s (ORNL’s) Future Technologies group have recently developed Memphis, a tool that analyzes memory access patterns in scientific applications on Non-Uniform Memory Access (NUMA) architectures. The authors have been using Memphis to find and fix performance problems in several major DOE applications. These improvements have, so far, led to performance increases on the Cray XT5 at Oak Ridge of 23 percent for runs at scale of XGC1, and of 24 percent and 13 percent for single node runs of CAM and HYCOM, respectively. The results were recently published at the 2010 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) as “Memphis: Finding and Fixing NUMA-related Performance Problems on Multi-core Platforms.”
Not surprisingly, high-end scientific applications have been immune to NUMA problems due to the uniform latency of memory accesses offered by earlier symmetric multiprocessing (SMP) platforms. However, current trends in microprocessor design, including on-chip memory controllers and multi-core processing, are pushing NUMA issues into small-scale systems. Several platforms, such as the AMD Istanbul and Intel Westmere, are currently NUMA between sockets, and upcoming processors will be NUMA within a socket. Memphis uses hardware performance monitoring to pinpoint memory accesses to data arrays that cause NUMA-related performance problems. The team is continuing to analyze and optimize other applications for NUMA performance problems. The paper is available at the FT PUB site.
Contact: Jeffrey Vetter,
SIAM Journal Article on ASCR Math Research Applied to the Power Grid
An article entitled “Optimization Strategies for the Vulnerability Analysis of the Electric Power Grid” by Ali Pinar (Sandia National Laboratories), Juan Meza Lawrence Berkeley National Laboratory), Vaibhav Donde (ABB Corporate Research) and Bernard Lesieutre (University of Wisconsin, Madison) has been published in the SIAM Journal on Optimization (Vol.20, No.4). The article provides mathematical formulations for analyzing vulnerabilities of the power grid, and investigates the connection between the topological structure of the power grid and the nonlinear power flow equations that govern the functioning of the system.
Contact: Ali Pinar,
Computer Scientists, Computational Physicist Collaborate on Scientific Breakthrough
Computer scientists Ewing “Rusty” Lusk (Mathematics and Computer Science Division, Argonne National Laboratory) and Ralph Butler (Middle Tennessee State University) and physicist Steven Pieper (Physics Division, Argonne National Laboratory) have achieved exciting new results in their collaborative work in the DOE SciDAC Universal Nuclear Energy Density Functional (UNEDF) project. A key part of this project has been the development of a new library, called ADLB, that enables the fine-grained parallelization essential for running a critical calculation of carbon-12 — carbon’s most important isotope and a specific target of the UNEDF project — on the Blue Gene/P.
In an article titled “More Scalability, Less Pain,” which appears in SciDAC Review 17, Spring 2010, pp. 30–37, the researchers tell the story of how the ADLB library has now been used successfully to run the GFMC (Green’s function Monte Carlo) nuclear physics code on over 131,072 cores of the Blue Gene/P at the Argonne Laboratory Computing Facility at 80 percent efficiency. Moreover, the ADLB library is general enough to be applicable to other scientific problems.
The ongoing collaboration relies heavily on the DOE SciDAC program for funding both the physics and computer science work and on the DOE Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program to provide the computer time to obtain the results.
CSCAPES Innovates with Hypergraphs for Mesh Partitioning
Researchers from the CSCAPES SciDAC Institute at Sandia have developed a new mathematical model for mesh partitioning based on hypergraphs. The model accurately represents memory and communication cost for “ghosting” in parallel computations. The Zoltan load-balancing toolkit provides a parallel hypergraph partitioner. Experiments performed by ITAPS collaborators at Rensselaer Polytechnic Institute using large meshes (up to 1 billion regions) show that the new mesh partitioner consistently gives higher quality mesh partitions, especially for large numbers of subdomains (cores). Application run-time for the PHASTA CFD solver was reduced by 5–10 percent on Intrepid BG/L.
Contact: Erik Boman,
Parallelism of an Ice Sheet Simulator using Trilinos
Sandia computational scientists are making an impact on the simulation of ice sheets, under the new SEACISM project funded by ASCR as part of the ISICLES initiative. By interfacing the Glimmer community ice sheet model code to the parallel data structures and solvers in Sandia’s Trilinos framework, researchers were able to parallelize the key computational kernel of the code that typically takes 70 percent of the execution time. Without any tuning, they were able to achieve a speed-up of a factor of 13 for this kernel for a key application problem: a discretization of the Greenland ice sheet on a 5 km grid. With their partners at ORNL, Los Alamos National Laboratory (LANL) and New York University, the Sandia team will continue to improve the parallelism and solver technology, with a direct line-of-sight to impacting climate assessments and sea-level rise predictions through the CCSM.
Contact: Andy Salinger,
Advanced Analysis Tools Aim to Reduce Uncertainty in Climate Data
PNNL researchers have developed a new, advanced data-reduction method that will greatly improve the capability to deal with uncertainty in high-dimensional noisy data from random simulations. Researchers have successfully used the minimum action method to detect the transition pathway and transition mechanism for the two-dimensional, noise-driven Ginzburgh-Laudau system and the Kuramoto-Sivashinsky equation.
Additionally, researchers have created a general approach for nonlinear bi-orthogonal decomposition of random fields to deal with uncertainty in climate model data as well as in measurements. An efficient algorithm based on this approach has been developed to find the optimal positions for placing sensors to obtain climate data, helping to further reduce uncertainty in the data. Data-driven simulations at the petascale level could lead to great advances in accurately predicting the performance of dynamic data-driven systems. Read more >
Blue Gene/P Aids in Precision Tests of the Standard Model
Using simulations of Quantum Chromodynamics (QCD), scientists are working to deepen the understanding of the interactions of quarks and gluons, the basic constituents of 99 percent of the visible matter in the universe. A major component of research in high energy physics involves the search for phenomena that cannot be explained by our current theory of the fundamental interactions (the Standard Model), and therefore, require new theoretical ideas for their explanation. Calculation of the decay properties of particles containing strange, charm, and bottom quarks plays an important role in these searches.
To study these decays, researchers have generated QCD field configurations on the Argonne Leadership Computing Facility’s Blue Gene/P with up, down and strange quarks on lattices that are sufficiently fine-grained and have sufficiently small up and down quark masses to enable the extrapolation of key quantities to their physical values found in nature. The generation of gauge configurations has been accelerated in many cases by a factor of 5 to 10 over what has been possible with other machines, allowing parts of their multi-year program to be completed in a little over a year.
These configurations are being generated with two different formulations quarks on a space-time lattice, each of which has advantages for different aspects of their research and provides important cross checks for consistency between the two formulations.  Recently, the neutral K meson mixing parameter, a quantity that plays an important role in precision tests of the Standard Model, has been calculated using both methods to within 4 percent accuracy.
Contact: Paul Mackenzie,


Berkeley Lab’s Dovi Poznanski Is a 2010 Einstein Fellow
Dovi Poznanski of the Lawrence Berkeley National Laboratory (LBNL) Computational Research Division’s Computational Cosmology Center (C3) is one of 10 NASA Einstein Postdoctoral Fellows for 2010. The fellowship enables Poznanski to continue investigating the use of core-collapse supernovae for cosmology, specifically as a tool to measure the expansion history of the Universe. The NASA Einstein Postdoctoral Fellowship is awarded to 10 astronomers who will conduct research related to its Physics of the Cosmos program, which aims to expand our knowledge of the origin, evolution, and fate of the Universe. The fellowship provides support to the awardees for three years, and the fellows may pursue their research at a host university or research center of their choosing in the United States. Poznanski has chosen to continue his research at Berkeley Lab’s C3.
ASCR Researcher Tamara Kolda Co-Organizes Workshop on Tensor Decompositions
Tamara G. Kolda, Sandia National Labs, co-organized an AIM- and NSF-sponsored workshop on “Computational optimization for tensor decompositions” at the American Institute of Mathematics (AIM) in Palo Alto, California from March 29 to April 2, 2010, with Rasmus Bro, Michael Friedlander, and Stephen Wright. During the past decade, there has been an explosion of interest in tensor decompositions as an important mathematical tool in fields such as psychometrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, and graph analysis. In many circumstances, N-way tensor representations allow a much more natural framework for representing relationships between elements in a data set than do traditional matrix representations. Tensor decompositions have the potential to revolutionize our scientific capabilities in such applications as environmental monitoring, medical diagnostics, cybersecurity, anti-doping testing, telecommunications, and more. Further advances in tensor decompositions depend critically on advanced optimization algorithms. This workshop brings together optimization and tensor decomposition experts to advance the field.
Contact: Tammy Kolda,
Sandian John Shadid Leads Three Mini-Symposia on Scalable Solvers
John Shadid, a Distinguished Member of the Technical Staff at Sandia National Laboratories and a DOE AMR PI, has recently co-organized three different mini-symposia, comprising seven sessions, at major computational science and numerical mathematics conferences. The title of the mini-symposia are: “Physics-based/Block-Oriented Preconditioners for the Scalable Solution of Multiphysics Systems,” (two sessions), SIAM Parallel Processing for Scientific Computation, Seattle, Wash., Feb. 24–29, 2010; “Robust and Efficient Solution of Implicit MHD Fluid Models” (two sessions) and “Robust and Efficient Solution of Large-scale Multiphysics Systems” (three sessions), both at the Eleventh Copper Mountain Conference on Iterative Methods, Copper Mountain, Colo., April 4–9, 2010.
Contact: John Shadid,
SIAM Session to Focus on ASCR Research in Complex, Distributed, Interconnected Systems
At the SIAM annual meeting to be held July 12–16 in Pittsburgh, a minisymposium will highlight the mathematics of complex distributed interconnected systems. In particular, the session organized by Juan Meza (LBNL), Ali Pinar (SNL) and David Bailey (LBNL) will showcase four projects recently funded by ASCR, and the mathematical challenges of modeling and simulating these systems. Complex, distributed, interconnected systems arise in many applications, including familiar examples such as the Internet and the electric power grid. Because of the large scale and dynamic nature of these critical systems, developing methods for analyzing and understanding their long-term behavior is a challenging task.
ORNL’s Hauck Organizing SIAM Mini-Symposium
Cory Hauck, a mathematician in the Computer Science and Mathematics Division at Oak Ridge National Laboratory, is co-organizer of a large mini-symposium on “Numerical Methods for Kinetic Equations and Related Models” at the SIAM annual meeting in Pittsburgh in July 2010. The goal of the symposium is to bring together researchers from the applied mathematics, computational science, physics, and engineering communities to discuss current issues and challenges in the numerical solution of kinetic equations. There will be five sessions and approximately 20 speakers from institutions in the U.S. and Europe. Other organizers are Ryan McClarren of Texas A&M and Jing-Mei Qiu from the Colorado School of Mines. Hauck’s research is supported by the ASCR program.
Sandia’s John Shadid Gives Invited Lecture on Scalable MHD Solvers
John Shadid, a Distinguished Member of the Technical Staff at Sandia National Laboratories and a DOE AMR PI, delivered an invited lecture at the Mathematics Department of the University of Washington on Feb. 23, 2010. The title of the talk was “Progress on the development of a scalable fully implicit stabilized unstructured finite element capability for resistive MHD.” John also met with a number of faculty members engaged in research on applied mathematics, multiscale methods and plasma physics simulations.
Contact: John Shadid,
Berkeley Lab’s David Bailey Adds Expertise to Pi Day
March 14 was Pi Day, when geeks everywhere celebrated the ratio of the circumference to the diameter of a circle. In an article titled “On Pi Day, one number ‘reeks of mystery,’” CNN’s Elizabeth Landau quotes David Bailey, Chief Technologist for LBNL’s Computational Research Division, on the frustrations and benefits of pi. Bailey also has his own page dedicated to all things pi.


OLCF Gets a Visit from Energy Secretary Steven Chu
On March 23, Energy Secretary Steven Chu toured the Oak Ridge Leadership Computing Facility (OLCF), housed in the National Center for Computational Sciences (NCCS) at Oak Ridge National Laboratory (ORNL). The stop was part of a two-day visit to Oak Ridge to dedicate a new uranium storehouse at the Y-12 National Security Complex and to announce a noncompetitive extension of the contract to operate ORNL. Chu’s tour of the OLCF included stops at the world’s fastest supercomputer, the Cray XT5 known as Jaguar, and the EVEREST PowerWall, a state-of-the-art visualization and data interpretation facility.
Accompanying Chu on his tour of the OLCF were Tennessee Gov. Phil Bredesen, U.S. Reps. Lincoln Davis (D-Tenn.) and Zach Wamp (R-Tenn.), and ORNL Director Thom Mason. Leading the tour and providing information about the facility and its goals were Associate Laboratory Directors Jeff Nichols and Martin Keller, visualization expert Ross Toedte, NCCS Director Jim Hack, and NCCS Director of Science Doug Kothe.
PetaTools Project Pushes Open|SpeedShop to 12,000 Processor Cores on Jaguar
The PetaTools project on Building a Community Infrastructure around Open|SpeedShop targets the extension of the current Open|SpeedShop tool to the petascale regime and its transformation into reusable components. As part of these efforts, researchers completed a port of the existing framework to the Cray XT5 machine and successfully completed a run on 12,000 processor cores, the largest run of Open|SpeedShop to date (about 3x larger then the largest run before). This run not only demonstrates the project’s progress in targeting DOE/ASCR leadership class machines, but also provided valuable feedback on current performance bottlenecks within the tool itself, in particular the storage technology used to manage the acquired performance data. These results will help the project further refine their strategy of creating the component infrastructure, which is currently in its final design stage.
Contact: Martin Schulz,
PNNL Secondary Network Connection Now Provides Seamless Data Transfer
Pacific Northwest National Laboratory (PNNL) has completed installation of a secondary high-speed connection to the world’s premiere research and education networks. This includes the Department of Energy-sponsored ESnet, which serves DOE laboratories and collaborators worldwide. The new configuration offers PNNL several pathway options for moving data between Richland, Seattle and Boise. With 10 and 40 gigabit per second channel options available now and a 100 gigabit option available soon, PNNL is poised to develop as data intensive computing grows.
Reliable, speedy data transfer with redundant connections is crucial to the conduct of science, according to Kevin Regimbal, who manages PNNL’s Scientific Data Management group. “In science, it’s all about the data,” he said as he explained that many research projects generate trillions of bytes, or terabytes, of data. “The transfer of a 1 terabyte data set over a 10-gigabit connection takes about 20 minutes. Compare that to over three hours to transfer that same data set over a typical network connection in an office setting, and you can see the importance of a dependable, fast connection in conducting science,” he said. Read more >
Contact: Kevin Regimbal ,
ESnet Collaborates with Dutch, Scandinavian Networks on Research Framework
The Energy Sciences Network (ESnet), DOE’s high-speed network connecting thousands of scientists and managed by Lawrence Berkeley National Laboratory, is embarking on a cooperative research effort with SURFnet (, the national computer network for higher education and research in the Netherlands, and the Scandinavian research network NORDUnet ( to explore potential synergies between the unique technical knowledge and expertise that each organization has to offer. ESnet is working with SURFnet and NORDUnet to explore common methodologies for reserving end-to-end bandwidth to accelerate such data transfers. This capability is essential to provide the high speed connectivity that can support extreme scale science. ESnet has a history of successful international and industry collaborations, including the PerfSONAR collaboration with Internet2 and U.S. and European national research and educational networks, and Fenius with the Global Lambda Integrated Facility (
OLCF Document Maps the Road to Exascale
The Oak Ridge Leadership Computing Facility is preparing for the exascale age by encompassing the expertise of its staff and user community in a 136-page blueprint entitled “Preparing for Exascale: ORNL Leadership Computing Facility Application Requirements and Strategy.” The specific challenges addressed in the document are drawn from four general research areas pursued by DOE Office of Science researchers, including energy security, nuclear security, scientific discovery and innovation, and environmental responsibility.
Specific research goals include such things as developing high-resolution models capable of analyzing climate change at the regional level; developing realistic simulations of lignocellulose (used in producing biofuels) and next-generation materials, efforts that will require a fortyfold increase in the number of atoms and a tenfold shrinking in the units of time being simulated; and developing accurate simulations of a full nuclear reactor core and working from the fundamental laws of nature to calculate nuclear fission. The document was produced by OLCF staff scientists Wayne Joubert and Hai Ah Nam and OLCF Director of Science Douglas Kothe.
Argonne Staff Introduced to Magellan Cloud
The first in a series of introductory workshops to the Argonne Magellan cloud computing resource was held Tuesday, March 23, 2010, at the lab. Open primarily to Argonne staff, the one-day workshop was attended by about 60 laboratory and University of Chicago employees. Argonne recently completed an initial deployment of cloud services on its 40-teraflop Magellan system to begin testing the potential for cloud computing to meet the computing and data storage needs of the nation’s scientific community.
Topics covered included an overview of the mid-range computing project given by Susan Coghlan, associate division director of the Argonne Leadership Computing Facility, and Narayan Desai, software architect for Magellan. Rick Bradshaw, senior systems administrator for MCS, led a hands-on session for the 19 individuals who had registered in advance for a Magellan account. The second Magellan Workshop, geared for a broader audience, will be held at Argonne, September 30 – October 1, 2010. Presentation slides from the March 23rd event can be viewed at the following link:


NERSC’s Broughton and Canon Speak at OpenFabrics Alliance Workshop
The OpenFabrics Alliance (OFA), an organization that develops, tests, licenses and distributes multi-platform, open-source software for high-performance computing and datacenter applications, held its Sixth Annual International Sonoma Workshop on March 14–17 under the theme of “Exascale to Enterprise.” Jeff Broughton, head of NERSC’s Systems Department, gave a talk on “Hybrid and Multicore Architectures” at the workshop. Shane Canon, leader of NERSC’s Technology Integration Group, spoke about “Cloud Computing and Networking.”
NERSC’s Nick Cardo, Shreyas Cholia Give Conference Presentations
Two NERSC staff members gave presentations at two difference conferences in March. Nick Cardo gave a talk on “Managing Production Scientific Computing Resources” at ISUM2010, the First International Supercomputing Conference in Mexico, March 2–5 in Guadalajara. Shreyas Cholia gave a presentation on “Building Transparent User Gateways with Globus and the MyProxy CA” at GlobusWorld 2010, March 2 - 4 at Argonne National Laboratory.
Contact: Philippe Pébay,