ASCR Monthly Computing News Report - May 2009

In this issue...


2009 SciDAC Meeting to Showcase Computational Science across Disciplines

Between June 14-18, about 400 of the leading experts in computational science will convene in San Diego for the Scientific Discovery through Advanced Computing (SciDAC) 2009 conference for four days of invited talks, technical presentations and posters, and a special evening devoted to scientific visualization. Kicking off the conference program on Monday, June 15, will be a keynote address by Dr. Ray Orbach, the former DOE Under Secretary of Science, who was a strong advocate for scientific computing during his tenure. The conference program will feature more than 50 talks by scientists from DOE national labs and other research institutions in the U.S., Japan and Germany. There will two evening poster sessions and “Vis Night,” during which top visualizations will vie to win one of the coveted OASCR awards for excellence in scientific visualization.

SciDAC 2009 is strategically co-located with the Department of Defense High Performance Computing Modernization Program (HPCMP) 2009 Users Group Conference. A joint plenary session and a joint poster session with speakers and presenters of interest to both communities are being planned. Full details of the 2009 SciDAC Conference can be found at:

New General Model for Detecting MPI Deadlocks Developed

Researchers from Lawrence Livermore National Laboratory (Bronis R. de Supinski, Martin Schulz) and Germany’s Dresden University of Technology (Tobias Hilbrich and Matthias Mueller) have developed the first general deadlock model for MPI using a graph-based approach. The MPI standard defines several usage patterns that can lead to deadlock, some of which involve collective communications or non-deterministic operations such as wildcard receives. Further, some MPI programming deadlocks only occur for some MPI implementations or certain configurations. Many tools to detect MPI deadlocks exist; however, none precisely handles the increased complexity of deadlock detection created by the richness of the MPI standard, which requires a general deadlock model.

The new general deadlock model for MPI includes a novel necessary and sufficient criterion, the OR-Knot, for deadlock in MPI programs. This model enables visualization of MPI deadlocks and motivates the design of a new deadlock detection mechanism. Compared to the ad hoc mechanism previously available in Umpire, implementation of this new mechanism reflected MPI nondeterminism and, thus, more completely detected MPI deadlocks than any other existing MPI deadlock detection tool. Overall, the results demonstrate that the mechanism improves performance by as much as two orders of magnitude while providing precise characterization of deadlocks.

Contact. Bronis de Supinski,
ORNL and Sun Researchers Demonstrate Parallel NFS Over Lustre at IPDPS 2009

Researchers in the Future Technologies group at ORNL and at Sun Microsystems have designed and implemented the first prototype of Parallel NFS over the scalable Lustre file system. Parallel NFS (pNFS) is an emergent open standard for parallelizing data transfer over a variety of I/O protocols. In a paper presented at the 23rd IEEE International Parallel and Distributed Processing Symposium, the team presented the design, implementation, and evaluation of pNFS, a Lustre-based parallel NFS. The benefits of using pNFS include portability across a range of back-end file systems with no changes to the client-side operating system. The initial performance evaluation shows that the performance of pNFS is comparable to that of original Lustre. Given these results, pNFS appears to be a promising approach to providing a scalable, high performance, portable file system for DOE computing facilities. For more information about the symposium, see the following link:

SNL, OSU Develop New Hypergraph Partitioning Algorithms

Researchers from Sandia National Laboratories (SNL) and Ohio State University (OSU) have developed a new algorithm for dynamic load balancing of scientific computing tasks based on hypergraph partitioning. Their paper “A Hypergraph Repartitioning Model for Dynamic Load Balancing” has been accepted for publication in the Journal of Parallel and Distributed Computing. A preliminary version of this paper won a Best Paper award at the IPDPS07 conference. The new partitioning approach has been implemented into the Zoltan software toolkit, and can be used on leadership class computers. This work was funded by the SciDAC Institute for Combinatorial Scientific Computing and Petascale Simulations (CSCAPES).

Contact. Erik Boman,
PNNL Scientists Demonstrate Leadership in HPC Focus Area for TASCS Project

The recent mid-term review of the Center for Technology for Advanced Scientific Components Software (TASCS) showed that Pacific Northwest National Laboratory (PNNL) scientists have made important contributions. In addition to PNNL’s collaboration with other research groups across multiple disciplines, such as subsurface science and computational biology, they are also leading one of the project’s four focus areas. PNNL’s contributions under the project’s high performance computing (HPC) initiative resulted in development of the capability to perform multiple-component multiple-data execution, which enables multiple parallel applications to be joined together. This allows researchers to run several parallel simulations simultaneously using different starting parameters, resulting in more precise analysis. PNNL also contributed to the effort to develop a hybrid computing capability that allows for different types of processing elements to be used together to solve problems.

Contacts: Daniel Chavarria - 509-372-6964 or
Manoj Krishnan - 509-372-4206
LANL Team Develops Monotonicity-Preserving Momentum Remap for Staggered Grids

A Los Alamos National Laboratory (LANL) team (L. Margolin and M. Shashkov) have constructed and verified modifications to a one-dimensional algorithm for remapping momentum on a staggered grid. The original algorithm (L.G. Margolin and M. Shashkov, “Remapping, Recovery and Repair on Staggered Grid,” Computer Methods in Applied Mechanics and Engineering, 193 (2004), pp. 4139–4155) conserved momentum, nearly conserved kinetic energy while ensuring local dissipativity, and was computationally efficient. Their enhancements to this scheme maintain all these advantages while, in addition, preserving monotonicity of the remapped velocities. This eliminates the need for the separate repair (of velocity) step that has been previously employed to enforce monotonicity. The construction is based on ideas of Sweby developed in the context of flux-limited advection algorithms, but contains some novel features since velocity is not a conserved quantity. Unit tests verify that the algorithm maintains conservation and preserves monotonicity. The researchers are currently performing convergence tests, based on their previously documented technique of cyclic remapping, to assess the accuracy of the modified algorithm.

This work was done as part of ASCR Applied Mathematics Research Project “Mimetic Finite Difference Methods for Partial Differential Equations.”

Contact. Len Margolin, or
Mikhail Shashkov,


41 DOE-Affiliated Researchers Named Fellows of SIAM

The Society for Industrial and Applied Mathematics (SIAM) announced its first class of Fellows on Friday, May 1, and the group included 14 current and retired mathematicians from DOE national laboratories, as well as 27 others affiliated with DOE. This represents about 20 percent of the 200 Fellows selected by the society from its international membership of more than 12,000 mathematicians.

Of the DOE-affiliated fellows, five are from Lawrence Berkeley National Laboratory, four are from Los Alamos National Laboratory, three from Argonne National Laboratory and one from Oak Ridge National Laboratory. Additionally, 24 university math researchers who have received DOE funding and three current and former members of ASCR’s Advisory Committee, ASCAC, were among the Fellows named.

The DOE-affiliated SIAM Fellows are:
Argonne National Laboratory
  • Hans Kaper (retired), for contributions to differential equations and dynamics
  • Sven Leyffer, for contributions to large-scale nonlinear optimization
  • Jorge Moré, for advances in algorithms and software for continuous optimization
Brookhaven National Laboratory
  • James Glimm (joint appointment at State University of New York at Stony Brook), for contributions to operator algebras, partial differential equations, mathematical physics, and especially shock wave theory
Lawrence Berkeley National Laboratory
  • John Bell, for contributions to numerical methods for the partial differential equations of computational science
  • Alexandre Chorin (joint appointment at University of California, Berkeley), for contributions to computational fluid dynamics
  • Phillip Colella, for contributions to adaptive and numerical methods for partial differential equations in science and engineering
  • James Demmell (joint appointment at UC Berkeley), for contributions to numerical linear algebra, including the LAPACK project
  • James Sethian (joint appointment at UC Berkeley), for contributions to the numerical solution of partial differential equations, especially level set methods
Los Alamos National Laboratory
  • James “Mac” Hyman, for contributions to the numerical solution of partial differential equations and modeling of biological systems
  • Alan Perelson, for contributions to viral dynamics and other problems of mathematical biology
  • David Sharp, for contributions to dynamics and biology
  • Burton Wendroff (retired), for contributions to the numerical solution of partial differential equations
Oak Ridge National Laboratory
  • Jack Dongarra (joint appointment at University of Tennessee), for contributions to numerical linear algebra, including EISPACK, LINPACK, and LAPACK, and high-performance computing
Advanced Scientific Computing Advisory Committee (ASCAC)
  • Marsha Berger, Courant Institute, New York University, for the development of adaptive algorithms and software for partial differential equations
  • Thomas Manteuffel, University of Colorado at Boulder, for contributions to iterative methods for linear systems and numerical methods for partial differential equations
  • Margaret H. Wright, Courant Institute of Mathematical Sciences, New York University, for contributions to numerical optimization and service to the profession
University researchers supported by DOE
  • Russel E. Caflisch, University of California, Los Angeles, for advances in physical applied mathematics and in mathematics applied to physical systems
  • Weinan E, Princeton University, for analysis of multiscale and stochastic problems
  • Howard C. Elman, University of Maryland-College Park, for contributions to numerical linear algebra and applications to finite elements and computational fluid dynamics
  • Paul Garabedian, Courant Institute, for contributions to partial differential equations and fluid dynamics
  • C. William Gear, NEC Research Institute, for contributions to numerical methods and software for ordinary differential equations and differential-algebraic equations.
  • Leslie F. Greengard, New York University, for creation of the Fast Multipole Method and other fast algorithms
  • John Guckenheimer, Cornell University, for contributions to theoretical and computational dynamical systems and mathematical neuroscience
  • Max D. Gunzberger, Florida State University, for contributions to control of fluids and scientific computing
  • Thomas Yizhao Hou, California Institute of Technology, for contributions to fluid mechanics and multiscale analysis
  • Arthur Jaffe, Harvard University, for contributions to the mathematical foundations of quantum field theory
  • Christopher R. Johnson, Scientific Computing and Imaging (SCI) Institute, University of Utah, for contributions to scientific computing and visualization
  • Donald E. Knuth, Stanford University, for contributions to the analysis of algorithms, exceptionally influential books, and the creation of TeX
  • Gregory A. Kriegsmann, New Jersey Institute of Technology, for contributions to the analysis of problems of electromagnetics and heating
  • Peter Lax, New York University Emeritus, for contributions to conservation laws, scattering theory, integrable systems, and numerical analysis
  • Dianne P. O’Leary, University of Maryland-College Park, for contributions to linear algebra, regularization, and applications
  • J. Tinsley Oden, University of Texas at Austin, for advances in finite element analysis and computational mechanics
  • Michael L. Overton, New York University, for advances in numerical optimization and eigenvalue problems
  • Linda R. Petzold, University of California, Santa Barbara, for contributions to numerical ordinary differential equations and differential-algebraic equations and computational science
  • Michael J. Shelley, New York University, for the analysis of vortices and flows
  • Chi-Wang Shu, Brown University, for contributions to the numerical solution of partial differential equations including discontinuous Galerkin methods
  • Gilbert Strang, MIT, for contributions to finite elements and wavelets as well as influential textbooks in applied and numerical mathematics.
  • Richard Tapia, Rice University, for contributions to linear and nonlinear programming and tireless efforts to increase diversity
  • Lloyd N. Trefethan, University of Oxford, for contributions to numerical analysis and scientific computing
  • Srinivasa R.S. Varadhan, New York University, for advances in probability theory, including the study of large deviations
PNNL's Terence Critchlow Inducted into World's Largest Computing Society

Terence Critchlow has been elected a Senior Member of the Association for Computing Machinery (ACM). Critchlow is the Associate Director of Scientific Data Management within the Computational Sciences and Mathematics Division at Pacific Northwest National Laboratory. With about 92,000 members, ACM is the world’s largest educational and scientific computing society. They delivers resources that advance computing as a science and a profession, including the computing field’s premier digital library, as well as leading-edge publications, conferences, and career resources.

Critchlow is one of 395 members inducted into the ACM Senior Member program this year. The program, initiated in 2006, includes members with at least 10 years of professional experience who have demonstrated performance that sets them apart from their peers through technical leadership, and technical or professional contributions. ACM Senior Member status recognizes the top 25 percent of ACM Professional Members for their demonstrated excellence in the computing field. ACM’s Senior Members join a distinguished list of colleagues to whom ACM and its members look for guidance and leadership in computing and information technology.

LLNL's Bronis de Supinski Named Chair of the OpenMP Language Committee

Bronis de Supinski was named Chair of the OpenMP Language Committee. OpenMP is expected to be the primary mechanism for SciDAC applications to exploit threading on future large scale systems that will feature chips with very large numbers of cores. As Chair of the Language Committee, de Supinski will oversee extensions of the specification that better support those applications. De Supinski is the co-leader of the Advanced Simulation and Computing (ASC) program’s Application Development Environment and Performance Team (ADEPT) at Lawrence Livermore National Laboratory (LLNL). He also leads the Petascale Computing Enabling Technologies LDRD project and is the LLNL principal point of contact for the Scientific Discovery through Advanced Computing (SciDAC) program’s Performance Engineering Research Institute (PERI).

Contact. Bronis de Supinski,


Speeding Up Science Data Transfers between Department of Energy Facilities

As scientists conduct cutting-edge research with ever more sophisticated techniques, instruments, and supercomputers, the data sets that they must move, analyze, and manage are increasing in size to unprecedented levels. The ability to move and share data is essential to scientific collaboration, and in support of this activity network and systems engineers from DOE’s Energy Sciences Network (ESnet), National Energy Research Scientific Computing Center (NERSC) and Oak Ridge Leadership Computing Facility (OLCF) are teaming up to optimize wide-area network data transfers.

OLCF, located at Oak Ridge National Laboratory in Tennessee, and NERSC, located at Lawrence Berkeley National Laboratory in California, are home to some of the fastest supercomputers in the world. OLCF is one of two DOE Leadership Computing Facilities, and NERSC provides computing resources to 3,000 researchers supported by the DOE Office of Science. A number of research groups use resources at both centers. ESnet, DOE’s high-speed network, connects the two centers, as well as other national labs and universities around the country. With the installation and deployment of new dedicated data transfer nodes at NERSC and OLCF linked by ESnet, researchers are now able to move large data sets between each facility’s mass storage systems at a rate of 200 megabytes per second (MB/sec). Read more at

ESnet Installs New 10-Gigabit Connections

From New England to Nashville, and across the Great Plains, the Energy Sciences Network (ESnet) added a several 10-gigabit lines to their nationwide backbone in the past three months to enhance international science collaborations. In March, a new 10-gigabit line was added to the ESnet hub in Boston, Mass., where the DOE network peers with the Northern Crossroads network, which connects research institutions across New England. This new connection gives researchers at Boston University and Harvard access to Brookhaven National Laboratory (BNL), where data collected by the ATLAS particle physics experiment in the Large Hadron Collider (LHC) in Switzerland will be stored for U.S. researchers. In early May, a second 10-gigabit link was added to ESnet’s Science Data Network (SDN) between the facility’s Nashville Hub and t Oak Ridge National Laboratory (ORNL). This doubles the ORNL capacity to 20 gigabits to ESnet’s backbone at Nashville.

Most recently, a new 10-gigabit port was added to ESnet’s Kansas City Hub, which is a peering point with the Great Plains Network. This new capability will support a 1-gigabit Virtual Local Area Network (VLAN) to the Deep Underground Science and Engineering Lab’s Homestake facility in South Dakota. The new 10-gigbit port at the Kansas City Hub will also support a 1-gigabit connection to the Oklahoma Telecommunications Network, giving ATLAS researchers at the University of Oklahoma access to LHC and ATLAS data stored at BNL.

National Ignition Facility at LLNL Certified for Operation

The National Ignition Facility (NIF), the world’s largest and highest-energy laser, was certified to operate by the U.S. Department of Energy on March 27, 2009. In 2010, NIF will focus the intense energy of 192 laser beams (two million joules of ultraviolet energy) on a BB-sized target filled with hydrogen fuel — fusing, or igniting, the hydrogen atoms’ nuclei in the world’s first controlled thermonuclear reaction. This is 60 times more energy than any previous laser system. The NIF dedication took place on May 29. More information can be found at

IMG ER Goes Primetime: Provides Expert-Driven QC for Microbial Genome Information

After a genome is sequenced and automatically annotated, researchers often manually review the predicted genes and their functions in order to improve accuracy and coverage across the vast genetic code of the particular target organism or community of organisms. These annotations drive the publication of high-profile science relevant to advancing bioenergy research and our understanding of biogeochemistry—the biological, chemical, physical, and geological processes that regulate our environment.

Scientists at DOE’s Joint Genome Institute (JGI) and the Biological Data Management and Technology Center in Lawrence Berkeley National Laboratory’s Computational Research Division have launched the Expert Review (ER) version of the Integrated Microbial Genomes (IMG) system. IMG ER supports and enhances the review and revision of annotations for both publicly available genome datasets and those newly released from private institutions. The IMG system contains a rich collection of genomes from all three domains of life: as of April 2009, IMG included 1,284 bacterial, 59 archaeal, 49 eukaryotic genomes, as well as 2,524 viruses and 924 plasmids. IMG and its companion metagenome system, IMG/M, have been cited in over 200 publications and have been used in the analysis of dozen of genomes and metagenomes. Read more at

Craig Lant Shares NERSC Security Project Details at DOE Cyber Security Conference

Craig Lant of NERSC’s Networking, Security, Workstations and Servers Group gave a talk at the 30th DOE Cyber Security Conference held May 11-15 in Henderson, Nevada. In his talk, Lant described a group project to gain access to SSH related data, including authentication data such as user names and key fingerprints, interactive session data such as keystrokes and responses, and information about non-interactive sessions such as commands executed and files transferred. Previously, this type of information could be easily accessed by intrusion detection systems to quickly and easily identify hostile activity. However, with the advent of SSH, all of this data is encrypted before it is sent over any network, rendering intrusion detection systems blind to it. NERSC’s solution was to modify the SSH daemon so that it sends copies of the relevant data to an intrusion detection system for analysis. Lant described the changes that the NERSC security team made to SSH and the mechanisms they used to get the data securely from the SSH daemons into Bro, Lawrence Berkeley National Laboratory’s (LBNL’s) intrusion detection system.

The purpose of the Cyber Security Conference is to promote DOE-wide cyber security awareness of current threats and protection measures, disseminating DOE-specific policy and operations information, and sharing state-of-the-art practices to improve the Department's overall cyber security posture. Read more at



SciDAC Outreach Center Offers Tutorials in Conjunction with SciDAC'09 Conference

On June 19, at the end of the SciDAC 2009 meeting in San Diego, the SciDAC Outreach Center will present the third SciDAC Tutorials Day at the San Diego Supercomputer Center. Tutorials Day provides open and free tutorials on a wide range of subjects in scientific computing. The focus is on bringing the benefits of DOE’s investments in SciDAC to new researchers in academia and industry. This year, 11 sessions are planned to help participants learn how to parallelize their current computational efforts or to improve their computation by implementing the latest computing algorithms and methods. For more information, goto the following link -

INCITE Proposal Writing Webinar Offered in June at ALCF

Over one billion processing hours are available through DOE’s INCITE program for 2010, which is jointly managed by the Argonne and Oak Ridge Leadership Computing Facilities. A June 2 Proposal Writing webinar sponsored by the program will help candidates craft the most effective INCITE proposals. Katherine Riley, team lead of the Argonne Leadership Computing Facility’s (ALCF’s) Catalyst group, and Bronson Messer of Oak Ridge’s Scientific Computing group will provide suggestions to improve the quality of the proposal submissions. Employees and visitors also may attend the webinar in person at the ALCF at Argonne National Laboratory.

Don’t forget: INCITE proposals are due July 1, 2009.

Contact: Chel Lancaster,
LLNL's Rob Falgout Presents Multigrid Lecture at International School in France

Rob Falgout presented a multigrid lecture at an international school for solving large algebraic systems on modern high performance computing systems in Sophia-Antipolis, France. The school was organized by CEA (the French Atomic Energy Commission), EDF (operator of 58 nuclear power plants) and INRIA (the French national institute for research in computer science and control). Falgout, one of nine invited speakers, gave a three-hour lecture on “Multigrid methods for ultra-parallel computers” to 85 attendees. Falgout is a computer scientist who leads the hypre team at LLNL. More information can be found at:

ORNL Hosts CUG 09

“Compute the Future,” the 51st Cray User Group meeting, was held May 4-7 in Atlanta, GA. More than 150 supercomputer users attended the CUG 09 annual meeting for organizations which use Cray HPC systems and support their use for science research. It was hosted this year by Oak Ridge National Laboratory (ORNL). James Hack, director of the National Center for Computational Sciences and an atmospheric scientist who leads ORNL’s Climate Change Initiative, gave the keynote address. Hack explained why the DOE’s Cray XT5, 1.64 petascale system Jaguar and the NSF’s Cray XT5, Kraken, both located at ORNL, are so important for accurately predicting global climate.

Arthur (Buddy) Bland, leader of the Oak Ridge Leadership Computing Facility, who supervised installation of the Cray XT5 Jaguar, introduced the world’s most powerful computer for open science to the assembly, describing several applications with speeds exceeding one petaflop.

Cray executive and technical staff presented Cray’s roadmap for future systems and discussed current software and hardware. More than 30 ORNL staff attended, alongside the systems managers and technical support staff from national and international centers, scientific researchers, and vendors to Cray systems such as AMD.

LBNL Team Wins Best Paper Award at 2009 CUG Meeting

At the 51st Cray User Group (CUG) annual meeting, held May 4-7 in Atlanta, a framework developed by Berkeley Lab computer scientists for speeding up the performance of scientific computing applications on multicore processors was recognized as the best paper presented at the conference. The paper, “A Generalized Framework for Auto-tuning Stencil Computations,” was written by Shoaib Kamil, Cy Chan, Samuel Williams, Leonid Oliker, John Shalf, Mark Howison, E. Wes Bethel and Prabhat, who all have positions in LBNL’s Computational Research or National Energy Research Scientific Computing Division. Kamil is a graduate student at UC Berkeley and Chan is a grad student at MIT.

Andrew Uselton of NERSC was a finalist for the best paper award for his submission on “Deploying Server-side File System Monitoring at NERSC.”

LBNL's James Sethian Gives Invited Talk at EMC2009

James Sethian, head of the Mathematics Group at Berkeley Lab, traveled to Hong Kong in May to give an invited talk on “Advances in Advancing Interfaces: Tracking Propagating Interfaces in Fluid Mechanics, Materials Sciences, and Medical Imaging” at the International Conference on Engineering and Computational Mathematics (EMC2009), held May 27-29 at the Hong Kong Polytechnic University. Sethian gave the plenary presentation in the session on Applications of Engineering Mathematics.

Contact. James Sethian,
Berkeley Lab Hosts Tenth Bay Area Scientific Computing Day

The Tenth Bay Area Scientific Computing Day (BASCD) was held Saturday, May 9, at Lawrence Berkeley National Laboratory, which organized the first BASCD in 2000. Attended by nearly 50 participants, the Bay Area Scientific Computing Day is an annual informal gathering to encourage the interaction and collaboration of researchers in the fields of scientific computing and computational science/engineering from the San Francisco Bay Area. This event provides a venue for junior researchers to present their work to the local community, and for the Bay Area scientific and computational science/engineering communities at large to interchange views on today’s multidisciplinary computational challenges and state-of-the-art developments. This year’s agenda included speakers from LBNL, Sandia National Laboratories, Stanford University and the University of California campuses in Berkeley and Davis. For more information, visit:

Berkeley Lab Researchers Share Expertise at ParCFD 2009

At the 21st International Conference on Parallel Computational Fluid Dynamics (ParCFD) held May 18-22 at Moffett Field in California, LBNL staff contributed an invited talk, a tutorial and participated in a panel discussion. ParCFD 2009 is the 21st in a series of annual international meetings dedicated to the discussion of recent developments and applications of parallel computing in the field of computational fluid dynamics and related disciplines. At the 2009 meeting:

  • Alice Koniges of NERSC was a co-presenter of the tutorial on “Hybrid OpenMP/MPI Programming and Other Models for Multi-Core Architectures”
  • Associate Lab Director for Computing Sciences Horst Simon gave an invited talk, “Future Directions in High Performance Computing (HPC) 2009—2018,”
  • John Shalf of NERSC was a panelist for a discussion on “PetaFLOPS and Beyond”

For more information, go to: