Getting the Most from Ultra-Large Climate Datasets

Project provides tools for advanced data analysis and visualization.

The Science

Fueled by exponential increases in the computational and storage capabilities of high-performance computing platforms, climate model simulations are evolving toward higher numerical fidelity, complexity, volume, and dimensionality. Climate data holdings are projected to reach hundreds of exabytes (10¹⁸ bytes) worldwide by 2020. Such explosive growth presents both challenges and opportunities for scientific breakthroughs.

The Impact

To overcome these challenges and accelerate knowledge discovery for climate “big data,” the U.S. Department of Energy (DOE) is supporting the Ultra-scale Visualization Climate Data Analysis Tools (UV-CDAT) project. This project provides software tools that address big data analytics, the need for reproducibility; requirements for advancing model ensemble analysis, uncertainty quantification, and metrics computation; and integration of heterogeneous data sources (simulations, observations, and re-analysis). The overall UV-CDAT architecture will enable incorporation of existing and future software components.

Summary

The UV-CDAT team has designed a Python-based framework that combines several disparate technologies under one infrastructure. United by standard, common protocols and application programming interfaces, UV-CDAT integrates more than 40 different software components. The primary goal of this nationally coordinated effort is to build an ultra-scale data analysis and visualization system empowering scientists to engage in new, exciting data exchanges that will enable breakthrough climate science. The framework is designed to evolve and incorporate new software tools as required by the science and science community. This large project includes teams from four DOE national laboratories (Lawrence Berkeley, Lawrence Livermore, Los Alamos, and Oak Ridge), two universities (Polytechnic Institute of New York University and University of Utah), the National Aeronautics and Space Administration (NASA) at Goddard Space Flight Center, and two private companies (Kitware and Tech-X).

Contact

Dean N. Williams
Lawrence Livermore National Laboratory
P.O. Box 808
Livermore, CA 94550
williams13@llnl.gov

Funding

This work is supported by the Office of Biological and Environmental Research within the DOE Office of Science under contract numbers DE-AC02-05CH11231 and DE-AC52-07NA27344 and by NASA.

Publications

Williams, D.N., et al. “The Ultra-scale Visualization Climate Data Analysis Tools: Data analysis and visualization for geoscience data.” Computer, in press.