Panorama 360: Performance Data Capture and Analysis for End-to-end Scientific Workflows

— Panorama 360 Project Vision:
Provide a resource for the collection, analysis, and sharing of performance data about end-to-end scientific workflows executing on DOE facilities
— A Collaborative Project between:

IMPACT ON DOE SCIENCE

Diamonds that deliver!

Neutrons, simulation analysis of tRNA-nanodiamond combo could transform drug delivery design principles.

A unique combination of experimentation and simulation was used to shed light on the design principles for improved delivery of RNA drugs, which are promising candidates in the treatment of a number of medical conditions including cancers and genetic disorders.

READ THE PRESS RELEASE

The DOE Panorama project has developed an SNS workflow to confirm that nanodiamonds enhance the dynamics of tRNA in presence of water. The workflow, enacted by the Pegasus Workflow Management System, calculates the epsilon that best matches experimental data. These calculations were for 10 ns each and the workflows used almost 400,000 CPU hours of time on DOE leadership class systems.

Water is seen as small red and white molecules on large nanodiamond spheres. The colored tRNA can be seen on the nanodiamond surface (Image by Michael Mattheson, OLCF, ORNL)

PROJECT GOALS AND ARCHITECTURAL OVERVIEW

Panorama 360 Overview

DATA CAPTURE

Characterization of instrument data capture, data summarization, and publication

REPOSITORY

An open access common repository for storing end-to-end workflow performance and resource data captured using a variety of tools

LEARNING

Development of ML techniques for workflow performance analysis and infrastructure troubleshooting

RECENT PUBLICATIONS

Our Scientific Contributions

Measuring the Impact of Burst Buffers on Data-Intensive Scientific Workflows
R. Ferreira da Silva, S. Callaghan, T. M. A. Do, G. Papadimitriou, and E. Deelman

Future Generation Computer Systems, vol. 101, pp. 208-220, 2019.

Using simple PID-inspired controllers for online resilient resource management of distributed scientific workflows
R. Ferreira da Silva, R. Filgueira, E. Deelman, E. Pairo-Castineira, I. M. Overton, M. Atkinson

Future Generation Computer Systems, vol. 95, pp. 615-628, 2019.

Detecting Outliers in Network Transfers with Feature Extraction
M. Kiran, C. Wang, N. S. V. Rao, and A. Mandal

ICML Joint Workshop on Deep (or Machine) Learning for Safety-Critical Applications in Engineering, 2018

IoT-Hub: New IoT data-platform for Virtual Research Environments
R. Filgueira, R. Ferreira da Silva, E. Deelman, V. Christodoulou, and A. Krause

10th International Workshop on Science Gateways (IWSG 2018), 2018

Distributed Workflows for Modeling Experimental Data
V. Lynch, J. B. Calvo, E. Deelman, R. Ferreira da Silva, M. Goswami, Y. Hui, E. Lingerfelt, and J. Vetter

IEEE Xplore, Nov. 2017

PANORAMA: An Approach to Performance Modeling and Diagnosis of Extreme Scale Workflows
E. Deelman, C. Carothers, A. Mandal, B. Tierney, J. S. Vetter, I. Baldin, C. Castillo, G. Juve, D. Krol, V. Lynch, B. Mayer, J. Meredith, T. Proffen, P. Ruth, R. Ferreira da Silva

International Journal of High Performance Computing Applications, vol. 31, iss. 1, pp. 4-18, 2017

View All Publications