Cross-geography scientific data transferring trends and behavior
- Argonne National Laboratory (ANL)
- ORNL
Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.
- Research Organization:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE
- DOE Contract Number:
- AC05-00OR22725;
- OSTI ID:
- 1468117
- Resource Type:
- Conference paper/presentation
- Conference Information:
- 27th International Symposium on High-Performance Parallel and Distributed Computing - Tempe, Arizona, United States of America - 6/12/2018-6/15/2018
- Country of Publication:
- United States
- Language:
- English
Similar Records
Scientific User Behavior and Data-Sharing Trends in a Petascale File System
XRootD popularity on hadoop clusters
Characterization and identification of HPC applications at leadership computing facility
Conference
·
2017
·
OSTI ID:1427663
XRootD popularity on hadoop clusters
Journal Article
·
2017
· Journal of Physics. Conference Series
·
OSTI ID:1831862
Characterization and identification of HPC applications at leadership computing facility
Conference
·
2020
·
OSTI ID:1649007
Related Subjects
