Big Data Analytics on Large-Scale Scientific Datasets in the INDIGO-DataCloud Project

University of Siena, Palazzo del Rettorato, Banchi di Sotto, 55, 53100 Siena (SI), Italy

EGI Foundation, IBM Research

In the context of the EU H2020 INDIGO-DataCloud project several use case on large scale scientfic data analysis regarding different research communities have been implemented. All of them require the availability of large amount of data related to either output of imulations or observed data from sensors and need scientic (big) data solutions to run data analysis experiments. More specically,the paper presents the case studies related to the following research communities: (i) the European Multidisciplinary Seaoor and water column Observatory (INGV-EMSO), (ii) the Large Binocular Tele-scope, (iii) LifeWatch, and (iv) the European Network for Earth System Modelling (ENES).

[1] L. Cinquini, D. J. Crichton, C. Mattmann, J. Harney, G. M. Shipman, F. Wang, R. Ananthakrishnan, N. Miller, S. Denvil, M. Morgan, et al. The earth system grid federation: An open infrastructure for access to distributed geospatial data.
Future Generation Comp. Syst., 36:400–417, 2014.

[2] B. Domenico, J. Caron, E. Davis, R. Kambic, and S. Nativi. Thematic real-time environmental distributed data services (thredds): Incorporating interactive analysis tools into nsdl. Journal of Digital Information, 2(4), 2006.

[3] D. Elia, S. Fiore, A. D’Anca, C. Palazzo, I. Foster, D. N. Williams, and G. Aloisio.
An in-memory based framework for scientic data analytics. In Proceedings of the ACM International Conference on Computing Frontiers, pages 424–429. ACM, 2016.

[4] ENES - European Network for Earth System Modelling. https://verc.enes.org. Accessed March 28, 2017.

[5] European Multidisciplinary Seaoor and water column Observatory - ERIC.
http://www.emso-eu.org. Accessed March 28, 2017.

[6] P. Favali, F. Chierici, G. Marinaro, G. Giovanetti, A. Azzarone, L. Beranzoli, A. De Santis, D. Embriaco, S. Monna, N. L. Bue, et al. Nemo-sn1 abyssal cabled observatory in the western ionian sea. IEEE Journal of Oceanic Engineering, 38(2):358–374, 2013.

[7] S. Fiore, A. D’Anca, C. Palazzo, I. T. Foster, D. N. Williams, and G. Aloisio.
Ophidia: Toward big data analytics for escience. In Proceedings of the International Conference on Computational Science, ICCS 2013, Barcelona, Spain, 5-7 June, 2013, pages 2376–2385. Elsevier, 2013.

[8] S. Fiore, M. Płóciennik, C. Doutriaux, C. Palazzo, J. Boutte, T. Żok, D. Elia, M. Owsiak, A. D’Anca, Z. Shaheen, et al. Distributed and cloud-based multi-
model analytics experiments on large volumes of climate change data in the earth system grid federation co-system. In Workshop on "Big Data Challenges,
Research, and Technologies in the Earth and Planetary Sciences", Big Data (Big Data), 2016 IEEE International Conference on, pages 2911–2918. IEEE, 2016.

[9] FITS (Flexible Image Transport System). https://ts.gsfc.nasa.gov/iaufwg/. Accessed March 28, 2017.

[10] fv (FITS viewer). https://heasarc.gsfc.nasa.gov/docs/software/ftools/fv/. Accessed March 28, 2017.

[11] IRAF (Image Reduction and Analysis Facility). http://iraf.noao.edu/. Accessed March 28, 2017.

[12] MOIST - Multidisciplinary Oceanic Information System. http://www.moist.it. Accessed March 28, 2017.

[13] C. Palazzo, A. Mariello, S. Fiore, A. D’Anca, D. Elia, D. N. Williams, and G. Aloisio.
A workow-enabled big data analytics software stack for escience. In 2015 International Conference on High Performance Computing & Simulation, HPCS 2015, Amsterdam, Netherlands, July 20-24, 2015, pages 545–552, 2015.

[14] M. Plóciennik, T. Zok, I. Altintas, J. Wang, D. Crawl, D. Abramson, F. Imbeaux, B. Guillerminet, M. López-Caniego, I. C. Plasencia, et al. Approaches to dis-
tributed execution of scientic workows in kepler. Fundam. Inform., 128(3):281–302, 2013.

[15] R. K. Rew and G. P. Davis. The unidata netcdf: Software for scientic data access. In Sixth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, pages 33–40, 1990.

[16] SAC - Seismic Analysis Code. http://ds.iris.edu/ds/nodes/dmc/software/downloads/sac/. Accessed March 28, 2017.

[17] SAC User Manual - Using SAC: Introduction. https://ds.iris.edu/les/sac-manual-draft/manual/intro.html. Accessed March 28, 2017.

[18] Saint Louis University - Department of Earth and Atmospheric Sciences. http://www.slu.edu/department-of-earth-and-atmospheric-sciences-home. Accessed March 28, 2017.

[19] D. P. van Vuuren, J. Edmonds, M. Kainuma, K. Riahi, A. Thomson, K. Hibbard, G. C. Hurtt, T. Kram, V. Krey, J.-F. Lamarque, et al. The representative concentration pathways: an overview. Climatic Change, 109(1):5, 2011.

[20] D. N. Williams, T. Bremer, C. Doutriaux, J. Patchett, S. Williams, G. Shipman, R. Miller, D. R. Pugmire, B. Smith, C. Steed, et al. Ultrascale visualization of climate data. Computer, 46(9):68–76, 2013.