Skip to main content
The search returned 9 results
Now showing 1 - 9 of 9
  • Some of the metrics are blocked by your 
    Item type:Publication, Access status: Open Access ,

    DARE to Perform Seismological Workflows

    The DARE e-science platform (http://project-dare.eu) offers innovative tools to ease scientific workflow development and execution exploiting efficient Cloud resources. It aims to enable on-demand numerical computations and analyses, fast large dataset handling, flexible and customisable workflow pipelines and complete provenance tracking. It also integrates available e-infrastructure services (e.g. EUDAT, EIDA) and can be linked to user developed interfaces. DARE is validated via two domain-specific pilots, one from the climate modelling community and one from the seismological research field. Focusing on the latter, the EPOS Use Case is driven by urgent issues and general user needs of solid Earth Science community, following developments and application standards in the computational seismology research society. This Use Case also benefits from the pioneering experience of previous European projects (e.g. VERCE, EPOS-IP) in this framework. We present here the development of a scientific workflow to perform a quick calculation of seismic source parameters after an earthquake. The workflow requirements include HPC calculations (on local-institutional or Cloud resources), fast data-intensive processing, provenance exploitation and seismic source inverse modelling tools. The DARE platform automatically conducts the required actions optimally mapped to computational resources, linking them together by managing intermediate data. It automatically deploys the necessary environment to perform on-demand transparent computations executing a dockerised version of the numerical simulation code on a Kubernetes cluster via a web API. Other API calls allow for remote, distributed execution of dispel4py workflows, used to describe the steps for data analysis and download of seismic recorded data via EIDA Research Infrastructure services. Well established scientific python codes, such as those for waveform misfit calculation and source inversion, are thus easily implemented in this flexible and modular structure, and executed at scale. Moreover, the pilot requirement of searching and reusing multiple simulations for the same earthquake strongly benefits from customisable management of metadata and lineage through the DARE platform exploiting the integration of S-ProvFlow with dispel4py.
      85  121
  • Some of the metrics are blocked by your 
    Item type:Publication, Access status: Open Access ,

    Comprehensible Control for Researchers and Developers Facing Data Challenges

    The DARE platform enables researchers and their developers to exploit more capabilities to handle complexity and scale in data, computation and collaboration. Today’s challenges pose increasing and urgent demands for this combination of capabilities. To meet technical, economic and governance constraints, application communities must use use shared digital infrastructure principally via virtualisation and mapping. This requires precise abstractions that retain their meaning while their implementations and infrastructures change. Giving specialists direct control over these capabilities with detail relevant to each discipline is necessary for adoption. Research agility, improved power and retained return on intellectual investment incentivise that adoption. We report on an architecture for establishing and sustaining the necessary optimised mappings and early evaluations of its feasibility with two application communities.
      67  133
  • Some of the metrics are blocked by your 
    Item type:Publication, Access status: Open Access ,

    S-ProvFlow. Storing and Exploring Lineage Data as a Service

    We present a set of configurable Web service and interactive tools, s-ProvFlow, for managing and exploiting records tracking data lineage during workflow runs. It facilitates detailed analysis of single executions. It helps users manage complex tasks by exposing the relationships between data, people, equipment and workflow runs intended to combine productively. Its logical model extends the PROV standard to precisely record parallel data-streaming applications. Its metadata handling encourages users to capture the application context by specifying how application attributes, often using standard vocabularies, should be added. These metadata records immediately help productivity as the interactive tools support their use in selection and bulk operations. Users rapidly appreciate the power of the encoded semantics as they reap the benefits. This improves the quality of provenance for users and management. Which in turn facilitates analysis of collections of runs, enabling users to manage results and validate procedures. It fosters reuse of data and methods and facilitates diagnostic investigations and optimisations. We present S-ProvFlow’s use by scientists, research engineers and managers as part of the DARE hyper-platform as they create, validate and use their data-driven scientific workflows.
      68  92
  • Some of the metrics are blocked by your 
    Item type:Publication, Access status: Open Access ,

    Data integration and FAIR data management in Solid Earth Science

    Integrated use of multidisciplinary data is nowadays a recognized trend in scientific research, in particular in the domain of solid Earth science where the understanding of a physical process is improved and made complete by different types of measurements – for instance, ground acceleration, SAR imaging, crustal deformation – describing a physical phenomenon. FAIR principles are recognized as a means to foster data integration by providing a common set of criteria for building data stewardship systems for Open Science. However, the implementation of FAIR principles raises issues along dimensions like governance and legal beyond, of course, the technical one. In the latter, in particular, the development of FAIR data provision systems is often delegated to Research Infrastructures or data providers, with support in terms of metrics and best practices offered by cluster projects or dedicated initiatives. In the current work, we describe the approach to FAIR data management in the European Plate Observing System (EPOS), a distributed research infrastructure in the solid Earth science domain that includes more than 250 individual research infrastructures across 25 countries in Europe. We focus in particular on the technical aspects, but including also governance, policies and organizational elements, by describing the architecture of the EPOS delivery framework both from the organizational and technical point of view and by outlining the key principles used in the technical design. We describe how a combination of approaches, namely rich metadata and service-based systems design, are required to achieve data integration. We show the system architecture and the basic features of the EPOS data portal, that integrates data from more than 220 services in a FAIR way. The construction of such a portal was driven by the EPOS FAIR data management approach, that by defining a clear roadmap for compliance with the FAIR principles, produced a number of best practices and technical approaches for complying with the FAIR principles. Such a work, that spans over a decade but concentrates the key efforts in the last 5 years with the EPOS Implementation Phase project and the establishment of EPOS-ERIC, was carried out in synergy with other EU initiatives dealing with FAIR data. On the basis of the EPOS experience, future directions are outlined, emphasizing the need to provide i) FAIR reference architectures that can ease data practitioners and engineers from the domain communities to adopt FAIR principles and build FAIR data systems; ii) a FAIR data management framework addressing FAIR through the entire data lifecycle, including reproducibility and provenance; and iii) the extension of the FAIR principles to policies and governance dimensions. .
      439  366
  • Some of the metrics are blocked by your 
    Item type:Publication, Access status: Open Access ,

    VERCE delivers a productive e-Science environment for seismology research

    The VERCE project has pioneered an e-Infrastructure to support researchers using established simulation codes on high-performance computers in conjunction with multiple sources of observational data. This is accessed and organised via the VERCE science gateway that makes it convenient for seismologists to use these resources from any location via the Internet. Their data handling is made flexible and scalable by two Python libraries, ObsPy and dispel4py and by data services delivered by ORFEUS and EUDAT. Provenance driven tools enable rapid exploration of results and of the relationships between data, which accelerates understanding and method improvement. These powerful facilities are integrated and draw on many other e-Infrastructures. This paper presents the motivation for building such systems, it reviews how solid-Earth scientists can make significant research progress using them and explains the architecture and mechanisms that make their construction and operation achievable. We conclude with a summary of the achievements to date and identify the crucial steps needed to extend the capabilities for seismologists, for solid-Earth scientists and for similar disciplines.
      106  234
  • Some of the metrics are blocked by your 
    Item type:Publication, Access status: Open Access ,

    SWIRRL. Managing Provenance-aware and Reproducible Workspaces

    Modern interactive tools for data analysis and visualisation are designed to expose their functionalities as a service through the Web. We present in this paper a Web API (SWIRRL) that allows Virtual Research Environments (VREs) to easily integrate such tools in their websites and re-purpose them to their users. The API deals, on behalf of the clients, with the underlying complexity of allocating and managing resources within a target cloud platform. By combining storage and containerised services, offering analysis notebooks and other visualisation software, the API creates dedicated working sessions on-demand, which can be accessed collaboratively. Thanks to the API's support for workflow execution, SWIRRL workspaces can be automatically populated with data of interest collected from external data providers. The system keeps track of updates and changes affecting the data and the tools by adopting versioning and standard provenance technologies. Users are provided with interactive controls enabling traceability and recovery actions, including the possibility of creating executable snapshots of their environments. SWIRRL is built in cooperation with two research infrastructures in the field of solid earth science and climate data modeling. We report on the particular adoptions and use cases.
      32  59
  • Some of the metrics are blocked by your 
    Item type:Publication, Access status: Open Access ,

    DARE: A Reflective Platform Designed to Enable Agile Data-Driven Research on the Cloud

    The DARE platform has been designed to help research developers deliver user-facing applications and solutions over diverse underlying e-infrastructures, data and computational contexts. The platform is Cloud-ready, and relies on the exposure of APIs, which are suitable for raising the abstraction level and hiding complexity. At its core, the platform implements the cataloguing and execution of fine-grained and Python-based dispel4py workflows as services. Reflection is achieved via a logical knowledge base, comprising multiple internal catalogues, registries and semantics, while it supports persistent and pervasive data provenance. This paper presents design and implementation aspects of the DARE platform, as well as it provides directions for future development.
      76  205
  • Some of the metrics are blocked by your 
    Item type:Publication, Access status: Open Access ,

    Active provenance for Data-Intensive workflows: engaging users and developers

    We present a practical approach for provenance capturing in Data-Intensive workflow systems. It provides contextualisation by recording injected domain metadata with the provenance stream. It offers control over lineage precision, combining automation with specified adaptations. We address provenance tasks such as extraction of domain metadata, injection of custom annotations, accuracy and integration of records from multiple independent workflows running in distributed contexts. To allow such flexibility, we introduce the concepts of programmable Provenance Types and Provenance Configuration.Provenance Types handle domain contextualisation and allow developers to model lineage patterns by re-defining API methods, composing easy-to-use extensions. Provenance Configuration, instead, enables users of a Data-Intensive workflow execution to prepare it for provenance capture, by configuring the attribution of Provenance Types to components and by specifying grouping into semantic clusters. This enables better searches over the lineage records. Provenance Types and Provenance Configuration are demonstrated in a system being used by computational seismologists. It is based on an extended provenance model, S-PROV.
      56  174
  • Some of the metrics are blocked by your 
    Item type:Publication, Access status: Open Access ,
      49  367