DASF: A data analytics software framework for distributed environments

Cite as:

Eggert, Daniel; Dransch, Doris (2021): DASF: A data analytics software framework for distributed environments. GFZ Data Services. https://doi.org/10.5880/GFZ.1.4.2021.004

Status

I N R E V I E W : Eggert, Daniel; Dransch, Doris (2021): DASF: A data analytics software framework for distributed environments. GFZ Data Services. https://doi.org/10.5880/GFZ.1.4.2021.004

There is a new version of this Software:

https://dx.doi.org/10.5880/GFZ.1.4.2021.008

Abstract

The success of scientific projects increasingly depends on using data analysis tools and data in distributed IT infrastructures. Scientists need to use appropriate data analysis tools and data, extract patterns from data using appropriate computational resources, and interpret the extracted patterns. Data analysis tools and data reside on different machines because the volume of the data often demands specific resources for their storage and processing, and data analysis tools usually require specific computational resources and run-time environments. The data analytics software framework DASF, developed at the GFZ German Research Centre for Geosciences (https://www.gfz-potsdam.de) and funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/), provides a framework for scientists to conduct data analysis in distributed environments.

The data analytics software framework DASF supports scientists to conduct data analysis in distributed IT infrastructures by sharing data analysis tools and data. For this purpose, DASF defines a remote procedure call (RPC) messaging protocol that uses a central message broker instance. Scientists can augment their tools and data with this protocol to share them with others. DASF supports many programming languages and platforms since the implementation of the protocol uses WebSockets. It provides two ready-to-use language bindings for the messaging protocol, one for Python and one for the Typescript programming language. In order to share a python method or class, users add an annotation in front of it. In addition, users need to specify the connection parameters of the message broker. The central message broker approach allows the method and the client calling the method to actively establish a connection, which enables using methods deployed behind firewalls. DASF uses Apache Pulsar (https://pulsar.apache.org/) as its underlying message broker. The Typescript bindings are primarily used in conjunction with web frontend components, which are also included in the DASF-Web library. They are designed to attach directly to the data returned by the exposed RPC methods. This supports the development of highly exploratory data analysis tools. DASF also provides a progress reporting API that enables users to monitor long-running remote procedure calls.

One application using the framework is the Digital Earth Flood Event Explorer (https://git.geomar.de/digital-earth/flood-event-explorer). The Digital Earth Flood Event Explorer integrates several exploratory data analysis tools and remote procedures deployed at various Helmholtz centers across Germany.

Authors

Eggert, Daniel;GFZ German Research Centre for Geosciences, Potsdam, Germany
Dransch, Doris;GFZ German Research Centre for Geosciences, Potsdam, Germany

Contact

Eggert, Daniel; GFZ German Research Centre for Geosciences, Potsdam, Germany; ➦

Keywords

DASF, RPC, Python, Progress, Data Analytics Software Framework

GCMD Science Keywords

More Metadata

datacite: / download xml
- resource (xsi:schemaLocation=http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4.3/metadata.xsd)
  - identifier (identifierType=DOI): 10.5880/GFZ.1.4.2021.004
  - creators
    - creator
      - creatorName (nameType=Personal): Eggert, Daniel
      - givenName: Daniel
      - familyName: Eggert
      - nameIdentifier (nameIdentifierScheme=ORCID): 0000-0003-0251-4390
      - affiliation: GFZ German Research Centre for Geosciences, Potsdam, Germany
    - creator
      - creatorName (nameType=Personal): Dransch, Doris
      - givenName: Doris
      - familyName: Dransch
      - affiliation: GFZ German Research Centre for Geosciences, Potsdam, Germany
  - titles
    - title: DASF: A data analytics software framework for distributed environments
  - publisher: GFZ Data Services
  - publicationYear: 2021
  - subjects
    - subject: DASF
    - subject: RPC
    - subject: Python
    - subject: Progress
    - subject: Data Analytics Software Framework
    - subject (subjectScheme=NASA/GCMD Earth Science Keywords): EARTH SCIENCE SERVICES > DATA ANALYSIS AND VISUALIZATION
    - subject (subjectScheme=NASA/GCMD Earth Science Keywords): EARTH SCIENCE SERVICES > DATA ANALYSIS AND VISUALIZATION > STATISTICAL APPLICATIONS
    - subject (subjectScheme=NASA/GCMD Earth Science Keywords): EARTH SCIENCE SERVICES > DATA MANAGEMENT/DATA HANDLING
    - subject (subjectScheme=NASA/GCMD Earth Science Keywords): EARTH SCIENCE SERVICES > DATA MANAGEMENT/DATA HANDLING > DATA NETWORKING/DATA TRANSFER TOOLS
  - contributors
    - contributor (contributorType=ContactPerson)
      - contributorName: Eggert, Daniel
      - affiliation: GFZ German Research Centre for Geosciences, Potsdam, Germany
  - resourceType (resourceTypeGeneral=Software): Software
  - relatedIdentifiers
    - relatedIdentifier (relatedIdentifierType=DOI relationType=HasPart): 10.5880/GFZ.1.4.2021.005
    - relatedIdentifier (relatedIdentifierType=DOI relationType=HasPart): 10.5880/GFZ.1.4.2021.006
    - relatedIdentifier (relatedIdentifierType=DOI relationType=HasPart): 10.5880/GFZ.1.4.2021.007
    - relatedIdentifier (relatedIdentifierType=URL relationType=Cites): https://www.digitalearth-hgf.de/
    - relatedIdentifier (relatedIdentifierType=URL relationType=Cites): https://git.geomar.de/digital-earth/flood-event-explorer
    - relatedIdentifier (relatedIdentifierType=DOI relationType=IsPreviousVersionOf): 10.5880/GFZ.1.4.2021.008
  - sizes:
  - formats:
  - rightsList
    - rights (rightsURI=http://www.apache.org/licenses/LICENSE-2.0): Apache License, Version 2.0; Copyright (C) 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences
  - descriptions
    - description (descriptionType=Abstract)
      - br:
      - br:
      - br:
      - br:
      - br:
  - fundingReferences
    - fundingReference
      - funderName: Helmholtz-Gemeinschaft
      - funderIdentifier (funderIdentifierType=Crossref Funder ID): http://doi.org/10.13039/501100001656
      - awardTitle: Impuls- und Vernetzungsfond der Helmholtz Gemeinschaft
iso19115: / download xml
- MD_Metadata (xsi:schemaLocation=http://www.isotc211.org/2005/gmd http://www.isotc211.org/2005/gmd/gmd.xsd)
  - fileIdentifier
    - CharacterString: 10.5880/GFZ.1.4.2021.004
  - language
    - LanguageCode (codeList=http://www.loc.gov/standards/iso639-2/ codeListValue=eng): eng
  - characterSet
    - MD_CharacterSetCode (codeListValue=utf8 codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_CharacterSetCode):
  - hierarchyLevel
    - MD_ScopeCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#MD_ScopeCode codeListValue=dataset): dataset
  - hierarchyLevelName
    - CharacterString:
  - contact
    - CI_ResponsibleParty
      - organisationName
        CharacterString: GFZ German Research Centre for Geosciences
      - contactInfo
        CI_Contact
        address
        CI_Address
        electronicMailAddress
        CharacterString: datapub(_at_)gfz-potsdam.de
        onlineResource
        CI_OnlineResource
        linkage
        URL: http://www.gfz-potsdam.de
        name
        CharacterString: Helmholtz-Centre Potsdam - GFZ German Research Centre for Geosciences
        description
        CharacterString: Helmholtz-Centre Potsdam - GFZ German Research Centre for Geosciences
      - role
        CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=pointOfContact): pointOfContact
  - dateStamp
    - Date: 2022-09-27
  - referenceSystemInfo
    - MD_ReferenceSystem
      - referenceSystemIdentifier
        RS_Identifier
        code
        CharacterString: urn:ogc:def:crs:EPSG:4326
  - identificationInfo
    - MD_DataIdentification
      - citation
        CI_Citation
        title
        CharacterString: DASF: A data analytics software framework for distributed environments
        date
        CI_Date
        date
        Date: 2022-09-27
        dateType
        CI_DateTypeCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_DateTypeCode codeListValue=revision): revision
        identifier
        MD_Identifier
        code
        CharacterString: doi:10.5880/GFZ.1.4.2021.004
        citedResponsibleParty (xlink:href=http://orcid.org/0000-0003-0251-4390)
        CI_ResponsibleParty
        individualName
        CharacterString: Eggert, Daniel
        organisationName
        CharacterString: GFZ German Research Centre for Geosciences, Potsdam, Germany
        role
        CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=author): author
        citedResponsibleParty
        CI_ResponsibleParty
        individualName
        CharacterString: Dransch, Doris
        organisationName
        CharacterString: GFZ German Research Centre for Geosciences, Potsdam, Germany
        role
        CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=author): author
        citedResponsibleParty
        CI_ResponsibleParty
        organisationName
        CharacterString: GFZ Data Services
        role
        CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=publisher): publisher
      - abstract
        CharacterString: The success of scientific projects increasingly depends on using data analysis tools and data in distributed IT infrastructures. Scientists need to use appropriate data analysis tools and data, extract patterns from data using appropriate computational resources, and interpret the extracted patterns. Data analysis tools and data reside on different machines because the volume of the data often demands specific resources for their storage and processing, and data analysis tools usually require specific computational resources and run-time environments. The data analytics software framework DASF, developed at the GFZ German Research Centre for Geosciences (https://www.gfz-potsdam.de) and funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/), provides a framework for scientists to conduct data analysis in distributed environments. The data analytics software framework DASF supports scientists to conduct data analysis in distributed IT infrastructures by sharing data analysis tools and data. For this purpose, DASF defines a remote procedure call (RPC) messaging protocol that uses a central message broker instance. Scientists can augment their tools and data with this protocol to share them with others. DASF supports many programming languages and platforms since the implementation of the protocol uses WebSockets. It provides two ready-to-use language bindings for the messaging protocol, one for Python and one for the Typescript programming language. In order to share a python method or class, users add an annotation in front of it. In addition, users need to specify the connection parameters of the message broker. The central message broker approach allows the method and the client calling the method to actively establish a connection, which enables using methods deployed behind firewalls. DASF uses Apache Pulsar (https://pulsar.apache.org/) as its underlying message broker. The Typescript bindings are primarily used in conjunction with web frontend components, which are also included in the DASF-Web library. They are designed to attach directly to the data returned by the exposed RPC methods. This supports the development of highly exploratory data analysis tools. DASF also provides a progress reporting API that enables users to monitor long-running remote procedure calls. One application using the framework is the Digital Earth Flood Event Explorer (https://git.geomar.de/digital-earth/flood-event-explorer). The Digital Earth Flood Event Explorer integrates several exploratory data analysis tools and remote procedures deployed at various Helmholtz centers across Germany.
      - pointOfContact
        CI_ResponsibleParty
        individualName
        CharacterString: Eggert, Daniel
        organisationName
        CharacterString: GFZ German Research Centre for Geosciences, Potsdam, Germany
        contactInfo
        CI_Contact
        address
        CI_Address
        electronicMailAddress
        CharacterString: daniel.eggert(_at_)gfz-potsdam.de
        onlineResource
        CI_OnlineResource
        linkage
        URL: https://www.gfz-potsdam.de/staff/daniel-eggert/?
        function
        CI_OnLineFunctionCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode codeListValue=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode_information): informtion
        role
        CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=pointOfContact): pointOfContact
      - descriptiveKeywords
        MD_Keywords
        keyword
        CharacterString: DASF
        keyword
        CharacterString: RPC
        keyword
        CharacterString: Python
        keyword
        CharacterString: Progress
        keyword
        CharacterString: Data Analytics Software Framework
      - descriptiveKeywords
        MD_Keywords
        keyword
        CharacterString: EARTH SCIENCE SERVICES > DATA ANALYSIS AND VISUALIZATION
        keyword
        CharacterString: EARTH SCIENCE SERVICES > DATA ANALYSIS AND VISUALIZATION > STATISTICAL APPLICATIONS
        keyword
        CharacterString: EARTH SCIENCE SERVICES > DATA MANAGEMENT/DATA HANDLING
        keyword
        CharacterString: EARTH SCIENCE SERVICES > DATA MANAGEMENT/DATA HANDLING > DATA NETWORKING/DATA TRANSFER TOOLS
        thesaurusName
        CI_Citation
        title
        CharacterString: NASA/GCMD Earth Science Keywords
        date
        CI_Date
        date (gco:nilReason=missing):
        dateType
        CI_DateTypeCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_DateTypeCode codeListValue=publication): publication
      - resourceConstraints (xlink:href=http://www.apache.org/licenses/LICENSE-2.0)
        MD_Constraints
        useLimitation
        CharacterString: Apache License, Version 2.0; Copyright (C) 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences
      - resourceConstraints
        MD_LegalConstraints
        accessConstraints
        MD_RestrictionCode (codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_RestrictionCode codeListValue=otherRestrictions):
        otherConstraints
        CharacterString: Apache License, Version 2.0; Copyright (C) 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences
      - resourceConstraints
        MD_SecurityConstraints
        classification
        MD_ClassificationCode (codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_ClassificationCode codeListValue=unclassified):
      - aggregationInfo
        MD_AggregateInformation
        aggregateDataSetIdentifier
        RS_Identifier
        code
        CharacterString: 10.5880/GFZ.1.4.2021.005
        codeSpace
        CharacterString: DOI
        associationType
        DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=HasPart): HasPart
      - aggregationInfo
        MD_AggregateInformation
        aggregateDataSetIdentifier
        RS_Identifier
        code
        CharacterString: 10.5880/GFZ.1.4.2021.006
        codeSpace
        CharacterString: DOI
        associationType
        DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=HasPart): HasPart
      - aggregationInfo
        MD_AggregateInformation
        aggregateDataSetIdentifier
        RS_Identifier
        code
        CharacterString: 10.5880/GFZ.1.4.2021.007
        codeSpace
        CharacterString: DOI
        associationType
        DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=HasPart): HasPart
      - aggregationInfo
        MD_AggregateInformation
        aggregateDataSetIdentifier
        RS_Identifier
        code
        CharacterString: https://www.digitalearth-hgf.de/
        codeSpace
        CharacterString: URL
        associationType
        DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=Cites): Cites
      - aggregationInfo
        MD_AggregateInformation
        aggregateDataSetIdentifier
        RS_Identifier
        code
        CharacterString: https://git.geomar.de/digital-earth/flood-event-explorer
        codeSpace
        CharacterString: URL
        associationType
        DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=Cites): Cites
      - aggregationInfo
        MD_AggregateInformation
        aggregateDataSetIdentifier
        RS_Identifier
        code
        CharacterString: 10.5880/GFZ.1.4.2021.008
        codeSpace
        CharacterString: DOI
        associationType
        DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=IsPreviousVersionOf): IsPreviousVersionOf
      - language
        CharacterString: eng
      - characterSet
        MD_CharacterSetCode (codeListValue=utf8 codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_CharacterSetCode):
      - topicCategory
        MD_TopicCategoryCode: geoscientificInformation
  - distributionInfo
    - MD_Distribution
      - transferOptions
        MD_DigitalTransferOptions
        onLine
        CI_OnlineResource
        linkage
        URL: http://doi.org/10.5880/GFZ.1.4.2021.004
        protocol
        CharacterString: WWW:LINK-1.0-http--link
        name
        CharacterString: Data Access - DOI
        description
        CharacterString: Data Access - DOI
        function
        CI_OnLineFunctionCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode codeListValue=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode_information): information
  - dataQualityInfo
    - DQ_DataQuality
      - scope
        DQ_Scope
        level
        MD_ScopeCode (codeListValue=dataset codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_ScopeCode):

Files

Related Work

Cites

Includes

Previous Version of

https://dx.doi.org/10.5880/GFZ.1.4.2021.008