Released
Software

DASF: Messaging Python: A python RPC wrapper for the data analytics software framework

Cite as:

Eggert, Daniel; Sommer, Philipp; Dransch, Doris (2021): DASF: Messaging Python: A python RPC wrapper for the data analytics software framework. GFZ Data Services. https://doi.org/10.5880/GFZ.1.4.2021.005

Status

I   N       R   E   V   I   E   W : Eggert, Daniel; Sommer, Philipp; Dransch, Doris (2021): DASF: Messaging Python: A python RPC wrapper for the data analytics software framework. GFZ Data Services. https://doi.org/10.5880/GFZ.1.4.2021.005

There is a new version of this Software:

Abstract

DASF: Messaging Python is part of the Data Analytics Software Framework (DASF, https://git.geomar.de/digital-earth/dasf), developed at the GFZ German Research Centre for Geosciences. It is funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/).

DASF: Messaging Python is a RPC (remote procedure call) wrapper library for the python programming language. As part of the data analytics software framework DASF, it implements the DASF RPC messaging protocol. This message broker based RPC implementation supports the integration of algorithms and methods implemented in python in a distributed environment. It utilizes pydantic (https://pydantic-docs.helpmanual.io/) for data and model validation using python type annotations. Currently the implementation relies on Apache Pulsar (https://pulsar.apache.org/) as a central message broker instance.

Additional Information

The data analytics software framework DASF, developed at the GFZ German Research Centre for Geosciences (https://www.gfz-potsdam.de) and funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/), provides a framework for scientists to conduct data analysis in distributed environments.

DASF supports scientists to conduct data analysis in distributed IT infrastructures by sharing data analysis tools and data. For this purpose, DASF defines a remote procedure call (RPC) messaging protocol that uses a central message broker instance. Scientists can augment their tools and data with this protocol to share them with others. DASF supports many programming languages and platforms since the implementation of the protocol uses WebSockets. It provides two ready-to-use language bindings for the messaging protocol, one for Python and one for the Typescript programming language. In order to share a python method or class, users add an annotation in front of it. In addition, users need to specify the connection parameters of the message broker. The central message broker approach allows the method and the client calling the method to actively establish a connection, which enables using methods deployed behind firewalls. DASF uses Apache Pulsar (https://pulsar.apache.org/) as its underlying message broker. The Typescript bindings are primarily used in conjunction with web frontend components, which are also included in the DASF-Web library. They are designed to attach directly to the data returned by the exposed RPC methods. This supports the development of highly exploratory data analysis tools. DASF also provides a progress reporting API that enables users to monitor long-running remote procedure calls.

Technical Information

Copyright 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences, Potsdam, Germany / DASF Data Analytics Software Framework

Licensed under the Apache License, Version 2.0 (the "License"); you may not use these files except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Authors

  • Eggert, Daniel;GFZ German Research Centre for Geosciences, Potsdam, Germany
  • Sommer, Philipp;Helmholtz-Zentrum Hereon, Geesthacht, Germany
  • Dransch, Doris;GFZ German Research Centre for Geosciences, Potsdam, Germany

Contact

    Funders

  • Alfred Wegener Institute Helmholtz Centre for Polar and Marine Research: Digital Earth (Impuls- und Vernetzungsfond)
  • Helmholtz-Gemeinschaft: Digital Earth (Impuls- und Vernetzungsfond)
  • Helmholtz-Zentrum Potsdam - Deutsches GeoForschungsZentrum GFZ: Digital Earth (Impuls- und Vernetzungsfond)

Keywords

DASF, Data Analytics Software Framework, RPC, remote procedure call, message broker, distributed analysis, python

GCMD Science Keywords

More Metadata

  • datacite: download xml
    • resource (xsi:schemaLocation=http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4.3/metadata.xsd)
      • identifier (identifierType=DOI): 10.5880/GFZ.1.4.2021.005
      • creators
        • creator
          • creatorName (nameType=Personal): Eggert, Daniel
          • givenName: Daniel
          • familyName: Eggert
          • nameIdentifier (nameIdentifierScheme=ORCID): 0000-0003-0251-4390
          • affiliation: GFZ German Research Centre for Geosciences, Potsdam, Germany
        • creator
          • creatorName (nameType=Personal): Sommer, Philipp
          • givenName: Philipp
          • familyName: Sommer
          • nameIdentifier (nameIdentifierScheme=ORCID): 0000-0001-6171-7716
          • affiliation: Helmholtz-Zentrum Hereon, Geesthacht, Germany
        • creator
          • creatorName (nameType=Personal): Dransch, Doris
          • givenName: Doris
          • familyName: Dransch
          • affiliation: GFZ German Research Centre for Geosciences, Potsdam, Germany
      • titles
        • title: DASF: Messaging Python: A python RPC wrapper for the data analytics software framework
      • publisher: GFZ Data Services
      • publicationYear: 2021
      • subjects
        • subject: DASF
        • subject: Data Analytics Software Framework
        • subject: RPC
        • subject: remote procedure call
        • subject: message broker
        • subject: distributed analysis
        • subject: python
        • subject (subjectScheme=NASA/GCMD Earth Science Keywords): EARTH SCIENCE SERVICES > DATA MANAGEMENT/DATA HANDLING
        • subject (subjectScheme=NASA/GCMD Earth Science Keywords): EARTH SCIENCE SERVICES > DATA MANAGEMENT/DATA HANDLING > DATA NETWORKING/DATA TRANSFER TOOLS
      • contributors
        • contributor (contributorType=ContactPerson)
          • contributorName: Eggert, Daniel
          • affiliation: GFZ German Research Centre for Geosciences
      • resourceType (resourceTypeGeneral=Software): Software
      • relatedIdentifiers
        • relatedIdentifier (relatedIdentifierType=DOI relationType=IsPartOf): 10.5880/GFZ.1.4.2021.004
        • relatedIdentifier (relatedIdentifierType=DOI relationType=References): 10.5880/GFZ.1.4.2021.006
        • relatedIdentifier (relatedIdentifierType=DOI relationType=References): 10.5880/GFZ.1.4.2021.007
        • relatedIdentifier (relatedIdentifierType=URL relationType=IsPartOf): https://git.geomar.de/digital-earth/dasf
        • relatedIdentifier (relatedIdentifierType=URL relationType=References): https://www.digitalearth-hgf.de/
        • relatedIdentifier (relatedIdentifierType=DOI relationType=IsPreviousVersionOf): 10.5880/GFZ.1.4.2021.008
      • sizes: 
      • formats: 
      • rightsList
        • rights (rightsURI=http://www.apache.org/licenses/LICENSE-2.0):  Apache License, Version 2.0; Copyright (C) 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences
      • descriptions
        • description (descriptionType=Abstract)
          • br: 
          • br: 
          • br: 
        • description (descriptionType=TechnicalInfo)
          • br: 
          • br: 
          • br: 
          • br: 
          • br: 
          • br: 
          • br: 
          • br: 
        • description (descriptionType=Other)
          • br: 
          • br: 
          • br: 
      • fundingReferences
        • fundingReference
          • funderName: Alfred Wegener Institute Helmholtz Centre for Polar and Marine Research
          • funderIdentifier (funderIdentifierType=Crossref Funder ID): http://doi.org/10.13039/501100003207
          • awardTitle: Digital Earth (Impuls- und Vernetzungsfond)
        • fundingReference
          • funderName: Helmholtz-Gemeinschaft
          • funderIdentifier (funderIdentifierType=Crossref Funder ID): http://doi.org/10.13039/501100001656
          • awardTitle: Digital Earth (Impuls- und Vernetzungsfond)
        • fundingReference
          • funderName: Helmholtz-Zentrum Potsdam - Deutsches GeoForschungsZentrum GFZ
          • funderIdentifier (funderIdentifierType=Crossref Funder ID): https://doi.org/10.13039/501100010956
          • awardTitle: Digital Earth (Impuls- und Vernetzungsfond)
  • iso19115: download xml
    • MD_Metadata (xsi:schemaLocation=http://www.isotc211.org/2005/gmd http://www.isotc211.org/2005/gmd/gmd.xsd)
      • fileIdentifier
        • CharacterString: 10.5880/GFZ.1.4.2021.005
      • language
        • LanguageCode (codeList=http://www.loc.gov/standards/iso639-2/ codeListValue=eng): eng
      • characterSet
        • MD_CharacterSetCode (codeListValue=utf8 codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_CharacterSetCode): 
      • hierarchyLevel
        • MD_ScopeCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#MD_ScopeCode codeListValue=dataset): dataset
      • hierarchyLevelName
        • CharacterString: 
      • contact
        • CI_ResponsibleParty
          • organisationName
            • CharacterString: GFZ German Research Centre for Geosciences
          • contactInfo
            • CI_Contact
              • address
                • CI_Address
                  • electronicMailAddress
                    • CharacterString: datapub(_at_)gfz-potsdam.de
              • onlineResource
                • CI_OnlineResource
                  • linkage
                    • URL: http://www.gfz-potsdam.de
                  • name
                    • CharacterString: Helmholtz-Centre Potsdam - GFZ German Research Centre for Geosciences
                  • description
                    • CharacterString: Helmholtz-Centre Potsdam - GFZ German Research Centre for Geosciences
          • role
            • CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=pointOfContact): pointOfContact
      • dateStamp
        • Date: 2022-09-27
      • referenceSystemInfo
        • MD_ReferenceSystem
          • referenceSystemIdentifier
            • RS_Identifier
              • code
                • CharacterString: urn:ogc:def:crs:EPSG:4326
      • identificationInfo
        • MD_DataIdentification
          • citation
            • CI_Citation
              • title
                • CharacterString: DASF: Messaging Python: A python RPC wrapper for the data analytics software framework
              • date
                • CI_Date
                  • date
                    • Date: 2022-09-27
                  • dateType
                    • CI_DateTypeCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_DateTypeCode codeListValue=revision): revision
              • identifier
                • MD_Identifier
                  • code
                    • CharacterString: doi:10.5880/GFZ.1.4.2021.005
              • citedResponsibleParty (xlink:href=http://orcid.org/0000-0003-0251-4390)
                • CI_ResponsibleParty
                  • individualName
                    • CharacterString: Eggert, Daniel
                  • organisationName
                    • CharacterString: GFZ German Research Centre for Geosciences, Potsdam, Germany
                  • role
                    • CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=author): author
              • citedResponsibleParty (xlink:href=http://orcid.org/0000-0001-6171-7716)
                • CI_ResponsibleParty
                  • individualName
                    • CharacterString: Sommer, Philipp
                  • organisationName
                    • CharacterString: Helmholtz-Zentrum Hereon, Geesthacht, Germany
                  • role
                    • CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=author): author
              • citedResponsibleParty
                • CI_ResponsibleParty
                  • individualName
                    • CharacterString: Dransch, Doris
                  • organisationName
                    • CharacterString: GFZ German Research Centre for Geosciences, Potsdam, Germany
                  • role
                    • CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=author): author
              • citedResponsibleParty
                • CI_ResponsibleParty
                  • organisationName
                    • CharacterString: GFZ Data Services
                  • role
                    • CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=publisher): publisher
          • abstract
            • CharacterString: DASF: Messaging Python is part of the Data Analytics Software Framework (DASF, https://git.geomar.de/digital-earth/dasf), developed at the GFZ German Research Centre for Geosciences. It is funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/). DASF: Messaging Python is a RPC (remote procedure call) wrapper library for the python programming language. As part of the data analytics software framework DASF, it implements the DASF RPC messaging protocol. This message broker based RPC implementation supports the integration of algorithms and methods implemented in python in a distributed environment. It utilizes pydantic (https://pydantic-docs.helpmanual.io/) for data and model validation using python type annotations. Currently the implementation relies on Apache Pulsar (https://pulsar.apache.org/) as a central message broker instance.
          • pointOfContact
            • CI_ResponsibleParty
              • individualName
                • CharacterString: Eggert, Daniel
              • organisationName
                • CharacterString: GFZ German Research Centre for Geosciences
              • contactInfo
                • CI_Contact
                  • address
                    • CI_Address
                      • electronicMailAddress
                        • CharacterString: daniel.eggert(_at_)gfz-potsdam.de
                  • onlineResource
                    • CI_OnlineResource
                      • linkage
                        • URL: https://www.gfz-potsdam.de/staff/daniel-eggert/?
                      • function
                        • CI_OnLineFunctionCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode codeListValue=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode_information): informtion
              • role
                • CI_RoleCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_RoleCode codeListValue=pointOfContact): pointOfContact
          • descriptiveKeywords
            • MD_Keywords
              • keyword
                • CharacterString: DASF
              • keyword
                • CharacterString: Data Analytics Software Framework
              • keyword
                • CharacterString: RPC
              • keyword
                • CharacterString: remote procedure call
              • keyword
                • CharacterString: message broker
              • keyword
                • CharacterString: distributed analysis
              • keyword
                • CharacterString: python
          • descriptiveKeywords
            • MD_Keywords
              • keyword
                • CharacterString: EARTH SCIENCE SERVICES > DATA MANAGEMENT/DATA HANDLING
              • keyword
                • CharacterString: EARTH SCIENCE SERVICES > DATA MANAGEMENT/DATA HANDLING > DATA NETWORKING/DATA TRANSFER TOOLS
              • thesaurusName
                • CI_Citation
                  • title
                    • CharacterString: NASA/GCMD Earth Science Keywords
                  • date
                    • CI_Date
                      • date (gco:nilReason=missing): 
                      • dateType
                        • CI_DateTypeCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_DateTypeCode codeListValue=publication): publication
          • resourceConstraints (xlink:href=http://www.apache.org/licenses/LICENSE-2.0)
            • MD_Constraints
              • useLimitation
                • CharacterString:  Apache License, Version 2.0; Copyright (C) 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences
          • resourceConstraints
            • MD_LegalConstraints
              • accessConstraints
                • MD_RestrictionCode (codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_RestrictionCode codeListValue=otherRestrictions): 
              • otherConstraints
                • CharacterString:  Apache License, Version 2.0; Copyright (C) 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences
          • resourceConstraints
            • MD_SecurityConstraints
              • classification
                • MD_ClassificationCode (codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_ClassificationCode codeListValue=unclassified): 
          • aggregationInfo
            • MD_AggregateInformation
              • aggregateDataSetIdentifier
                • RS_Identifier
                  • code
                    • CharacterString: 10.5880/GFZ.1.4.2021.004
                  • codeSpace
                    • CharacterString: DOI
              • associationType
                • DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=IsPartOf): IsPartOf
          • aggregationInfo
            • MD_AggregateInformation
              • aggregateDataSetIdentifier
                • RS_Identifier
                  • code
                    • CharacterString: 10.5880/GFZ.1.4.2021.006
                  • codeSpace
                    • CharacterString: DOI
              • associationType
                • DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=References): References
          • aggregationInfo
            • MD_AggregateInformation
              • aggregateDataSetIdentifier
                • RS_Identifier
                  • code
                    • CharacterString: 10.5880/GFZ.1.4.2021.007
                  • codeSpace
                    • CharacterString: DOI
              • associationType
                • DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=References): References
          • aggregationInfo
            • MD_AggregateInformation
              • aggregateDataSetIdentifier
                • RS_Identifier
                  • code
                    • CharacterString: https://git.geomar.de/digital-earth/dasf
                  • codeSpace
                    • CharacterString: URL
              • associationType
                • DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=IsPartOf): IsPartOf
          • aggregationInfo
            • MD_AggregateInformation
              • aggregateDataSetIdentifier
                • RS_Identifier
                  • code
                    • CharacterString: https://www.digitalearth-hgf.de/
                  • codeSpace
                    • CharacterString: URL
              • associationType
                • DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=References): References
          • aggregationInfo
            • MD_AggregateInformation
              • aggregateDataSetIdentifier
                • RS_Identifier
                  • code
                    • CharacterString: 10.5880/GFZ.1.4.2021.008
                  • codeSpace
                    • CharacterString: DOI
              • associationType
                • DS_AssociationTypeCode (codeList=http://datacite.org/schema/kernel-4 codeListValue=IsPreviousVersionOf): IsPreviousVersionOf
          • language
            • CharacterString: eng
          • characterSet
            • MD_CharacterSetCode (codeListValue=utf8 codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_CharacterSetCode): 
          • topicCategory
            • MD_TopicCategoryCode: geoscientificInformation
      • distributionInfo
        • MD_Distribution
          • transferOptions
            • MD_DigitalTransferOptions
              • onLine
                • CI_OnlineResource
                  • linkage
                    • URL: http://doi.org/10.5880/GFZ.1.4.2021.005
                  • protocol
                    • CharacterString: WWW:LINK-1.0-http--link
                  • name
                    • CharacterString: Data Access - DOI
                  • description
                    • CharacterString: Data Access - DOI
                  • function
                    • CI_OnLineFunctionCode (codeList=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode codeListValue=http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#CI_OnLineFunctionCode_information): information
      • dataQualityInfo
        • DQ_DataQuality
          • scope
            • DQ_Scope
              • level
                • MD_ScopeCode (codeListValue=dataset codeList=http://www.isotc211.org/2005/resources/codeList.xml#MD_ScopeCode): 
  • schema.org download

Files

License: Apache License, Version 2.0; Copyright (C) 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences

Related Work

Part of

Previous Version of

References