Released
Software

DASF: Messaging Python: A python RPC wrapper for the data analytics software framework

Cite as:

Eggert, Daniel; Sommer, Philipp; Dransch, Doris (2021): DASF: Messaging Python: A python RPC wrapper for the data analytics software framework. GFZ Data Services. https://doi.org/10.5880/GFZ.1.4.2021.005

Status

I   N       R   E   V   I   E   W : Eggert, Daniel; Sommer, Philipp; Dransch, Doris (2021): DASF: Messaging Python: A python RPC wrapper for the data analytics software framework. GFZ Data Services. https://doi.org/10.5880/GFZ.1.4.2021.005

There is a new version of this Software:

Abstract

DASF: Messaging Python is part of the Data Analytics Software Framework (DASF, https://git.geomar.de/digital-earth/dasf), developed at the GFZ German Research Centre for Geosciences. It is funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/).

DASF: Messaging Python is a RPC (remote procedure call) wrapper library for the python programming language. As part of the data analytics software framework DASF, it implements the DASF RPC messaging protocol. This message broker based RPC implementation supports the integration of algorithms and methods implemented in python in a distributed environment. It utilizes pydantic (https://pydantic-docs.helpmanual.io/) for data and model validation using python type annotations. Currently the implementation relies on Apache Pulsar (https://pulsar.apache.org/) as a central message broker instance.

Additional Information

The data analytics software framework DASF, developed at the GFZ German Research Centre for Geosciences (https://www.gfz-potsdam.de) and funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/), provides a framework for scientists to conduct data analysis in distributed environments.

DASF supports scientists to conduct data analysis in distributed IT infrastructures by sharing data analysis tools and data. For this purpose, DASF defines a remote procedure call (RPC) messaging protocol that uses a central message broker instance. Scientists can augment their tools and data with this protocol to share them with others. DASF supports many programming languages and platforms since the implementation of the protocol uses WebSockets. It provides two ready-to-use language bindings for the messaging protocol, one for Python and one for the Typescript programming language. In order to share a python method or class, users add an annotation in front of it. In addition, users need to specify the connection parameters of the message broker. The central message broker approach allows the method and the client calling the method to actively establish a connection, which enables using methods deployed behind firewalls. DASF uses Apache Pulsar (https://pulsar.apache.org/) as its underlying message broker. The Typescript bindings are primarily used in conjunction with web frontend components, which are also included in the DASF-Web library. They are designed to attach directly to the data returned by the exposed RPC methods. This supports the development of highly exploratory data analysis tools. DASF also provides a progress reporting API that enables users to monitor long-running remote procedure calls.

Technical Information

Copyright 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences, Potsdam, Germany / DASF Data Analytics Software Framework

Licensed under the Apache License, Version 2.0 (the "License"); you may not use these files except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Authors

  • Eggert, Daniel;GFZ German Research Centre for Geosciences, Potsdam, Germany
  • Sommer, Philipp;Helmholtz-Zentrum Hereon, Geesthacht, Germany
  • Dransch, Doris;GFZ German Research Centre for Geosciences, Potsdam, Germany

Contact

Keywords

DASF, Data Analytics Software Framework, RPC, remote procedure call, message broker, distributed analysis, python

GCMD Science Keywords

Files

License: Apache License, Version 2.0; Copyright (C) 2021 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences

Related Work

Part of

Previous Version of

References