Weaver
| Summary | Features | Links | Configuration | Documentation | Extra Details & Sponsors |
Implementations
OGC API - Processes (WPS-REST bindings)
Web Processing Services (WPS)
Earth System Grid Federation (ESGF) processes
Execution Management Service (EMS) for dispatching distrubted workflow processing
Application, Deployment and Execution Service (ADES) for processing close to the data
- Weaver (the nest-builder)
Weaver birds build exquisite and elaborate nest structures that are a rival to any human feat of engineering. Some of these nests are the largest structures to be built by birds. [Eden].
Although weavers are named for their elaborately woven nests, some are notable for their selective parasitic nesting habits instead. [Wikipedia]
Weaver is an OGC-API flavored Execution Management Service that allows the execution of workflows chaining various applications and Web Processing Services inputs and outputs. Remote execution is deferred by the EMS to one or many Application, Deployment and Execution Service or remote service providers, and employs Common Workflow Language (CWL) configurations to define an OGC Application Package deployed for each process.
dependencies |
|
---|---|
license |
|
build status |
|
tests status |
|
releases |
Summary
Weaver is primarily an Execution Management Service that allows the execution of workflows chaining various applications and Web Processing Services inputs and outputs. Remote execution of each process in a workflow chain is dispatched by the EMS to one or many registered Application, Deployment and Execution Service by ensuring the transfer of files accordingly between instances when located across multiple remote locations.
Weaver can also accomplish the ADES role in order to perform application deployment at the data source using the application definition provided by Common Workflow Language (CWL) configuration. It can then directly execute a registered process OGC Application Package with received inputs from a WPS request to expose output results for a following ADES in a EMS workflow execution chain.
Weaver extends OGC API - Processes by providing additional functionalities such as more detailed job logs endpoints, adding more process management and search request options than required by the standard, and supporting remote providers registration for dynamic process definitions, to name a few. Because of this, not all features offered in Weaver are guaranteed to be applicable on other similarly behaving ADES and/or EMS instances. The reference specification is tracked to preserve the minimal conformance requirements and provide feedback to Open Geospatial Consortium (OGC) in this effect.
Weaver can be launched either as an EMS, an ADES or an HYBRID of both according to its configuration. For more details, see Configuration and Documentation sections.
Features
Following videos present some of the features and potential capabilities of servicing and executing processes offered by Application, Deployment and Execution Service and Execution Management Service instances like Weaver.
Keywords: Big Data, software architecture, Earth Observation, satellite data, processing, climate change, machine learning, climate services.
Applications
The video shares the fundamental ideas behind the architecture, illustrates how application stores for Earth Observation data processing can evolve, and illustrates the advantages with applications based on machine learning.
Platform
The video shares the fundamental ideas behind the architecture, illustrates how platform managers can benefit from application stores, and shows the potential for multidisciplinary workflows in thematic platforms.
Links
Docker image repositories:
Name |
Reference |
Access |
---|---|---|
DockerHub |
||
CRIM registry |
||
CRIM OGC Processes |
For a prebuilt image, pull as follows:
docker pull pavics/weaver:5.8.0
For convenience, following tags are also available:
weaver:5.8.0-manager
: Weaver image that will run the API for WPS process and job management.weaver:5.8.0-worker
: Weaver image that will run the process job runner application.
Following links correspond to existing servers with Weaver configured as EMS or ADES instances respectively.
Institution & Partners |
Project & Description |
Version |
Entrypoint |
---|---|---|---|
Demonstration Services Portal |
|||
PAVICS Server |
|||
Marble Climate - RedOak Instance |
|||
ClimateData.ca / DonneesClimatiques.ca Portal |
Note
The test servers will not necessarily be up-to-date with the latest version.
Configuration
All configuration settings can be overridden using a weaver.ini
file that will be picked during
instantiation of the application. An example of such file is provided here: weaver.ini.example.
Setting the operational mode of Weaver (EMS/ADES/HYBRID) is accomplished using the
weaver.configuration
field of weaver.ini
. For more configuration details, please refer to Documentation.
Documentation
The REST API documentation is auto-generated and served under any running Weaver application on route
{WEAVER_URL}/api/
. This documentation will correspond to the version of the executed Weaver application.
For the latest documentation, you can refer to the OpenAPI Specification served directly on ReadTheDocs.
More ample details about installation, configuration and usage are also provided on ReadTheDocs. These are generated from corresponding information provided in docs source directory.
Extra Details & Sponsors
The project was initially developed upon OGC Testbed-14 – ESA Sponsored Threads – Exploitation Platform findings and improvements following from previous OGC Testbed-13 - Cloud Engineering Report architecture designs. It was also built upon sponsorship from the U.S. Department of Energy to support common API of the Earth System Grid Federation (ESGF). The findings are reported on the OGC Testbed-14 thread, and more explicitly in the ADES & EMS Results and Best Practices Engineering Report.
The project has been employed for OGC Testbed-15 - Machine Learning Thread to demonstrate the use of Machine Learning interactions with OGC web standards in the context of natural resources applications. The advancements are reported through the OGC Testbed-15: Machine Learning Engineering Report.
Developments are continued in OGC Testbed-16 to improve methodologies in order to provide better interoperable geospatial data processing in the areas of Earth Observation Application Packages. Findings and recommendations are presented in the OGC Testbed-16: Data Access and Processing Engineering Report.
Todo
deploy from ipynb, add OGC Testbed-16: Earth Observation Application Packages with Jupyter Notebooks Engineering Report (https://github.com/crim-ca/weaver/issues/63)
Videos and more functionalities were introduced in Weaver following OGC Earth Observation Applications Pilot. Corresponding developments are reported in the OGC Earth Observation Applications Pilot: CRIM Engineering Report.
Weaver has been used to participate in interoperability testing effort that lead to OGC Best Practice for Earth Observation Application Package technical report. This resulted, along with previous efforts, in the definition of OGC API - Processes - Part 2: Deploy, Replace, Undeploy (DRU) backed by validated test cases using Common Workflow Language (CWL) as the representation method for the deployment and execution of OGC Application Package close to the data.
Weaver is employed in the OGC Open Science Persistent Demonstrator initiative to demonstrate reusability, portability, and transparency in the context of open science in Earth Observation, using OGC Application Package encoded as CWL for interoperability and distributed processing workflows. Its related developments and demonstrations were presented at the OGC 129th Member’s Meeting (2024, Montréal) and the Earth Science Information Partners (ESIP) 2024 Meeting.
Weaver is employed in OGC Testbed-20 - GeoDataCubes to improve and work on the alignment of multiple community standards involved in workflow design, such as CWL, openEO and OGC API - Processes - Part 3: Workflows and Chaining, for processing of multidimensional data involved through GeoDataCube interactions.
The project is furthermore developed through the Data Analytics for Canadian Climate Services (DACCS) initiative and is employed by the ClimateData.ca / DonneesClimatiques.ca portal.
Weaver is implemented in Python with the Pyramid web framework. It is part of PAVICS and Birdhouse ecosystems and is available within the birdhouse-deploy server stack.
Table of Contents
- Installation
- Configuration
- Configuration Settings
- Configuration of Celery with MongoDB Backend
- Configuration of AWS S3 Buckets
- Configuration of Data Sources
- Configuration of WPS Processes
- Configuration of CWL Processes
- Configuration of Request Options
- Configuration of Quotation Estimation
- Configuration of File Vault
- Starting the Application
- Running Weaver
- Tutorial
- Processes
- Type of Processes
- Managing processes included in Weaver ADES/EMS
- Register a new process (Deploy)
- Access registered processes (GetCapabilities, DescribeProcess)
- Modify an existing process (Update, Replace, Undeploy)
- Execution of a process (Execute)
- Execution Body
- Execution Mode
- Execution Steps
- Process Operations
- Workflow Step Operations
- File Reference Types
- File Reference Names
- File Vault Inputs
- AWS S3 Bucket References
- OpenSearch Data Source
- BoundingBox Inputs
- Collection Inputs
- Collection Outputs
- Multiple Inputs
- Multiple Outputs
- Outputs Location
- Notification Subscribers
- Monitoring of a process execution (GetStatus)
- Obtaining results, outputs, logs or errors
- Uploading File to the Vault
- WPS Endpoint
- Special Weaver EMS use-cases
- Application Package
- Quotation and Billing
- Examples
- OpenAPI Specification
- Weaver CLI and Client
- Source Code Documentation
- Changes
- Unreleased (latest)
- 5.8.0 (2024-09-05)
- 5.7.0 (2024-07-16)
- 5.6.1 (2024-06-14)
- 5.6.0 (2024-06-11)
- 5.5.0 (2024-06-06)
- 5.4.2 (2024-06-05)
- 5.4.1 (2024-06-03)
- 5.4.0 (2024-05-27)
- 5.3.0 (2024-05-13)
- 5.2.0 (2024-05-08)
- 5.1.1 (2024-03-19)
- 5.1.0 (2024-03-19)
- 5.0.0 (2023-12-12)
- 4.38.0 (2023-11-24)
- 4.37.0 (2023-11-22)
- 4.36.0 (2023-11-06)
- 4.35.0 (2023-11-03)
- 4.34.0 (2023-10-16)
- 4.33.0 (2023-10-06)
- 4.32.0 (2023-09-25)
- 4.31.0 (2023-09-14)
- 4.30.1 (2023-07-07)
- 4.30.0 (2023-03-24)
- 4.29.0 (2023-03-07)
- 4.28.0 (2022-12-06)
- 4.27.0 (2022-11-22)
- 4.26.0 (2022-10-31)
- 4.25.0 (2022-10-05)
- 4.24.0 (2022-09-29)
- 4.23.0 (2022-09-12)
- 4.22.0 (2022-08-18)
- 4.21.0 (2022-08-15)
- 4.20.0 (2022-07-15)
- 4.19.0 (2022-07-05)
- 4.18.0 (2022-06-09)
- 4.17.0 (2022-05-30)
- 4.16.1 (2022-05-12)
- 4.16.0 (2022-05-11)
- 4.15.0 (2022-04-20)
- 4.14.0 (2022-03-14)
- 4.13.0 (2022-03-09)
- 4.12.0 (2022-02-28)
- 4.11.0 (2022-02-24)
- 4.10.0 (2022-02-22)
- 4.9.1 (2022-02-21)
- 4.9.0 (2022-02-17)
- 4.8.0 (2022-01-11)
- 4.7.0 (2021-12-21)
- 4.6.0 (2021-12-15)
- 4.5.0 (2021-11-25)
- 4.4.0 (2021-11-19)
- 4.3.0 (2021-11-16)
- 4.2.1 (2021-10-20)
- 4.2.0 (2021-10-19)
- 4.1.2 (2021-10-13)
- 4.1.1 (2021-10-12)
- 4.1.0 (2021-09-29)
- 4.0.0 (2021-09-21)
- 3.5.0 (2021-08-19)
- 3.4.0 (2021-08-11)
- 3.3.0 (2021-07-16)
- 3.2.1 (2021-06-08)
- 3.2.0 (2021-06-08)
- 3.1.0 (2021-04-23)
- 3.0.0 (2021-03-16)
- 2.2.0 (2021-03-03)
- 2.1.0 (2021-02-26)
- 2.0.0 (2021-02-22)
- 1.14.0 (2021-01-11)
- 1.13.1 (2020-07-17)
- 1.13.0 (2020-07-15)
- 1.12.0 (2020-07-03)
- 1.11.0 (2020-07-02)
- 1.10.1 (2020-06-03)
- 1.10.0 (2020-06-03)
- 1.9.0 (2020-06-01)
- 1.8.1 (2020-05-22)
- 1.8.0 (2020-05-21)
- 1.7.0 (2020-05-15)
- 1.6.0 (2020-05-07)
- 1.5.1 (2020-03-26)
- 1.5.0 (2020-03-25)
- 1.4.0 (2020-03-18)
- 1.3.0 (2020-03-10)
- 1.2.0 (2020-03-06)
- 1.1.0 (2020-02-17)
- 1.0.0 (2020-01-28)
- 0.2.2 (2019-05-31)
- 0.2.1 (2019-05-29)
- 0.2.0 (2019-03-26)
- 0.1.3 (2019-03-07)
- 0.1.2 (2019-03-05)
- 0.1.1 (2019-03-04)
- 0.1.0 (2019-02-26)
- FAQ
- How to specify the Docker image reference?
- Fixing permission error on input files
- Problem connecting workflow steps together
- Where can I find references to CWL specification and examples?
- Where can I find examples of Process Deployment, Execution and Application Package definitions?
- Is there an easier way to interact with Weaver for its multiple requests and operations?
- Glossary
- Useful Links