Changes

Unreleased (latest)

1.7.0 (2020-05-15)

Changes:

• Add additional status log for EOImage input modification with OpenSearch during process execution.
• Add captured stderr/stdout logging of underlying CWL application being executed to resulting Job logs (addresses first step of #131).
• Use weaver.utils.request_retry in even more places and extend convenience arguments offered by it to adapt it to specific use cases.

Fixes:

• Fix handling of WPS-REST output matching a JSON file for multiple-output format specified with a relative local path as specified by job output location. Only remote HTTP references where correctly parsed. Also avoid failing the job if the reference JSON parsing fails. It will simply return the original reference URL in this case without expanded data (relates to #25).
• Fix CWL job logs to be timezone aware, just like most other logs that will report UTC time.
• Fix JSON response parsing of remote provider processes.
• Fix parsing of CWL ordered parsing when I/O is specified as shorthand "<id>":"<type>" directly under the ``inputs or outputs dictionary instead of extended sub-dictionary variant (fixes #137).

1.6.0 (2020-05-07)

Changes:

• Reuse weaver.utils.request_retry function across a few locations that where essentially reimplementing the core functionality.
• Add even more failure-permissive request attempts when validating a MIME-type against IANA website.
• Add auto-resolution of common extensions known under PyWPS as well as employing their specific encoding.
• Add geotiff format type support via PyWPS (#100).
• Make WPS status check more resilient to failing WPS outputs location not found in case the directory path can be resolved to a valid local file representing the XML status (i.e.: don’t depend as much on the HTTP WPS output route).
• Ensure backward support of generic/default text/plain I/O when extracted from a referenced WPS-1/2 XML remote process which provides insufficient format details. For CWL output generated from it, replace the glob pattern to match anything (<id>.*) instead of <id>.txt extracted from text/plain to simulate MIME-type as */*. Issue log warning message for future use cases.

Fixes:

• Fix invalid AllowedValue parsing when using LiteralData inputs that resulted in AnyValue being parsed as a "None" string. This was transparent in case of string inputs and breaking for other types like integer when they attempted conversion.
• Fix erroneous Metadata keywords passed down to owslib.wps.Metadata objects in case of more verbose detailed not allowed by this implementation.
• Fix parsing of explicitly-typed optional array CWL I/O notation that was not considered (i.e.: using type as list with additional "null" instead of type: "<type>?" shorthand).
• Fix parsing of MIME-type from format field to exclude additional parameters (e.g.: ; charset=UTF-8 for remote IANA validation.

1.5.1 (2020-03-26)

Changes:

• Add unittest of utility function fetch_file.
• Split some unittest utility functions to allow more reuse.

Fixes:

• Fix invalid retry parameter not handled automatically by request.

1.5.0 (2020-03-25)

Changes:

• Adjust incorrectly parsed href file reference as WPS complex input which resulted in failing location retrieval.
• Partially address unnecessary fetch of file that has to be passed down to CWL, which will in turn request the file as required. Need update from PyWPS to resolve completely (#91, geopython/pywps#526).
• Adjust WPS output results to use relative HTTP path in order to recompose the output URL if server settings change.
• Support WPS output results as value (WPS literal data). Everything was considered an href file beforehand.
• Add additional timeout and retry during fetching of remote file for process jsonarray2netcdf to avoid unnecessary failures during edge case connexion problems.
• Add support of title and version field of builtin processes.

Fixes:

• Patch builtin process execution failing since cwltool 2.x update.
• Avoid long fetch operation using streamed request that defaulted to chuck size of 1. Now, we use an appropriate size according to available memory.

1.4.0 (2020-03-18)

Changes:

• Update owslib to 0.19.2
• Drop support for python 3.5

1.3.0 (2020-03-10)

Changes:

• Provide a way to override the external URL reported by WPS-1/2 and WPS-REST via configuration settings allowing for more advanced server-side results in response bodies.

1.2.0 (2020-03-06)

Changes:

• Add wps languages for other wps requests types: DescribeProcess and GetCapabilities.

Fixes:

• Fix a bug where the validation of OneOf items was casting the value to the first valid possibility. Now, it doesn’t change the value if it’s valid without casting it (and still casts it if it’s necessary to make it valid).

1.1.0 (2020-02-17)

Changes:

• Simplify docker image generation and make base/manager/worker variants all available under the same docker repo docker-registry.crim.ca/ogc/weaver with different tags (#5).
• Add planned future support of Accept-Language header for WPS-1/2 (geopython/OWSLib 0.20.0) (#74).
• Improved job logs update with message and progress to allow better tracking of internal operations and/or problems.
• Allow WPS builtin process jsonarray2netcdf to fetch a remote file.
• Change doc to point to DockerHub pavics/weaver images.
• Adjust CI rule long-lasting failures until it gets patched by original reference (gitleaks-actions#3).

Fixes:

• Fix readthedocs documentation generation.
• Fix .travis docker image build condition.
• Fix geopython/OWSLib>=0.19.1 requirement for Python 3.8 support (#62).
• Fix job update filling due to status location incorrectly resolved according to configured PyWPS output path.

1.0.0 (2020-01-28)

New Features:

• Add notification_email field to Job datatype that stores an encrypted email (according to settings) when provided in the job submission body (#44).
• Add ability to filter jobs with notification_email query parameter (#44).
• Add jobs statistics grouping by specific fields using comma-separated list groups query parameter (#46).
• Add some tests to evaluate new job search methods / grouping results and responses (#44, #46).
• Add handling of multiple CWL field format for File type.
• Add missing ontology reference support for CWL field format by defaulting to IANA namespace.
• Add support for I/O array of enum (ie: multiple values of AllowedValues for a given input) (#30).
• Add support of label synonym as title for inputs and process description (CWL specifying a label will set it in WPS process) (#31)
• Add support of input minOccurs and maxOccurs as int while maintaining str support (#14).
• Add conformance route with implementation links (#53).
• Add additional landing page link details (#54).
• Add weaver.wps_restapi.colander_extras.DropableNoneSchema to auto-handle some schema JSON deserialization.
• Add weaver.wps_restapi.colander_extras.VariableMappingSchema to auto-handle some schema JSON deserialization.
• Add more functional tests (#11, #17).

Changes:

• Use bump2version and move all config under setup.cfg.
• Remove enforced text/plain for CWL File when missing format field.
• Replace bubbling up of too verbose unhandled exceptions (500 Internal Server Error) by summary message and additional internal logging for debugging the cause using an utility exception log decorator.
• Use the same exception log decorator to simplify function definitions when HTTP exceptions are already handled.
• Make null reference a singleton so that multiple instantiation calls all refer to the same instance and produce the expected behaviour of <x> is null instead of hard-to-identify errors because of english syntax.
• Remove unused function weaver.utils.replace_caps_url and corresponding tests.
• Remove weaver.processes.utils.jsonify_value duplicated by weaver.processes.wps_package.complex2json.
• Use more JSON body schema validation using API schema definitions deserialization defined by weaver.datatype.
• Enforce builtin processes registration on startup to receive applicable updates.
• Provide 2 separate docker images for Weaver manager and worker, corresponding to the EMS/ADES API and the celery job runner respectively.
• Update Apache license.

Fixes:

• Adjust some typing definitions incorrectly specified.
• Fix some failing functionality tests (#11, #17).
• Fix I/O field ordering preserved as specified in payload or loaded reference file.
• Fix setting minOccurs=0 when a default is specified in the corresponding CWL I/O (#17, #25).
• Fix incorrectly overridden maxOccurs="unbounded" by maxOccurs="1" when a partial array input definition is specified without explicit maxOccurs in WPS payload (#17, #25).
• Fix case where omitted format[s] in both CWL and WPS deploy bodies generated a process description with complex I/O (file) without required formats field. Default text/plain format is now automatically added.
• Fix case where format[s] lists between CWL and WPS where incorrectly merged.
• Fix metadata field within a WPS I/O incorrectly parsed when provided by a WPS-1/2 XML process definition.
• Fix invalid JSON response formatting on failing schema validation of process deployment body.
• Fix docker images to support pserve when using gunicorn>=20.x dropping support of --paste config feature.
• Fix multiple Python 2/3 compatibility issues.

0.2.2 (2019-05-31)

• Support notification email subject template.

0.2.1 (2019-05-29)

• Add per-process email notification template.

0.2.0 (2019-03-26)

• Fixes to handle invalid key characters "$" and "." during CWL package read/write operations to database.
• Fixes some invalid CWL package generation from WPS-1 references.
• More cases handled for WPS-1 to CWL WPS1Requirement conversion (AllowedValues, Default, SupportedFormats, minOccurs, maxOccurs).
• Add file format validation to generated CWL package from WPS-1 MIME-types.
• Allow auto-deployment of WPS-REST processes from WPS-1 references specified by configuration.
• Add many deployment and execution validation tests for WPS1Requirement.
• Add builtin application packages support for common operations.

0.1.3 (2019-03-07)

• Add useful Makefile targets for deployment.
• Add badges indications in README.rst for tracking from repo landing page.
• Fix security issue of PyYAML requirement.
• Fix some execution issues for Wps1Process.
• Fix some API schema erroneous definitions.
• Additional logging of unhandled errors.
• Improve some typing definitions.

0.1.2 (2019-03-05)

• Introduce WPS1Requirement and corresponding Wps1Process to run a WPS-1 process under CWL.
• Remove mongodb requirement, assume it is running on an external service or docker image.
• Add some typing definitions.
• Fix some problematic imports.
• Fix some PEP8 issues and PyCharm warnings.

0.1.1 (2019-03-04)

• Modify Dockerfile to use lighter debian:latest instead of birdhouse/bird-base:latest.
• Modify Dockerfile to reduce build time by reusing built image layers (requirements installation mostly).
• Make some buildout dependencies optional to also reduce build time and image size.
• Some additional striping of deprecated or invalid items from Twitcher.

0.1.0 (2019-02-26)

• Initial Release. Based off Twitcher tag ogc-0.4.7.