Add AWS S3 bucket support for process input reference files.
weaver.wps_output_s3_bucketsetting to upload results to AWS S3 bucket instead of local directory.
weaver.wps_output_s3_regionsetting to allow override parameter extracted from AWS profile otherwise.
Add more documentation about supported file reference schemes.
Add documentation references to ESGF-CWT Compute API.
Add conditional input file reference fetching (depending on ADES/EMS, process type from CWL
hints) to take advantage of request-options and all supported scheme formats by Weaver, instead of relying on
CWLwherever how far downstream the URL reference was reaching.
Adjust some docstrings to better indicate raised errors.
weaver.processes.wps_package.WpsPackageto use its internal logger when running the process in order to preserve log entries under its job execution. They were otherwise lost over time across all process executions.
Add multiple CWL ESGF processes and workflows, namely
SubsetNASAESGFand many more.
Add tests for ESGF processes and workflows.
Add documentation for
Wps1Processextension, to handle
ESGF-CWTRequirementprocesses and workflows.
MongoDatabaseconnection when we are in a forked process.
Generate Weaver OpenAPI specification for readthedocs publication.
Add some sections for documentation (#61).
Add support of documentation RST file redirection to generated HTML for reference resolution in both Github source and Readthedocs served pages.
Improve documentation links, ReadTheDocs format and TOC references.
Add tests to make sure processes
Remove Python 2.7 version as not officially supported.
Move and update WPS status location and status check functions into
Fix reported WPS status location to handle when starting with
/although not representing an absolute path.
celery==4.4.2to avoid import error on missing
futures.utilscalled internally in following versions.
Add support of value-typed metadata fields for process description.
relfield when specifying an
hrefJSON link to match corresponding XML requirement.
weaver.wps_workdirconfiguration setting to define the location where the underlying
cwltoolapplication should be executed under. This can allow more control over the scope of the mounted volumes for Application Package running a docker image.
Add mapping of WPS results from the
Job’s UUID to generated PyWPS UUID for outputs, status and log locations.
Add experimental configuration settings
weaver.cwl_egidto provide effective user/group identifiers to employ when running the CWL Application Package. Using these require good control of the directory and process I/O locations as invalid permissions could break a previously working job execution.
Add more logging configuration and apply them to
cwltoolbefore execution of Application Package.
RuntimeContextto ensure that docker application is executed with same user as
weaverand that process input files are not modified inplace (readonly) where potentially inaccessible (according to settings). Definition of CWL package will need to add InitialWorkDirRequirement as per defined by reference specification to stage those files if they need to be accessed with write permissions (see: example). Addresses some issues listed in #155.
Enforce removal of some invalid CWL hints/requirements that would break the behaviour offered by
weaver.request_optionsfor WPS GetCapabilities and WPS Check Status requests under the running job.
DOCKER_REPOvalue defined in
Makefileto point to reference mentioned in
README.mdand considered as official deployment location.
application/x-cwlMIME-type supported with updated
application/x-yamlMIME-type to known formats.
application/tar+gzipMIME-type (not official) but resolved as synonym
application/gzip(official) to preserve compressed file support during CWL format validation.
Falseto retrieve original default behaviour of the function. Since CWL usually doesn’t need to add
File.formatfield when no corresponding reference actually exists, this default also makes more sense.
Add Travis-CI smoke test of built docker images for early detection of invalid setup or breaking code to boot them.
Add Travis-CI checks for imports. This check was not validated previously although available.
weaver.ini.exampleto reflect working demo server configuration (employed by smoke test).
weaverweb application to
weaver.appto reduce chances of breaking
setup.pyinstallation from import errors due to
weaverdependencies not yet installed. Redirect to new location makes this change transparent when loaded with the usual
Fix base docker image to install Python 3 development dependencies in order to compile requirements with expected environment Python version. Package
python-devfor Python 2 was being installed instead.
Fix failing docker image boot due to incorrectly placed
yamlimport during setup installation.
Fix imports according to
Fix parsing of
PyWPSmetadata to correctly employ values provided by
weaver.utils.request_extrato include more requests functionality and reuse it across the whole code base.
requests_extraSSL verification option using specific URL regex(es) matches from configuration settings.
file://transport scheme support directly to utility
requests_extrato handle local file paths.
weaver.request_optionsINI configuration setting to specify per-request method/URL options.
Retry-Afterresponse header (if any available on
429status) which indicates how long to wait until next request to avoid automatically defined response right after.
weaver.wps_workdirconfiguration setting with allow setting corresponding
Dockerfile-managerto run web application using
gunicorndoesn’t correctly handles worker options anymore when loaded form
--pasteargument. Also simplifies the command which already required multiple patches such as reapplying the host/port binding from INI file.
Fix handling of Literal Data I/O
typewhen retrieved from
OWSLib.wpsobject with remote WPS XML body.
make starttarget to use new
make install-runtarget which installs the dependencies and package in edition mode so that configuration files present locally can be employed for running the application. Previously, one would have to move their configurations to the
site-packageinstall location of the active Python.
celery>4.2not found because of application path modification.
Fix invalid handling of
weaver.iniwhen specified as relative path to configuration directory.
Fix handling of
WPS<->CWLI/O merge of
Fix installation of
yaml-related packages for Python 2 backward compatibility.
Add additional status log for
EOImageinput modification with OpenSearch during process execution.
stderr/stdoutlogging of underlying CWL application being executed to resulting
Joblogs (addresses first step of #131).
weaver.utils.request_retryin even more places and extend convenience arguments offered by it to adapt it to specific use cases.
Fix handling of WPS-REST output matching a JSON file for multiple-output format specified with a relative local path as specified by job output location. Only remote HTTP references where correctly parsed. Also avoid failing the job if the reference JSON parsing fails. It will simply return the original reference URL in this case without expanded data (relates to #25).
Fix CWL job logs to be timezone aware, just like most other logs that will report UTC time.
Fix JSON response parsing of remote provider processes.
Fix parsing of CWL ordered parsing when I/O is specified as shorthand
"<id>":"<type>" directly under the ``inputsor
outputsdictionary instead of extended sub-dictionary variant (fixes #137).
weaver.utils.request_retryfunction across a few locations that where essentially reimplementing the core functionality.
Add even more failure-permissive request attempts when validating a MIME-type against IANA website.
Add auto-resolution of common extensions known under PyWPS as well as employing their specific encoding.
geotiffformat type support via PyWPS (#100).
Make WPS status check more resilient to failing WPS outputs location not found in case the directory path can be resolved to a valid local file representing the XML status (i.e.: don’t depend as much on the HTTP WPS output route).
Ensure backward support of generic/default
text/plainI/O when extracted from a referenced WPS-1/2 XML remote process which provides insufficient format details. For CWL output generated from it, replace the glob pattern to match anything (
<id>.*) instead of
text/plainto simulate MIME-type as
*/*. Issue log warning message for future use cases.
AllowedValueparsing when using
LiteralDatainputs that resulted in
AnyValuebeing parsed as a
"None"string. This was transparent in case of string inputs and breaking for other types like integer when they attempted conversion.
Metadatakeywords passed down to
owslib.wps.Metadataobjects in case of more verbose detailed not allowed by this implementation.
Fix parsing of explicitly-typed optional array CWL I/O notation that was not considered (i.e.: using
typeas list with additional
Fix parsing of MIME-type from
formatfield to exclude additional parameters (e.g.:
; charset=UTF-8for remote IANA validation.
Add unittest of utility function
Split some unittest utility functions to allow more reuse.
retryparameter not handled automatically by request.
Adjust incorrectly parsed href file reference as WPS complex input which resulted in failing location retrieval.
Partially address unnecessary fetch of file that has to be passed down to CWL, which will in turn request the file as required. Need update from PyWPS to resolve completely (#91, geopython/pywps#526).
Adjust WPS output results to use relative HTTP path in order to recompose the output URL if server settings change.
Support WPS output results as value (WPS literal data). Everything was considered an href file beforehand.
retryduring fetching of remote file for process
jsonarray2netcdfto avoid unnecessary failures during edge case connexion problems.
Add support of
builtinprocess execution failing since
Avoid long fetch operation using streamed request that defaulted to chuck size of 1. Now, we use an appropriate size according to available memory.
Provide a way to override the external URL reported by WPS-1/2 and WPS-REST via configuration settings allowing for more advanced server-side results in response bodies.
Add wps languages for other wps requests types:
Fix a bug where the validation of
OneOfitems was casting the value to the first valid possibility. Now, it doesn’t change the value if it’s valid without casting it (and still casts it if it’s necessary to make it valid).
Add planned future support of
Accept-Languageheader for WPS-1/2 (
geopython/OWSLib 0.20.0) (#74).
Improved job logs update with message and progress to allow better tracking of internal operations and/or problems.
Allow WPS builtin process
jsonarray2netcdfto fetch a remote file.
Change doc to point to DockerHub pavics/weaver images.
Adjust CI rule long-lasting failures until it gets patched by original reference (gitleaks-actions#3).
Jobdatatype that stores an encrypted email (according to settings) when provided in the job submission body (#44).
Add ability to filter jobs with
notification_emailquery parameter (#44).
Add jobs statistics grouping by specific fields using comma-separated list
groupsquery parameter (#46).
Add handling of multiple CWL field
Add missing ontology reference support for CWL field
formatby defaulting to IANA namespace.
Add support for I/O
enum(ie: multiple values of
AllowedValuesfor a given input) (#30).
Add support of
titlefor inputs and process description (CWL specifying a
labelwill set it in WPS process) (#31)
Add support of input
Add conformance route with implementation links (#53).
Add additional landing page link details (#54).
weaver.wps_restapi.colander_extras.DropableNoneSchemato auto-handle some schema JSON deserialization.
weaver.wps_restapi.colander_extras.VariableMappingSchemato auto-handle some schema JSON deserialization.
bump2versionand move all config under
Replace bubbling up of too verbose unhandled exceptions (500 Internal Server Error) by summary message and additional internal logging for debugging the cause using an utility exception log decorator.
Use the same exception log decorator to simplify function definitions when HTTP exceptions are already handled.
nullreference a singleton so that multiple instantiation calls all refer to the same instance and produce the expected behaviour of
<x> is nullinstead of hard-to-identify errors because of english syntax.
Remove unused function
weaver.utils.replace_caps_urland corresponding tests.
Use more JSON body schema validation using API schema definitions deserialization defined by
builtinprocesses registration on startup to receive applicable updates.
Provide 2 separate docker images for Weaver manager and worker, corresponding to the EMS/ADES API and the
celeryjob runner respectively.
Update Apache license.
Adjust some typing definitions incorrectly specified.
Fix I/O field ordering preserved as specified in payload or loaded reference file.
Fix case where omitted
format[s]in both CWL and WPS deploy bodies generated a process description with complex I/O (file) without required
text/plainformat is now automatically added.
Fix case where
format[s]lists between CWL and WPS where incorrectly merged.
metadatafield within a WPS I/O incorrectly parsed when provided by a WPS-1/2 XML process definition.
Fix invalid JSON response formatting on failing schema validation of process deployment body.
Fix docker images to support
gunicorn>=20.xdropping support of
Fix multiple Python 2/3 compatibility issues.
Fixes to handle invalid key characters
"."during CWL package read/write operations to database.
Fixes some invalid CWL package generation from WPS-1 references.
More cases handled for WPS-1 to CWL
Add file format validation to generated CWL package from WPS-1 MIME-types.
Allow auto-deployment of WPS-REST processes from WPS-1 references specified by configuration.
Add many deployment and execution validation tests for
builtinapplication packages support for common operations.
Add useful Makefile targets for deployment.
Add badges indications in
README.rstfor tracking from repo landing page.
Fix security issue of PyYAML requirement.
Fix some execution issues for
Fix some API schema erroneous definitions.
Additional logging of unhandled errors.
Improve some typing definitions.
Wps1Processto run a WPS-1 process under CWL.
Remove mongodb requirement, assume it is running on an external service or docker image.
Add some typing definitions.
Fix some problematic imports.
Fix some PEP8 issues and PyCharm warnings.
Modify Dockerfile to use lighter
Modify Dockerfile to reduce build time by reusing built image layers (requirements installation mostly).
Make some buildout dependencies optional to also reduce build time and image size.
Some additional striping of deprecated or invalid items from Twitcher.