weaver.processes.wps_package

Functions and classes that offer interoperability and conversion between corresponding elements defined as CWL CommandLineTool/Workflow and WPS ProcessDescription in order to generate ADES/EMS Application Package.

See also

Module Contents

weaver.processes.wps_package.PACKAGE_EXTENSIONS[source]
weaver.processes.wps_package.DEFAULT_FORMAT[source]
weaver.processes.wps_package.retrieve_package_job_log(execution, job) → None[source]

Obtains the underlying WPS execution log from the status file to add them after existing job log entries.

weaver.processes.wps_package.get_process_location(process_id_or_url, data_source=None) → AnyStr[source]

Obtains the URL of a WPS REST DescribeProcess given the specified information.

Parameters:
  • process_id_or_url – process “identifier” or literal URL to DescribeProcess WPS-REST location.
  • data_source – identifier of the data source to map to specific ADES, or map to localhost if None.
Returns:

URL of EMS or ADES WPS-REST DescribeProcess.
weaver.processes.wps_package.get_package_workflow_steps(package_dict_or_url) → List[Dict[AnyStr, AnyStr]][source]
Parameters:package_dict_or_url – process package definition or literal URL to DescribeProcess WPS-REST location.
Returns:list of workflow steps as {“name”: <name>, “reference”: <reference>} where name is the generic package step name, and reference is the id/url of a registered WPS package.
weaver.processes.wps_package.complex2json(data) → Union[JSON, Any][source]

Obtains the JSON representation of a ComplexData or simply return the unmatched type.

weaver.processes.wps_package.metadata2json(meta, force=False) → Union[JSON, Any][source]

Obtains the JSON representation of a OwsMetadata or pywps.app.Common.Metadata. Otherwise, simply return the unmatched type. If requested, can enforce parsing a dictionary for the corresponding keys.

weaver.processes.wps_package.get_process_definition(process_offering, reference=None, package=None, data_source=None) → JSON[source]

Returns an updated process definition dictionary ready for storage using provided WPS process_offering and a package definition passed by reference or package CWL content. The returned process information can be used later on to load an instance of weaver.wps_package.WpsPackage.

Parameters:
  • process_offering – WPS REST-API (WPS-3) process offering as JSON.
  • reference – URL to CWL package definition, WPS-1 DescribeProcess endpoint or WPS-3 Process endpoint.
  • package – literal CWL package definition (YAML or JSON format).
  • data_source – where to resolve process IDs (default: localhost if None).
Returns:

updated process definition with resolved/merged information from package/reference.
class weaver.processes.wps_package.WpsPackage(**kw)[source]

Bases: pywps.Process

Creates a WPS-3 Process instance to execute a CWL package definition.

Process parameters should be loaded from an existing weaver.datatype.Process instance generated using weaver.wps_package.get_process_definition().

Provided kw should correspond to weaver.datatype.Process.params_wps()

package :Optional[CWL][source]
package_id :Optional[AnyStr][source]
package_log_hook_stderr :Optional[AnyStr][source]
package_log_hook_stdout :Optional[AnyStr][source]
percent :Optional[Number][source]
log_file :Optional[AnyStr][source]
log_level :int[source]
logger :Optional[logging.Logger][source]
step_packages :Optional[List[CWL]][source]
step_launched :Optional[List[AnyStr]][source]
request :Optional[WPSRequest][source]
response :Optional[ExecuteResponse][source]
setup_logger(self)[source]

Configures useful loggers to catch most of the common output and/or error messages during package execution.

See also

insert_package_log() retrieve_package_job_log()

insert_package_log(self, result)[source]

Retrieves additional CWL sub-process logs captures to retrieve internal application output and/or errors.

After execution of this method, the WPS output log (which can be obtained by retrieve_package_job_log()) will have additional stderr/stdout entries extracted from the underlying application package tool execution.

The outputs and errors are inserted as best as possible in the logical order to make reading of the merged logs appear as a natural and chronological order. In the event that both output and errors are available, they are appended one after another as merging in an orderly fashion cannot be guaranteed by outside CWL runner.

Parameters:result – output results returned from the CWL package instance execution.

Todo

improve for realtime updates when using async routine (https://github.com/crim-ca/weaver/issues/131)

See also

setup_logger() retrieve_package_job_log()

update_status(self, message, progress, status)[source]

Updates the PyWPS real job status from a specified parameters.

step_update_status(self, message, progress, start_step_progress, end_step_progress, step_name, target_host, status)[source]
log_message(self, status, message, progress=None, level=logging.INFO)[source]
exception_message(self, exception_type, exception=None, message='no message', status=STATUS_EXCEPTION, level=logging.ERROR)[source]
classmethod map_step_progress(cls, step_index, steps_total)[source]

Calculates the percentage progression of a single step of the full process.

Note

The step procession is adjusted according to delimited start/end of the underlying CWL execution to provide a continuous progress percentage over the complete execution. Otherwise, we would have values that jump around according to whichever progress the underlying remote WPS or monitored CWL employs, if any is provided.

static make_location_input(input_type, input_definition)[source]

Generates the JSON content required to specify a CWL File input definition from a location.

_handler(self, request, response)[source]
make_tool(self, toolpath_object, loading_context)[source]
get_job_process_definition(self, jobname, joborder, tool)[source]

This function is called before running an ADES job (either from a workflow step or a simple EMS dispatch). It must return a WpsProcess instance configured with the proper package, ADES target and cookies.

Parameters:
  • jobname – The workflow step or the package id that must be launch on an ADES string
  • joborder – The params for the job dict {input_name: input_value} input_value is one of input_object or array [input_object] input_object is one of string or dict {class: File, location: string} in our case input are expected to be File object
  • tool – Whole CWL config including hints requirement