weaver.processes.wps_process_base

Module Contents

weaver.processes.wps_process_base.LOGGER[source]
class weaver.processes.wps_process_base.WpsRemoteJobProgress[source]

Progress of a remotely monitored WPS-1 job process execution.

Note

Implementations can reuse same progress values or intermediate ones within the range of the relevant sections.

START = 1[source]
PREPARE = 2[source]
READY = 5[source]
STAGE_IN = 10[source]
FORMAT_IO = 12[source]
EXECUTE = 15[source]
MONITOR = 20[source]
RESULTS = 85[source]
STAGE_OUT = 90[source]
CLEANUP = 95[source]
COMPLETED = 100[source]
class weaver.processes.wps_process_base.WpsProcessInterface(request: weaver.typedefs.UpdateStatusPartialFunction, update_status)[source]

Common interface for WPS Process to be used for dispatching CWL jobs.

Multiple convenience methods are provide. Processes inheriting from this base should provide abstract method implementation as needed or required.

Note

For expected operations details and their execution order, please refer to Workflow Step Operations.

See also

execute() for complete details of the operations and ordering.

execute(self: weaver.typedefs.CWL_RuntimeInputsMap, workflow_inputs: str, out_dir: weaver.typedefs.CWL_ExpectedOutputs, expected_outputs) None[source]

Execute the core operation of the remote Process using the given inputs.

The function is expected to monitor the process and update the status. Retrieve the expected outputs and store them in the out_dir.

Parameters
  • workflow_inputsCWL job dict

  • out_dir – directory where the outputs must be written

  • expected_outputs – expected value outputs as {‘id’: ‘value’}

prepare(self) None[source]

Implementation dependent operations to prepare the Process for Job execution.

This is an optional step that can be omitted entirely if not needed.

format_inputs(self: weaver.typedefs.JobInputs, workflow_inputs) Union[weaver.typedefs.JobInputs, Any][source]

Implementation dependent operations to configure input values for Job execution.

This is an optional step that will simply pass down the inputs as is if no formatting is required. Otherwise, the implementing Process can override the step to reorganize workflow step inputs into the necessary format required for their dispatch() call.

format_outputs(self: weaver.typedefs.JobOutputs, workflow_outputs) weaver.typedefs.JobOutputs[source]

Implementation dependent operations to configure expected outputs for Job execution.

This is an optional step that will simply pass down the outputs as is if no formatting is required. Otherwise, the implementing Process can override the step to reorganize workflow step outputs into the necessary format required for their dispatch() call.

abstract dispatch(self: weaver.typedefs.JobInputs, process_inputs: weaver.typedefs.JobOutputs, process_outputs) weaver.typedefs.JobMonitorReference[source]

Implementation dependent operations to dispatch the Job execution to the remote Process.

Returns

reference details that will be passed to monitor().

abstract monitor(self: weaver.typedefs.JobMonitorReference, monitor_reference) bool[source]

Implementation dependent operations to monitor the status of the Job execution that was dispatched.

This step should block execute() until the final status of the remote Job (failed/success) can be obtained.

Returns

success status

abstract get_results(self: weaver.typedefs.JobMonitorReference, monitor_reference) weaver.typedefs.JobResults[source]

Implementation dependent operations to retrieve the results following a successful Job execution.

The operation should NOT fetch (stage) results, but only obtain the locations where they can be retrieved, based on the monitoring reference that was generated from the execution.

Returns

results locations

cleanup(self) None[source]

Implementation dependent operations to clean the Process or Job execution.

This is an optional step that doesn’t require any override if not needed by derived classes.

get_auth_headers(self) weaver.typedefs.AnyHeadersContainer[source]

Implementation dependent operation to retrieve applicable authorization headers.

This method is employed for every make_request() call to avoid manually providing them each time. Any overriding method should consider calling this method to retrieve authorization headers from WPS request.

get_auth_cookies(self) weaver.typedefs.CookiesTupleType[source]

Implementation dependent operation to retrieve applicable authorization cookies.

This method is employed for every make_request() call to avoid manually providing them each time. Any overriding method should consider calling this method to retrieve authorization cookies from WPS request.

make_request(self: Ellipsis, method: str, url: str, retry: Union[bool, int] = False, cookies: Optional[weaver.typedefs.AnyCookiesContainer] = None, headers: Optional[weaver.typedefs.AnyHeadersContainer] = None, **kwargs) weaver.typedefs.AnyResponseType[source]

Sends the request with additional parameter handling for the current process definition.

host_file(self, file_path)[source]

Hosts an intermediate file between Workflow steps for processes that require external or remote access.

Parameters

file_path – Intermediate file location (local path expected).

Returns

Hosted temporary HTTP file location.

stage_results(self: weaver.typedefs.JobResults, results: weaver.typedefs.CWL_ExpectedOutputs, expected_outputs: str, out_dir) None[source]

Retrieves the remote execution Job results for staging locally into the specified output directory.

This operation should be called by the implementing remote Process definition after execute().

Note

The CWL runner expects the output file(s) to be written matching definition in expected_outputs, but this definition could be a glob pattern to match multiple file and/or nested directories. We cannot rely on specific file names to be mapped, since glob can match many (eg: "*.txt").

stage_inputs(self: weaver.typedefs.CWL_WorkflowInputs, workflow_inputs) weaver.typedefs.JobInputs[source]

Retrieves inputs for local staging if required for the following Job execution.