weaver.wps_restapi.jobs.utils ============================= .. py:module:: weaver.wps_restapi.jobs.utils Module Contents --------------- .. py:data:: MultiPartFieldsParamsType .. py:data:: LOGGER .. py:function:: get_job(request: weaver.typedefs.PyramidRequest) -> weaver.datatype.Job Obtain a :term:`Job` from request parameters. .. versionchanged:: 4.20 When looking for :term:`Job` that refers to a local :term:`Process`, allow implicit resolution of the unspecified ``version`` portion to automatically resolve the identifier. Consider that validation of the expected :term:`Process` for this :term:`Job` is "good enough", since the specific ID is not actually required to obtain the :term:`Job` (could be queried by ID only on the ``/jobs/{jobId}`` endpoint. If the ``version`` is provided though (either query parameter or tagged representation), the validation will ensure that it matches explicitly. :param request: Request with path and query parameters to retrieve the desired job. :returns: Job information if found. :raise HTTPNotFound: with JSON body details on missing/non-matching job, process, provider IDs. .. py:function:: get_job_list_links(job_total: int, filters: Dict[str, weaver.typedefs.AnyValueType], request: weaver.typedefs.AnyRequestType) -> List[weaver.typedefs.JSON] Obtains a list of all relevant links for the corresponding job listing defined by query parameter filters. :raises IndexError: if the paging values are out of bounds compared to available total :term:`Job` matching search. .. py:function:: get_job_io_schema_query(schema: Optional[str], strict: bool = True, default: weaver.processes.constants.JobInputsOutputsSchemaType = None) -> weaver.processes.constants.JobInputsOutputsSchemaType .. py:function:: get_job_status_schema(request: weaver.typedefs.AnyRequestType) -> Tuple[weaver.processes.constants.JobStatusSchemaType, weaver.typedefs.HeadersType] Identifies if a :term:`Job` status response schema applies for the request. .. py:function:: make_result_link(job: weaver.datatype.Job, result: weaver.typedefs.ExecutionResultValue, output_id: str, output_mode: weaver.execute.AnyExecuteTransmissionMode, output_format: Optional[weaver.typedefs.JobValueFormat] = None, *, settings: weaver.typedefs.SettingsType) -> List[str] Convert a result definition as ``value`` into the corresponding ``reference`` for output transmission. .. seealso:: :rfc:`8288`: HTTP ``Link`` header specification. .. py:function:: get_results(job: weaver.datatype.Job, container: weaver.typedefs.AnySettingsContainer, value_key: Optional[str] = None, schema: Optional[weaver.processes.constants.JobInputsOutputsSchemaType] = JobInputsOutputsSchema.OLD, link_references: bool = False) -> Tuple[weaver.typedefs.ExecutionResults, weaver.typedefs.HeadersTupleType] Obtains the job results with extended full WPS output URL as applicable and according to configuration settings. :param job: job from which to retrieve results. :param container: any container giving access to instance settings (to resolve reference output location). :param value_key: If not specified, the returned values will have the appropriate ``data``/``href`` key according to the content. Otherwise, all values will have the specified key. :param schema: Selects which schema to employ for representing the output results (listing or mapping). :param link_references: If enabled, an output that was requested by reference instead of by value will be returned as ``Link`` header. :returns: Tuple with: - List or mapping of all outputs each with minimally an ID and value under the requested key. - List of ``Link`` headers for reference outputs when requested. Empty otherwise. .. py:function:: get_job_return(job: Optional[weaver.datatype.Job] = None, body: Optional[weaver.typedefs.JSON] = None, headers: Optional[weaver.typedefs.AnyHeadersContainer] = None) -> Tuple[weaver.execute.AnyExecuteResponse, weaver.execute.AnyExecuteReturnPreference] Obtain the :term:`Job` result representation based on the resolution order of preferences and request parameters. Body and header parameters are considered first, in case they provide 'overrides' for the active request. Then, if the :paramref:`job` was already parsed from the original request, and contains pre-resolved return, this format is employed. When doing the initial parsing, ``job=None`` MUST be used. .. py:function:: get_job_output_transmission(job: weaver.datatype.Job, output_id: str, is_reference: bool) -> Tuple[weaver.execute.AnyExecuteTransmissionMode, Optional[weaver.typedefs.JobValueFormat]] Obtain the requested :term:`Job` output ``transmissionMode`` and ``format``. .. py:function:: get_job_results_response(job: weaver.datatype.Job, *, container: weaver.typedefs.AnySettingsContainer, request_headers: Optional[weaver.typedefs.AnyHeadersContainer] = None, response_headers: Optional[weaver.typedefs.AnyHeadersContainer] = None, results_headers: Optional[weaver.typedefs.AnyHeadersContainer] = None, results_contents: Optional[weaver.typedefs.JSON] = None) -> weaver.typedefs.AnyResponseType Generates the :term:`OGC` compliant :term:`Job` results response according to submitted execution parameters. Parameters that impact the format of the response are: - Body parameter ``outputs`` with the amount of *requested outputs* to be returned. - Body parameter ``response: raw|document`` for content representation. - Body parameter ``transmissionMode: value|reference`` per output. - Header parameter ``Prefer: return=representation|minimal`` for content representation. - Overrides, for any of the previous parameters, allowing request of an alternate representation. Resolution order/priority: 1. :paramref:`override_contents` 2. :paramref:`override_headers` 3. :paramref:`job` definitions The logic of the resolution order is that any body parameters resolving to an equivalent information provided by header parameters will be more important, since ``Prefer`` are *soft* requirements, whereas body parameters are *hard* requirements. The parameters stored in the :paramref:`job` are defined during :term:`Job` submission, which become the "default" results representation if requested as is. If further parameters are provided to override during the results request, they modify the "default" results representation. In this case, an header provided in the results request overrides the body parameters from the original :term:`Job`, since their results request context is "closer" than the ones at the time of the :term:`Job` submission. .. seealso:: More details available for each combination: - https://docs.ogc.org/is/18-062r2/18-062r2.html#sc_execute_response - https://docs.ogc.org/is/18-062r2/18-062r2.html#_response_7 - :ref:`proc_op_job_results` - :ref:`proc_exec_results` :param job: Job for which to generate the results response, which contains the originally submitted parameters. :param container: Application settings. :param request_headers: Original headers submitted to the request that leads to this response. :param response_headers: Additional headers to provide in the response. :param results_headers: Headers that override originally submitted job parameters when requesting results. :param results_contents: Body contents that override originally submitted job parameters when requesting results. .. py:function:: generate_or_resolve_result(job: weaver.datatype.Job, result: weaver.typedefs.ExecutionResultObject, result_id: str, output_id: str, output_mode: weaver.execute.AnyExecuteTransmissionMode, output_format: Optional[weaver.typedefs.JobValueFormat], settings: weaver.typedefs.SettingsType) -> Tuple[weaver.typedefs.HeadersType, Optional[weaver.typedefs.AnyDataStream]] Obtains the local file path and the corresponding :term:`URL` reference for a given result, generating it as needed. :param job: Job with results details. :param result: The specific output value or reference (could be an item index within an array of a given output). :param result_id: Specific identifier of the result, including any array index as applicable. :param output_id: Generic identifier of the output containing the result. :param output_mode: Desired output transmission mode. :param output_format: Desired output transmission ``format``, with minimally the :term:`Media-Type`. :param settings: Application settings to resolve locations. :return: Resolved headers and data (as applicable) for the result. If only returned by reference, ``None`` data is returned. An empty-data contents would be an empty string. Therefore, the explicit check of ``None`` is important to identify a by-reference result. .. py:function:: resolve_result_json_literal(result: weaver.typedefs.ExecutionResultValue, output_format: Optional[weaver.typedefs.JobValueFormat], content_type: Optional[str] = None, content_encoding: Optional[str] = None) -> weaver.typedefs.ExecutionResultValue Generates a :term:`JSON` literal string or object representation according to requested format and result contents. If not a ``value`` structure, the result is returned unmodified. If no output ``format`` is provided, or that the extracted result :term:`Media-Type` does not correspond to a :term:`JSON` value, the result is also unmodified. Otherwise, string/object representation is resolved according to the relevant :term:`Media-Type`. :param result: Container with nested data. :param output_format: Desired output transmission ``format``, with minimally the :term:`Media-Type`. :param content_type: Explicit :term:`Media-Type` to employ instead of an embedded ``mediaType`` result property. :param content_encoding: Explicit data encoding to employ instead of an embedded ``encoding`` result property. :return: Converted :term:`JSON` data or the original result as applicable. .. py:function:: get_job_results_links(job: weaver.datatype.Job, references: Dict[str, weaver.typedefs.ExecutionResultValue], transmissions: Dict[str, Tuple[weaver.execute.AnyExecuteTransmissionMode, weaver.typedefs.JobValueFormat]], headers: weaver.typedefs.AnyHeadersContainer, *, settings: weaver.typedefs.SettingsType) -> weaver.typedefs.AnyHeadersContainer Generates ``Link`` headers for all specified result references and adds them to the specified header container. .. py:function:: get_job_results_single(job: weaver.datatype.Job, result: weaver.typedefs.ExecutionResultObject, output_id: str, output_format: Optional[weaver.typedefs.JobValueFormat], headers: weaver.typedefs.AnyHeadersContainer, *, settings: weaver.typedefs.AnySettingsContainer) -> Union[pyramid.httpexceptions.HTTPOk, pyramid.httpexceptions.HTTPNoContent] Generates a single result response according to specified or resolved output transmission and format. :param job: Job definition to obtain relevant path resolution. :param result: Result to be represented. :param output_id: Identifier of the corresponding result output. :param output_format: Desired output format for convertion, as applicable. :param headers: Additional headers to include in the response. :param settings: Application settings to resolve locations. :return: .. py:function:: get_job_results_document(job: weaver.datatype.Job, results: weaver.typedefs.ExecutionResults, *, settings: Any) -> weaver.typedefs.ExecutionResults Generates the :term:`Job` results document response from available or requested outputs with necessary conversions. Removes nested literal value definitions if qualified value representation is not needed. Qualified value representation is not needed if no other field than ``value`` is provided with the literal data, or when the specified :term:`Media-Type` is simply the plain text default for data literals. The simplification is applied for both literals on their own and nested array of literals. However, when processing an array, the qualified value representation is preserved if any of the items requires the explicit mention of another :term:`Media-Type` than plain text, to return a consistent structure. Uses the :paramref:`job` definition and submitted ``headers`` .. warning:: This function assumes that schema deserialization was applied beforehand. Therefore, it will not attempt matching every possible combination of the results representation. .. py:function:: get_job_results_multipart(job: weaver.datatype.Job, results: weaver.typedefs.ExecutionResults, *, headers: Any, settings: weaver.typedefs.AnyHeadersContainer) -> pyramid.httpexceptions.HTTPOk Generates the :term:`Job` results multipart response from available or requested outputs with necessary conversions. .. seealso:: - Function :func:`get_results` should be used to avoid re-processing all output format combinations. - Details of ``multipart`` (:rfc:`2046#section-5.1`) :term:`Media-Type` family. :param job: Job definition with potential metadata about requested outputs. :param results: Pre-filtered and pre-processed results in a normalized format structure. :param headers: Additional headers to include in the response. :param settings: Application settings to resolve locations. .. py:function:: get_job_submission_response(body: weaver.typedefs.JSON, headers: weaver.typedefs.AnyHeadersContainer, error: bool = False, response_class: Optional[Type[weaver.typedefs.AnyResponseClass]] = None) -> Union[weaver.typedefs.AnyResponseClass, pyramid.httpexceptions.HTTPBadRequest] Generates the response contents returned by :term:`Job` submission process. If :term:`Job` already finished processing within requested ``Prefer: wait=X`` seconds delay (and if allowed by the :term:`Process` ``jobControlOptions``), return the successful status immediately instead of created status. If the status is not successful, return the failed :term:`Job` status response. Otherwise, return the status monitoring location of the created :term:`Job` to be monitored asynchronously. .. seealso:: - :func:`weaver.processes.execution.submit_job` - :func:`weaver.processes.execution.submit_job_handler` - :ref:`proc_op_job_status` .. py:function:: validate_service_process(request: weaver.typedefs.PyramidRequest) -> Tuple[Optional[str], Optional[str]] Verifies that any :term:`Provider` or :term:`Process` specified by path or query are valid. :raises HTTPException: Relevant HTTP error with details if validation failed. :returns: Validated and existing service and process if specified. .. py:function:: raise_job_bad_status_locked(job: weaver.datatype.Job, container: Optional[weaver.typedefs.AnySettingsContainer] = None) -> None Raise the appropriate message for :term:`Job` unable to be modified. .. py:function:: raise_job_bad_status_success(job: weaver.datatype.Job, container: Optional[weaver.typedefs.AnySettingsContainer] = None) -> None Raise the appropriate message for :term:`Job` not ready or unable to retrieve output results due to status. .. py:function:: raise_job_dismissed(job: weaver.datatype.Job, container: Optional[weaver.typedefs.AnySettingsContainer] = None) -> None Raise the appropriate messages for dismissed :term:`Job` status. .. py:function:: dismiss_job_task(job: weaver.datatype.Job, container: weaver.typedefs.AnySettingsContainer) -> weaver.datatype.Job Cancels any pending or running :mod:`Celery` task and removes completed job artifacts. .. note:: The :term:`Job` object itself is not deleted, only its artifacts. Therefore, its inputs, outputs, logs, exceptions, etc. are still available in the database, but corresponding files that would be exposed by ``weaver.wps_output`` configurations are removed. :param job: Job to cancel or cleanup. :param container: Application settings. :return: Updated and dismissed job. .. py:function:: get_job_prov_response(request: weaver.typedefs.PyramidRequest) -> weaver.typedefs.AnyResponseType Retrieve specific :term:`Provenance` contents of a :term:`Job` based on the request. The specific request path is redirected to the relevant command from :mod:`cwlprov`. If applicable, request :term:`Media-Type` specifiers are considered to return alternate representations.