weaver.processes.esgf_process
=============================

.. py:module:: weaver.processes.esgf_process






Module Contents
---------------

.. py:data:: ESGFProcessInputs

.. py:data:: LAST_PERCENT_REGEX

.. py:class:: Percent



   .. py:attribute:: PREPARING
      :value: 2



   .. py:attribute:: SENDING
      :value: 3



   .. py:attribute:: COMPUTE_DONE
      :value: 98



   .. py:attribute:: FINISHED
      :value: 100



.. py:class:: InputNames



   .. py:attribute:: FILES
      :value: 'files'



   .. py:attribute:: VARIABLE
      :value: 'variable'



   .. py:attribute:: API_KEY
      :value: 'api_key'



   .. py:attribute:: TIME
      :value: 'time'



   .. py:attribute:: LAT
      :value: 'lat'



   .. py:attribute:: LON
      :value: 'lon'



.. py:class:: InputArguments



   .. py:attribute:: START
      :value: 'start'



   .. py:attribute:: END
      :value: 'end'



   .. py:attribute:: CRS
      :value: 'crs'



.. py:class:: ESGFProcess(provider: str, process: str, request: Optional[weaver.wps.service.WorkerRequest], update_status: weaver.typedefs.UpdateStatusPartialFunction)



   Common interface for :term:`WPS` :term:`Process` to be used for dispatching :term:`CWL` jobs.

   Multiple convenience methods are provided.
   Processes inheriting from this base should implement required abstract methods and override operations as needed.

   .. note::
       For expected operations details and their execution order, please refer to :ref:`proc_workflow_ops`.

   .. seealso::
       :meth:`execute` for complete details of the operations and ordering.


   .. py:attribute:: required_inputs


   .. py:attribute:: provider


   .. py:attribute:: process


   .. py:attribute:: wps_provider
      :type:  Optional[cwt.WPSClient]
      :value: None



   .. py:attribute:: wps_process
      :type:  Optional[cwt.Process]
      :value: None



   .. py:method:: _get_domain(workflow_inputs: weaver.typedefs.CWL_RuntimeInputsMap) -> Optional[cwt.Domain]
      :staticmethod:



   .. py:method:: _check_required_inputs(workflow_inputs)


   .. py:method:: _get_files_urls(workflow_inputs: weaver.typedefs.JSON) -> List[Tuple[str, str]]
      :staticmethod:


      Get all netcdf files from the cwl inputs.



   .. py:method:: _get_variable(workflow_inputs: weaver.typedefs.JSON) -> str
      :staticmethod:


      Get all netcdf files from the cwl inputs.



   .. py:method:: prepare(workflow_inputs: weaver.typedefs.CWL_RuntimeInputsMap, expected_outputs: weaver.typedefs.CWL_ExpectedOutputs) -> None

      Prepare the :term:`ESGF-CWT` :term:`WPS` client.



   .. py:method:: format_inputs(job_inputs: weaver.typedefs.JobInputs) -> ESGFProcessInputs

      Convert inputs from cwl inputs to :term:`ESGF-CWT` format.



   .. py:method:: dispatch(process_inputs: ESGFProcessInputs, process_outputs: weaver.typedefs.CWL_ExpectedOutputs) -> cwt.Process

      Run an :term:`ESGF-CWT` process.



   .. py:method:: monitor(esgf_process: cwt.Process, sleep_time: float = 2) -> bool

      Wait for an :term:`ESGF-CWT` process to finish, while reporting its status.



   .. py:method:: get_results(esgf_process: cwt.Process) -> weaver.typedefs.JobResults

      Process the result of the execution.



   .. py:method:: stage_results(results: weaver.typedefs.JobResults, expected_outputs: weaver.typedefs.CWL_ExpectedOutputs, out_dir: str) -> None

      Retrieves the remote execution :term:`Job` results for staging locally into the specified output directory.

      This operation should be called by the implementing remote :term:`Process` definition after :meth:`execute`.

      .. note::
          The :term:`CWL` runner expects the output file(s) to be written matching definition in ``expected_outputs``,
          but this definition could be a glob pattern to match multiple file and/or nested directories.
          We cannot rely on specific file names to be mapped, since glob can match many (eg: ``"*.txt"``).

      .. seealso::
          Function :func:`weaver.processes.convert._convert_any2cwl_io_complex` defines a generic glob pattern from
          the expected file extension based on Content-Type format. Since the remote :term:`WPS` :term:`Process`
          doesn't necessarily produce file names with the output ID as expected to find them (could be anything),
          staging must patch locations to let :term:`CWL` runtime resolve the files according to glob definitions.