weaver.datatype =============== .. py:module:: weaver.datatype .. autoapi-nested-parse:: Definitions of types used by tokens. Module Contents --------------- .. py:data:: AnyParams .. py:data:: LOGGER .. py:class:: DictBase Dictionary with extended attributes auto-``getter``/``setter`` for convenience. Explicitly overridden ``getter``/``setter`` attributes are called instead of ``dict``-key ``get``/``set``-item to ensure corresponding checks and/or value adjustments are executed before applying it to the sub-``dict``. Initialize self. See help(type(self)) for accurate signature. .. py:method:: properties(fget=True, fset=True) :classmethod: Get names of properties stored in the object, optionally filtered by read-only or write-only conditions. .. py:method:: dict() Generate a dictionary representation of the object, but with inplace resolution of attributes as applicable. .. py:class:: AutoBase Base that automatically converts literal class members to properties also accessible by dictionary keys. .. code-block:: python class Data(AutoBase): field = 1 other = None d = Data() d.other # returns None d.other = 2 # other is modified d.other # returns 2 dict(d) # returns {'field': 1, 'other': 2} d.field # returns 1 d["field"] # also 1 ! Initialize self. See help(type(self)) for accurate signature. .. py:class:: Base Base interface for all data-types. Initialize self. See help(type(self)) for accurate signature. .. py:property:: id :abstractmethod: .. py:property:: uuid .. py:method:: json() :abstractmethod: Obtain the JSON data representation for response body. .. note:: This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response. .. py:method:: params() :abstractmethod: Obtain the internal data representation for storage. .. note:: This method implementation should provide a JSON-serializable definition of all fields representing the object to store. .. py:class:: LocalizedDateTimeProperty(fget=None, fset=None, fdel=None, doc=None, default_now=False) Property that ensures date-time localization is applied on the stored/retrieved value as required. Initialize self. See help(type(self)) for accurate signature. .. py:attribute:: default_now :value: False .. py:class:: Service(*args, **kwargs) Dictionary that contains OWS services. It always has ``url`` key. Initialize self. See help(type(self)) for accurate signature. .. py:property:: id .. py:property:: url Service URL. .. py:property:: name Service name. .. py:property:: type Service type. .. py:property:: public Flag if service has public access. .. py:property:: auth Authentication method: public, token, cert. .. py:method:: json() Obtain the JSON data representation for response body. .. note:: This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response. .. py:method:: params() Obtain the internal data representation for storage. .. note:: This method implementation should provide a JSON-serializable definition of all fields representing the object to store. .. py:method:: wps(container=None, **kwargs) Obtain the remote WPS service definition and metadata. Stores the reference locally to avoid re-fetching it needlessly for future reference. .. py:method:: links(container, fetch=True, self_link=None) Obtains the links relevant to the service :term:`Provider`. :param container: object that helps retrieve instance details, namely the host URL. :param fetch: whether to attempt retrieving more precise details from the remote provider. :param self_link: name of a section that represents the current link that will be returned. .. py:method:: metadata(container) Obtains the metadata relevant to the service provider. .. py:method:: keywords(container=None) Obtains the keywords relevant to the service provider. .. py:method:: summary(container, fetch=True, ignore=False) Obtain the summary information from the provider service. When metadata fetching is disabled, the generated summary will contain only information available locally. :param container: Employed to retrieve application settings. :param fetch: Indicates whether metadata should be fetched from remote. :param ignore: Indicates if failing metadata retrieval/parsing should be silently discarded or raised. :return: generated summary information. :raises ServiceParsingError: If the target service provider is not reachable, content is not parsable or any other error related to validating the service that needs to be understood for summary creation. :raises colander.Invalid: If the generated response format is not valid according to schema definition. .. py:method:: processes(container, ignore=False) Obtains a list of remote service processes in a compatible :class:`weaver.datatype.Process` format. .. note:: Remote processes won't be stored to the local process storage. :param container: Employed to retrieve application settings. :param ignore: Indicates if failing service retrieval/parsing should be silently discarded or raised. :raises ServiceParsingError: If parsing failed and was NOT requested to be ignored. :return: If parsing was successful, list of converted remote service processes. If parsing failed and was requested to be ignored, returns ``None`` to distinguish from empty process list. .. py:method:: check_accessible(settings, ignore=True) Verify if the service URL is accessible. .. py:class:: Job(*args, **kwargs) Dictionary that contains :term:`Job` details for local :term:`Process` or remote :term:`OWS` execution. It always has ``id`` and ``task_id`` keys. Initialize self. See help(type(self)) for accurate signature. .. py:method:: update_from(job) Forwards any internal or control properties from the specified :class:`Job` to this one. .. py:method:: cleanup() .. py:property:: tmpdir Optional temporary directory available for the :term:`Job` to store files needed for its operation. It is up to the caller to remove the contents by calling :meth:`cleanup`. .. py:method:: _get_message(message, size_limit=None) :staticmethod: .. py:method:: _get_log_msg(msg=None, status=None, progress=None, size_limit=None) .. py:method:: _get_err_msg(error, size_limit=None) :staticmethod: .. py:method:: log(level, message, *args, **kwargs) Provides the :class:`LoggerHandler` interface, allowing to pass the :term:`Job` directly as a logger reference. The same parameters as :meth:`save_log` can be provided. .. py:method:: save_log(*, errors=None, logger=None, message=None, level=INFO, status=None, progress=None, size_limit=None) Logs the specified error and/or message, and adds the log entry to the complete job log. For each new log entry, additional :class:`Job` properties are added according to :meth:`Job._get_log_msg` and the format defined by :func:`get_job_log_msg`. :param errors: An error message or a list of WPS exceptions from which to log and save generated message stack. :param logger: An additional :class:`Logger` for which to propagate logged messages on top saving them to the job. :param message: Explicit string to be logged, otherwise use the current :py:attr:`Job.status_message` is used. :param level: Logging level to apply to the logged ``message``. This parameter is ignored if ``errors`` are logged. :param status: Override status applied in the logged message entry, but does not set it to the job object. Uses the current :attr:`Job.status` value if not specified. Must be one of :mod:`Weaver.status` values. :param progress: Override progress applied in the logged message entry, but does not set it to the job object. Uses the current :attr:`Job.progress` value if not specified. :param size_limit: Log message entries that individually exceed the limit will be clipped with a generic message. The parameter is provided for convenience, but take note that setting a too large value could cause the complete :term:`Job` to fail saving to the database if its total size exceeds the document limit. .. note:: The job object is updated with the log but still requires to be pushed to database to actually persist it. .. py:property:: id Job UUID to retrieve the details from storage. .. py:property:: task_id Reference Task UUID attributed by the ``Celery`` worker that monitors and executes this job. .. py:property:: wps_id Reference WPS Request/Response UUID attributed by the executed ``PyWPS`` process. This UUID matches the status-location, log and output directory of the WPS process. This parameter is only available when the process is executed on this local instance. .. seealso:: - :attr:`Job.request` - :attr:`Job.response` .. py:property:: wps_url Service URL reference for :term:`WPS` interface. .. seealso:: - :attr:`Process.processEndpointWPS1` - :attr:`Service.url` .. py:property:: service Service identifier of the corresponding remote process. .. seealso:: - :attr:`Service.id` .. py:property:: process Process identifier of the corresponding remote process. .. seealso:: - :attr:`Process.id` .. py:property:: processing_entity_type .. py:property:: type Obtain the type of the element associated to the creation of this job. .. seealso:: - Defined in https://schemas.opengis.net/ogcapi/processes/part1/1.0/openapi/schemas/statusInfo.yaml. - Queried with https://docs.ogc.org/is/18-062r2/18-062r2.html#toc49 (Parameter Type section). .. py:property:: title .. py:method:: _get_inputs() .. py:method:: _set_inputs(inputs) .. py:attribute:: inputs .. py:method:: _get_outputs() .. py:method:: _set_outputs(outputs) .. py:attribute:: outputs .. py:property:: user_id .. py:property:: success .. py:property:: status .. py:property:: status_message .. py:property:: status_location .. py:property:: subscribers .. py:property:: accept_type .. py:property:: accept_language .. py:property:: accept_profile .. py:property:: execute_async .. py:property:: execute_sync .. py:property:: execution_mode .. py:property:: execution_wait Execution time (in seconds) to wait for a synchronous response. .. py:property:: execution_response .. py:property:: execution_return .. py:property:: is_local .. py:property:: is_workflow .. py:property:: is_finished .. py:method:: mark_finished() .. py:method:: _get_updated() .. py:attribute:: created .. py:attribute:: started .. py:attribute:: finished .. py:attribute:: updated .. py:property:: duration .. py:property:: duration_str .. py:property:: progress .. py:property:: statistics Collected statistics about used memory and processing units if available. .. py:method:: _get_results() .. py:method:: _set_results(results) .. py:attribute:: results .. py:method:: _get_exceptions() .. py:method:: _set_exceptions(exceptions) .. py:attribute:: exceptions .. py:method:: _get_logs() .. py:method:: _set_logs(logs) .. py:attribute:: logs .. py:method:: _get_tags() .. py:method:: _set_tags(tags) .. py:attribute:: tags .. py:property:: access Job visibility access from execution. .. py:property:: context Job outputs context. .. py:property:: request XML request for WPS execution submission as string (binary). .. py:property:: response XML status response from WPS execution submission as string (binary). .. py:method:: process_url(container=None) .. py:method:: job_url(container=None, extra_path=None) .. py:method:: status_url(container=None) .. py:method:: logs_url(container=None) .. py:method:: exceptions_url(container=None) .. py:method:: inputs_url(container=None) .. py:method:: outputs_url(container=None) .. py:method:: results_url(container=None) .. py:method:: result_path(job_id=None, output_id=None, file_name=None) Obtains a *relative* result path, according to requested parameters and the :term:`Job` definition. The generated path will automatically apply the relative job context if defined. :param job_id: Override ID to employ for the job path. Otherwise, uses the usually job UUID by default. This should be used for cases where the ID is "not yet" established by the job, or that an alternate location based on a UUID established by another source must be employed. :param output_id: Output ID to refer to in the path. If omitted, the path prefix will stop at the job ID fragment. :param file_name: Output file name and extension to apply to the path. If omitted, the path prefix will stop at the output ID. :return: Resolved *relative* result path. .. py:method:: get_all_possible_formats_links(url, results) Get direct links to all outputs in any possible format. :param url: The base URL for constructing links. :param results: A list of result dictionaries containing "mimeType" and "identifier". :returns: A list of dictionaries representing the links to all possible output formats. .. py:method:: prov_url(container=None, extra_path=None) .. py:method:: prov_path(container=None, extra_path=None, prov_format=None) Obtain the relative path of the ``PROV`` contents. .. py:method:: prov_data(container=None, extra_path=None, prov_format=None) Read or retrieve data from the packaged provenance directory contents associated to the :term:`Job`. .. py:method:: links(container=None, self_link=None) Obtains the JSON links section of the response body for a :term:`Job`. If :paramref:`self_link` is provided (e.g.: `"outputs"`) the link for that corresponding item will also be added as `self` entry to the links. It must be a recognized job link field. :param container: object that helps retrieve instance details, namely the host URL. :param self_link: name of a section that represents the current link that will be returned. .. py:method:: summary(validate=True, **kwargs) .. py:method:: json(container=None, **kwargs) Obtains the :term:`JSON` data representation for :term:`Job` response body. .. note:: Settings are required to update API shortcut URLs to job additional information. Without them, paths will not include the API host, which will not resolve to full URI. .. py:method:: params() Obtain the internal data representation for storage. .. note:: This method implementation should provide a JSON-serializable definition of all fields representing the object to store. .. py:class:: AuthenticationTypes Generic enumeration. Derive from this class to define new enumerations. .. py:attribute:: DOCKER :value: 'docker' .. py:attribute:: VAULT :value: 'vault' .. py:class:: Authentication(auth_scheme, auth_token=None, auth_username=None, auth_password=None, auth_link=None, **kwargs) Authentication details to store details required for process operations. Initialize self. See help(type(self)) for accurate signature. .. py:property:: scheme .. py:property:: token .. py:property:: type :abstractmethod: .. py:property:: id .. py:property:: link .. py:method:: json() Obtain the JSON data representation for response body. .. note:: This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response. .. py:method:: params() Obtain the internal data representation for storage. .. note:: This method implementation should provide a JSON-serializable definition of all fields representing the object to store. .. py:method:: from_credentials(username, password, **params) :classmethod: .. py:method:: from_params(**params) :classmethod: Obtains the specialized :class:`Authentication` using loaded parameters from :meth:`params`. .. py:class:: DockerAuthentication(auth_link, auth_scheme='Basic', auth_token=None, **kwargs) Authentication associated to a :term:`Docker` image to retrieve from a private registry given by the reference link. .. seealso:: :ref:`app_pkg_docker` Initialize the authentication reference for pulling a Docker image from a protected registry. :param auth_link: Fully qualified Docker registry image link (``{registry-url}/{image}:{label}``). :param auth_scheme: Authentication scheme (Basic, Bearer, etc.) if required. :param kwargs: Additional parameters including authentication token, username/password credentials according to specified scheme, and other definitions to load contents already parsed from database. .. py:attribute:: DOCKER_LINK_REGEX .. py:attribute:: DOCKER_REGISTRY_DEFAULT_DOMAIN :value: 'index.docker.io' .. py:attribute:: DOCKER_REGISTRY_DEFAULT_URI :value: 'https://index.docker.io/v1/' .. py:attribute:: type .. py:property:: credentials Generates the credentials to submit the login operation based on the authentication token and scheme. .. py:property:: image Obtains the image portion of the reference without repository prefix. .. py:property:: registry Obtains the registry entry that must be used for ``docker login {registry}``. .. py:property:: reference Obtains the full reference required when doing :term:`Docker` operations such as ``docker pull {reference}``. .. py:attribute:: docker .. py:property:: repository Obtains the full :term:`Docker` repository reference without any tag. .. py:property:: tag Obtain the requested tag from the :term:`Docker` reference. .. py:method:: params() Obtain the internal data representation for storage. .. note:: This method implementation should provide a JSON-serializable definition of all fields representing the object to store. .. py:class:: VaultFile(file_name='', file_format=None, file_secret=None, auth_token=None, **kwargs) Dictionary that contains :term:`Vault` file and its authentication information. Initialize self. See help(type(self)) for accurate signature. .. py:attribute:: type .. py:attribute:: bytes :value: 32 .. py:method:: authorized(file, token) :classmethod: Determine whether the file access is authorized. This method should be employed to validate access and reduce impact of timing attack analysis. .. py:method:: encrypt(file) Encrypt file data using a secret to avoid plain text contents during temporary :term:`Vault` storage. .. note:: This is not intended to be a *strong* security countermeasure as contents can still be decrypted at any time if provided with the right secret. This is only to slightly obfuscate the contents while it transits between storage phases until destruction by the consuming process. .. py:method:: decrypt(file) Decrypt file contents using secret. .. py:property:: secret Secret associated to this :term:`Vault` file to hash contents back and forth. .. py:property:: id Vault file UUID to retrieve the details from storage. .. py:property:: name Name to retrieve the file. .. py:property:: format Format Media-Type of the file. .. py:property:: href Obtain the vault input reference corresponding to the file. This corresponds to the ``href`` value to be provided when submitting an input that should be updated using the vault file of specified UUID and using the respective authorization token in ``X-Auth-Vault`` header. .. py:method:: json() Obtain the JSON data representation for response body. .. note:: This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response. .. py:method:: params() Obtain the internal data representation for storage. .. note:: This method implementation should provide a JSON-serializable definition of all fields representing the object to store. .. py:class:: Process(*args, **kwargs) Dictionary that contains a process definition for db storage. It always has ``identifier`` (or ``id`` alias) and a ``package`` definition. Parameters can be accessed by key or attribute, and appropriate validators or default values will be applied. Initialize self. See help(type(self)) for accurate signature. .. py:method:: _get_id() .. py:method:: _set_id(_id) .. py:method:: split_version(process_id) :classmethod: Split the tagged version from the :term:`Process` identifier considering any required special handling. :returns: Process ID (only) and the version if any was available in tagged reference. .. py:property:: latest Checks if this :term:`Process` corresponds to the latest revision. .. py:property:: name Obtain only the :term:`Process` name portion of the unique identifier. .. py:property:: tag Full identifier including the version for a unique reference. .. py:property:: title .. py:method:: _get_desc() .. py:method:: _set_desc(description) .. py:property:: keywords .. py:property:: metadata .. py:property:: version .. py:property:: inputs Inputs of the process following backward-compatible conversion of stored parameters. According to `OGC-API`, ``maxOccurs`` and ``minOccurs`` representations should be: - ``maxOccurs``: ``int`` or ``"unbounded"`` - ``minOccurs``: ``int`` And, ``mediaType`` should be in description as: - ``mediaType``: ``string`` .. note:: Because of pre-registered/deployed/retrieved remote processes, inputs are formatted in-line to respect valid OGC-API schema representation and apply any required correction transparently. .. py:property:: outputs Outputs of the process following backward-compatible conversion of stored parameters. According to `OGC-API`, ``mediaType`` should be in description as: - ``mediaType``: ``string`` .. note:: Because of pre-registered/deployed/retrieved remote processes, inputs are formatted in-line to respect valid OGC-API schema representation and apply any required correction transparently. .. py:property:: jobControlOptions Control options that indicate which :term:`Job` execution modes are supported by the :term:`Process`. .. note:: There are no official mentions about the ordering of ``jobControlOptions``. Nevertheless, it is often expected that the first item can be considered the default mode when none is requested explicitly (at execution time). With the definition of execution mode through the ``Prefer`` header, `Weaver` has the option to decide if it wants to honor this header, according to available resources and :term:`Job` duration. For this reason, ``async`` is placed first by default when nothing was defined during deployment, since it is the preferred mode in `Weaver`. If deployment included items though, they are preserved as is. This allows to re-deploy a :term:`Process` to a remote non-`Weaver` :term:`ADES` preserving the original :term:`Process` definition. .. seealso:: Discussion about expected ordering of ``jobControlOptions``: https://github.com/opengeospatial/ogcapi-processes/issues/171#issuecomment-836819528 .. py:property:: outputTransmission .. py:property:: processDescriptionURL .. py:property:: processEndpointWPS1 .. py:property:: executeEndpoint .. py:property:: owsContext .. py:property:: type Type of process amongst :mod:`weaver.processes.types` definitions. .. py:property:: mutable Indicates if a process can be modified. .. py:property:: deployment_profile .. py:property:: package Package :term:`CWL` definition as :term:`JSON`. .. py:property:: payload Deployment specification as :term:`JSON`. .. py:attribute:: _character_codes :value: [('$', '＄'), ('.', '．')] .. py:method:: _recursive_replace(pkg, index_from, index_to) :staticmethod: .. py:method:: _encode(obj) :staticmethod: .. py:method:: _decode(obj) :staticmethod: .. py:property:: estimator .. py:property:: visibility .. py:property:: auth Authentication token required for operations with the process. .. py:method:: params() Obtain the internal data representation for storage. .. note:: This method implementation should provide a JSON-serializable definition of all fields representing the object to store. .. py:property:: params_wps Values applicable to create an instance of :class:`pywps.app.Process`. .. py:method:: dict() Generate a dictionary representation of the object, but with inplace resolution of attributes as applicable. .. py:method:: json() Obtains the JSON serializable complete representation of the process. .. py:attribute:: _links .. py:method:: links(container=None) Obtains the JSON links section of many response body for the :term:`Process`. :param container: object that helps retrieve instance details, namely the host URL. .. py:property:: additional_links .. py:method:: href(container=None) Obtain the reference URL for this :term:`Process`. .. py:method:: offering(schema=ProcessSchema.OGC, request=None) Obtains the :term:`JSON` or :term:`XML` serializable offering/description representation of the :term:`Process`. :param request: HTTP request that can provide more details on how to describe the process. :param schema: One of values defined by :class:`sd.ProcessDescriptionSchemaQuery` to select which process description representation to generate (see each schema for details). .. note:: Property name ``offering`` is employed to differentiate from the string process ``description`` field. The result of this JSON representation is still the ``ProcessDescription`` schema. .. py:method:: summary(revision=False, links=True, container=None) Obtains the JSON serializable summary representation of the process. :param revision: Replace the process identifier by the complete tag representation. :param links: Include process links in summary. :param container: Application settings or database container to retrieve links and avoid reconnections. .. py:method:: from_wps(wps_process, **extra_params) :staticmethod: Converts a :mod:`pywps` Process into a :class:`weaver.datatype.Process` using provided parameters. .. py:method:: from_ows(process, service, container, **kwargs) :staticmethod: Converts a :mod:`owslib.wps` Process to local storage :class:`weaver.datatype.Process`. .. py:property:: service Name of the parent service provider under which this process resides. .. seealso:: - :meth:`Service.processes` - :meth:`Process.convert` .. py:method:: convert(process, service=None, container=None, **kwargs) :staticmethod: Converts known process equivalents definitions into the formal datatype employed by Weaver. .. py:method:: xml(request) Obtain the raw :term:`XML` representation of the :term:`Process` using :term:`WPS` schema. .. py:method:: wps(container=None) Converts this :class:`Process` to a corresponding format understood by :mod:`pywps`. .. py:class:: PriceMixin Base interface for all data-types. Initialize self. See help(type(self)) for accurate signature. .. py:property:: amount Amount of the current quote. .. py:property:: currency Currency of the quote price. .. py:property:: price .. py:class:: Quote(*args, **kwargs) Dictionary that contains quote information. It always has ``id`` and ``process`` keys. Initialize the quote. .. note:: Although many parameters are required to render the final quote, they are not enforced at creation since the partial quote definition is needed before it can be processed. .. py:method:: setdefault(key, default) Insert key with a value of default if key is not in the dictionary. Return the value for key if key is in the dictionary, else default. .. py:method:: update(mapping, **__) D.update([E, ]**F) -> None. Update D from dict/iterable E and F. If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k] .. py:property:: paid .. py:property:: id Quote ID. .. py:property:: detail .. py:property:: status .. py:property:: user User ID requesting the quote. .. py:property:: process Process ID. .. py:property:: seconds Estimated time of the process execution in seconds. .. py:property:: duration Duration as delta time that can be converted to ISO-8601 format (``P[n]Y[n]M[n]DT[n]H[n]M[n]S``). .. py:property:: duration_str Human-readable duration in formatted as ``hh:mm:ss``. .. py:property:: parameters Process execution parameters for quote. This should include minimally the inputs and expected outputs, but could be extended as needed with relevant details for quoting algorithm. .. py:property:: results Process execution results following quote estimation. .. py:property:: outputs Quote estimation outputs for a following step quote estimation. .. py:attribute:: expire .. py:attribute:: created .. py:property:: steps Sub-quote IDs if applicable. .. py:method:: params() Obtain the internal data representation for storage. .. note:: This method implementation should provide a JSON-serializable definition of all fields representing the object to store. .. py:method:: partial() Submitted :term:`Quote` representation with minimal details until evaluation is completed. .. py:method:: json() Step :term:`Quote` with :term:`JSON` representation. .. note:: Does not include derived :term:`Quote` details if the associated :term:`Process` is a :term:`Workflow`. .. py:method:: links(container=None) .. py:method:: href(container=None) Obtain the reference URL for this :term:`Quote`. .. py:class:: Bill(*args, **kwargs) Dictionary that contains bill information. It always has ``id``, ``user``, ``quote`` and ``job`` keys. Initialize self. See help(type(self)) for accurate signature. .. py:property:: id Bill ID. .. py:property:: user User ID. .. py:property:: quote Quote ID. .. py:property:: job Job ID. .. py:property:: created Quote creation datetime. .. py:property:: title Quote title. .. py:property:: description Quote description. .. py:method:: params() Obtain the internal data representation for storage. .. note:: This method implementation should provide a JSON-serializable definition of all fields representing the object to store. .. py:method:: json() Obtain the JSON data representation for response body. .. note:: This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response.