weaver.datatype

Module Contents

weaver.datatype.AnyParams[source]
weaver.datatype.LOGGER[source]
class weaver.datatype.DictBase[source]

Dictionary with extended attributes auto-getter/setter for convenience.

Explicitly overridden getter/setter attributes are called instead of dict-key get/set-item to ensure corresponding checks and/or value adjustments are executed before applying it to the sub-dict.

Initialize self. See help(type(self)) for accurate signature.

classmethod properties(fget: bool = True, fset: bool = True) Iterator[str][source]

Get names of properties stored in the object, optionally filtered by read-only or write-only conditions.

dict() AnyParams[source]

Generate a dictionary representation of the object, but with inplace resolution of attributes as applicable.

class weaver.datatype.AutoBase[source]

Base that automatically converts literal class members to properties also accessible by dictionary keys.

class Data(AutoBase):
    field = 1
    other = None

d = Data()
d.other         # returns None
d.other = 2     # other is modified
d.other         # returns 2
dict(d)         # returns {'field': 1, 'other': 2}
d.field         # returns 1
d["field"]      # also 1 !

Initialize self. See help(type(self)) for accurate signature.

class weaver.datatype.Base[source]

Base interface for all data-types.

Initialize self. See help(type(self)) for accurate signature.

property id[source]
property uuid uuid.UUID[source]
abstract json() weaver.typedefs.JSON[source]

Obtain the JSON data representation for response body.

Note

This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response.

abstract params() AnyParams[source]

Obtain the internal data representation for storage.

Note

This method implementation should provide a JSON-serializable definition of all fields representing the object to store.

class weaver.datatype.LocalizedDateTimeProperty(fget: Ellipsis = None, fset: Callable[[Any, Union[datetime.datetime, str]], None] = None, fdel: Callable[[Any], None] = None, doc: str = None, default_now: bool = False)[source]

Property that ensures date-time localization is applied on the stored/retrieved value as required.

Initialize self. See help(type(self)) for accurate signature.

class weaver.datatype.Service(*args: Any, **kwargs: Any)[source]

Dictionary that contains OWS services.

It always has url key.

Initialize self. See help(type(self)) for accurate signature.

property id[source]
property url[source]

Service URL.

property name[source]

Service name.

property type[source]

Service type.

property public[source]

Flag if service has public access.

property auth[source]

Authentication method: public, token, cert.

json() weaver.typedefs.JSON[source]

Obtain the JSON data representation for response body.

Note

This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response.

params() AnyParams[source]

Obtain the internal data representation for storage.

Note

This method implementation should provide a JSON-serializable definition of all fields representing the object to store.

wps(container: weaver.typedefs.AnySettingsContainer = None, **kwargs: Any) owslib.wps.WebProcessingService[source]

Obtain the remote WPS service definition and metadata.

Stores the reference locally to avoid re-fetching it needlessly for future reference.

Obtains the links relevant to the service Provider.

Parameters
  • container – object that helps retrieve instance details, namely the host URL.

  • fetch – whether to attempt retrieving more precise details from the remote provider.

  • self_link – name of a section that represents the current link that will be returned.

metadata(container: weaver.typedefs.AnySettingsContainer) List[weaver.typedefs.Metadata][source]

Obtains the metadata relevant to the service provider.

keywords(container: weaver.typedefs.AnySettingsContainer = None) List[str][source]

Obtains the keywords relevant to the service provider.

summary(container: weaver.typedefs.AnySettingsContainer, fetch: bool = True, ignore: bool = False) Optional[weaver.typedefs.JSON][source]

Obtain the summary information from the provider service.

When metadata fetching is disabled, the generated summary will contain only information available locally.

Parameters
  • container – Employed to retrieve application settings.

  • fetch – Indicates whether metadata should be fetched from remote.

  • ignore – Indicates if failing metadata retrieval/parsing should be silently discarded or raised.

Returns

generated summary information.

Raises
  • ServiceParsingError – If the target service provider is not reachable, content is not parsable or any other error related to validating the service that needs to be understood for summary creation.

  • colander.Invalid – If the generated response format is not valid according to schema definition.

processes(container: weaver.typedefs.AnySettingsContainer, ignore: bool = False) Optional[List[Process]][source]

Obtains a list of remote service processes in a compatible weaver.datatype.Process format.

Note

Remote processes won’t be stored to the local process storage.

Parameters
  • container – Employed to retrieve application settings.

  • ignore – Indicates if failing service retrieval/parsing should be silently discarded or raised.

Raises

ServiceParsingError – If parsing failed and was NOT requested to be ignored.

Returns

If parsing was successful, list of converted remote service processes. If parsing failed and was requested to be ignored, returns None to distinguish from empty process list.

check_accessible(settings: weaver.typedefs.AnySettingsContainer, ignore: bool = True) bool[source]

Verify if the service URL is accessible.

class weaver.datatype.Job(*args: Any, **kwargs: Any)[source]

Dictionary that contains Job details for local Process or remote OWS execution.

It always has id and task_id keys.

Initialize self. See help(type(self)) for accurate signature.

inputs[source]
outputs[source]
created[source]
started[source]
finished[source]
updated[source]
results[source]
exceptions[source]
logs[source]
tags[source]
_get_log_msg(msg: Optional[str] = None, status: Optional[weaver.status.AnyStatusType] = None, progress: Optional[weaver.typedefs.Number] = None) str[source]
static _get_err_msg(error: owslib.wps.WPSException) str[source]
save_log(errors: Ellipsis = None, logger: Optional[logging.Logger] = None, message: Optional[str] = None, level: weaver.typedefs.AnyLogLevel = INFO, status: Optional[weaver.status.AnyStatusType] = None, progress: Optional[weaver.typedefs.Number] = None) None[source]

Logs the specified error and/or message, and adds the log entry to the complete job log.

For each new log entry, additional Job properties are added according to Job._get_log_msg() and the format defined by get_job_log_msg().

Parameters
  • errors – An error message or a list of WPS exceptions from which to log and save generated message stack.

  • logger – An additional Logger for which to propagate logged messages on top saving them to the job.

  • message – Explicit string to be logged, otherwise use the current Job.status_message is used.

  • level – Logging level to apply to the logged message. This parameter is ignored if errors are logged.

  • status – Override status applied in the logged message entry, but does not set it to the job object. Uses the current Job.status value if not specified. Must be one of Weaver.status values.

  • progress – Override progress applied in the logged message entry, but does not set it to the job object. Uses the current Job.progress value if not specified.

Note

The job object is updated with the log but still requires to be pushed to database to actually persist it.

property id uuid.UUID[source]

Job UUID to retrieve the details from storage.

property task_id Optional[weaver.typedefs.AnyUUID][source]

Reference Task UUID attributed by the Celery worker that monitors and executes this job.

property wps_id Optional[uuid.UUID][source]

Reference WPS Request/Response UUID attributed by the executed PyWPS process.

This UUID matches the status-location, log and output directory of the WPS process. This parameter is only available when the process is executed on this local instance.

property service Optional[str][source]

Service identifier of the corresponding remote process.

See also

property process Optional[str][source]

Process identifier of the corresponding remote process.

See also

  • Process.id

property type str[source]

Obtain the type of the element associated to the creation of this job.

_get_inputs() Optional[weaver.typedefs.ExecutionInputs][source]
_set_inputs(inputs: Optional[weaver.typedefs.ExecutionInputs]) None[source]
_get_outputs() Optional[weaver.typedefs.ExecutionOutputs][source]
_set_outputs(outputs: Optional[weaver.typedefs.ExecutionOutputs]) None[source]
property user_id Optional[str][source]
property status weaver.status.Status[source]
property status_message str[source]
property status_location Optional[str][source]
status_url(container: Optional[weaver.typedefs.AnySettingsContainer] = None) str[source]

Obtain the resolved endpoint where the Job status information can be obtained.

property notification_email Optional[str][source]
property accept_language Optional[str][source]
property execute_async bool[source]
property execute_sync bool[source]
property execution_mode weaver.execute.AnyExecuteMode[source]
property execution_response weaver.execute.AnyExecuteResponse[source]
property is_local bool[source]
property is_workflow bool[source]
property is_finished bool[source]
mark_finished() None[source]
_get_updated() datetime.datetime[source]
property duration Optional[datetime.timedelta][source]
property duration_str str[source]
property progress weaver.typedefs.Number[source]
property statistics Optional[weaver.typedefs.Statistics][source]

Collected statistics about used memory and processing units if available.

_get_results() List[Optional[Dict[str, weaver.typedefs.JSON]]][source]
_set_results(results: List[Optional[Dict[str, weaver.typedefs.JSON]]]) None[source]
_get_exceptions() List[Union[str, Dict[str, str]]][source]
_set_exceptions(exceptions: List[Union[str, Dict[str, str]]]) None[source]
_get_logs() List[str][source]
_set_logs(logs: List[str]) None[source]
_get_tags() List[Optional[str]][source]
_set_tags(tags: List[Optional[str]]) None[source]
property access weaver.visibility.Visibility[source]

Job visibility access from execution.

property context Optional[str][source]

Job outputs context.

property request Optional[str][source]

XML request for WPS execution submission as string (binary).

property response Optional[str][source]

XML status response from WPS execution submission as string (binary).

_job_url(base_url: Optional[str] = None) str[source]

Obtains the JSON links section of the response body for a Job.

If self_link is provided (e.g.: “outputs”) the link for that corresponding item will also be added as self entry to the links. It must be a recognized job link field.

Parameters
  • container – object that helps retrieve instance details, namely the host URL.

  • self_link – name of a section that represents the current link that will be returned.

json(container: Optional[weaver.typedefs.AnySettingsContainer] = None, self_link: Optional[str] = None) weaver.typedefs.JSON[source]

Obtains the JSON data representation for response body.

Note

Settings are required to update API shortcut URLs to job additional information. Without them, paths will not include the API host, which will not resolve to full URI.

params() AnyParams[source]

Obtain the internal data representation for storage.

Note

This method implementation should provide a JSON-serializable definition of all fields representing the object to store.

class weaver.datatype.AuthenticationTypes[source]

Generic enumeration.

Derive from this class to define new enumerations.

DOCKER = docker[source]
VAULT = vault[source]
class weaver.datatype.Authentication(auth_scheme: str, auth_token: str, auth_link: Optional[str], **kwargs: Any)[source]

Authentication details to store details required for process operations.

Initialize self. See help(type(self)) for accurate signature.

property type AuthenticationTypes[source]
property id uuid.UUID[source]
property token str[source]
property scheme str[source]
json()[source]

Obtain the JSON data representation for response body.

Note

This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response.

params() AnyParams[source]

Obtain the internal data representation for storage.

Note

This method implementation should provide a JSON-serializable definition of all fields representing the object to store.

classmethod from_params(**params: Any) AnyAuthentication[source]

Obtains the specialized Authentication using loaded parameters from params().

class weaver.datatype.DockerAuthentication(auth_scheme: str, auth_token: str, auth_link: str, **kwargs: Any)[source]

Authentication associated to a Docker image to retrieve from a private registry given by the reference link.

Initialize the authentication reference for pulling a Docker image from a protected registry.

Parameters
  • auth_scheme – Authentication scheme (Basic, Bearer, etc.)

  • auth_token – Applied token or credentials according to specified scheme.

  • auth_link – Fully qualified Docker registry image link ({registry-url}/{image}:{label}).

  • kwargs – Additional parameters for loading contents already parsed from database.

DOCKER_REGISTRY_DEFAULT_DOMAIN = index.docker.io[source]
DOCKER_REGISTRY_DEFAULT_URI[source]
type[source]
property credentials weaver.typedefs.JSON[source]

Generates the credentials to submit the login operation based on the authentication token and scheme.

property image str[source]

Obtains the image portion of the reference without repository prefix.

property registry str[source]

Obtains the registry entry that must used for docker login {registry}.

property docker str[source]

Obtains the full reference required when doing Docker operations such as docker pull <reference>.

property repository str[source]

Obtains the full Docker repository reference without any tag.

property tag Optional[str][source]

Obtain the requested tag from the Docker reference.

params() AnyParams[source]

Obtain the internal data representation for storage.

Note

This method implementation should provide a JSON-serializable definition of all fields representing the object to store.

class weaver.datatype.VaultFile(file_name: str = '', file_format: Optional[str] = None, file_secret: Optional[str] = None, auth_token: Optional[str] = None, **kwargs: Any)[source]

Dictionary that contains Vault file and its authentication information.

Initialize self. See help(type(self)) for accurate signature.

type[source]
bytes = 32[source]
classmethod authorized(file: Optional[VaultFile], token: Optional[str]) bool[source]

Determine whether the file access is authorized.

This method should be employed to validate access and reduce impact of timing attack analysis.

encrypt(file: IO[bytes | str]) io.BytesIO[source]

Encrypt file data using a secret to avoid plain text contents during temporary Vault storage.

Note

This is not intended to be a strong security countermeasure as contents can still be decrypted at any time if provided with the right secret. This is only to slightly obfuscate the contents while it transits between storage phases until destruction by the consuming process.

decrypt(file: IO[bytes | str]) io.BytesIO[source]

Decrypt file contents using secret.

property secret bytes[source]

Secret associated to this Vault file to hash contents back and forth.

property id uuid.UUID[source]

Vault file UUID to retrieve the details from storage.

property name str[source]

Name to retrieve the file.

property format Optional[str][source]

Format Media-Type of the file.

property href str[source]

Obtain the vault input reference corresponding to the file.

This corresponds to the href value to be provided when submitting an input that should be updated using the vault file of specified UUID and using the respective authorization token in X-Auth-Vault header.

json() weaver.typedefs.JSON[source]

Obtain the JSON data representation for response body.

Note

This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response.

params() AnyParams[source]

Obtain the internal data representation for storage.

Note

This method implementation should provide a JSON-serializable definition of all fields representing the object to store.

class weaver.datatype.Process(*args: Any, **kwargs: Any)[source]

Dictionary that contains a process definition for db storage.

It always has identifier (or id alias) and a package definition. Parameters can be accessed by key or attribute, and appropriate validators or default values will be applied.

Initialize self. See help(type(self)) for accurate signature.

_character_codes = [['$', '$'], ['.', '.']][source]
_get_id() str[source]
_set_id(_id: str) None[source]
classmethod split_version(process_id: str) Tuple[str, Optional[str]][source]

Split the tagged version from the Process identifier considering any required special handling.

Returns

Process ID (only) and the version if any was available in tagged reference.

property latest bool[source]

Checks if this Process corresponds to the latest revision.

property name str[source]

Obtain only the Process name portion of the unique identifier.

property tag str[source]

Full identifier including the version for an unique reference.

property title str[source]
_get_desc() str[source]
_set_desc(description: str) None[source]
property keywords List[str][source]
property metadata List[weaver.typedefs.Metadata][source]
property version Optional[str][source]
property inputs Optional[List[Dict[str, weaver.typedefs.JSON]]][source]

Inputs of the process following backward-compatible conversion of stored parameters.

According to OGC-API, maxOccurs and minOccurs representations should be:
  • maxOccurs: int or "unbounded"

  • minOccurs: int

And, mediaType should be in description as:
  • mediaType: string

Note

Because of pre-registered/deployed/retrieved remote processes, inputs are formatted in-line to respect valid OGC-API schema representation and apply any required correction transparently.

property outputs Optional[List[Dict[str, weaver.typedefs.JSON]]][source]

Outputs of the process following backward-compatible conversion of stored parameters.

According to OGC-API, mediaType should be in description as:
  • mediaType: string

Note

Because of pre-registered/deployed/retrieved remote processes, inputs are formatted in-line to respect valid OGC-API schema representation and apply any required correction transparently.

property jobControlOptions List[weaver.execute.AnyExecuteControlOption][source]

Control options that indicate which Job execution modes are supported by the Process.

Note

There are no official mentions about the ordering of jobControlOptions. Nevertheless, it is often expected that the first item can be considered the default mode when none is requested explicitly (at execution time). With the definition of execution mode through the Prefer header, Weaver has the option to decide if it wants to honor this header, according to available resources and Job duration.

For this reason, async is placed first by default when nothing was defined during deployment, since it is the preferred mode in Weaver. If deployment included items though, they are preserved as is. This allows to re-deploy a Process to a remote non-Weaver ADES preserving the original Process definition.

See also

Discussion about expected ordering of jobControlOptions: https://github.com/opengeospatial/ogcapi-processes/issues/171#issuecomment-836819528

property outputTransmission List[weaver.execute.AnyExecuteTransmissionMode][source]
property processDescriptionURL Optional[str][source]
property processEndpointWPS1 Optional[str][source]
property executeEndpoint Optional[str][source]
property owsContext Optional[weaver.typedefs.JSON][source]
property type weaver.processes.types.AnyProcessType[source]

Type of process amongst weaver.processes.types definitions.

property mutable bool[source]

Indicates if a process can be modified.

property deployment_profile str[source]
property package Optional[weaver.typedefs.CWL][source]

Package CWL definition as JSON.

property payload weaver.typedefs.JSON[source]

Deployment specification as JSON.

static _recursive_replace(pkg: weaver.typedefs.JSON, index_from: int, index_to: int) weaver.typedefs.JSON[source]
static _encode(obj: Optional[weaver.typedefs.JSON]) Optional[weaver.typedefs.JSON][source]
static _decode(obj: Optional[weaver.typedefs.JSON]) Optional[weaver.typedefs.JSON][source]
property visibility weaver.visibility.Visibility[source]
property auth Optional[AnyAuthentication][source]

Authentication token required for operations with the process.

params() AnyParams[source]

Obtain the internal data representation for storage.

Note

This method implementation should provide a JSON-serializable definition of all fields representing the object to store.

property params_wps AnyParams[source]

Values applicable to create an instance of pywps.app.Process.

dict() AnyParams[source]

Generate a dictionary representation of the object, but with inplace resolution of attributes as applicable.

json() weaver.typedefs.JSON[source]

Obtains the JSON serializable complete representation of the process.

Obtains the JSON links section of many response body for the Process.

Parameters

container – object that helps retrieve instance details, namely the host URL.

href(container: Optional[weaver.typedefs.AnySettingsContainer] = None) str[source]

Obtain the reference URL for this Process.

offering(schema: weaver.processes.constants.ProcessSchemaType = ProcessSchema.OGC, request: weaver.typedefs.AnyRequestType = None) weaver.typedefs.JSON[source]

Obtains the JSON or XML serializable offering/description representation of the Process.

Parameters
  • request – HTTP request that can provide more details on how to describe the process.

  • schema – One of values defined by sd.ProcessDescriptionSchemaQuery to select which process description representation to generate (see each schema for details).

Note

Property name offering is employed to differentiate from the string process description field. The result of this JSON representation is still the ProcessDescription schema.

summary(revision: bool = False) weaver.typedefs.JSON[source]

Obtains the JSON serializable summary representation of the process.

Parameters

revision – Replace the process identifier by the complete tag representation.

static from_wps(wps_process: pywps.Process, **extra_params: Any) Process[source]

Converts a pywps Process into a weaver.datatype.Process using provided parameters.

static from_ows(process: owslib.wps.Process, service: Service, container: weaver.typedefs.AnySettingsContainer, **kwargs: Any) Process[source]

Converts a owslib.wps Process to local storage weaver.datatype.Process.

property service Optional[str][source]

Name of the parent service provider under which this process resides.

static convert(process: weaver.typedefs.AnyProcess, service: Optional[Service] = None, container: Optional[weaver.typedefs.AnySettingsContainer] = None, **kwargs: Any) Process[source]

Converts known process equivalents definitions into the formal datatype employed by Weaver.

xml(request: weaver.typedefs.AnyRequestType) str[source]

Obtain the raw XML representation of the Process using WPS schema.

wps() pywps.Process[source]

Converts this Process to a corresponding format understood by pywps.

class weaver.datatype.Quote(*args: Any, **kwargs: Any)[source]

Dictionary that contains quote information.

It always has id and process keys.

Initialize the quote.

Note

Although many parameters are required to render the final quote, they are not enforced at creation since the partial quote definition is needed before it can be processed.

processParameters[source]
expire[source]
created[source]
property id uuid.UUID[source]

Quote ID.

property detail Optional[str][source]
property status weaver.quotation.status.QuoteStatus[source]
property user Optional[Union[str, int]][source]

User ID requesting the quote.

property process str[source]

Process ID.

property seconds int[source]

Estimated time of the process execution in seconds.

property duration datetime.timedelta[source]

Duration as delta time that can be converted to ISO-8601 format (P[n]Y[n]M[n]DT[n]H[n]M[n]S).

property duration_str str[source]

Human-readable duration in formatted as hh:mm:ss.

property parameters weaver.typedefs.QuoteProcessParameters[source]

Process execution parameters for quote.

This should include minimally the inputs and expected outputs, but could be extended as needed with relevant details for quoting algorithm.

property price float[source]

Price of the current quote.

property currency Optional[str][source]

Currency of the quote price.

property steps List[uuid.UUID][source]

Sub-quote IDs if applicable.

params() AnyParams[source]

Obtain the internal data representation for storage.

Note

This method implementation should provide a JSON-serializable definition of all fields representing the object to store.

partial() weaver.typedefs.JSON[source]

Submitted Quote representation with minimal details until evaluation is completed.

json() weaver.typedefs.JSON[source]

Step Quote with JSON representation.

Note

Does not include derived Quote details if the associated Process is a Workflow.

href(container: Optional[weaver.typedefs.AnySettingsContainer] = None) str[source]

Obtain the reference URL for this Quote.

class weaver.datatype.Bill(*args, **kwargs)[source]

Dictionary that contains bill information.

It always has id, user, quote and job keys.

Initialize self. See help(type(self)) for accurate signature.

property id[source]

Bill ID.

property user[source]

User ID.

property quote[source]

Quote ID.

property job[source]

Job ID.

property price[source]

Price of the current quote.

property currency[source]

Currency of the quote price.

property created[source]

Quote creation datetime.

property title[source]

Quote title.

property description[source]

Quote description.

params() AnyParams[source]

Obtain the internal data representation for storage.

Note

This method implementation should provide a JSON-serializable definition of all fields representing the object to store.

json() weaver.typedefs.JSON[source]

Obtain the JSON data representation for response body.

Note

This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response.