weaver.datatype
Definitions of types used by tokens.
Module Contents
- class weaver.datatype.DictBase[source]
Dictionary with extended attributes auto-
getter
/setter
for convenience.Explicitly overridden
getter
/setter
attributes are called instead ofdict
-keyget
/set
-item to ensure corresponding checks and/or value adjustments are executed before applying it to the sub-dict
.Initialize self. See help(type(self)) for accurate signature.
- class weaver.datatype.AutoBase[source]
Base that automatically converts literal class members to properties also accessible by dictionary keys.
class Data(AutoBase): field = 1 other = None d = Data() d.other # returns None d.other = 2 # other is modified d.other # returns 2 dict(d) # returns {'field': 1, 'other': 2} d.field # returns 1 d["field"] # also 1 !
Initialize self. See help(type(self)) for accurate signature.
- class weaver.datatype.Base[source]
Base interface for all data-types.
Initialize self. See help(type(self)) for accurate signature.
- class weaver.datatype.LocalizedDateTimeProperty(fget=None, fset=None, fdel=None, doc=None, default_now=False)[source]
Property that ensures date-time localization is applied on the stored/retrieved value as required.
Initialize self. See help(type(self)) for accurate signature.
- class weaver.datatype.Service(*args, **kwargs)[source]
Dictionary that contains OWS services.
It always has
url
key.Initialize self. See help(type(self)) for accurate signature.
- json()[source]
Obtain the JSON data representation for response body.
Note
This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response.
- params()[source]
Obtain the internal data representation for storage.
Note
This method implementation should provide a JSON-serializable definition of all fields representing the object to store.
- wps(container=None, **kwargs)[source]
Obtain the remote WPS service definition and metadata.
Stores the reference locally to avoid re-fetching it needlessly for future reference.
- links(container, fetch=True, self_link=None)[source]
Obtains the links relevant to the service Provider.
- Parameters:
container – object that helps retrieve instance details, namely the host URL.
fetch – whether to attempt retrieving more precise details from the remote provider.
self_link – name of a section that represents the current link that will be returned.
- summary(container, fetch=True, ignore=False)[source]
Obtain the summary information from the provider service.
When metadata fetching is disabled, the generated summary will contain only information available locally.
- Parameters:
container – Employed to retrieve application settings.
fetch – Indicates whether metadata should be fetched from remote.
ignore – Indicates if failing metadata retrieval/parsing should be silently discarded or raised.
- Returns:
generated summary information.
- Raises:
ServiceParsingError – If the target service provider is not reachable, content is not parsable or any other error related to validating the service that needs to be understood for summary creation.
colander.Invalid – If the generated response format is not valid according to schema definition.
- processes(container, ignore=False)[source]
Obtains a list of remote service processes in a compatible
weaver.datatype.Process
format.Note
Remote processes won’t be stored to the local process storage.
- Parameters:
container – Employed to retrieve application settings.
ignore – Indicates if failing service retrieval/parsing should be silently discarded or raised.
- Raises:
ServiceParsingError – If parsing failed and was NOT requested to be ignored.
- Returns:
If parsing was successful, list of converted remote service processes. If parsing failed and was requested to be ignored, returns
None
to distinguish from empty process list.
- class weaver.datatype.Job(*args, **kwargs)[source]
Dictionary that contains Job details for local Process or remote OWS execution.
It always has
id
andtask_id
keys.Initialize self. See help(type(self)) for accurate signature.
- update_from(job)[source]
Forwards any internal or control properties from the specified
Job
to this one.
- property tmpdir[source]
Optional temporary directory available for the Job to store files needed for its operation.
It is up to the caller to remove the contents by calling
cleanup()
.
- log(level, message, *args, **kwargs)[source]
Provides the
LoggerHandler
interface, allowing to pass the Job directly as a logger reference.The same parameters as
save_log()
can be provided.
- save_log(*, errors=None, logger=None, message=None, level=INFO, status=None, progress=None, size_limit=None)[source]
Logs the specified error and/or message, and adds the log entry to the complete job log.
For each new log entry, additional
Job
properties are added according toJob._get_log_msg()
and the format defined byget_job_log_msg()
.- Parameters:
errors – An error message or a list of WPS exceptions from which to log and save generated message stack.
logger – An additional
Logger
for which to propagate logged messages on top saving them to the job.message – Explicit string to be logged, otherwise use the current
Job.status_message
is used.level – Logging level to apply to the logged
message
. This parameter is ignored iferrors
are logged.status – Override status applied in the logged message entry, but does not set it to the job object. Uses the current
Job.status
value if not specified. Must be one ofWeaver.status
values.progress – Override progress applied in the logged message entry, but does not set it to the job object. Uses the current
Job.progress
value if not specified.size_limit – Log message entries that individually exceed the limit will be clipped with a generic message. The parameter is provided for convenience, but take note that setting a too large value could cause the complete Job to fail saving to the database if its total size exceeds the document limit.
Note
The job object is updated with the log but still requires to be pushed to database to actually persist it.
- property task_id[source]
Reference Task UUID attributed by the
Celery
worker that monitors and executes this job.
- property wps_id[source]
Reference WPS Request/Response UUID attributed by the executed
PyWPS
process.This UUID matches the status-location, log and output directory of the WPS process. This parameter is only available when the process is executed on this local instance.
See also
- property process[source]
Process identifier of the corresponding remote process.
See also
Process.id
- property type[source]
Obtain the type of the element associated to the creation of this job.
See also
Defined in https://schemas.opengis.net/ogcapi/processes/part1/1.0/openapi/schemas/statusInfo.yaml.
Queried with https://docs.ogc.org/is/18-062r2/18-062r2.html#toc49 (Parameter Type section).
- property statistics[source]
Collected statistics about used memory and processing units if available.
- result_path(job_id=None, output_id=None, file_name=None)[source]
Obtains a relative result path, according to requested parameters and the Job definition.
The generated path will automatically apply the relative job context if defined.
- Parameters:
job_id – Override ID to employ for the job path. Otherwise, uses the usually job UUID by default. This should be used for cases where the ID is “not yet” established by the job, or that an alternate location based on a UUID established by another source must be employed.
output_id – Output ID to refer to in the path. If omitted, the path prefix will stop at the job ID fragment.
file_name – Output file name and extension to apply to the path. If omitted, the path prefix will stop at the output ID.
- Returns:
Resolved relative result path.
- prov_path(container=None, extra_path=None, prov_format=None)[source]
Obtain the relative path of the
PROV
contents.
- prov_data(container=None, extra_path=None, prov_format=None)[source]
Read or retrieve data from the packaged provenance directory contents associated to the Job.
- links(container=None, self_link=None)[source]
Obtains the JSON links section of the response body for a Job.
If
self_link
is provided (e.g.: “outputs”) the link for that corresponding item will also be added as self entry to the links. It must be a recognized job link field.- Parameters:
container – object that helps retrieve instance details, namely the host URL.
self_link – name of a section that represents the current link that will be returned.
- class weaver.datatype.AuthenticationTypes[source]
Generic enumeration.
Derive from this class to define new enumerations.
- class weaver.datatype.Authentication(auth_scheme, auth_token=None, auth_username=None, auth_password=None, auth_link=None, **kwargs)[source]
Authentication details to store details required for process operations.
Initialize self. See help(type(self)) for accurate signature.
- json()[source]
Obtain the JSON data representation for response body.
Note
This method implementation should validate the JSON schema against the API definition whenever applicable to ensure integrity between the represented data type and the expected API response.
- params()[source]
Obtain the internal data representation for storage.
Note
This method implementation should provide a JSON-serializable definition of all fields representing the object to store.
- classmethod from_params(**params)[source]
Obtains the specialized
Authentication
using loaded parameters fromparams()
.
- class weaver.datatype.DockerAuthentication(auth_link, auth_scheme='Basic', auth_token=None, **kwargs)[source]
Authentication associated to a Docker image to retrieve from a private registry given by the reference link.
See also
Initialize the authentication reference for pulling a Docker image from a protected registry.
- Parameters:
auth_link – Fully qualified Docker registry image link (
{registry-url}/{image}:{label}
).auth_scheme – Authentication scheme (Basic, Bearer, etc.) if required.
kwargs – Additional parameters including authentication token, username/password credentials according to specified scheme, and other definitions to load contents already parsed from database.
- property credentials[source]
Generates the credentials to submit the login operation based on the authentication token and scheme.
- property registry[source]
Obtains the registry entry that must be used for
docker login {registry}
.
- class weaver.datatype.VaultFile(file_name='', file_format=None, file_secret=None, auth_token=None, **kwargs)[source]
Dictionary that contains Vault file and its authentication information.
Initialize self. See help(type(self)) for accurate signature.
- classmethod authorized(file, token)[source]
Determine whether the file access is authorized.
This method should be employed to validate access and reduce impact of timing attack analysis.
- encrypt(file)[source]
Encrypt file data using a secret to avoid plain text contents during temporary Vault storage.
Note
This is not intended to be a strong security countermeasure as contents can still be decrypted at any time if provided with the right secret. This is only to slightly obfuscate the contents while it transits between storage phases until destruction by the consuming process.
- property href[source]
Obtain the vault input reference corresponding to the file.
This corresponds to the
href
value to be provided when submitting an input that should be updated using the vault file of specified UUID and using the respective authorization token inX-Auth-Vault
header.
- class weaver.datatype.Process(*args, **kwargs)[source]
Dictionary that contains a process definition for db storage.
It always has
identifier
(orid
alias) and apackage
definition. Parameters can be accessed by key or attribute, and appropriate validators or default values will be applied.Initialize self. See help(type(self)) for accurate signature.
- classmethod split_version(process_id)[source]
Split the tagged version from the Process identifier considering any required special handling.
- Returns:
Process ID (only) and the version if any was available in tagged reference.
- property inputs[source]
Inputs of the process following backward-compatible conversion of stored parameters.
- According to OGC-API,
maxOccurs
andminOccurs
representations should be: maxOccurs
:int
or"unbounded"
minOccurs
:int
- And,
mediaType
should be in description as: mediaType
:string
Note
Because of pre-registered/deployed/retrieved remote processes, inputs are formatted in-line to respect valid OGC-API schema representation and apply any required correction transparently.
- According to OGC-API,
- property outputs[source]
Outputs of the process following backward-compatible conversion of stored parameters.
- According to OGC-API,
mediaType
should be in description as: mediaType
:string
Note
Because of pre-registered/deployed/retrieved remote processes, inputs are formatted in-line to respect valid OGC-API schema representation and apply any required correction transparently.
- According to OGC-API,
- property jobControlOptions[source]
Control options that indicate which Job execution modes are supported by the Process.
Note
There are no official mentions about the ordering of
jobControlOptions
. Nevertheless, it is often expected that the first item can be considered the default mode when none is requested explicitly (at execution time). With the definition of execution mode through thePrefer
header, Weaver has the option to decide if it wants to honor this header, according to available resources and Job duration.For this reason,
async
is placed first by default when nothing was defined during deployment, since it is the preferred mode in Weaver. If deployment included items though, they are preserved as is. This allows to re-deploy a Process to a remote non-Weaver ADES preserving the original Process definition.See also
Discussion about expected ordering of
jobControlOptions
: https://github.com/opengeospatial/ogcapi-processes/issues/171#issuecomment-836819528
- property type[source]
Type of process amongst
weaver.processes.types
definitions.
- params()[source]
Obtain the internal data representation for storage.
Note
This method implementation should provide a JSON-serializable definition of all fields representing the object to store.
- dict()[source]
Generate a dictionary representation of the object, but with inplace resolution of attributes as applicable.
- links(container=None)[source]
Obtains the JSON links section of many response body for the Process.
- Parameters:
container – object that helps retrieve instance details, namely the host URL.
- offering(schema=ProcessSchema.OGC, request=None)[source]
Obtains the JSON or XML serializable offering/description representation of the Process.
- Parameters:
request – HTTP request that can provide more details on how to describe the process.
schema – One of values defined by
sd.ProcessDescriptionSchemaQuery
to select which process description representation to generate (see each schema for details).
Note
Property name
offering
is employed to differentiate from the string processdescription
field. The result of this JSON representation is still theProcessDescription
schema.
- summary(revision=False, links=True, container=None)[source]
Obtains the JSON serializable summary representation of the process.
- Parameters:
revision – Replace the process identifier by the complete tag representation.
links – Include process links in summary.
container – Application settings or database container to retrieve links and avoid reconnections.
- static from_wps(wps_process, **extra_params)[source]
Converts a
pywps
Process into aweaver.datatype.Process
using provided parameters.
- static from_ows(process, service, container, **kwargs)[source]
Converts a
owslib.wps
Process to local storageweaver.datatype.Process
.
- property service[source]
Name of the parent service provider under which this process resides.
See also
- class weaver.datatype.PriceMixin[source]
Base interface for all data-types.
Initialize self. See help(type(self)) for accurate signature.
- class weaver.datatype.Quote(*args, **kwargs)[source]
Dictionary that contains quote information.
It always has
id
andprocess
keys.Initialize the quote.
Note
Although many parameters are required to render the final quote, they are not enforced at creation since the partial quote definition is needed before it can be processed.
- setdefault(key, default)[source]
Insert key with a value of default if key is not in the dictionary.
Return the value for key if key is in the dictionary, else default.
- update(mapping, **__)[source]
D.update([E, ]**F) -> None. Update D from dict/iterable E and F. If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
- property duration[source]
Duration as delta time that can be converted to ISO-8601 format (
P[n]Y[n]M[n]DT[n]H[n]M[n]S
).
- property parameters[source]
Process execution parameters for quote.
This should include minimally the inputs and expected outputs, but could be extended as needed with relevant details for quoting algorithm.
- params()[source]
Obtain the internal data representation for storage.
Note
This method implementation should provide a JSON-serializable definition of all fields representing the object to store.
- class weaver.datatype.Bill(*args, **kwargs)[source]
Dictionary that contains bill information.
It always has
id
,user
,quote
andjob
keys.Initialize self. See help(type(self)) for accurate signature.