:mod:`weaver.utils` =================== .. py:module:: weaver.utils Module Contents --------------- .. data:: LOGGER .. py:class:: _Singleton Bases: :class:`type` .. py:class:: _NullType Bases: :class:`six.with_metaclass()` Represents a ``null`` value to differentiate from ``None``. .. data:: null .. function:: get_weaver_url(container) -> AnyStr Retrieves the home URL of the `weaver` application. .. function:: get_any_id(info) -> Union[AnyStr, None] Retrieves a dictionary `id-like` key using multiple common variations ``[id, identifier, _id]``. :param info: dictionary that potentially contains an `id-like` key. :returns: value of the matched `id-like` key or ``None`` if not found. .. function:: get_any_value(info) -> AnyValue Retrieves a dictionary `value-like` key using multiple common variations ``[href, value, reference]``. :param info: dictionary that potentially contains a `value-like` key. :returns: value of the matched `value-like` key or ``None`` if not found. .. function:: get_any_message(info) -> AnyStr Retrieves a dictionary 'value'-like key using multiple common variations [message]. :param info: dictionary that potentially contains a 'message'-like key. :returns: value of the matched 'message'-like key or an empty string if not found. .. function:: get_registry(container) -> Registry Retrieves the application ``registry`` from various containers referencing to it. .. function:: get_settings(container) -> SettingsType Retrieves the application ``settings`` from various containers referencing to it. .. function:: get_header(header_name, header_container) -> Union[AnyStr, None] Searches for the specified header by case/dash/underscore-insensitive ``header_name`` inside ``header_container``. .. function:: get_cookie_headers(header_container, cookie_header_name='Cookie') -> HeadersType Looks for ``cookie_header_name`` header within ``header_container``. :returns: new header container in the form ``{'Cookie': }`` if it was matched, or empty otherwise. .. function:: get_url_without_query(url) -> AnyStr Removes the query string part of an URL. .. function:: is_valid_url(url) -> bool .. function:: parse_extra_options(option_str) Parses the extra options parameter. The option_str is a string with coma separated ``opt=value`` pairs. Example:: tempdir=/path/to/tempdir,archive_root=/path/to/archive :param option_str: A string parameter with the extra options. :return: A dict with the parsed extra options. .. function:: fully_qualified_name(obj) -> str Obtains the ``'.'`` full path definition of the object to allow finding and importing it. .. function:: now() -> datetime .. function:: now_secs() -> int Return the current time in seconds since the Epoch. .. function:: wait_secs(run_step=-1) .. function:: expires_at(hours=1) -> int .. function:: localize_datetime(dt, tz_name='UTC') -> datetime Provide a timezone-aware object for a given datetime and timezone name .. function:: get_base_url(url) -> AnyStr Obtains the base URL from the given ``url``. .. function:: xml_path_elements(path) -> List[AnyStr] .. function:: xml_strip_ns(tree) -> None .. function:: ows_context_href(href, partial=False) -> JSON Returns the complete or partial dictionary defining an ``OWSContext`` from a reference. .. function:: pass_http_error(exception, expected_http_error) -> None Given an `HTTPError` of any type (pyramid, requests), ignores (pass) the exception if the actual error matches the status code. Other exceptions are re-raised. :param exception: any `Exception` instance ("object" from a `try..except exception as "object"` block). :param expected_http_error: single or list of specific pyramid `HTTPError` to handle and ignore. :raise exception: if it doesn't match the status code or is not an `HTTPError` of any module. .. function:: raise_on_xml_exception(xml_node) Raises an exception with the description if the XML response document defines an ExceptionReport. :param xml_node: instance of :class:`etree.Element` :raise Exception: on found ExceptionReport document. .. function:: str2bytes(string) -> bytes Obtains the bytes representation of the string. .. function:: bytes2str(string) -> str Obtains the unicode representation of the string. .. function:: islambda(func) -> bool .. data:: first_cap_re .. data:: all_cap_re .. function:: convert_snake_case(name) -> AnyStr .. function:: parse_request_query(request) -> Dict[AnyStr, Dict[AnyKey, AnyStr]] :param request: :return: dict of dict where k=v are accessible by d[k][0] == v and q=k=v are accessible by d[q][k] == v, lowercase .. function:: get_log_fmt() -> AnyStr .. function:: get_log_date_fmt() -> AnyStr .. function:: get_log_monitor_msg(job_id, status, percent, message, location) -> AnyStr .. function:: get_job_log_msg(status, message, progress=0, duration=None) -> AnyStr .. function:: make_dirs(path, mode=493, exist_ok=False) Alternative to ``os.makedirs`` with ``exists_ok`` parameter only available for ``python>3.5``. Also using a reduced set of permissions ``755`` instead of original default ``777``. .. note:: The method employed in this function is safer then ``if os.pat.exists`` or ``if os.pat.isdir`` pre-check to calling ``os.makedirs`` as this can result in race condition (between evaluation and actual creation). .. function:: get_caller_name(skip=2, base_class=False) Returns the name of a caller in the format ``module.class.method``. :param skip: specifies how many levels of stack to skip while getting the caller. :param base_class: Specified if the base class should be returned or the top-most class in case of inheritance If the caller is not a class, this doesn't do anything. :returns: An empty string if skipped levels exceed stack height; otherwise, the requested caller name. .. function:: get_ssl_verify_option(method, url, settings, request_options=None) -> bool Obtains the SSL verification option from combined settings from ``weaver.ssl_verify`` and parsed ``weaver.request_options`` file for the corresponding request. :param method: request method (GET, POST, etc.). :param url: request URL. :param settings: application setting container with pre-loaded *request options* specifications. :param request_options: pre-processed *request options* for method/URL to avoid re-parsing the settings. :returns: SSL ``verify`` option to be passed down to some ``request`` function. .. function:: get_request_options(method, url, settings) -> SettingsType Obtains the *request options* corresponding to the request according to configuration file specified by pre-loaded setting ``weaver.request_options``. If no file was pre-loaded or no match is found for the request, an empty options dictionary is returned. .. seealso:: - :func:`get_ssl_verify_option` - `config/request_options.yml.example <../../config/config/request_options.yml.example>`_ :param method: request method (GET, POST, etc.). :param url: request URL. :param settings: application setting container with pre-loaded *request options* specifications. :returns: dictionary with keyword options to be applied to the corresponding request if matched. .. function:: request_extra(method, url, retries=None, backoff=None, intervals=None, retry_after=True, allowed_codes=None, only_server_errors=True, ssl_verify=None, settings=None, **request_kwargs) -> AnyResponseType Standard library :mod:`requests` with additional functional utilities. Retry operation --------------- Implements request retry if the previous request failed, up to the specified number of retries. Using :paramref:`backoff` factor, you can control the interval between request attempts such as:: delay = backoff * (2 ^ retry) Alternatively, you can explicitly define ``intervals=[...]`` with the list values being the number of seconds to wait between each request attempt. In this case, :paramref:`backoff` is ignored and :paramref:`retries` is overridden accordingly with the number of items specified in the list. Furthermore, :paramref:`retry_after` (default: ``True``) indicates if HTTP status code ``429 (Too Many Requests)`` should be automatically handled during retries. If enabled and provided in the previously failed request response through the ``Retry-After`` header, the next request attempt will be executed only after the server-specified delay instead of following the calculated delay from :paramref:`retries` and :paramref:`backoff`, or from corresponding index of :paramref:`interval`, accordingly to specified parameters. This will avoid uselessly calling the server and automatically receive a denied response. You can disable this feature by passing ``False``, which will result into requests being retried blindly without consideration of the called server instruction. Because different request implementations use different parameter naming conventions, all following keywords are looked for: - Both variants of ``backoff`` and ``backoff_factor`` are accepted. - All variants of ``retires``, ``retry`` and ``max_retries`` are accepted. .. note:: Total amount of executed request attempts will be +1 the number of :paramref:`retries` or :paramref:`intervals` items as first request is done immediately, and following attempts are done with the appropriate delay. File Transport Scheme --------------------- Any request with ``file://`` scheme or empty scheme (no scheme specified) will be automatically handled as potential local file path. The path should be absolute to ensure it to be correctly resolved. All access errors due to file permissions return 403 status code, and missing file returns 404. Any other :py:exc:`IOError` types are converted to a 400 responses. .. seealso:: - :class:`FileAdapter` SSL Verification ---------------- Allows SSL verify option to be enabled or disabled according to configuration settings or explicit parameters. Any variation of ``verify`` or ``ssl_verify`` keyword arguments are considered. If they all resolve to ``True``, then application settings are retrieved from ``weaver.ini`` to parse additional SSL options that could disable it. Following :mod:`weaver` settings are considered : - `weaver.ssl_verify = True|False` - `weaver.request_options = request_options.yml` .. note:: Argument :paramref:`settings` must also be provided through any supported container by :func:`get_settings` to retrieve and apply any :mod:`weaver`-specific configurations. .. seealso:: - :func:`get_request_options` - :func:`get_ssl_verify_option` :param method: HTTP method to set request. :param url: URL of the request to execute. :param retries: Number of request retries to attempt if first attempt failed (according to allowed codes or error). :param backoff: Factor by which to multiply delays between retries. :param intervals: Explicit intervals in seconds between retries. :param retry_after: If enabled, honor ``Retry-After`` response header of provided by a failing request attempt. :param allowed_codes: HTTP status codes that are considered valid to stop retrying (default: any non-4xx/5xx code). :param ssl_verify: Explicit parameter to disable SSL verification (overrides any settings, default: True). :param settings: Additional settings from which to retrieve configuration details for requests. :param only_server_errors: Only HTTP status codes in the 5xx values will be considered for retrying the request (default: True). This catches sporadic server timeout, connection error, etc., but 4xx errors are still considered valid results. This parameter is ignored if allowed codes are explicitly specified. :param request_kwargs: All other keyword arguments are passed down to the request call. .. function:: fetch_file(file_reference, file_outdir, settings=None, **request_kwargs) -> AnyStr Fetches a file from a local path or remote URL and dumps it's content to the specified output directory. The output directory is expected to exist prior to this function call. :param file_reference: Local filesystem path or remote URL file reference. :param file_outdir: Output directory path of the fetched file. :param settings: Additional request setting details from the application configuration. :param request_kwargs: Additional keywords to forward to request call (if needed). :return: Path of the local copy of the fetched file. .. data:: REGEX_SEARCH_INVALID_CHARACTERS .. data:: REGEX_ASSERT_INVALID_CHARACTERS .. function:: get_sane_name(name, min_len=3, max_len=None, assert_invalid=True, replace_character='_') -> Union[AnyStr, None] Returns a cleaned-up version of the input name, replacing invalid characters matched with ``REGEX_SEARCH_INVALID_CHARACTERS`` by ``replace_character``. :param name: value to clean :param min_len: Minimal length of ``name`` to be respected, raises or returns ``None`` on fail according to ``assert_invalid``. :param max_len: Maximum length of ``name`` to be respected, raises or returns trimmed ``name`` on fail according to ``assert_invalid``. If ``None``, condition is ignored for assertion or full ``name`` is returned respectively. :param assert_invalid: If ``True``, fail conditions or invalid characters will raise an error instead of replacing. :param replace_character: Single character to use for replacement of invalid ones if ``assert_invalid=False``. .. function:: assert_sane_name(name, min_len=3, max_len=None) Asserts that the sane name respects conditions. .. seealso:: - argument details in :func:`get_sane_name` .. function:: clean_json_text_body(body) -> AnyStr Cleans a textual body field of superfluous characters to provide a better human-readable text in a JSON response.