ae.deep

easy handling of deeply nested data structures

this ae namespace portion is pure python, depends only on the Python runtime and the ae.base portion, and provides functions for to read, update and delete values of deep data structures. more helper function to prepare and convert data structures between different systems are available in the ae.sys_data module.

the root and node objects of deep data structures consisting of sequences (like list, tuple, …), mappings (dict, …) and data (class) objects. the leaf data objects are mostly simple types like int, float or string.

deep data structure example

the following deep data structure is composed of the data class Person, a member list and two dictionaries:

>>> from dataclasses import dataclass
>>> @dataclass
... class Person:
...     first_name: str
...     hobbies: List[str]
>>> member_hobbies = [
...         "dancing",
...         "music",            # ...
...         ]
>>> member_peter = Person(
...     first_name="Peter",
...     hobbies=member_hobbies,
...     # ...
...     )
>>> member_list = [
...     member_peter,           # ...
...     ]
>>> club_data = {
...     'city': "Madrid",
...     'members': member_list, # ...
...     }
>>> clubs_mapping = {
...     'fun-club': club_data,  # ...
...     }

putting the above data structures together, results in a deep data structure, in where clubs_mapping represents the root object.

the nodes of this deep data structure get referenced by club_data, member_list, member_peter and member_hobbies.

the fields city, first_name and the items 0 and 1 of member_hobbies (referencing the values "dancing" and "music") are finally representing the leafs of this data structure:

>>> clubs_mapping == {
...     'fun-club': {
...         'city': "Madrid",
...         'members': [
...             Person(
...                 first_name="Peter",
...                 hobbies=[
...                     "dancing",
...                     "music",
...                 ]
...             ),
...         ]
...     }
... }
True

referencing a deep data object

there are two types of paths to reference the data items within a deep data structure: object key lists and key path strings.

to get any node object or leaf value within a deep data structure, referenced by a key path string, call the functions key_path_object(), which expects a data structure in its first argument obj and a key path string in its key_path second argument.

in the following example, the function key_path_object() determines the first name object from the member_list data node:

>>> key_path_object(member_list, '0.first_name')
'Peter'

to determine the same object via an object key list, use the function key_list_object():

>>> key_list_object([(member_list, 0),
...                  (member_peter, 'first_name')])
'Peter'

use the function key_path_string() to convert an object key list into a key path string. the following example determines the same first_name data leaf object with an object key list:

>>> key_path = key_path_string([(member_list, 0),
...                             (member_peter, 'first_name')])
>>> print(repr(key_path))
'0.first_name'
>>> key_path_object(member_list, key_path)
'Peter'

e.g. the more deep/complex key path string 'fun-club.members.0.first_name.4', references the 5th character of the leaf object “Peter”, this time from the root node of the example data structure (clubs_mapping):

>>> key_path_object(clubs_mapping, 'fun-club.members.0.first_name.4')
'r'

the same char object, referenced above with a key path string, can also be referenced with an object key list, with the help of the function key_path_string():

>>> key_path_string([
...     (clubs_mapping, 'fun-club'),    # clubs_mapping['fun-club'] == club_data
...     (club_data, 'members'),         # club_data['members'] == member_list
...     (member_list, 0),               # member_list[0] == member_peter
...     (member_peter, 'first_name'),   # member_peter.first_name == "Peter"
...     ("Peter", 4),                   # "Peter"[4] == "r"
...     ])
'fun-club.members.0.first_name.4'

helpers to examine deep data structures

the deep_search() function allows to scan and inspect all the elements of any deep data structure. deep_search() can also be very useful for discovering internals of the Python language/libraries or to debug and test deep and complex data structures.

object_items() is another useful helper function which is returning a list of key-value pairs of any type of data node object.

helpers to change data in deep data structures

use the function replace_object() to change/replace a single node or leaf object within a deep data structure. alternatively you could use key_path_object(), by passing the new value as additional argument to it.

for multiple/bulk changes use the function deep_replace(), which is traversing/scanning the entire data structure.

the function deep_update() merges two deep data structures.

to wipe any node/leaf from a deep data structure use the function pop_object(), which returns the old/removed node, item or attribute value. another option to remove objects from a data structure is to use deep_update() with data:~ae.base.UNSET values in its updating_obj argument.

more details you find in the respective docstring of these functions.

Module Attributes

DataLeafTypesType

list/tuple of types of deep data leaves

KeyType

index/attribute types of deep data structures

KeyFilterCallable

callable to filter item key/index and attribute names

ObjKeysType

object key list of tuples of: object, key/index/attribute

ObjCheckCallable

deep_replace()/deep_search() parameter type

MutableDataTypes

deep data structure node types

DEFAULT_LEAF_TYPES

default leaf types of deep data structures

KEY_PATH_SEPARATORS

separator characters in key path string

Functions

deep_replace(obj, replace_with[, ...])

replace values (bottom up) within the passed (deeply nested) data structure.

deep_search(obj, found[, leaf_types, ...])

search key and/or value within the passed (deeply nested) data object structure.

deep_update(obj, updating_obj[, leaf_types, ...])

merge the updating_obj data structure into the similar structured node object obj.

key_filter_default(index_or_attr)

default key filter callable, returning True to filter out key/index/attribute id/name.

key_list_object(obj_keys)

determine object in a deep nested data structure via an object key list.

key_path_object(obj, key_path[, new_value])

determine object in a deep nested data structure via a key path string, and optionally assign a new value to it.

key_path_string(obj_keys)

convert obj keys path into deep object key path string.

key_value(key_str)

convert key string (mapping key, sequence index, obj attribute) into its value (str, int, float, tuple, ..).

next_key_of_path(key_path)

parse key_path to determine the next item key/index, the path separator and the rest of the key path.

object_item_value(obj, key[, default_value])

determine value of a data object item/attribute.

object_items(obj[, leaf_types, key_filter])

determine the items of a data object, with mapping keys, sequence indexes or attribute names and its values.

pop_object(obj_keys)

delete sub-attribute/item of a data object.

replace_object(obj_keys, new_value)

set sub-attribute/item with a mutable parent object to a new value within a deeply nested object/data structure.

value_filter_default(value, obj_keys)

default item value filter callable, returning True to filter out values to be scanned deeper.

DataLeafTypesType

list/tuple of types of deep data leaves

alias of Tuple[Type, …]

KeyType

index/attribute types of deep data structures

alias of Union[str, int, tuple]

KeyFilterCallable

callable to filter item key/index and attribute names

alias of Callable[[Union[str, int, tuple]], bool]

ObjKeysType

object key list of tuples of: object, key/index/attribute

alias of List[Tuple[Any, Any]]

ObjCheckCallable

deep_replace()/deep_search() parameter type

alias of Callable[[List[Tuple[Any, Any]], Any, Any], Any]

MutableDataTypes = (<class 'collections.abc.MutableMapping'>, <class 'collections.abc.MutableSequence'>)

deep data structure node types

DEFAULT_LEAF_TYPES = (<class 'bytes'>, <class 'int'>, <class 'float'>, <class 'set'>, <class 'str'>, <class 'type'>)

default leaf types of deep data structures

KEY_PATH_SEPARATORS = ('.', '[', ']')

separator characters in key path string

key_filter_default(index_or_attr)[source]

default key filter callable, returning True to filter out key/index/attribute id/name.

this function is the default value for the key_filter parameter of the deep scanning and traversing functions deep_replace(), deep_search(), deep_update() and object_items().

if you are using your own filter callable you could call this function from it to prevent endless recursion, especially if your deep data structures contains circular- or self-referencing objects, like e.g.: :rtype: bool

  • self- or doubly-linked data structures (e.g. kivy.app.App.proxy_ref (self-linked) or ae.kivy.apps.KivyMainApp.framework_app and ae.kivy.apps.FrameworkApp.main_app).

  • back-linked data structures, like e.g. the parent property in Kivy widget trees.

  • magic (two leading underscore characters) or internal (one leading underscore) attributes (e.g. via WindowSDL._WindowBase__instance).

value_filter_default(value, obj_keys)[source]

default item value filter callable, returning True to filter out values to be scanned deeper.

this function is the default value for the value_filter parameter of the deep scanning and traversing functions deep_replace(), deep_search() and deep_update().

by using your own filter callable make sure to prevent endless processing or recursion, especially if your deep data structures contains circular- or self-referencing objects. e.g. some data value types have to be excluded from to be deeper processed to prevent RecursionError (endless recursion, e.g. on str values because str[i] is str, on int because int.denominator is int).

Return type:

bool

deep_replace(obj, replace_with, leaf_types=(<class 'bytes'>, <class 'int'>, <class 'float'>, <class 'set'>, <class 'str'>, <class 'type'>), key_filter=<function key_filter_default>, value_filter=<function value_filter_default>, obj_keys=None)[source]

replace values (bottom up) within the passed (deeply nested) data structure.

Parameters:
  • obj (Any) – mutable sequence or mapping data structure to be deep searched and replaced. can contain any combination of deep nested data objects. mutable node objects (e.g. dict/list) as well as the immutable types not included in leaf_types will be recursively deep searched (top down) by passing their items one by one to the callback function specified by replace_with.

  • replace_with (Callable[[List[Tuple[Any, Any]], Any, Any], Any]) – called for each item with the 3 arguments object key list, key in parent data-structure, and the object/value. any return value other than UNSET will be used to overwrite the node/leaf object in the data-structure.

  • leaf_types (Tuple[Type, ...]) – tuple of leaf types to skip from to be searched deeper. the default value of this parameter is specified in the modul constant DEFAULT_LEAF_TYPES.

  • key_filter (Callable[[Union[str, int, tuple]], bool]) – called for each sub-item/-attribute of the data structure specified by obj. return True for item-key/item-index/attribute-name to be filtered out. by default all attribute/key names starting with an underscore character will be filtered out (see default callable key_filter_default()).

  • value_filter (Callable[[Any, List[Tuple[Any, Any]]], bool]) – called for each sub-item/-attribute of the data structure specified by obj. return True for items/attributes values to be filtered out. by default empty values, excluded values (see EXCLUDED_VALUE_TYPES) and already scanned objects will be filtered out (see default callable value_filter_default()).

  • obj_keys (Optional[List[Tuple[Any, Any]]]) – used (internally only) to pass the parent data-struct path in recursive calls.

Return type:

int

Returns:

the number of levels of the first mutable data objects above the changed data object, or 0 of the changed data object is mutable.

Raises:

ValueError if no mutable parent object is in data structure (obj).

Raises:

AttributeError e.g. if obj is of type int, and int is missing in leaf_types.

Note

make sure to prevent overwrites on internal objects of the Python runtime, on some of them the Python interpreter could even crash (e.g. with: exit code 134 (interrupted by signal 6: SIGABRT)).

search key and/or value within the passed (deeply nested) data object structure.

Parameters:
  • obj (Any) –

    root object to start the top-down deep search from, which can contain any combination of deep nested elements/objects. for each sub-element the callable passed into found will be executed. if the callable returns True then the data path, the key and the value will be stored in a tuple and added to the search result list (finally returned to the caller of this function).

    for iterable objects of type dict/tuple/list, the sub-items will be searched, as well as the attributes determined via the Python dir() function. to reduce the number of items/attributes to be searched use the parameters leaf_types and/or key_filter.

  • found (Callable[[List[Tuple[Any, Any]], Any, Any], Any]) – called for each item with 3 arguments (data-struct-path, key in data-structure, value), and if the return value is True then the data/object path, the last key and value will be added as a new item to the returned list.

  • leaf_types (Tuple[Type, ...]) – tuple of leaf types to skip from to be searched deeper. the default value of this parameter is specified in the modul constant DEFAULT_LEAF_TYPES.

  • key_filter (Callable[[Union[str, int, tuple]], bool]) – called for each sub-item/-attribute of the data structure specified by obj. return True for item-key/item-index/attribute-name to be filtered out. by default all attribute/key names starting with an underscore character will be filtered out (see default callable key_filter_default()).

  • value_filter (Callable[[Any, List[Tuple[Any, Any]]], bool]) – called for each sub-item/-attribute of the data structure specified by obj. return True for items/attributes values to be filtered out. by default empty values, excluded values (see EXCLUDED_VALUE_TYPES) and already scanned objects will be filtered out (see default callable value_filter_default()).

  • obj_keys (Optional[List[Tuple[Any, Any]]]) – used (internally only) to pass the parent data-struct path in recursive calls.

Return type:

List[Tuple[List[Tuple[Any, Any]], Any, Any]]

Returns:

list of tuples (data-struct-path, key, value); one tuple for each found item within the passed obj argument. an empty list will be returned if no item got found.

deep_update(obj, updating_obj, leaf_types=(<class 'bytes'>, <class 'int'>, <class 'float'>, <class 'set'>, <class 'str'>, <class 'type'>), key_filter=<function key_filter_default>, value_filter=<function value_filter_default>, obj_keys=None)[source]

merge the updating_obj data structure into the similar structured node object obj.

Parameters:
  • obj (Any) – deep data object to update.

  • updating_obj (Any) – data structure similar structured like the obj argument with update values. a UNSET value will delete the item from the obj argument.

  • leaf_types (Tuple[Type, ...]) – tuple of leaf types to skip from to be searched deeper. the default value of this parameter is specified in the modul constant DEFAULT_LEAF_TYPES.

  • key_filter (Callable[[Union[str, int, tuple]], bool]) – called for each sub-item/-attribute of the data structure specified by obj. return True for item-key/item-index/attribute-name to be filtered out. by default all attribute/key names starting with an underscore character will be filtered out (see default callable key_filter_default()).

  • value_filter (Callable[[Any, List[Tuple[Any, Any]]], bool]) – called for each sub-item/-attribute of the data structure specified by obj. return True for items/attributes values to be filtered out. by default empty values, excluded values (see EXCLUDED_VALUE_TYPES) and already scanned objects will be filtered out (see default callable value_filter_default()).

  • obj_keys (Optional[List[Tuple[Any, Any]]]) – used (internally only) to pass the parent data-struct path in recursive calls.

key_list_object(obj_keys)[source]

determine object in a deep nested data structure via an object key list.

Parameters:

obj_keys (List[Tuple[Any, Any]]) – object key list.

Return type:

Any

Returns:

recalculated object referenced by the first object and the keys of obj_keys or ~ae.base.UNSET if not found.

Raises:

TypeError if key does not match the object type in any item of obj_keys. ValueError if obj_keys is not of type ObjKeysType.

Hint

to include changes on immutable data structures, the returned object value gets recalculated, starting from the first object (obj_keys`[0][0]), going deeper via the key only (while ignoring all other child objects in the object key list specified by :paramref:`obj_keys).

key_path_object(obj, key_path, new_value=<ae.base.UnsetType object>)[source]

determine object in a deep nested data structure via a key path string, and optionally assign a new value to it.

Parameters:
  • obj (Any) – initial data object to search in (and its sub-objects).

  • key_path (str) –

    composed key string containing dict keys, tuple/list/str indexes and object attribute names, separated by a dot character, like shown in the following examples:

    >>> class AClass:
    ...     str_attr_name_a = "a_attr_val"
    ...     dict_attr = {'a_str_key': 3, 999: "value_with_int_key", '999': "..str_key"}
    
    >>> class BClass:
    ...     str_attr_name_b = "b_b_b_b_b"
    ...     b_obj = AClass()
    
    >>> b = BClass()
    
    >>> assert key_path_object(b, 'str_attr_name_b') == "b_b_b_b_b"
    >>> assert key_path_object(b, 'b_obj.str_attr_name_a') == "a_attr_val"
    >>> assert key_path_object(b, 'b_obj.str_attr_name_a.5') == "r"  # 6th chr of a_attr_val
    >>> assert key_path_object(b, 'b_obj.dict_attr.a_str_key') == 3
    

    the item key or index value of lists and dictionaries can alternatively be specified in Python syntax, enclosed in [ and ]:

    >>> assert key_path_object(b, 'b_obj.dict_attr["a_str_key"]') == 3
    >>> assert key_path_object(b, 'b_obj.dict_attr[\'a_str_key\']') == 3
    >>> assert key_path_object(b, 'b_obj.dict_attr[999]') == "value_with_int_key"
    >>> assert key_path_object(b, 'b_obj.dict_attr["999"]') == "..str_key"
    

    only dict key strings that are not can be misinterpreted as number can be specified without the high commas (enclosing the key string), like e.g.:

    >>> assert key_path_object(b, 'b_obj.dict_attr[a_str_key]') == 3
    

  • new_value (Optional[Any]) –

    optional new value - replacing the found object. the old object value will be returned.

    Note

    immutable objects, like tuples, that are embedding in obj will be automatically updated/replaced up in the data tree structure until a mutable object (list, dict or object) get found.

Return type:

Any

Returns:

specified object/value (the old value if new_value got passed) or UNSET if not found/exists (key path string is invalid).

key_path_string(obj_keys)[source]

convert obj keys path into deep object key path string.

Parameters:

obj_keys (List[Tuple[Any, Any]]) – object key list to convert.

Return type:

str

Returns:

key path string of the object keys path specified by the obj_keys argument.

key_value(key_str)[source]

convert key string (mapping key, sequence index, obj attribute) into its value (str, int, float, tuple, ..).

Return type:

Any

next_key_of_path(key_path)[source]

parse key_path to determine the next item key/index, the path separator and the rest of the key path.

Parameters:

key_path (str) – data object key/index path string to parse.

Return type:

Tuple[Any, str, str]

Returns:

tuple of key/index, the separator character and the (unparsed) rest of the key path (possibly an empty string).

Raises:

IndexError if the argument of key_path is an empty string.

object_items(obj, leaf_types=(<class 'bytes'>, <class 'int'>, <class 'float'>, <class 'set'>, <class 'str'>, <class 'type'>), key_filter=<function key_filter_default>)[source]

determine the items of a data object, with mapping keys, sequence indexes or attribute names and its values.

Parameters:
  • obj (Any) – data structure/node object (list, dict, set, tuple, data object, …).

  • leaf_types (Tuple[Type, ...]) – tuple of leaf types to skip from to be searched deeper. the default value of this parameter is specified in the modul constant DEFAULT_LEAF_TYPES.

  • key_filter (Callable[[Union[str, int, tuple]], bool]) – called for each sub-item/-attribute of the data structure specified by obj. return True for item-key/item-index/attribute-name to be filtered out. by default all attribute/key names starting with an underscore character will be filtered out (see default callable key_filter_default()).

Return type:

List[Tuple[Any, Any]]

Returns:

items view of the data object specified in the obj argument.

object_item_value(obj, key, default_value=<ae.base.UnsetType object>)[source]

determine value of a data object item/attribute.

Parameters:
  • obj (Any) – data structure object to get item/attribute value from.

  • key (Any) – mapping key, attribute name or sequence index of the item/attribute.

  • default_value (Any) – default value to return if the item/attribute does not exist in obj.

Return type:

Any

Returns:

data object item/attribute value or the default_value if not found.

pop_object(obj_keys)[source]

delete sub-attribute/item of a data object.

Parameters:

obj_keys (List[Tuple[Any, Any]]) –

list of (object, key) tuples identifying an element within a deeply nested data structure or object hierarchy. the root of the data/object structure is the object at list index 0 and the element to be deleted is identified by the object and key in the last list item of this argument.

the referenced data structure can contain even immutable node objects (like tuples) which will be accordingly changed/replaced if affected/needed. for that at least one object/element above the immutable object in the deep data structure has to be mutable, else a ValueError will be raised.

Return type:

List[Any]

Returns:

list of deleted values or UNSET if not found.

Raises:

ValueError if no immutable parent object got found or if the obj_keys' is empty. IndexError if the one of the specified indexes in :paramref:`obj_keys does not exist.

replace_object(obj_keys, new_value)[source]

set sub-attribute/item with a mutable parent object to a new value within a deeply nested object/data structure.

Parameters:
  • obj_keys (List[Tuple[Any, Any]]) –

    list of (object, key) tuples identifying an element within a deeply nested data structure or object hierarchy. the root of the data/object structure is the object at list index 0 and the element to be changed is identified by the object and key in the last list item of this argument.

    the referenced data structure can contain immutable data node objects (like tuples) which will be accordingly changed/replaced if affected/needed.

    at least one object/element, situated above of replaced data object within the deep data structure, has to be mutable, else a ValueError will be raised.

  • new_value (Any) – value to be assigned to the element referenced by the last list item of the argument in obj_keys.

Return type:

int

Returns:

the number of levels of the first mutable data objects above the changed data object, or 0 of the changed data object is mutable.

Raises:

ValueError if no immutable parent object got found or if the object key list in the obj_keys argument is empty.