ae.deep
easy handling of deeply nested data structures
this ae
namespace portion is pure python, depends only on the Python runtime and the ae.base
portion,
and provides functions for to read, update and delete values of deep data structures. more helper function to
prepare and convert data structures between different systems are available in the ae.sys_data
module.
the root and node objects of deep data structures consisting of sequences (like list, tuple, …), mappings (dict, …) and data (class) objects. the leaf data objects are mostly simple types like int, float or string.
deep data structure example
the following deep data structure is composed of the data class Person
, a member list and two dictionaries:
>>> from dataclasses import dataclass
>>> @dataclass
... class Person:
... first_name: str
... hobbies: List[str]
>>> member_hobbies = [
... "dancing",
... "music", # ...
... ]
>>> member_peter = Person(
... first_name="Peter",
... hobbies=member_hobbies,
... # ...
... )
>>> member_list = [
... member_peter, # ...
... ]
>>> club_data = {
... 'city': "Madrid",
... 'members': member_list, # ...
... }
>>> clubs_mapping = {
... 'fun-club': club_data, # ...
... }
putting the above data structures together, results in a deep data structure, in where clubs_mapping
represents
the root object.
the nodes of this deep data structure get referenced by club_data
, member_list
, member_peter
and
member_hobbies
.
the fields city
, first_name
and the items 0
and 1
of member_hobbies
(referencing the values
"dancing"
and "music"
) are finally representing the leafs of this data structure:
>>> clubs_mapping == {
... 'fun-club': {
... 'city': "Madrid",
... 'members': [
... Person(
... first_name="Peter",
... hobbies=[
... "dancing",
... "music",
... ]
... ),
... ]
... }
... }
True
referencing a deep data object
there are two types of paths to reference the data items within a deep data structure:
object key lists
and key path strings
.
to get any node object or leaf value within a deep data structure, referenced by a key path string, call the functions
key_path_object()
, which expects a data structure in its first argument obj
and a key path string in its key_path
second argument.
in the following example, the function key_path_object()
determines the first name object from the
member_list
data node:
>>> key_path_object(member_list, '0.first_name')
'Peter'
to determine the same object via an object key list, use the function key_list_object()
:
>>> key_list_object([(member_list, 0),
... (member_peter, 'first_name')])
'Peter'
use the function key_path_string()
to convert an object key list into a key path string. the following example
determines the same first_name
data leaf object with an object key list:
>>> key_path = key_path_string([(member_list, 0),
... (member_peter, 'first_name')])
>>> print(repr(key_path))
'0.first_name'
>>> key_path_object(member_list, key_path)
'Peter'
e.g. the more deep/complex key path string 'fun-club.members.0.first_name.4'
, references the
5th character of the leaf object “Peter”, this time from the root node of the example
data structure (clubs_mapping
):
>>> key_path_object(clubs_mapping, 'fun-club.members.0.first_name.4')
'r'
the same char object, referenced above with a key path string, can also be referenced with an object key list, with
the help of the function key_path_string()
:
>>> key_path_string([
... (clubs_mapping, 'fun-club'), # clubs_mapping['fun-club'] == club_data
... (club_data, 'members'), # club_data['members'] == member_list
... (member_list, 0), # member_list[0] == member_peter
... (member_peter, 'first_name'), # member_peter.first_name == "Peter"
... ("Peter", 4), # "Peter"[4] == "r"
... ])
'fun-club.members.0.first_name.4'
helpers to examine deep data structures
the deep_search()
function allows to scan and inspect all the elements of any deep data structure.
deep_search()
can also be very useful for discovering internals of the Python language/libraries or to debug
and test deep and complex data structures.
object_items()
is another useful helper function which is returning a list of key-value pairs of any type
of data node object.
helpers to change data in deep data structures
use the function replace_object()
to change/replace a single node or leaf object within a deep data structure.
alternatively you could use key_path_object()
, by passing the new value as additional argument to it.
for multiple/bulk changes use the function deep_replace()
, which is traversing/scanning the entire data structure.
the function deep_update()
merges two deep data structures.
to wipe any node/leaf from a deep data structure use the function pop_object()
, which returns the old/removed
node, item or attribute value. another option to remove objects from a data structure is to use deep_update()
with data:~ae.base.UNSET values in its updating_obj
argument.
more details you find in the respective docstring of these functions.
Module Attributes
list/tuple of types of deep data leaves |
|
index/attribute types of deep data structures |
|
callable to filter item key/index and attribute names |
|
object key list of tuples of: object, key/index/attribute |
|
|
|
deep data structure node types |
|
default leaf types of deep data structures |
|
separator characters in key path string |
Functions
|
replace values (bottom up) within the passed (deeply nested) data structure. |
|
search key and/or value within the passed (deeply nested) data object structure. |
|
merge the |
|
default key filter callable, returning True to filter out key/index/attribute id/name. |
|
determine object in a deep nested data structure via an object key list. |
|
determine object in a deep nested data structure via a key path string, and optionally assign a new value to it. |
|
convert obj keys path into deep object key path string. |
|
convert key string (mapping key, sequence index, obj attribute) into its value (str, int, float, tuple, ..). |
|
parse key_path to determine the next item key/index, the path separator and the rest of the key path. |
|
determine value of a data object item/attribute. |
|
determine the items of a data object, with mapping keys, sequence indexes or attribute names and its values. |
|
delete sub-attribute/item of a data object. |
|
set sub-attribute/item with a mutable parent object to a new value within a deeply nested object/data structure. |
|
default item value filter callable, returning True to filter out values to be scanned deeper. |
- KeyFilterCallable
callable to filter item key/index and attribute names
- ObjKeysType
object key list of tuples of: object, key/index/attribute
- ObjCheckCallable
deep_replace()
/deep_search()
parameter type
- MutableDataTypes = (<class 'collections.abc.MutableMapping'>, <class 'collections.abc.MutableSequence'>)
deep data structure node types
- DEFAULT_LEAF_TYPES = (<class 'bytes'>, <class 'int'>, <class 'float'>, <class 'set'>, <class 'str'>, <class 'type'>)
default leaf types of deep data structures
- KEY_PATH_SEPARATORS = ('.', '[', ']')
separator characters in key path string
- key_filter_default(index_or_attr)[source]
default key filter callable, returning True to filter out key/index/attribute id/name.
this function is the default value for the
key_filter
parameter of the deep scanning and traversing functionsdeep_replace()
,deep_search()
,deep_update()
andobject_items()
.if you are using your own filter callable you could call this function from it to prevent endless recursion, especially if your deep data structures contains circular- or self-referencing objects, like e.g.: :rtype:
bool
self- or doubly-linked data structures (e.g.
kivy.app.App.proxy_ref
(self-linked) orae.kivy.apps.KivyMainApp.framework_app
andae.kivy.apps.FrameworkApp.main_app
).back-linked data structures, like e.g. the
parent
property in Kivy widget trees.magic (two leading underscore characters) or internal (one leading underscore) attributes (e.g. via WindowSDL._WindowBase__instance).
- value_filter_default(value, obj_keys)[source]
default item value filter callable, returning True to filter out values to be scanned deeper.
this function is the default value for the
value_filter
parameter of the deep scanning and traversing functionsdeep_replace()
,deep_search()
anddeep_update()
.by using your own filter callable make sure to prevent endless processing or recursion, especially if your deep data structures contains circular- or self-referencing objects. e.g. some data value types have to be excluded from to be deeper processed to prevent RecursionError (endless recursion, e.g. on str values because str[i] is str, on int because int.denominator is int).
- Return type:
- deep_replace(obj, replace_with, leaf_types=(<class 'bytes'>, <class 'int'>, <class 'float'>, <class 'set'>, <class 'str'>, <class 'type'>), key_filter=<function key_filter_default>, value_filter=<function value_filter_default>, obj_keys=None)[source]
replace values (bottom up) within the passed (deeply nested) data structure.
- Parameters:
obj¶ (
Any
) – mutable sequence or mapping data structure to be deep searched and replaced. can contain any combination of deep nested data objects. mutable node objects (e.g. dict/list) as well as the immutable types not included inleaf_types
will be recursively deep searched (top down) by passing their items one by one to the callback function specified byreplace_with
.replace_with¶ (
Callable
[[List
[Tuple
[Any
,Any
]],Any
,Any
],Any
]) – called for each item with the 3 arguments object key list, key in parent data-structure, and the object/value. any return value other thanUNSET
will be used to overwrite the node/leaf object in the data-structure.leaf_types¶ (
Tuple
[Type
,...
]) – tuple of leaf types to skip from to be searched deeper. the default value of this parameter is specified in the modul constantDEFAULT_LEAF_TYPES
.key_filter¶ (
Callable
[[Union
[str
,int
,tuple
]],bool
]) – called for each sub-item/-attribute of the data structure specified byobj
. return True for item-key/item-index/attribute-name to be filtered out. by default all attribute/key names starting with an underscore character will be filtered out (see default callablekey_filter_default()
).value_filter¶ (
Callable
[[Any
,List
[Tuple
[Any
,Any
]]],bool
]) – called for each sub-item/-attribute of the data structure specified byobj
. return True for items/attributes values to be filtered out. by default empty values, excluded values (seeEXCLUDED_VALUE_TYPES
) and already scanned objects will be filtered out (see default callablevalue_filter_default()
).obj_keys¶ (
Optional
[List
[Tuple
[Any
,Any
]]]) – used (internally only) to pass the parent data-struct path in recursive calls.
- Return type:
- Returns:
the number of levels of the first mutable data objects above the changed data object, or 0 of the changed data object is mutable.
- Raises:
ValueError if no mutable parent object is in data structure (
obj
).- Raises:
AttributeError e.g. if
obj
is of type int, and int is missing inleaf_types
.
Note
make sure to prevent overwrites on internal objects of the Python runtime, on some of them the Python interpreter could even crash (e.g. with: exit code 134 (interrupted by signal 6: SIGABRT)).
- deep_search(obj, found, leaf_types=(<class 'bytes'>, <class 'int'>, <class 'float'>, <class 'set'>, <class 'str'>, <class 'type'>), key_filter=<function key_filter_default>, value_filter=<function value_filter_default>, obj_keys=None)[source]
search key and/or value within the passed (deeply nested) data object structure.
- Parameters:
root object to start the top-down deep search from, which can contain any combination of deep nested elements/objects. for each sub-element the callable passed into
found
will be executed. if the callable returnsTrue
then the data path, the key and the value will be stored in a tuple and added to the search result list (finally returned to the caller of this function).for iterable objects of type dict/tuple/list, the sub-items will be searched, as well as the attributes determined via the Python
dir()
function. to reduce the number of items/attributes to be searched use the parametersleaf_types
and/orkey_filter
.found¶ (
Callable
[[List
[Tuple
[Any
,Any
]],Any
,Any
],Any
]) – called for each item with 3 arguments (data-struct-path, key in data-structure, value), and if the return value isTrue
then the data/object path, the last key and value will be added as a new item to the returned list.leaf_types¶ (
Tuple
[Type
,...
]) – tuple of leaf types to skip from to be searched deeper. the default value of this parameter is specified in the modul constantDEFAULT_LEAF_TYPES
.key_filter¶ (
Callable
[[Union
[str
,int
,tuple
]],bool
]) – called for each sub-item/-attribute of the data structure specified byobj
. return True for item-key/item-index/attribute-name to be filtered out. by default all attribute/key names starting with an underscore character will be filtered out (see default callablekey_filter_default()
).value_filter¶ (
Callable
[[Any
,List
[Tuple
[Any
,Any
]]],bool
]) – called for each sub-item/-attribute of the data structure specified byobj
. return True for items/attributes values to be filtered out. by default empty values, excluded values (seeEXCLUDED_VALUE_TYPES
) and already scanned objects will be filtered out (see default callablevalue_filter_default()
).obj_keys¶ (
Optional
[List
[Tuple
[Any
,Any
]]]) – used (internally only) to pass the parent data-struct path in recursive calls.
- Return type:
- Returns:
list of tuples (data-struct-path, key, value); one tuple for each found item within the passed
obj
argument. an empty list will be returned if no item got found.
- deep_update(obj, updating_obj, leaf_types=(<class 'bytes'>, <class 'int'>, <class 'float'>, <class 'set'>, <class 'str'>, <class 'type'>), key_filter=<function key_filter_default>, value_filter=<function value_filter_default>, obj_keys=None)[source]
merge the
updating_obj
data structure into the similar structured node objectobj
.- Parameters:
updating_obj¶ (
Any
) – data structure similar structured like theobj
argument with update values. a UNSET value will delete the item from theobj
argument.leaf_types¶ (
Tuple
[Type
,...
]) – tuple of leaf types to skip from to be searched deeper. the default value of this parameter is specified in the modul constantDEFAULT_LEAF_TYPES
.key_filter¶ (
Callable
[[Union
[str
,int
,tuple
]],bool
]) – called for each sub-item/-attribute of the data structure specified byobj
. return True for item-key/item-index/attribute-name to be filtered out. by default all attribute/key names starting with an underscore character will be filtered out (see default callablekey_filter_default()
).value_filter¶ (
Callable
[[Any
,List
[Tuple
[Any
,Any
]]],bool
]) – called for each sub-item/-attribute of the data structure specified byobj
. return True for items/attributes values to be filtered out. by default empty values, excluded values (seeEXCLUDED_VALUE_TYPES
) and already scanned objects will be filtered out (see default callablevalue_filter_default()
).obj_keys¶ (
Optional
[List
[Tuple
[Any
,Any
]]]) – used (internally only) to pass the parent data-struct path in recursive calls.
- key_list_object(obj_keys)[source]
determine object in a deep nested data structure via an object key list.
- Parameters:
- Return type:
- Returns:
recalculated object referenced by the first object and the keys of
obj_keys
or ~ae.base.UNSET if not found.- Raises:
TypeError if key does not match the object type in any item of
obj_keys
. ValueError ifobj_keys
is not of typeObjKeysType
.
Hint
to include changes on immutable data structures, the returned object value gets recalculated, starting from the first object (
obj_keys`[0][0]), going deeper via the key only (while ignoring all other child objects in the object key list specified by :paramref:`obj_keys
).
- key_path_object(obj, key_path, new_value=<ae.base.UnsetType object>)[source]
determine object in a deep nested data structure via a key path string, and optionally assign a new value to it.
- Parameters:
obj¶ (
Any
) – initial data object to search in (and its sub-objects).composed key string containing dict keys, tuple/list/str indexes and object attribute names, separated by a dot character, like shown in the following examples:
>>> class AClass: ... str_attr_name_a = "a_attr_val" ... dict_attr = {'a_str_key': 3, 999: "value_with_int_key", '999': "..str_key"}
>>> class BClass: ... str_attr_name_b = "b_b_b_b_b" ... b_obj = AClass()
>>> b = BClass()
>>> assert key_path_object(b, 'str_attr_name_b') == "b_b_b_b_b" >>> assert key_path_object(b, 'b_obj.str_attr_name_a') == "a_attr_val" >>> assert key_path_object(b, 'b_obj.str_attr_name_a.5') == "r" # 6th chr of a_attr_val >>> assert key_path_object(b, 'b_obj.dict_attr.a_str_key') == 3
the item key or index value of lists and dictionaries can alternatively be specified in Python syntax, enclosed in [ and ]:
>>> assert key_path_object(b, 'b_obj.dict_attr["a_str_key"]') == 3 >>> assert key_path_object(b, 'b_obj.dict_attr[\'a_str_key\']') == 3 >>> assert key_path_object(b, 'b_obj.dict_attr[999]') == "value_with_int_key" >>> assert key_path_object(b, 'b_obj.dict_attr["999"]') == "..str_key"
only dict key strings that are not can be misinterpreted as number can be specified without the high commas (enclosing the key string), like e.g.:
>>> assert key_path_object(b, 'b_obj.dict_attr[a_str_key]') == 3
optional new value - replacing the found object. the old object value will be returned.
Note
immutable objects, like tuples, that are embedding in
obj
will be automatically updated/replaced up in the data tree structure until a mutable object (list, dict or object) get found.
- Return type:
- Returns:
specified object/value (the old value if
new_value
got passed) orUNSET
if not found/exists (key path string is invalid).
- key_value(key_str)[source]
convert key string (mapping key, sequence index, obj attribute) into its value (str, int, float, tuple, ..).
- Return type:
- next_key_of_path(key_path)[source]
parse key_path to determine the next item key/index, the path separator and the rest of the key path.
- object_items(obj, leaf_types=(<class 'bytes'>, <class 'int'>, <class 'float'>, <class 'set'>, <class 'str'>, <class 'type'>), key_filter=<function key_filter_default>)[source]
determine the items of a data object, with mapping keys, sequence indexes or attribute names and its values.
- Parameters:
obj¶ (
Any
) – data structure/node object (list, dict, set, tuple, data object, …).leaf_types¶ (
Tuple
[Type
,...
]) – tuple of leaf types to skip from to be searched deeper. the default value of this parameter is specified in the modul constantDEFAULT_LEAF_TYPES
.key_filter¶ (
Callable
[[Union
[str
,int
,tuple
]],bool
]) – called for each sub-item/-attribute of the data structure specified byobj
. return True for item-key/item-index/attribute-name to be filtered out. by default all attribute/key names starting with an underscore character will be filtered out (see default callablekey_filter_default()
).
- Return type:
- Returns:
items view of the data object specified in the
obj
argument.
- object_item_value(obj, key, default_value=<ae.base.UnsetType object>)[source]
determine value of a data object item/attribute.
- Parameters:
- Return type:
- Returns:
data object item/attribute value or the
default_value
if not found.
- pop_object(obj_keys)[source]
delete sub-attribute/item of a data object.
- Parameters:
obj_keys¶ (
List
[Tuple
[Any
,Any
]]) –list of (object, key) tuples identifying an element within a deeply nested data structure or object hierarchy. the root of the data/object structure is the object at list index 0 and the element to be deleted is identified by the object and key in the last list item of this argument.
the referenced data structure can contain even immutable node objects (like tuples) which will be accordingly changed/replaced if affected/needed. for that at least one object/element above the immutable object in the deep data structure has to be mutable, else a
ValueError
will be raised.- Return type:
- Returns:
list of deleted values or UNSET if not found.
- Raises:
ValueError if no immutable parent object got found or if the
obj_keys' is empty. IndexError if the one of the specified indexes in :paramref:`obj_keys
does not exist.
- replace_object(obj_keys, new_value)[source]
set sub-attribute/item with a mutable parent object to a new value within a deeply nested object/data structure.
- Parameters:
obj_keys¶ (
List
[Tuple
[Any
,Any
]]) –list of (object, key) tuples identifying an element within a deeply nested data structure or object hierarchy. the root of the data/object structure is the object at list index 0 and the element to be changed is identified by the object and key in the last list item of this argument.
the referenced data structure can contain immutable data node objects (like tuples) which will be accordingly changed/replaced if affected/needed.
at least one object/element, situated above of replaced data object within the deep data structure, has to be mutable, else a
ValueError
will be raised.new_value¶ (
Any
) – value to be assigned to the element referenced by the last list item of the argument inobj_keys
.
- Return type:
- Returns:
the number of levels of the first mutable data objects above the changed data object, or 0 of the changed data object is mutable.
- Raises:
ValueError if no immutable parent object got found or if the object key list in the
obj_keys
argument is empty.