ae.sys_data

external system data structures

this module allows your application to easily manage data structures to interface data flows from and onto external systems.

an external system can be nearly anything: like e.g.: a database or web server or an application or software suite with an interface for data exchanges; even data files can be used as an external system.

the same data can be represented differently on different systems. this module allows you to specify individual data structures for each system and provides hooks to easily integrate system-specific data conversion functions.

data structures

use the classes of this module to define any kind of simple data structures - like lists, mappings and trees:

• a list can get represented by an instance of one of the classes Records or Values. each Records instance is a sequence of 0..n Record instances. a Values instance is a sequence of 1..n Value instances.

• a mapping get represented by an instance of the class Record, whereas each mapping item gets represented by an instance of the private class _Field.

• a node of a tree structure can be represented by an instance of the classes Values, Records or Record.

by combining these simple data structures you can build any complex data structures. the root of such a complex data structure can be defined by an instance of either Records or Record.

the leaves of any simple or complex data structure are represented by instances of the Value class.

the following diagram is showing a complex data structure with all the possible combinations (of a single system):

digraph { node [shape=record] rec1 [label="{<rec1>Record (root) | { <A>Field A | <B>Field B | <C>Field C | <D>Field D } }"] "Records (root)" -> rec1 [arrowhead=crow style=tapered penwidth=3] rec1:A -> "Value (of Field A)" [minlen=3] rec1:B -> "Values" "Values" -> "Value (of one Values item)" [minlen=2 arrowhead=crow style=tapered penwidth=3] rec2 [label="{<rec2>Record (sub-record) | { <CA>Field CA | <CB>Field CB | <CN>... } }"] rec1:C -> rec2 rec2:CA -> "Value (of Field CA)" [minlen=2] rec3 [label="{<rec3>Record (sub-records-sub-record) | { <DA>Field DA | <DB>Field DB | <DN>... } }"] rec1:D -> "Records (sub-records)" "Records (sub-records)" -> rec3 [arrowhead=crow style=tapered penwidth=3] rec3:DA -> "Value (of Field DA)" }

data reference index paths

an index path is a tuple of indexes/keys that is referencing one part or a value of a data structure.

the items of an index path are either of type int (specifying a list index in a Values or Records instance) or of type str (specifying a field name of a Record instance).

e.g. the index path to reference in the above graph example the field DA from the record Record (root) would be ('D', 0, 'DA'). the first item is referencing the field D in the root record. the second item 0 is referencing the first item of the Records instance/value of field D, which is a Record instance. and finally the last item DA specifies the field name in this Record instance.

the hierarchically highest placed Record instance in a data structure is called the root record, and the Value instances at the lowest level in a data structure are the leaves of a data structure.

each _Field is providing apart from its field value also a reference to its root record as well as a root index. the root index is an index path to reference the _Field instance from the root record.

if the format/representation of the data value or of any references differs at one of the used systems then a _Field instance allows you to store a separate system-specific value. to access a system-specific value or reference in your data structure you have to specify additionally to the index path also a system identifier and optionally a direction identifier.

system and direction identifiers

a system id is a user-defined string that is uniquely identifying a system and should consist of at least two characters (see _ASP_SYS_MIN_LEN).

a direction id is specifying the data flow direction. the two pre-defined direction ids are:

• FAD_FROM for pulling data from a system or

• FAD_ONTO for pushing data onto a system.

the methods pull() and push() of the Record class are used to pull/push and convert system-specific data from/onto a system.

Module Attributes

ACTION_INSERT	insert action
ACTION_UPDATE	update action
ACTION_UPSERT	insert or update (if already exists) action
ACTION_DELETE	delete action
ACTION_SEARCH	search action
ACTION_PARSE	parse action
ACTION_BUILD	build action
ACTION_PULL	pull-from-system action
ACTION_PUSH	push-to-system action
ACTION_COMPARE	compare action
FAT_IDX	main/system field name within parent Record or list index within Records/Values
FAT_VAL	main/system field value - storing one of the VALUE_TYPES instance
FAT_CLEAR_VAL	field default/clear value (init by _Field.set_clear_val(), used by clear_leaves())
FAT_REC	root Record instance
FAT_RCX	field index path (idx_path) from the root Record instance
FAT_CAL	calculator callable
FAT_CHK	validator callable
FAT_CNV	system value converter callable
FAT_FLT	field filter callable
FAT_SQE	SQL expression to fetch field value from db
ALL_FATS	tuple of all pre-defined field aspect types/prefixes
FAD_FROM	FROM field aspect direction
FAD_ONTO	ONTO field aspect direction
IDX_PATH_SEP	separator character used for idx_path values (especially if field has a Record value).
ALL_FIELDS	special key of fields_patches argument of Record.copy() to allow aspect value patching for all fields.
CALLABLE_SUFFIX	suffix for aspect keys - used by _Field.set_aspects()
IDX_TYPES	tuple of classes/types used for system data index path items
PATH_TYPES	path items, plus field name path, plus index path
AspectKeyType	type of _Field aspect key
IdxItemType	types of idx_path items
IdxPathType	idx_path type
IdxTypes	types of either idx_path or idx_path items
NodeType	node types (NODE_TYPES)
ListType	list/sequence types (LIST_TYPES)
ValueType	value types (VALUE_TYPES)
NodeChildType	node child types
FieldCallable	callable with _Field argument
FieldValCallable	callable with _Field and additional argument
VALUE_TYPES	tuple of classes/types used for system data values
LIST_TYPES	tuple of classes/types used for system data lists
NODE_TYPES	tuple of classes/types used for system data nodes
NODE_CHILD_TYPES	tuple of classes/types used for system data node children
PARENT_TYPES	value types that can have children (Value excluded)

Functions

aspect_key(type_or_key[, system, direction])	compiles an aspect key from the given args
aspect_key_direction(key)	determines the direction id string from an aspect key.
aspect_key_system(key)	determines the system id string from an aspect key.
compose_current_index(node, idx_path, ...)	determine tuple with the current indexes.
deeper(deepness, instance)	check and calculate resulting/remaining deepness for Record/_Field/Records.copy() when going one level deeper.
field_name_idx_path(field_name[, ...])	converts a field name path string into an index path tuple.
field_names_idx_paths(field_names)	return list of the full idx paths names for all the fields specified in the field_names argument.
get_current_index(node)	get current index of passed node.
idx_path_field_name(idx_path[, add_sep])	convert index path tuple/list into field name string.
init_current_index(node, idx_path, use_curr_idx)	determine current index of node and if not set then initialize to the first index path item.
set_current_index(node[, idx, add])	set current index of node.
string_to_records(str_val, field_names[, ...])	convert formatted string into a Records instance containing several Record instances.
template_idx_path(idx_path[, is_sub_rec])	check/determine if idx_path is referring to template item.
use_current_index(node, idx_path, use_curr_idx)	determine index path of node by using current index of node if exists and is enabled by use_curr_idx arg.
use_rec_default_root_rec_idx(rec, root_rec)	helper function to determine resulting root record and root index.
use_rec_default_sys_dir(rec, system, direction)	helper function to determine resulting system/direction.

Classes

Record([template, fields, system, ...])	instances of this mapping class are used to represent record-like data structures.
Records([seq])	Records class.
Value([iterable])	represents a value.
Values([seq])	ordered/mutable sequence/list, which contains 0..n instances of the class Value.

ACTION_INSERT = 'INSERT'

insert action

ACTION_UPDATE = 'UPDATE'

update action

ACTION_UPSERT = 'UPSERT'

insert or update (if already exists) action

ACTION_DELETE = 'DELETE'

delete action

ACTION_SEARCH = 'SEARCH'

search action

ACTION_PARSE = 'PARSE'

parse action

ACTION_BUILD = 'BUILD'

build action

ACTION_PULL = 'PULL'

pull-from-system action

ACTION_PUSH = 'PUSH'

push-to-system action

ACTION_COMPARE = 'COMPARE'

compare action

FAT_IDX = 'idx'

main/system field name within parent Record or list index within Records/Values

FAT_VAL = 'vle'

main/system field value - storing one of the VALUE_TYPES instance

FAT_CLEAR_VAL = 'vwc'

field default/clear value (init by _Field.set_clear_val(), used by clear_leaves())

FAT_REC = 'rrd'

root Record instance

FAT_RCX = 'rrx'

field index path (idx_path) from the root Record instance

FAT_CAL = 'clc'

calculator callable

FAT_CHK = 'chk'

validator callable

FAT_CNV = 'cnv'

system value converter callable

FAT_FLT = 'flt'

field filter callable

FAT_SQE = 'sqc'

SQL expression to fetch field value from db

ALL_FATS = ('idx', 'vle', 'vwc', 'rrd', 'rrx', 'clc', 'chk', 'cnv', 'flt', 'sqc')

tuple of all pre-defined field aspect types/prefixes

FAD_FROM = 'From'

FROM field aspect direction

FAD_ONTO = 'Onto'

ONTO field aspect direction

IDX_PATH_SEP = '/'

separator character used for idx_path values (especially if field has a Record value).

don’t use dot char because this is used e.g. to separate system field names in xml element name paths.

ALL_FIELDS = '**'

special key of fields_patches argument of Record.copy() to allow aspect value patching for all fields.

CALLABLE_SUFFIX = '()'

suffix for aspect keys - used by _Field.set_aspects()

_ASP_TYPE_LEN = 3

aspect key type string length

_ASP_DIR_LEN = 4

aspect key direction string length

_ASP_SYS_MIN_LEN = 2

aspect key system id string length

IDX_TYPES: Tuple[Type, ...] = (<class 'int'>, <class 'str'>)

tuple of classes/types used for system data index path items

PATH_TYPES = (<class 'int'>, <class 'str'>, <class 'tuple'>)

path items, plus field name path, plus index path

AspectKeyType

type of _Field aspect key

IdxItemType

types of idx_path items

alias of Union[int, str]

IdxPathType

idx_path type

alias of Tuple[Union[int, str], …]

IdxTypes

types of either idx_path or idx_path items

alias of Union[int, str, Tuple[Union[int, str], …]]

NodeType

node types (NODE_TYPES)

alias of Union[Record, Records]

ListType

list/sequence types (LIST_TYPES)

alias of Union[Values, Records]

ValueType

value types (VALUE_TYPES)

alias of Union[Value, Values, Record, Records]

NodeChildType

node child types

alias of Union[_Field, Record]

FieldCallable

callable with _Field argument

alias of Callable[[_Field], Any]

FieldValCallable

callable with _Field and additional argument

alias of Callable[[_Field, Any], Any]

aspect_key(type_or_key, system='', direction='')

compiles an aspect key from the given args

Parameters:

• type_or_key (str) – either FAT_* type or full key (including already the system and direction)-

• system (str) – system id string (if type_or_key is a pure FAT_* constant).

• direction (str) – direction string FAD_* constant (if type_or_key is a pure FAT_* constant).

Return type:

str

Returns:

compiled aspect key as string.

aspect_key_system(key)

determines the system id string from an aspect key.

Parameters:

key (str) – aspect key string.

Return type:

str

Returns:

system id (SDI_* constant).

aspect_key_direction(key)

determines the direction id string from an aspect key.

Parameters:

key (str) – aspect key string.

Return type:

str

Returns:

direction id (FAD_* constant).

deeper(deepness, instance)

check and calculate resulting/remaining deepness for Record/_Field/Records.copy() when going one level deeper.

Parameters:

• deepness (int) – <0 will be returned unchanged until last level is reached: -1==full deep copy, -2==deep copy until deepest Value, -3==deep copy until deepest _Field.

• instance (Any) – instance to be processed/copied (if this method is returning != 0/zero).

Return type:

int

Returns:

if deepness == 0 then return 0, if deepness < 0 then return 0 if the deepest level is reached, else (deepness > 0) return deepness - 1.

field_name_idx_path(field_name, return_root_fields=False)

converts a field name path string into an index path tuple.

Parameters:

• field_name (Union[int, str, Tuple[Union[int, str], ...]]) – field name str or field name index/path string or field index tuple or int (for Records index).

• return_root_fields (bool) – pass True to also return len()==1-tuple for fields with no deeper path (def=False).

Return type:

Tuple[Union[int, str], ...]

Returns:

index path tuple (idx_path) or empty tuple if the field has no deeper path and return_root_fields==False.

field_names_idx_paths(field_names)

return list of the full idx paths names for all the fields specified in the field_names argument.

Parameters:

field_names (Sequence[Tuple[Union[int, str], ...]]) – sequence/list/tuple of field (main or system) names.

Return type:

List[Tuple[Union[int, str], ...]]

Returns:

list of their idx paths names.

idx_path_field_name(idx_path, add_sep=False)

convert index path tuple/list into field name string.

Parameters:

• idx_path (Tuple[Union[int, str], ...]) – index path to convert.

• add_sep (bool) – pass True to always separate index with IDX_PATH_SEP. the default value False will only put a separator char if the field value is a Record (to separate the root field name from the subfield name).

Return type:

str

Returns:

field name string.

compose_current_index(node, idx_path, use_curr_idx)

determine tuple with the current indexes.

Parameters:

• node (Union[Values, Records, Record]) – root node/list (Record or Records/Values instance) to process.

• idx_path (Tuple[Union[int, str], ...]) – index path relative to root node passed in node arg.

• use_curr_idx (List) – list of index counters within idx_path where the current index has to be used.

Return type:

Tuple[Union[int, str], ...]

Returns:

tuple of current indexes.

get_current_index(node)

get current index of passed node.

Parameters:

node (Union[Values, Records, Record]) – instance of Record or Records (real node) or Values (simple list).

Return type:

Union[int, str, None]

Returns:

current index value.

init_current_index(node, idx_path, use_curr_idx)

determine current index of node and if not set then initialize to the first index path item.

Parameters:

• node (Union[Values, Records, Record]) – root node/list (Record or Records/Values instance) to process.

• idx_path (Tuple[Union[int, str], ...]) – index path relative to root node passed in node arg.

• use_curr_idx (Optional[List]) – list of index counters within idx_path where the current index has to be used.

Return type:

Tuple[Union[int, str], ...]

Returns:

tuple of current indexes.

set_current_index(node, idx=None, add=None)

set current index of node.

Parameters:

• node (Union[Values, Records, Record]) – root node/list (Record or Records/Values instance) to process.

• idx (Union[int, str, None]) – index value to set (str for field name; int for list index); if given add will be ignored.

• add (Optional[int]) – value to add to list index; will be ignored if the idx argument is specified.

Return type:

Union[int, str]

Returns:

the finally set/new index value.

use_current_index(node, idx_path, use_curr_idx, check_idx_type=False, delta=1)

determine index path of node by using current index of node if exists and is enabled by use_curr_idx arg.

Parameters:

• node (Union[Values, Records, Record]) – root node/list (Record or Records/Values instance) to process.

• idx_path (Tuple[Union[int, str], ...]) – index path relative to root node passed in node arg.

• use_curr_idx (Optional[List]) – list of index counters within idx_path where the current index has to be used.

• check_idx_type (bool) – pass True to additionally check if the index type is correct (def=False).

• delta (int) – value to decrease the list index counters within use_curr_idx (def=1).

Return type:

Tuple[Union[int, str], ...]

Returns:

tuple of current indexes.

string_to_records(str_val, field_names, rec_sep=',', fld_sep='=', root_rec=None, root_idx=())

convert formatted string into a Records instance containing several Record instances.

Parameters:

• str_val (str) – formatted string to convert.

• field_names (Sequence) – list/tuple of field names of each record

• rec_sep (str) – character(s) used in str_val to separate the records.

• fld_sep (str) – character(s) used in str_val to separate the field values of each record.

• root_rec (Optional[Record]) – root to which the returned records will be added.

• root_idx (Tuple[Union[int, str], ...]) – index path from root where the returned records will be added.

Return type:

Records

Returns:

converted Records instance.

template_idx_path(idx_path, is_sub_rec=False)

check/determine if idx_path is referring to template item.

Parameters:

• idx_path (Tuple[Union[int, str], ...]) – index path to check.

• is_sub_rec (bool) – pass True to only check sub-record-fields (will then return always False for root-fields).

Return type:

bool

Returns:

True if idx_path is referencing a template item (with index zero/0), else False.

use_rec_default_root_rec_idx(rec, root_rec, idx=(), root_idx=(), met='')

helper function to determine resulting root record and root index.

Parameters:

• rec (Record) – current Record instance.

• root_rec (Optional[Record]) – default root record (def=`rec`).

• idx (Optional[Tuple[Union[int, str], ...]]) – current index of rec.

• root_idx (Optional[Tuple[Union[int, str], ...]]) – default root index.

• met (str) – calling method/function name (used only for assert error message, def=’’).

Return type:

Tuple[Record, Tuple[Union[int, str], ...]]

Returns:

resulting root record and root index (as tuple).

use_rec_default_sys_dir(rec, system, direction)

helper function to determine resulting system/direction.

Parameters:

• rec (Optional[Record]) – current Record instance.

• system (Optional[str]) – default system id.

• direction (Optional[str]) – default direction (see FAD_* constants).

Return type:

Tuple[str, str]

Returns:

resulting system and direction (as tuple).

class Value(iterable=(), /)

Bases: list

represents a value.

this class inherits directly from the Python list class. each instance can hold either a (single/atomic) value (which can be anything: numeric, char/string or any object) or a list of these single/atomic values.

__getitem__(key)

determine atomic value.

Parameters:

key (Union[slice, int]) – list index if value is a list.

Return type:

Union[Any, MutableSequence[Any]]

Returns:

list item value.

__setitem__(key, value)

set/initialize list item identified by key to the value passed in value.

Parameters:

• key (Union[slice, int]) – list index if value is a list.

• value (Any) – the new value of the list item.

Return type:

None

__repr__()

representation which can be used to serialize and re-create Value instance.

Return type:

str

Returns:

Value representation string.

__str__()

string representation of this Value instance.

Return type:

str

Returns:

Value string.

property initialized

flag if this Value instance got already initialized.

Returns:

True if already set to a value, else False.

node_child(idx_path, moan=False, **__)

check if idx_path is correct (has to be empty) and if yes then return self.

this method is to simplify the data structure hierarchy implementation.

Parameters:

• idx_path (Tuple[Union[int, str], ...]) – this argument has to be an empty tuple/list.

• moan (bool) – pass True to raise AssertionError if idx_path is not empty.

Return type:

Optional[Value]

Returns:

self or None (if idx_path is not empty and moan == False).

value(*idx_path, **__)

check if idx_path is correct (has to be empty) and if yes then return self.

this method is to simplify the data structure hierarchy implementation.

Parameters:

idx_path (Union[int, str]) – this argument has to be an empty tuple.

Return type:

Optional[Value]

Returns:

self or None (if idx_path is not empty and moan == False).

val(*idx_path, **__)

check if idx_path is correct (either empty or contain one int) and if yes then return list item.

this method is to simplify the data structure hierarchy implementation.

Parameters:

idx_path – this argument is either empty or contains a list index.

Returns:

atomic/single value or list item value or empty string.

set_val(val, *idx_path, **__)

set a/the value of this instance.

Parameters:

• val (Any) – simple/atomic value to be set.

• idx_path (Union[int, str]) – this argument is either empty or contains a list index.

Returns:

self.

copy(*_, **__)

copy the value of this Value instance into a new one.

Returns:

new Value instance containing the same immutable value.

clear_leaves(**__)

clear/reset the value of this instance.

use self[-1] = ‘’ to clear only the newest/top val.

Returns:

self.

class Values(seq=())

Bases: list

ordered/mutable sequence/list, which contains 0..n instances of the class Value.

__init__(seq=())

create new Values instance.

Parameters:

seq (Iterable) – Iterable used to initialize the new instance (pass list, tuple or other iterable).

__repr__()

Return repr(self).

Return type:

str

__str__()

Return str(self).

Return type:

str

node_child(idx_path, use_curr_idx=None, moan=False, selected_sys_dir=None)

determine and return node instance specified by idx_path if exists in this instance or underneath.

Parameters:

• idx_path (Union[int, str, Tuple[Union[int, str], ...]]) – index path to the node, relative to this instance.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

• moan (bool) – flag to check data integrity; pass True to raise AssertionError if not.

• selected_sys_dir (Optional[dict]) – optional dict to return the currently selected system/direction.

Return type:

Union[Value, Values, Record, Records, None]

Returns:

found node instance or None if not found.

value(*idx_path, system='', direction='', **kwargs)

determine the ValueType instance referenced by idx_path of this Values/Records instance.

Parameters:

• idx_path (Union[int, str]) – index path items.

• system (str) – system id.

• direction (str) – direction id.

• kwargs – extra args (will be passed to underlying data structure).

Return type:

Union[Value, Values, Record, Records, None]

Returns:

found Value or Values instance, or None if not found.

set_value(value, *idx_path, system='', direction='', protect=False, root_rec=None, root_idx=(), use_curr_idx=None)

set the ValueType instance referenced by idx_path of this Values/Records instance.

Parameters:

• value (Union[Value, Values, Record, Records]) – ValueType instance to set/change.

• idx_path (Union[int, str]) – index path items.

• system (str) – system id.

• direction (str) – direction id.

• protect (bool) – pass True to prevent replacement of already existing ValueType.

• root_rec (Optional[Record]) – root record.

• root_idx (Tuple[Union[int, str], ...]) – root index.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

Return type:

Union[Values, Records]

Returns:

self (this instance of Values or Records).

val(*idx_path, system='', direction='', flex_sys_dir=True, use_curr_idx=None, **kwargs)

determine the user/system value referenced by idx_path of this Values/Records instance.

Parameters:

• idx_path (Union[int, str]) – index path items.

• system (str) – system id.

• direction (str) – direction id.

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

• kwargs – extra args (will be passed to underlying data structure).

Return type:

Any

Returns:

found user/system value, or None if not found or empty string if value was not set yet.

set_val(val, *idx_path, protect=False, extend=True, use_curr_idx=None)

set the user/system value referenced by idx_path of this Values instance.

Parameters:

• val (Any) – user/system value to set/change.

• idx_path (Union[int, str]) – index path items.

• protect (bool) – pass True to prevent replacement of already existing ValueType.

• extend (bool) – pass True to allow extension of data structure.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

Return type:

Values

Returns:

self (this instance of Values).

copy(deepness=0, root_rec=None, root_idx=(), **kwargs)

copy the values/records of this Values/Records instance.

Parameters:

• deepness (int) – deep copy levels: <0==see deeper(), 0==only copy current instance, >0==deep copy to deepness value - _Field occupies two deepness: 1st=_Field, 2nd=Value.

• root_rec (Optional[Record]) – destination root record.

• root_idx (Tuple[Union[int, str], ...]) – destination index path (tuple of field names and/or list/Records indexes).

• kwargs – additional arguments (will be passed on - most of them used by Record.copy()).

Return type:

Union[Values, Records]

Returns:

new instance of self (which is an instance of Values or Records).

clear_leaves(system='', direction='', flex_sys_dir=True, reset_lists=True)

clear/reset the user/system values of all the leaves of this Values/Records instance.

Parameters:

• system (str) – system id.

• direction (str) – direction id (FAD_* constants).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• reset_lists (bool) – pass False to prevent the reset of all the underlying lists.

Return type:

Union[Values, Records]

Returns:

class Record(template=None, fields=None, system='', direction='', action='', root_rec=None, root_idx=(), field_items=False)

Bases: OrderedDict

instances of this mapping class are used to represent record-like data structures.

isinstance(…, dict) not working if using MutableMapping instead of OrderedDict as super class. and dict cannot be used as super class because instance as kwarg will then not work: see the Groxx’s answer in stackoverflow question at https://stackoverflow.com/questions/3387691/how-to-perfectly-override-a-dict/47361653#47361653.

__init__(template=None, fields=None, system='', direction='', action='', root_rec=None, root_idx=(), field_items=False)

create new Record instance, which is an ordered collection of _Field items.

Parameters:

• template (Optional[Records]) – pass Records instance to use first item/[0] as template (after deep copy / values cleared).

• fields (Optional[Iterable]) – OrderedDict/dict of _Field instances (field order is not preserved when using dict) or Record instance (fields will be referenced, not copied!) or list of (field_name, fld_or_val) tuples.

• system (str) – main/current system of this record,

• direction (str) – interface direction of this record.

• action (str) – current action (see ACTION_INSERT, ACTION_SEARCH, ACTION_DELETE, …)

• root_rec (Optional[Record]) – root record of this record (def=self will be a root record).

• root_idx (Optional[Tuple[Union[int, str], ...]]) – root index of this record (def=()).

• field_items (bool) – pass True to get Record items - using __getitem__() - as of type _Field (not as val()).

__repr__()

Return repr(self).

Return type:

str

__str__()

Return str(self).

Return type:

str

__contains__(idx_path)

True if the dictionary has the specified key, else False.

Return type:

bool

__getitem__(key)

x.__getitem__(y) <==> x[y]

Return type:

Any

__setitem__(key, value)

Set self[key] to value.

node_child(idx_path, use_curr_idx=None, moan=False, selected_sys_dir=None)

get the node child specified by idx_path relative to this Record instance.

Parameters:

• idx_path (Union[int, str, Tuple[Union[int, str], ...]]) – index path or field name index string.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

• moan (bool) – flag to check data integrity; pass True to raise AssertionError if not.

• selected_sys_dir (Optional[dict]) – optional dict to return the currently selected system/direction.

Return type:

Union[Value, Values, Record, Records, _Field, None]

Returns:

found node instance or None if not found.

set_node_child(fld_or_val, *idx_path, system=None, direction=None, protect=False, root_rec=None, root_idx=(), use_curr_idx=None, to_value_type=False)

set/replace the child of the node specified by idx_path with the value of fld_or_val.

Parameters:

• fld_or_val (Any) – field or value - to be set.

• idx_path (Union[int, str]) – index path of the node child to set/replace.

• system (Optional[str]) – system id (pass None to use default system id of this Record instance).

• direction (Optional[str]) – direction id (pass None to use default direction id of this Record instance).

• protect (bool) – pass True to prevent the replacement of a node child.

• root_rec (Optional[Record]) – root record of this data structure.

• root_idx (Optional[Tuple[Union[int, str], ...]]) – root index path of this node.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

• to_value_type (bool) – pass True ensure conversion of fld_or_val to a Value instance.

Return type:

Record

Returns:

self (this Record instance).

value(*idx_path, system=None, direction=None, **kwargs)

search the Value specified by idx_path and return it if found.

Parameters:

• idx_path (Union[int, str]) – index path of Value.

• system (Optional[str]) – system id (pass None to use default system id of this Record instance).

• direction (Optional[str]) – direction id (pass None to use default direction id of this Record instance).

• kwargs – additional keyword args (to be passed onto underlying data structure).

Return type:

Union[Value, Values, Record, Records, None]

Returns:

found Value instance of None if not found.

set_value(value, *idx_path, system=None, direction=None, protect=False, root_rec=None, root_idx=(), use_curr_idx=None)

set/replace the Value instance of the node specified by idx_path.

Parameters:

• value (Union[Value, Values, Record, Records]) – Value instance to be set/replaced.

• idx_path (Union[int, str]) – index path of the Value to be set.

• system (Optional[str]) – system id (pass None to use default system id of this Record instance).

• direction (Optional[str]) – direction id (pass None to use default direction id of this Record instance).

• protect (bool) – pass True to protect existing node value from to be changed/replaced.

• root_rec (Optional[Record]) – root Record instance of this data structure.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/Record instance.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

Return type:

Record

Returns:

self (this Record instance).

val(*idx_path, system=None, direction=None, flex_sys_dir=True, use_curr_idx=None, **kwargs)

determine the user/system value referenced by idx_path of this Record instance.

Parameters:

• idx_path (Union[int, str]) – index path items.

• system (Optional[str]) – system id (pass None to use default system id of this Record instance).

• direction (Optional[str]) – direction id (pass None to use default direction id of this Record instance).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

• kwargs – extra args (will be passed to underlying data structure).

Return type:

Any

Returns:

found user/system value, or None if not found or empty string if value was not set yet.

set_val(val, *idx_path, system=None, direction=None, flex_sys_dir=True, protect=False, extend=True, converter=None, root_rec=None, root_idx=(), use_curr_idx=None, to_value_type=False)

set the user/system value referenced by idx_path of this Record instance.

Parameters:

• val (Any) – user/system value to be set/replaced.

• idx_path (Union[int, str]) – index path of the Value to be set.

• system (Optional[str]) – system id (pass None to use default system id of this Record instance).

• direction (Optional[str]) – direction id (pass None to use default direction id of this Record instance).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• protect (bool) – pass True to protect existing node value from to be changed/replaced.

• extend (bool) – pass False to prevent extension of data structure.

• converter (Optional[Callable[[_Field, Any], Any]]) – converter callable to convert user values between systems.

• root_rec (Optional[Record]) – root Record instance of this data structure.

• root_idx (Optional[Tuple[Union[int, str], ...]]) – root index to this node/Record instance.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

• to_value_type (bool) – pass True ensure conversion of val to a Value instance.

Return type:

Record

Returns:

self (this Record instance).

_add_field(field, idx='')

add _Field instance to this Record instance.

Parameters:

• field (_Field) – _Field instance to add.

• idx (str) – name/key/idx string to map and identify this field (mostly identical to field.name()).

Return type:

Record

Returns:

self.

add_fields(fields, root_rec=None, root_idx=())

adding fields to this Record instance.

Parameters:

• fields (Iterable) – either a dict, a Record or a list with (key/field_name, val/_Field) tuples. key strings containing digits/numbers are interpreted as name/idx paths (then also the specified sub-Records/-Fields will be created). values can be either _Field instances or field values.

• root_rec (Optional[Record]) – root record of this record (def=self will be a root record).

• root_idx (Tuple[Union[int, str], ...]) – root index of this record (def=()).

Return type:

Record

Returns:

self

add_system_fields(system_fields, sys_fld_indexes=None, system=None, direction=None, extend=True)

add/set fields to this Record instance from the field definition passed into system_fields.

make sure before you call this method that this Record instance has the system and direction attributes specified/set.

Parameters:

• system_fields (Iterable[Iterable[Any]]) – tuple/list of tuples/lists with system and main field names and optional field aspects. the index of the field names and aspects within the inner tuples/lists get specified by sys_fld_indexes.

• sys_fld_indexes (Optional[Dict]) – mapping/map-item-indexes of the inner tuples of system_fields. keys are field aspect types (FAT_* constants), optionally extended with a direction (FAD_* constant) and a system (SDI_* constant). if the value aspect key (FAT_VAL) contains a callable then it will be set as the calculator (FAT_CAL) aspect; if contains a field value then also the clear_val of this field will be set to the specified value.

• system (Optional[str]) – system of the fields to be added - if not specified system will be used.

• direction (Optional[str]) – direction (FAD constants) of the fields to be added - if not passed used self.direction.

• extend (bool) – True=add not existing fields, False=apply new system aspects only on existing fields.

Return type:

Record

Returns:

self

collect_system_fields(sys_fld_name_path, path_sep)

compile list of system _Field instances of this Record instance.

Parameters:

• sys_fld_name_path (Sequence) – sequence/tuple/list of system field names/keys.

• path_sep (str) – system field name/key separator character(s).

Return type:

List[_Field]

Returns:

list of :class:’_Field` instances which are having system field name/keys set.

compare_leaves(rec, field_names=(), exclude_fields=())

compare the leaf ‘compare’ values of this Record instance with the one passed into rec.

‘Compare values’ are simplified user/system values generated by Record.compare_val().

Parameters:

• rec (Record) – other Record instance to compare to.

• field_names (Iterable) – field names to include in compare; pass empty tuple (==default) to include all fields.

• exclude_fields (Iterable) – field names to exclude from compare.

Return type:

List[str]

Returns:

list of str with the differences between self and rec.

compare_val(*idx_path)

determine normalized/simplified user/system value of the node specified by idx_path.

Parameters:

idx_path (Union[int, str]) – index path to the node.

Return type:

Any

Returns:

normalized/simplified compare value.

copy(deepness=0, root_rec=None, root_idx=(), onto_rec=None, filter_fields=None, fields_patches=None)

copy the fields of this record.

Parameters:

• deepness (int) – deep copy level: <0==see deeper(), 0==only copy this record instance, >0==deep copy to deepness value - _Field occupies two deepness: 1st=_Field, 2nd=Value.

• root_rec (Optional[Record]) – destination root record - using onto_rec/new record if not specified.

• root_idx (Tuple[Union[int, str], ...]) – destination root index (tuple/list with index path items: field names, list indexes).

• onto_rec (Optional[Record]) – destination record; pass None to create new Record instance.

• filter_fields (Optional[Callable[[_Field], Any]]) – method called for each copied field (return True to filter/hide/not-include into copy).

• fields_patches (Optional[Dict[str, Dict[str, Union[str, Value, Values, Record, Records, Callable[[_Field], Any]]]]]) – dict[field_name_or_ALL_FIELDS:dict[aspect_key:val_or_callable]] to set/overwrite aspect values in each copied _Field instance. the keys of the outer dict are either field names or the ALL_FIELDS value; aspect keys ending with the CALLABLE_SUFFIX have a callable in the dict item that will be called for each field with the field instance as argument; the return value of the callable will then be used as the (new) aspect value. set the aspect value that stores the field value (aspect key == FAT_VAL) by passing a data structure instance (of type ValueType).

Return type:

Record

Returns:

new/extended record instance.

clear_leaves(system='', direction='', flex_sys_dir=True, reset_lists=True)

clear the leaf values including this Record instance and all deeper data structures.

Parameters:

• system (str) – system id (pass None to use default system id of this Record instance).

• direction (str) – direction id (pass None to use default direction id of this Record instance).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• reset_lists (bool) – pass False to prevent reset of sub-lists.

Return type:

Record

Returns:

leaves(system=None, direction=None, flex_sys_dir=True)

generate leaves/_Field-instances of this Record instance.

Parameters:

• system (Optional[str]) – system id (pass None to use default system id of this Record instance).

• direction (Optional[str]) – direction id (pass None to use default direction id of this Record instance).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

Return type:

Generator[_Field, None, None]

Returns:

leaf/_Field-instance generator.

leaf_indexes(*idx_path, system=None, direction=None, flex_sys_dir=True)

generate leaf-/_Field-index paths for all fields of this Record instance.

Parameters:

• idx_path (Union[int, str]) – index path to be added as base index path (index path to this Record instance).

• system (Optional[str]) – system id (pass None to use default system id of this Record instance).

• direction (Optional[str]) – direction id (pass None to use default direction id of this Record instance).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

Return type:

Generator[Tuple[Union[int, str], ...], None, None]

Returns:

leaf/_Field-instance generator.

leaf_names(system='', direction='', col_names=(), field_names=(), exclude_fields=(), name_type=None)

compile a tuple of name types (specified by name_type) for this Record instance.

Parameters:

• system (str) – system id (pass None to use default system id of this Record instance).

• direction (str) – direction id (pass None to use default direction id of this Record instance).

• col_names (Iterable) – system field/column names to include; pass empty tuple (==default) to include all.

• field_names (Iterable) – field names to include; pass empty tuple (==default) to include all fields.

• exclude_fields (Iterable) – field names to exclude.

• name_type (Optional[str]) – type of name to be included/returned - see available name types underneath.

Return type:

Tuple[Union[int, str, Tuple[Union[int, str], ...]], ...]

Returns:

tuple of field names/indexes of the included/found leaves.

possible values for the name_type argument are:

• ‘s’: user/system field/column name.

• ‘f’: field name.

• ‘r’: root name (index path converted into string by idx_path_field_name()).

• ‘S’: index path with user/system field/column name of leaf.

• ‘F’: index path tuple.

• None: names depends on each leaf:

  • root name if leaf is not a root field.

  • user/system name if system is not empty/None.

  • field name.

merge_leaves(rec, system='', direction='', flex_sys_dir=True, extend=True)

merge the leaves of the other record in rec into this Record instance.

Parameters:

• rec (Record) – other Record to merge into this one.

• system (str) – system id (pass None to use default system id of this Record instance).

• direction (str) – direction id (pass None to use default direction id of this Record instance).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• extend (bool) – pass False to prevent extension of this data structure.

Return type:

Record

Returns:

self (this Record instance).

match_key(match_fields)

make tuple of user/system values for all the fields in match_fields.

Parameters:

match_fields (Iterable) – Iterable with field names/index paths.

Return type:

tuple

Returns:

tuple of user/system values.

merge_values(rec, system='', direction='', flex_sys_dir=True, extend=True)

merge user/system values of the other record in rec into this Record instance.

Parameters:

• rec (Record) – other Record to merge into this one.

• system (str) – system id (pass None to use default system id of this Record instance).

• direction (str) – direction id (pass None to use default direction id of this Record instance).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• extend (bool) – pass False to prevent extension of this data structure.

Return type:

Record

Returns:

self (this Record instance).

missing_fields(required_fields=())

check field-names/index-paths specified in required_fields.

Parameters:

required_fields (Iterable) – list/tuple/iterable of field names or index paths of required fields.

Return type:

List[Tuple[Union[int, str], ...]]

Returns:

list of index paths for the ones that are missing or having an empty/None value.

pop(idx, default=None)

check if field name exists and if yes then remove the _Field instance from this Record instance.

Parameters:

• idx (str) – field name.

• default (Optional[_Field]) – return value if a field with the name specified by idx does not exist in this Record.

Return type:

Optional[_Field]

Returns:

removed _Field instance if found, else None.

this method got added because the OrderedDict.pop() method does not call __delitem__() (see also __contains__())

pull(from_system)

pull all user/system values and convert them into field values.

Parameters:

from_system (str) – system id of the system to pull from.

Return type:

Record

Returns:

self.

push(onto_system)

push field values of this Record instance to the related user/system values (converted).

Parameters:

onto_system (str) – system id of the system to push to.

Return type:

Record

Returns:

self.

set_current_system_index(sys_fld_name_prefix, path_sep, idx_val=None, idx_add=1)

check and if possible set the current system index of this Record instance.

Parameters:

• sys_fld_name_prefix (str) – user/system field name prefix.

• path_sep (str) – user/system name path separator.

• idx_val (Optional[int]) – new current system index value to set (if passed then idx_add will be ignored).

• idx_add (Optional[int]) – increment current system index value with the passed int value.

Return type:

Optional[Record]

Returns:

self (this Record instance) if current index got changed, else None.

set_env(system=None, direction=None, action=None, root_rec=None, root_idx=())

set the environment (system/direction/action) of this Record instance.

Parameters:

• system (Optional[str]) – system id (pass None to leave unchanged).

• direction (Optional[str]) – direction id (pass None to leave unchanged).

• action (Optional[str]) – action id (pass None to leave unchanged).

• root_rec (Optional[Record]) – root Record instance of this data structure.

• root_idx (Optional[Tuple[Union[int, str], ...]]) – root index to this node/Record instance.

Return type:

Record

Returns:

self.

sql_columns(from_system, col_names=())

return list of sql column names for given system.

Parameters:

• from_system (str) – system from which the data will be selected/fetched.

• col_names (Iterable) – optionally restrict to select columns to names given in this list.

Return type:

List[str]

Returns:

list of sql column names.

sql_select(from_system, col_names=())

return list of sql column names/expressions for given system.

Parameters:

• from_system (str) – system from which the data will be selected/fetched.

• col_names (Iterable) – optionally restrict to select columns to names given in this list.

Return type:

List[str]

Returns:

list of sql column names/expressions.

to_dict(filter_fields=None, key_type=<class 'str'>, push_onto=True, use_system_key=True, put_system_val=True, put_empty_val=False, system=None, direction=None)

copy Record leaf values into a dict.

Parameters:

• filter_fields (Optional[Callable[[_Field], Any]]) – callable returning True for each field that need to be excluded in returned dict, pass None to include all fields (if put_empty_val == True).

• key_type (Union[Type[str], Type[tuple], None]) – type of dict keys: None=field name, tuple=index path tuple, str=index path string (def).

• push_onto (bool) – pass False to prevent self.push(system).

• use_system_key (bool) – pass False to put leaf field name/index; def=True to use system field name/keys, specified by the system/direction args.

• put_system_val (bool) – pass False to include/use main field val; def=True to include system val specified by the system/direction args.

• put_empty_val (bool) – pass True to also include fields with an empty value (None/’’).

• system (Optional[str]) – system id to determine included leaf and field val (if put_system_val == True).

• direction (Optional[str]) – direction id to determine included leaf and field val (if put_system_val == True).

Return type:

Dict[Union[int, str, Tuple[Union[int, str], ...]], Any]

Returns:

dict of filtered leaf user/system values with field names/idx_path-tuples as their key.

update(mapping=(), **kwargs)

update this Record instance - overwriting/extending OrderedDict/super().update() to return self.

Parameters:

• mapping – mapping to use to update this Record instance.

• kwargs – optional keyword arguments.

Returns:

self (this Record instance).

class Records(seq=())

Bases: Values

Records class.

each instance of this Records class is a list of 0..n Record instances.

the not overwritten methods of the inherited Values class are also available - like e.g. Values.node_child() or Values.val().

__init__(seq=())

create new Records instance.

Parameters:

seq (Iterable) – Iterable used to initialize the new instance (pass list, tuple or other iterable).

__getitem__(key)

x.__getitem__(y) <==> x[y]

Return type:

Union[Value, Values, Record, Records, List[Union[Value, Values, Record, Records]]]

__setitem__(key, value)

Set self[key] to value.

set_node_child(rec_or_fld_or_val, *idx_path, system='', direction='', protect=False, root_rec=None, root_idx=(), use_curr_idx=None)

set/replace the child of the node specified by idx_path with the value of rec_or_fld_or_val.

Parameters:

• rec_or_fld_or_val – record, field or value - to be set.

• idx_path (Union[int, str]) – index path of the node child to set/replace.

• system (str) – system id (pass None to use default system id of this Records instance).

• direction (str) – direction id (pass None to use default direction id of this Records instance).

• protect (bool) – pass True to prevent the replacement of a node child.

• root_rec (Optional[Record]) – root record of this data structure.

• root_idx (Optional[Tuple[Union[int, str], ...]]) – root index path of this node.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

Return type:

Records

Returns:

self (this Records instance).

set_val(val, *idx_path, system='', direction='', flex_sys_dir=True, protect=False, extend=True, converter=None, root_rec=None, root_idx=(), use_curr_idx=None)

set the user/system value referenced by idx_path of this Records instance.

Parameters:

• val (Any) – user/system value to be set/replaced.

• idx_path (Union[int, str]) – index path of the Value to be set.

• system (str) – system id (pass None to use default system id of this Records instance).

• direction (str) – direction id (pass None to use default direction id of this Records instance).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• protect (bool) – pass True to protect existing node value from to be changed/replaced.

• extend (bool) – pass False to prevent extension of data structure.

• converter (Optional[Callable[[_Field, Any], Any]]) – converter callable to convert user values between systems.

• root_rec (Optional[Record]) – root Record instance of this data structure.

• root_idx (Optional[Tuple[Union[int, str], ...]]) – root index to this node/Records instance.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

Return type:

Records

Returns:

self (this Records instance).

append_record(root_rec, root_idx=(), from_rec=None, clear_leaves=True)

add/append Record to this Records instance.

Parameters:

• root_rec (Record) – root Record instance of this data structure.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/Records instance.

• from_rec (Optional[Record]) – Record instance to append, if not passed then use a copy of the first/template Record of this Records instance, else create new Record instance.

• clear_leaves (bool) – pass False to not clear the leaf values.

Return type:

Record

Returns:

added/appended Record instance.

compare_records(records, match_fields, field_names=(), exclude_fields=(), record_comparator=None)

compare the records of this instance with the ones passed in the records argument.

Parameters:

• records (Records) – other instance of Records to be compared against self.

• match_fields (Iterable) – iterable with field names/index paths to determine each Record id/pkey.

• field_names (Iterable) – iterable with field names/index paths that get compared.

• exclude_fields (Iterable) – iterable with field names/index-paths that get excluded from to be compared.

• record_comparator (Optional[Callable[[Record, Record], List[str]]]) – optional callable for additional compare (called for each Record).

Return type:

List[str]

Returns:

list of differences.

index_match_fields(match_fields)

create/initialize match index for this Records instance (stored in match_index attribute).

Parameters:

match_fields (Iterable) – iterable with field names/index paths to determine each Record id/pkey.

Return type:

Records

Returns:

self (this Records instance).

leaves(system='', direction='', flex_sys_dir=True)

generate leaves/_Field-instances of this Records instance.

Parameters:

• system (str) – system id (pass None to use default system id of the underlying Record instances).

• direction (str) – direction id (pass None to use default direction of the underlying Record instances).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

Return type:

Generator[_Field, None, None]

Returns:

leaf/_Field-instance generator.

leaf_indexes(*idx_path, system='', direction='', flex_sys_dir=True)

generate leaf-/_Field-index paths for all fields of this Records instance.

Parameters:

• idx_path (Union[int, str]) – index path to be added as base index path (index path to this Records instance).

• system (str) – system id (pass None to use default system id of the underlying Record instances).

• direction (str) – direction id (pass None to use default direction of the underlying Record instances).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

Return type:

Generator[Tuple[Union[int, str], ...], None, None]

Returns:

leaf/_Field-instance generator.

merge_records(records, match_fields=())

merge the records passed in records into this Records instance.

Parameters:

• records (Records) – records to be merged in.

• match_fields (Iterable) – match fields used to identify already existing records (merge values in these cases).

Return type:

Records

Returns:

self.

set_env(system='', direction='', root_rec=None, root_idx=())

set the environment (system/direction/action) of each record underneath this Records instance.

Parameters:

• system (Optional[str]) – system id (pass None to leave unchanged).

• direction (Optional[str]) – direction id (pass None to leave unchanged).

• root_rec (Optional[Record]) – root Record instance of this data structure.

• root_idx (Optional[Tuple[Union[int, str], ...]]) – root index to this node/Record instance.

Return type:

Records

Returns:

self.

class _Field(root_rec=None, root_idx=(), allow_values=False, **aspects)

Bases: object

Internal/Private class used by Record to create record field instances.

an instance of _Field is representing one record field. the field properties are internally stored within a private dict (_Field._aspects) and are called the ‘aspects’ of a field.

field aspects get used by a field instance e.g. to: * store field value(s) * define callable(s) to convert, filter or validate field values * associate the root record and root index * store any other user-defined field properties (like sql column expressions, comments, …)

the FAT_ALL constant contains all pre-defined aspects (see the other FAT_* constants defined at the top of this module). these values are called ‘aspect keys’ and are used as dict keys in the private dict.

each aspect can additionally have a separate property for each system/direction - in this case the aspect key gets extended with direction/system ids. the two available direction ids are pre-defined by the constants FAD_FROM and FAD_ONTO. the system ids are not defined in this module, they have to be defined by the application. aspect keys can be compiled with the function aspect_key().

__init__(root_rec=None, root_idx=(), allow_values=False, **aspects)

__repr__()

Return repr(self).

__str__()

Return str(self).

__getitem__(key)

node_child(idx_path, use_curr_idx=None, moan=False, selected_sys_dir=None)

get the node child specified by idx_path relative to this _Field instance.

Parameters:

• idx_path (Union[int, str, Tuple[Union[int, str], ...]]) – index path or field name index string.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

• moan (bool) – flag to check data integrity; pass True to raise AssertionError if not.

• selected_sys_dir (Optional[dict]) – optional dict to return the currently selected system/direction.

Return type:

Union[Value, Values, Record, Records, _Field, None]

Returns:

found node instance or None if not found.

value(*idx_path, system='', direction='', flex_sys_dir=False)

search the Value specified by idx_path and return it if found.

Parameters:

• idx_path (Union[int, str]) – index path of Value.

• system (str) – system id (’’ stands for the main/system-independent value).

• direction (str) – direction id (’’ stands for the main/system-independent value).

• flex_sys_dir (bool) – pass True to allow fallback to main (non-user/-system) value.

Return type:

Union[Value, Values, Record, Records, None]

Returns:

found Value instance of None if not found.

set_value(value, *idx_path, system='', direction='', protect=False, root_rec=None, root_idx=(), use_curr_idx=None)

set/replace the Value instance of the node specified by idx_path.

Parameters:

• value (Union[Value, Values, Record, Records]) – Value instance to be set/replaced.

• idx_path (Union[int, str]) – index path of the Value to be set.

• system (str) – system id (’’ stands for the main/system-independent value).

• direction (str) – direction id (’’ stands for the main/system-independent value).

• protect (bool) – pass True to protect existing node value from to be changed/replaced.

• root_rec (Optional[Record]) – root Record instance of this data structure.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/Record instance.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

Return type:

_Field

Returns:

self (this _Field instance).

val(*idx_path, system='', direction='', flex_sys_dir=True, use_curr_idx=None, **kwargs)

determine the user/system value referenced by idx_path.

Parameters:

• idx_path (Union[int, str]) – index path items.

• system (str) – system id (’’ stands for the main/system-independent value).

• direction (str) – direction id (’’ stands for the main/system-independent value).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

• kwargs – extra args (will be passed to underlying data structure).

Return type:

Any

Returns:

found user/system value, or None if not found or empty string if value was not set yet.

set_val(val, *idx_path, system='', direction='', flex_sys_dir=True, protect=False, extend=True, converter=None, root_rec=None, root_idx=(), use_curr_idx=None, to_value_type=False)

set the user/system value referenced by idx_path.

Parameters:

• val (Any) – user/system value to be set/replaced.

• idx_path (Union[int, str]) – index path of the Value to be set.

• system (str) – system id (’’ stands for the main/system-independent value).

• direction (str) – direction id (’’ stands for the main/system-independent value).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• protect (bool) – pass True to protect existing node value from to be changed/replaced.

• extend (bool) – pass False to prevent extension of data structure.

• converter (Optional[Callable[[_Field, Any], Any]]) – converter callable to convert user values between systems.

• root_rec (Optional[Record]) – root Record instance of this data structure.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/Record instance.

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

• to_value_type (bool) – pass True ensure conversion of val to a Value instance.

Return type:

_Field

Returns:

self (this _Field instance).

leaf_value(system='', direction='', flex_sys_dir=False)

determine the leaf value of this field (and optionally system/direction).

Parameters:

• system (str) – system id (’’ stands for the main/system-independent value).

• direction (str) – direction id (’’ stands for the main/system-independent value).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

Return type:

Union[Value, Values, Record, Records, None]

Returns:

the main or user/system value of this leaf/field or None if deeper value exists and flex_sys_dir is False.

used also to check if a deeper located sys field exists in the current data structure.

leaves(system='', direction='', flex_sys_dir=True)

generate all sub-leaves/_Field-instances underneath this _Field instance.

Parameters:

• system (str) – system id (’’ stands for the main/system-independent value).

• direction (str) – direction id (’’ stands for the main/system-independent value).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

Return type:

Generator[_Field, None, None]

Returns:

leaf/_Field-instance generator.

leaf_indexes(*idx_path, system='', direction='', flex_sys_dir=True)

generate leaf-/_Field-index paths for all subfields underneath (if exist) or this _Field instance.

Parameters:

• idx_path (Union[int, str]) – index path to this _Field instance.

• system (str) – system id (’’ stands for the main/system-independent value).

• direction (str) – direction id (’’ stands for the main/system-independent value).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

Return type:

Generator[Tuple[Union[int, str], ...], None, None]

Returns:

leaf/_Field-instance generator.

find_aspect_key(*aspect_types, system='', direction='')

search for the passed aspect_types in this _Field instance.

Parameters:

• aspect_types (str) – aspect types (dict key prefixes) to search for.

• system (Optional[str]) – system id (’’ stands for the main/system-independent value).

• direction (Optional[str]) – direction id (’’ stands for the main/system-independent value).

Return type:

Optional[str]

Returns:

the full aspect key of the first found aspect that is matching the passed aspect type and optionally also the passed system and direction ids.

the search will be done in the following order: * all passed aspect types including the passed system/direction ids. * all passed aspect types including the passed system and both directions (if direction id get not passed). * all passed aspect types including the passed system and without direction id. * all passed aspect types without system and direction ids.

set_env(system='', direction='', root_rec=None, root_idx=())

set the environment (system/direction/action) of each record underneath this _Field instance.

Parameters:

• system (Optional[str]) – system id (don’t pass or pass None to leave unchanged).

• direction (Optional[str]) – direction id (don’t pass or pass None to leave unchanged).

• root_rec (Optional[Record]) – root Record instance of this field, system and direction.

• root_idx (Optional[Tuple[Union[int, str], ...]]) – root index to this node/_Field instance.

Return type:

_Field

Returns:

self (this _Field instance).

set_system_root_rec_idx(system=None, direction=None, root_rec=None, root_idx=())

set the root record and index of the data structure where this _Field instance is included.

Parameters:

• system (Optional[str]) – system id (don’t pass or pass None to leave unchanged).

• direction (Optional[str]) – direction id (don’t pass or pass None to leave unchanged).

• root_rec (Optional[Record]) – new root Record instance of this data structure (pass None to leave unchanged).

• root_idx (Optional[Tuple[Union[int, str], ...]]) – root index to this node/Record instance (pass None to determine).

Return type:

_Field

Returns:

self (this _Field instance).

aspect_exists(*aspect_types, system='', direction='', flex_sys_dir=False)

check if aspect exists and return full aspect key (including system/direction) if yes.

Parameters:

• aspect_types (str) – aspect types (dict key prefixes) to search for.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• flex_sys_dir (Optional[bool]) – pass False to prevent fallback to system-independent value.

Return type:

Optional[str]

Returns:

the full aspect key of the first found aspect that is matching the passed aspect type and optionally also the passed system and direction ids.

Returns:

aspect_value(*aspect_types, system='', direction='', flex_sys_dir=False)

Parameters:

• aspect_types (str) – aspect types (dict key prefixes) to search for.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• flex_sys_dir (Optional[bool]) – pass False to prevent fallback to system-independent value.

Return type:

Any

Returns:

the value of the first found aspect that is matching the passed aspect type and optionally also the passed system and direction ids or None if not found.

del_aspect(type_or_key, system='', direction='')

remove aspect from this field.

Parameters:

• type_or_key (str) – either FAT_* type or full key (including already the system and direction)-

• system (str) – system id string (if type_or_key is a pure FAT_* constant).

• direction (str) – direction string FAD_* constant (if type_or_key is a pure FAT_* constant).

Return type:

Any

Returns:

the aspect value of the removed aspect.

set_aspect(aspect_value, type_or_key, system='', direction='', protect=False, allow_values=False)

set/change the value of an aspect identified by type_or_key, system and direction.

Parameters:

• aspect_value (Any) – the value to set on the aspect.

• type_or_key (str) – either FAT_* type or full key (including already the system and direction)-

• system (str) – system id string (if type_or_key is a pure FAT_* constant).

• direction (str) – direction string FAD_* constant (if type_or_key is a pure FAT_* constant).

• protect (bool) – pass True to prevent overwrite of already existing/set aspect value.

• allow_values (bool) – pass True to allow change of field value aspect (FAT_VAL aspect key).

Return type:

_Field

Returns:

self (this _Field instance).

set_aspects(allow_values=False, **aspects)

set multiple aspects provided in aspects.

Parameters:

• allow_values (bool) – pass True to allow change of field value aspect (FAT_VAL aspect key).

• aspects (Any) – dict of aspects where the dict key is the full/complete aspect key.

Return type:

_Field

Returns:

self (this _Field instance).

add_aspects(allow_values=False, **aspects)

add multiple aspects provided in aspects.

Parameters:

• allow_values (bool) – pass True to allow change of field value aspect (FAT_VAL aspect key).

• aspects (Any) – dict of aspects where the dict key is the full/complete aspect key.

Return type:

_Field

Returns:

self (this _Field instance).

name(system='', direction='', flex_sys_dir=True)

determine one of the names of this field.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

Return type:

str

Returns:

main or system-specific name of this field.

del_name(system, direction='')

remove system-specific name from this field.

Parameters:

• system (str) – system id (has to be non-empty system id).

• direction (str) – direction id (def=’’ stands for both directions).

Return type:

_Field

Returns:

self (this _Field instance).

has_name(name, selected_sys_dir=None)

check if this field has a name identical to the one passed in name.

Parameters:

• name (str) – name to search for.

• selected_sys_dir (Optional[dict]) – pass dict to get back the system and direction ids of the found name.

Return type:

Optional[str]

Returns:

full aspect key of the found name (including system/direction) or None if not found.

set_name(name, system='', direction='', protect=False)

set/change one of the names of this field.

Parameters:

• name (str) – the new name of this field.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• protect (bool) – pass True to prevent overwrite of already set/existing name.

Return type:

_Field

Returns:

self (this _Field instance).

root_rec(system='', direction='')

determine and return root record of this field with given system and direction ids.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Optional[Record]

Returns:

root record instance or None if not set.

set_root_rec(rec, system='', direction='')

set/change the root record of this field, system and direction.

Parameters:

• rec (Record) – root record instance.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

_Field

Returns:

self (this _Field instance).

root_idx(system='', direction='')

return the root index of this field for the specified system and direction ids.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Tuple[Union[int, str], ...]

Returns:

root index of this field.

set_root_idx(idx_path, system='', direction='')

set/change the root record of this field, system and direction.

Parameters:

• idx_path (Tuple[Union[int, str], ...]) – root index for this field/system/direction.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

_Field

Returns:

self (this _Field instance).

calculator(system='', direction='')

return the calculation callable for this field, system and direction.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Optional[Callable[[_Field], Any]]

Returns:

callable used to calculate the value of this field or None if not set/exists/found.

set_calculator(calculator, system='', direction='', protect=False)

set/change the field value calculator of this field, system and direction.

Parameters:

• calculator (Callable[[_Field], Any]) – new callable used to calculate the value of this field.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• protect (bool) – pass True to prevent overwrite of already set/existing calculator callable.

Return type:

_Field

Returns:

self (this _Field instance).

clear_val(system='', direction='')

return the initial field value (the clear value) of this field, system and direction.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Any

Returns:

the found clear/init value or None if not set/found.

set_clear_val(clr_val, system='', direction='')

set/change the clear/init value of this field, system and direction.

Parameters:

• clr_val – new clear/init value of this field.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

_Field

Returns:

self (this _Field instance).

_ensure_system_value(system, direction='', root_rec=None, root_idx=())

check if a field value for the specified system/direction exists and if not then create it.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• root_rec (Optional[Record]) – root Record instance of this field, system and direction.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/_Field instance.

converter(system, direction='')

return the converter callable for this field, system and direction.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Optional[Callable[[_Field, Any], Any]]

Returns:

callable used to convert the field value between systems or None if not set.

separate system-specific representations of the field value can be (automatically) converted by specifying a converter callable aspect.

set_converter(converter, system, direction='', protect=True, root_rec=None, root_idx=())

set/change the field value converter of this field, system and direction.

Parameters:

• converter (Callable[[_Field, Any], Any]) – new callable used to convert the value of this field between systems.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• protect (bool) – pass False to allow the overwriting of an already set converter callable.

• root_rec (Optional[Record]) – root Record instance of this field, system and direction.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/_Field instance.

Return type:

_Field

Returns:

self (this _Field instance).

convert(val, system, direction)

convert field value from/onto system.

Parameters:

• val (Any) – field value to convert.

• system (str) – system to convert from/onto.

• direction (str) – conversion direction (from or onto - see FAD_FROM and FAD_ONTO).

Return type:

Any

Returns:

converted field value.

filterer(system='', direction='')

return the filter callable for this field, system and direction.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Optional[Callable[[_Field], Any]]

Returns:

callable used to filter this field/parent-record or None if not set.

set_filterer(filterer, system='', direction='', protect=False)

set/change the filterer callable of this field, system and direction.

Parameters:

• filterer (Callable[[_Field], Any]) – new callable used to filter this field or the parent record.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• protect (bool) – pass True to prevent overwrite of already set/existing filter callable.

Return type:

_Field

Returns:

self (this _Field instance).

sql_expression(system='', direction='')

return the sql column expression for this field, system and direction.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Optional[str]

Returns:

sql column expression string if set or None if not set.

set_sql_expression(sql_expression, system='', direction='', protect=False)

set/change sql column expression of this field, system and direction.

Parameters:

• sql_expression (str) – new sql column expression used to fetch associated db column of this field.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• protect (bool) – pass True to prevent overwrite of already set/existing filter callable.

Return type:

_Field

Returns:

self (this _Field instance).

validator(system='', direction='')

return the validation callable for this field, system and direction.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Optional[Callable[[_Field, Any], Any]]

Returns:

validation callable if set or None if not set.

set_validator(validator, system='', direction='', protect=False, root_rec=None, root_idx=())

set/change the field value validator of this field, system and direction.

Parameters:

• validator (Callable[[_Field, Any], Any]) – new callable used to validate the value of this field, systems and direction.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• protect (bool) – pass False to allow the overwriting of an already set converter callable.

• root_rec (Optional[Record]) – root Record instance of this field, system and direction.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/_Field instance.

Return type:

_Field

Returns:

self (this _Field instance).

validate(val, system='', direction='')

validate field value for specified system and direction.

Parameters:

• val (Any) – field value to validate (if ok to be set as new field value).

• system (str) – system id of new field value.

• direction (str) – direction id of new field value.

Return type:

bool

Returns:

True if val is ok to be set as new field value else False.

append_record(system='', direction='', flex_sys_dir=True, root_rec=None, root_idx=())

append new record to the Records value of this field/system/direction.

Parameters:

• system (str) – system id of the field value to extend.

• direction (str) – direction id of the field value to extend.

• flex_sys_dir (bool) – pass False to prevent fallback to system-independent value.

• root_rec (Optional[Record]) – root Record instance of this field, system and direction.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/_Field instance.

Return type:

Record

Returns:

added/appended Record instance.

clear_leaves(system='', direction='', flex_sys_dir=True, reset_lists=True)

clear/reset field values and if reset_lists == True also Records/Values lists to one item.

Parameters:

• system (str) – system of the field value to clear, pass None to clear all field values.

• direction (str) – direction of the field value to clear.

• flex_sys_dir (bool) – if True then also clear field value if system is given and field has no system value.

• reset_lists (bool) – if True/def then also clear Records/lists to one item.

Return type:

_Field

Returns:

self (this _Field instance).

copy(deepness=0, root_rec=None, root_idx=(), **kwargs)

copy the aspects (names, indexes, values, …) of this field.

Parameters:

• deepness – deep copy level: <0==see deeper(), 0==only copy current instance, >0==deep copy to deepness value - _Field occupies two deepness: 1st=_Field, 2nd=Value.

• root_rec (Optional[Record]) – destination root record.

• root_idx (Tuple[Union[int, str], ...]) – destination index path (tuple of field names and/or list/Records/Values indexes).

• kwargs – additional arguments (will be passed on - most of them used by Record.copy()).

Return type:

_Field

Returns:

new/copied _Field instance.

parent(system='', direction='', value_types=None)

determine one of the parent ValueType instances in this data structure above of this field.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• value_types (Optional[Tuple[Type[Union[Value, Values, Record, Records]], ...]]) – pass tuple of ValueType to restrict search to one of the passed types.

Return type:

Union[Value, Values, Record, Records, None]

Returns:

found parent instance or None if not set or if type is not of passed value_types.

pull(from_system, root_rec, root_idx)

pull the system-specific value (specified by from_system) into the main value of this field.

Parameters:

• from_system (str) – system id of the system to pull from.

• root_rec (Record) – root Record instance of this field, system and direction.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/_Field instance.

Return type:

_Field

Returns:

self (this _Field instance).

push(onto_system, root_rec, root_idx)

push the main value of this field onto the system-specific value (specified by onto_system).

Parameters:

• onto_system (str) – system id of the system to pull from.

• root_rec (Record) – root Record instance of this field, system and direction.

• root_idx (Tuple[Union[int, str], ...]) – root index to this node/_Field instance.

Return type:

_Field

Returns:

self (this _Field instance).

string_to_records(str_val, field_names, rec_sep=',', fld_sep='=', system='', direction='')

convert formatted string into a Records instance containing several Record instances.

Parameters:

• str_val (str) – formatted string to convert.

• field_names (Sequence) – list/tuple of field names of each record

• rec_sep (str) – character(s) used in str_val to separate the records.

• fld_sep (str) – character(s) used in str_val to separate the field values of each record.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Records

Returns:

converted Records instance.

record_field_val(*idx_path, system='', direction='')

get/determine the value of any field specified via idx_path within this data structure.

Parameters:

• idx_path (Union[int, str]) – index path of the field.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Any

Returns:

the field value if found else None.

this method has an alias named rfv().

rfv(*idx_path, system='', direction='')

alias of method record_field_val()

Return type:

Any

system_record_val(*idx_path, system='', direction='', use_curr_idx=None)

get/determine the current value of a/this field within this data structure.

Parameters:

• idx_path (Union[int, str]) – index path of the field.

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

• use_curr_idx (Optional[list]) – list of counters to specify if and which current indexes have to be used.

Return type:

Any

Returns:

the currently selected field value if found else None.

this method has an alias named srv().

srv(*idx_path, system='', direction='', use_curr_idx=None)

alias of method system_record_val()

Return type:

Any

in_actions(*actions, system='', direction='')

determine if current data structure is in one of the passed actions.

Parameters:

• actions (str) – tuple of actions (see ACTION_ constants).

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

bool

Returns:

True if the data structure has set one of the passed actions else False.

this method has an alias named ina().

ina(*actions, system='', direction='')

alias of method in_actions()

Return type:

bool

current_records_idx(system='', direction='')

determine current index of Records instance, situated above of this field in this data structure.

Parameters:

• system (str) – system id (def=’’ stands for the main/system-independent value).

• direction (str) – direction id (def=’’ stands for the main/system-independent value).

Return type:

Union[int, str, None]

Returns:

full index path or None if no current index exists above this field.

this method has an alias named crx().

crx(system='', direction='')

alias of method current_records_idx()

Return type:

Union[int, str, None]

VALUE_TYPES = (<class 'ae.sys_data.Value'>, <class 'ae.sys_data.Values'>, <class 'ae.sys_data.Record'>, <class 'ae.sys_data.Records'>)

tuple of classes/types used for system data values

LIST_TYPES = (<class 'ae.sys_data.Values'>, <class 'ae.sys_data.Records'>)

tuple of classes/types used for system data lists

NODE_TYPES = (<class 'ae.sys_data.Record'>, <class 'ae.sys_data.Records'>)

tuple of classes/types used for system data nodes

NODE_CHILD_TYPES = (<class 'ae.sys_data._Field'>, <class 'ae.sys_data.Record'>)

tuple of classes/types used for system data node children

PARENT_TYPES = (<class 'ae.sys_data.Values'>, <class 'ae.sys_data.Record'>, <class 'ae.sys_data.Records'>)

value types that can have children (Value excluded)