ae.console

console application environment

an instance of the ConsoleApp class is representing a python application with dynamically configurable logging, debugging features (inherited from AppBase), command line arguments and config files and options.

the helper function sh_exec() provided by this portion simplifies the execution of shell/console commands.

define command line arguments and options

the methods add_argument() and add_option() are defining command line arguments and config options, finally parsed/loaded by calling run_app():

ca = ConsoleApp(app_title="command line arguments demo", app_version="3.6.9")
ca.add_argument('argument_name_or_id', help="Help text for this command line argument")
ca.add_option('option_name_or_id', "help text for this command line option", "default_value")
...
ca.run_app()

the values of the commend line arguments and options are determined via the methods get_argument() and get_option(). additional configuration values, stored in INI/CFG files, are accessible via the get_variable() method.

auto-collected app name, title and version

if one of the kwargs app_title or app_version is not specified in the init call of the ConsoleApp class instance, then they will automatically get determined from your app main module: the app title from the docstring title, and the application version string from the __version__ variable:

""" module docstring title """
from ae.console import ConsoleApp

__version__ = '1.2.3'

ca = ConsoleApp()

assert ca.app_title == "module docstring title"
assert ca.app_version == '1.2.3'

ConsoleApp also determines on instantiation the name/id of your application, if not explicitly specified in app_name. other application environment vars/options (like e.g. the application startup folder path and the current working directory path) will be automatically initialized and provided via the app instance.

configuration files, sections, variables and options

a config file consists of config sections, each section provides config variables and config options to parametrize your application at run-time.

config files

configuration files can be shared between apps or used exclusively by one app. the following file names are recognized and loaded automatically on app initialization:

config file	used for …. config variables and options
<any_path_name_and_ext>	application/domain specific
<app_name>.ini	application specific (read-/write-able)
<app_name>.cfg	application specific (read-only)
.app_env.cfg	application/suite specific (read-only)
.sys_env.cfg	general system (read-only)
.sys_env<SYS_ENV_ID>.cfg	the system with SYS_ID (read-only)

the above table is ordered by the preference to search/get the value of a config variable/option. so the values stored in the domain/app specific config file will always precede/overwrite any application and system specific values.

app/domain-specific config files have to be specified explicitly, either on initialization of the ConsoleApp instance via the kwarg additional_cfg_file, or by calling the method add_cfg_files(). they can have any file extension and can be placed into any accessible folder.

all the other config files have to have the specified name with a .ini or .cfg file extension, and get recognized in the current working directory, in the user data directory (see ae.paths.user_data_path()) and in the application installation directory.

config sections

this module is supporting the config file format of Pythons built-in ConfigParser class, extended by more complex config value types. the following examples shows a config file with two config sections containing one config option (named log_file) and two config variables (configVar1 and configVar2):

[aeOptions]
log_file = './logs/your_log_file.log'
configVar1 = ['list-element1', ('list-element2-1', 'list-element2-2', ), {}]

[YourSectionName]
configVar2 = {'key1': 'value 1', 'key2': 2222, 'key3': datetime.datetime.now()}

the config section aeOptions (defined by MAIN_SECTION_NAME) is the default or main section, storing the values of any pre-defined config option and of some config variables.

config variables

config variables can store complex data types. in the example config file above the config variable configVar1 holds a list with 3 elements: the first element is a string, the second element a tuple, and the third element an empty dict.

all the values, of which its repr string can be evaluated with the built-in eval() function, can be stored in a config variable, by calling the set_variable() method. to read/fetch their value, call the method get_variable() with the name and section names of the config variable. you can specify the type of an config variable via the value passed into value argument or by the see special encapsulated strings, respectively the config value literal.

the following config variables are pre-defined in the main config section and recognized by this module, some of them also by the module/portion ae.core:

• debug_level: debug logging verbosity level config option

• log_file: ae logging file name (this is also a config option - set-able as command line arg)

• logging_params: general ae logging configuration parameters (py and ae logging)

• py_logging_params: python logging configuration

• onboarding_tour_started: count the onboarding tour starts since the installation of the app. will be reset to zero after a user registration (by calling register_user())

• registered_users: users registered with their OS user name as user id (see register_user())

• user_id: id of the app user (default is determined from the system user name <ae.base.os_user_name>)

• user_specific_cfg_vars: list of config variables storing an individual value for each registered user (see section user specific config variables)

Note

the value of a config variable can be overwritten by defining an OS environment variable with a name that is equal to the snake+upper-case converted names of the config-section and -variable. e.g. declare an OS environment variable with the name AE_OPTIONS_LOG_FILE to overwrite the value of the pre-defined config option/variable log_file.

config options

config options are config variables, defined persistently in the config section aeOptions. specifying them on the command line, preceding the option name with two leading hyphen characters, and using an equal character between the name and the option value, overwrites the value stored in the config file:

$ your_application --log_file='your_new_log_file.log'

the default value of a not specified config option gets searched first in the config files (the exact search order is documented in the doc-string of the method add_cfg_files()), or if not found then the default value will be used, that is specified in the definition of the config option (the call of add_option()).

the method get_option() determines the value of a config option:

my_log_file_name = ca.get_option('log_file')

use the set_option() if you want to change the value of a configuration option at run-time. to read the default value of a config option or variable directly from the available configuration files use the method get_variable(). the default value of a config option or variable can also be set or changed directly from within your application by calling the set_variable() method.

pre-defined configuration options

for a more verbose logging to the console output specify, either on the command line or in a config files, the config option debug_level (or as short option -D) with a value of 2 (for verbose). the supported config option values are documented here.

the value of the second pre-defined config option log_file specifies the log file path/file_name. also this option can be abbreviated on the command line with the short -L option id.

Note

after an explicit definition of the optional config option user_id via add_option() it will be automatically used to initialize the user_id attribute.

user specific config variables

config variables specified in the set user_specific_cfg_vars get automatically recognized as user-specific. override the method :meth`~ConsoleApp._init_default_user_cfg_vars` in your main app instance to define or revoke which config variables the app is storing individually for each user.

Hint

to permit individual sets of user-specific config variables for a user (or group) add the config variable user_specific_cfg_vars in the user-specific config file section(s). don’t forget in this special case to also add there also this config variable, e.g. as (‘aeOptions’, ‘user_specific_cfg_vars’).

Module Attributes

MAIN_SECTION_NAME

default name of main config section

Functions

config_value_string(value)	convert passed value to a string to store them in a config/ini file.
sh_exec(command_line[, extra_args, ...])	execute command in the current working directory of the OS console/shell.

Classes

ConsoleApp([app_title, app_name, ...])

provides command line arguments and options, config options, logging and debugging for your application.

MAIN_SECTION_NAME: str = 'aeOptions'

default name of main config section

config_value_string(value)

convert passed value to a string to store them in a config/ini file.

Parameters:

value (Any) – value to convert to ini variable string/literal.

Return type:

str

Returns:

ini variable literal string.

Note

Literal converts the returned string format back into the representing value.

sh_exec(command_line, extra_args=(), console_input='', lines_output=None, cae=None, shell=False)

execute command in the current working directory of the OS console/shell.

Parameters:

• command_line (str) – command line string to execute on the console/shell. could contain command line args separated by whitespace characters (alternatively use extra_args).

• extra_args (Sequence) – optional sequence of extra command line arguments.

• console_input (str) – optional string to be sent to the stdin stream of the console/shell.

• lines_output (Optional[List[str]]) – optional list to return the lines printed to stdout/stderr on execution.

• cae (Optional[Any]) – optional ConsoleApp instance, only used for logging. to suppress any logging output pass UNSET.

• shell (bool) – pass True to execute command in the default OS shell (see subprocess.run()).

Return type:

int

Returns:

return code of the executed command or 126 if execution raised any other exception.

class ConsoleApp(app_title='', app_name='', app_version='', sys_env_id='', debug_level=0, multi_threading=False, suppress_stdout=False, cfg_opt_eval_vars=None, additional_cfg_files=(), cfg_opt_val_stripper=None, formatter_class=None, epilog='', **logging_params)

Bases: AppBase

provides command line arguments and options, config options, logging and debugging for your application.

most applications only need a single instance of this class. each instance is encapsulating a ConfigParser and a ArgumentParser instance. so only apps with threads and different sets of config options for each thread could create a separate instance of this class.

instance attributes (ordered alphabetically - ignoring underscore characters):

• _arg_parser ArgumentParser instance.

• cfg_opt_choices valid choices for pre-/user-defined options.

• cfg_opt_eval_vars additional dynamic variable values that are getting set via the cfg_opt_eval_vars argument of the method ConsoleApp.__init__() and get then used in the evaluation of evaluable config option values.

• _cfg_files iterable of config file names that are getting loaded and parsed (specify additional configuration/INI files via the additional_cfg_files argument).

• cfg_options pre-/user-defined options (dict of Literal instances defined via add_option()).

• _cfg_parser ConfigParser instance.

• _main_cfg_fnam main config file name.

• _main_cfg_mod_time last modification datetime of main config file.

• _cfg_opt_val_stripper callable to strip option values.

• _parsed_arguments ArgumentParser.parse_args() return.

__init__(app_title='', app_name='', app_version='', sys_env_id='', debug_level=0, multi_threading=False, suppress_stdout=False, cfg_opt_eval_vars=None, additional_cfg_files=(), cfg_opt_val_stripper=None, formatter_class=None, epilog='', **logging_params)

initialize a new ConsoleApp instance.

Parameters:

• app_title (str) –

  application title/description to set the instance attribute app_title.

  if not specified then the docstring of your app’s main module will be used (see example).

• app_name (str) –

  application instance name to set the instance attribute app_name.

  if not specified then base name of the main module file name will be used.

• app_version (str) –

  application version string to set the instance attribute app_version.

  if not specified then value of a global variable with the name __version__` will be used (if declared in the actual call stack).

• sys_env_id (str) –

  system environment id to set the instance attribute sys_env_id.

  this value is also used as file name suffix to load all the system config variables in sys_env<suffix>.cfg. pass e.g. ‘LIVE’ to init this ConsoleApp instance with config values from sys_envLIVE.cfg.

  the default value of this argument is an empty string.

  Note

  if the argument value results as empty string then the value of the optionally defined OS environment variable AE_OPTIONS_SYS_ENV_ID will be used as default.

• debug_level (int) –

  default debug level to set the instance attribute debug_level.

  the default value of this argument is DEBUG_LEVEL_DISABLED.

• multi_threading (bool) – pass True if instance is used in multi-threading app.

• suppress_stdout (bool) – pass True (for wsgi apps) to prevent any python print outputs to stdout.

• cfg_opt_eval_vars (Optional[dict]) – dict of additional application specific data values that are used in eval expressions (e.g. AcuSihotMonitor.ini).

• additional_cfg_files (Iterable) – iterable of additional CFG/INI file names (opt. incl. abs/rel. path).

• cfg_opt_val_stripper (Optional[Callable]) – callable to strip/reformat/normalize the option choices values.

• formatter_class (Optional[Any]) – alternative formatter class passed onto ArgumentParser instantiation.

• epilog (str) – optional epilog text for command line arguments/options help text (passed onto ArgumentParser instantiation).

• logging_params – all other kwargs are interpreted as logging configuration values - the supported kwargs are all the method kwargs of init_logging().

_cfg_parser

ConfigParser instance

cfg_options: Dict[str, Literal]

all config options

cfg_opt_choices: Dict[str, Iterable]

all valid config option choices

cfg_opt_eval_vars: dict

app-specific vars for init of cfg options

_cfg_files: list

list of all found INI/CFG files

_main_cfg_fnam: str

def main config file name

_main_cfg_mod_time: float

main config file modification datetime

_cfg_opt_val_stripper: Optional[Callable]

callable to strip or normalize config option choice values

_parsed_arguments: Optional[Namespace]

storing returned namespace of ArgumentParser.parse_args() call, used to retrieve command line args

_arg_parser: ArgumentParser

ArgumentParser instance

_init_default_user_cfg_vars()

init user default config variables.

override this method to add module-/app-specific config vars that can be set individually per user.

_init_logging(logging_params)

determine and init logging config.

Parameters:

logging_params (Dict[str, Any]) – logging config dict passed as args by user that will be amended with cfg values.

Return type:

Optional[str]

Returns:

None if py logging is active, log file name if ae logging is set in cfg or args or empty string if no logging got configured in cfg/args.

the logging configuration can be specified in several alternative places. the precedence on various existing configurations is (the highest precedence first):

• log_file configuration option specifies the name of the used ae log file (will be read after initialisation of this app instance)

• logging_params configuration variable dict with a py_logging_params key to activate python logging

• logging_params configuration variable dict with the ae log file name in the key log_file_name

• py_logging_params configuration variable to use the python logging module

• log_file configuration variable specifying ae log file

• logging_params dict passing the python logging configuration in the key py_logging_params to this method

• logging_params dict passing the ae log file in the logging key log_file_name to this method

__del__()

deallocate this app instance by calling ae.core.AppBase.shutdown().

property debug_level: int

debug level property:

Getter:

return the current debug level of this app instance.

Setter:

change the debug level of this app instance.

add_cfg_files(*additional_cfg_files)

extend list of available and additional config files (in _cfg_files).

Parameters:

additional_cfg_files (str) – domain/app-specific config file names to be defined/registered additionally.

Return type:

str

Returns:

empty string on success else line-separated list of error message text.

underneath the search order of the config files variable value - the first found one will be returned:

1. the domain/app-specific config files added in your app code by this method. these files will be searched for the config option value in reversed order - so the last added config file will be the first one where the config value will be searched.

2. config files added via additional_cfg_files argument of ConsoleApp.__init__() (searched in the reversed order).

3. <app_name>.INI file in the <app_dir>

4. <app_name>.CFG file in the <app_dir>

5. <app_name>.INI file in the <usr_dir>

6. <app_name>.CFG file in the <usr_dir>

7. <app_name>.INI file in the <cwd>

8. <app_name>.CFG file in the <cwd>

9. .sys_env.cfg in the <app_dir>

10. .sys_env<sys_env_id>.cfg in the <app_dir>

11. .app_env.cfg in the <app_dir>

12. .sys_env.cfg in the <usr_dir>

13. .sys_env<sys_env_id>.cfg in the <usr_dir>

14. .app_env.cfg in the <usr_dir>

15. .sys_env.cfg in the <cwd>

16. .sys_env<sys_env_id>.cfg in the <cwd>

17. .app_env.cfg in the <cwd>

18. .sys_env.cfg in the parent folder of the <cwd>

19. .sys_env<sys_env_id>.cfg in the parent folder of the <cwd>

20. .app_env.cfg in the parent folder of the <cwd>

21. .sys_env.cfg in the parent folder of the parent folder of the <cwd>

22. .sys_env<sys_env_id>.cfg in the parent folder of the parent folder of the <cwd>

23. .app_env.cfg in the parent folder of the parent folder of the <cwd>

24. value argument passed into the add_opt() method call (defining the option)

25. default_value argument passed into this method (only if add_option didn’t get called)

legend of the placeholders in the above search order lists (see also ae.paths.PATH_PLACEHOLDERS):

• <cwd> is the current working directory of your application (determined with os.getcwd())

• <app_name> is the base app name without extension of your main python code file.

• <app_dir> is the application data directory (APPDATA/<app_name> in Windows, ~/.config/<app_name> in Linux).

• <usr_dir> is the user data directory (APPDATA in Windows, ~/.config in Linux).

• <sys_env_id> is the specified argument of ConsoleApp.__init__().

cfg_section_variable_names(section, cfg_parser=None)

determine current config variable names/keys of the passed config file section.

Parameters:

• section (str) – config file section name.

• cfg_parser (Optional[ConfigParser]) – ConfigParser instance to use (def=self._cfg_parser).

Return type:

Tuple[str, ...]

Returns:

tuple of all config variable names.

_get_cfg_parser_val(name, section, default_value=None, cfg_parser=None)

determine thread-safe the value of a config variable from the config file.

Parameters:

• name (str) – name/option_id of the config variable.

• section (str) – name of the config section.

• default_value (Optional[Any]) – default value to return if config value is not specified in any config file.

• cfg_parser (Optional[ConfigParser]) – ConfigParser instance to use (def=self._cfg_parser).

Return type:

Any

load_cfg_files(config_modified=True)

(re)load and parse all config files.

Parameters:

config_modified (bool) – pass False to prevent the refresh/overwrite the initial config file modified date.

is_main_cfg_file_modified()

determine if main config file got modified.

Return type:

bool

Returns:

True if the content of the main config file got modified/changed.

get_variable(name, section=None, default_value=None, cfg_parser=None, value_type=None)

get value of config option, OS environ or config variable.

Parameters:

• name (str) – name of a config option or of an existing/declared config variable.

• section (Optional[str]) – name of the config section. defaulting to the app options section (MAIN_SECTION_NAME) if not specified or if None or empty string passed.

• default_value (Optional[Any]) – default value to return if config value is not specified in any config file.

• cfg_parser (Optional[ConfigParser]) – optional ConfigParser instance to use (def= _cfg_parser).

• value_type (Optional[Type]) – optional type of the config value. only used for config variables and ignored for config options.

Return type:

Any

Returns:

variable value which will be searched in the config options, the OS environment and in the config variables in the following order and manner:

• config option with a name equal to the name argument (only if the passed section value is either empty, None or equal to MAIN_SECTION_NAME).

• OS environment variable with a matching snake+upper-cased name, compiled from the section and name arguments, separated by an underscore character.

• config variable with a name and section equal to the values passed into the name and section arguments.

if no variable could be found then a None value will be returned.

this method has an alias named get_var().

get_var(name, section=None, default_value=None, cfg_parser=None, value_type=None)

alias of method get_variable()

Return type:

Any

set_variable(name, value, cfg_fnam=None, section=None, old_name='')

set/change the value of a config variable and if exists the related config option.

if the passed string in name is the id of a defined config option and section is either empty or equal to the value of MAIN_SECTION_NAME then the value of this config option will be changed too.

if the section does not exist it will be created (in contrary to Pythons ConfigParser).

Parameters:

• name (str) – name/option_id of the config value to set.

• value (Any) – value to assign to the config value, specified by the name argument.

• cfg_fnam (Optional[str]) – file name (def= _main_cfg_fnam) to save the new option value to.

• section (Optional[str]) – name of the config section. defaulting to the app options section (MAIN_SECTION_NAME) if not specified or if None or empty string passed.

• old_name (str) – old name/option_id that has to be removed (used to rename config option name/key).

Return type:

str

Returns:

empty string on success else error message text.

this method has an alias named set_var().

set_var(name, value, cfg_fnam=None, section=None, old_name='')

alias of method set_variable()

Return type:

str

add_argument(*args, **kwargs)

define new command line argument.

original/underlying args/kwargs of argparse.ArgumentParser are used - please see the description/definition of add_argument().

this method has an alias named add_arg().

add_arg(*args, **kwargs)

alias of method add_argument()

get_argument(name)

determine the command line parameter value.

Parameters:

name (str) – argument id of the parameter.

Return type:

Any

Returns:

value of the parameter.

this method has an alias named get_arg().

get_arg(name)

alias of method get_argument()

Return type:

Any

add_option(name, desc, value, short_opt='', choices=None, multiple=False)

defining and adding a new config option for this app.

Parameters:

• name (str) – string specifying the option id and short description of this new option. the name value will also be available as long command line argument option (case-sens.).

• desc (str) – description and command line help string of this new option.

• value (Any) – default value and type of the option. returned by get_option() if this option is not specified as command line argument nor exists as config variable in any config file. pass UNSET to define a boolean flag option, specified without a value on the command line. the resulting value will be True if the option will be specified on the command line, else False. specifying a value on the command line results in a SystemExit on parsing.

• short_opt (Union[str, UnsetType]) – short option character. if not passed or passed as ‘’ then the first character of the name will be used. passing UNSET or None prevents the declaration of a short option. please note that the short options ‘h’, ‘D’ and ‘L’ are already used internally by the classes ArgumentParser and ConsoleApp.

• choices (Optional[Iterable]) – list of valid option values (optional, default=allow all values).

• multiple (bool) – True if option can be added multiple times to command line (optional, default=False).

the value of a config option can be of any type and gets represented by an instance of the Literal class. supported value types and literals are documented here.

this method has an alias named add_opt().

add_opt(name, desc, value, short_opt='', choices=None, multiple=False)

alias of method add_option()

_change_option(name, value)

change config option and any references to it.

get_option(name, default_value=None)

determine the value of a config option specified by its name (option id).

Parameters:

• name (str) – name/id of the config option.

• default_value (Optional[Any]) – default value of the option (if not defined with add_option).

Return type:

Any

Returns:

first found value of the option identified by name. the returned value has the same type as the value specified in the add_option() call. if not given on the command line, then it gets search next in default config section (MAIN_SECTION_NAME) of the collected config files (the exact search order is documented in the doc-string of the method add_cfg_files()). if not found in the config file then the default value specified of the option definition (the add_option() call) will be used. the other default value, specified in the default_value kwarg of this method, will be returned only if the option name/id never got defined.

this method has an alias named get_opt().

get_opt(name, default_value=None)

alias of method get_option()

Return type:

Any

set_option(name, value, cfg_fnam=None, save_to_config=True)

set or change the value of a config option.

Parameters:

• name (str) – id of the config option to set.

• value (Any) – value to assign to the option, identified by name.

• cfg_fnam (Optional[str]) – config file name to save new option value. if not specified then the default file name of set_variable() will be used.

• save_to_config (bool) – pass False to prevent to save the new option value also to a config file. the value of the config option will be changed in any case.

Return type:

str

Returns:

‘’/empty string on success else error message text.

this method has an alias named set_opt().

set_opt(name, value, cfg_fnam=None, save_to_config=True)

alias of method set_option()

Return type:

str

parse_arguments()

parse all command line args.

this method get normally only called once and after all the options have been added with add_option(). add_option() will then set the determined config file value as the default value and then the following call of this method will overwrite it with command line argument value, if given.

property user_id

id of the user of this app.

load_user_cfg()

load users configuration.

register_user(**user_data)

register the current user and create/copy a new set of user specific config vars.

Parameters:

user_data – user data dict.

Note

this method will overwrite an existing user with the same user id, with the passed user data and the config variable values of the current/default user.

user_section(section, name)

return the user section name if the passed (section, name) setting id is user-specific.

Parameters:

• section (str) – section name.

• name (str) – variable name.

Return type:

str

Returns:

passed section name or user-specific section name.

app_env_dict()

collect run-time app environment data and settings - for app logging and debugging.

Return type:

Dict[str, Any]

Returns:

dict with app environment data/settings.

run_app()

prepare app run. call after definition of command line arguments/options and before run of app code.

show_help()

print help message, listing defined command line args and options, to console output/stream.

includes command line args defined with add_argument(), options defined with add_option() and the args/kwargs defined with the respective ArgumentParser methods (see description/definition of print_help() of ArgumentParser).