ae.paths

generic file path helpers

this pure python namespace portion is providing generic file paths together with useful helper functions and classes that are independent from the operating system. the currently support operating systems are:

  • android OS

  • iOS

  • linux

  • MacOS

  • Windows

the classes Collector and FilesRegister collect and group available files. Collector is more for temporary quick searches, while the class FilesRegister is used to permanent registers in which you later can find the best fitting match for a requested purpose - useful for dynamic selection of image/font/audio/… resource files that have to be selected depending on the current user preferences, hardware and/or software environment.

this portion is also encapsulating helper functions from Python’s shutil module as aliases (see copy_file(), copy_tree(), move_file() and move_tree()).

the only external hard dependency of this module are the ae namespace portions ae.base and ae.files. optional dependencies are:

generic system paths

some generic system paths are determined by the following helper functions:

these system paths together with additional generic paths like e.g. the current working directory, as well as file path parts (like e.g. the user or application name) are provided as path placeholders, which are stored within the PATH_PLACEHOLDERS dict.

the helper function norm_path() converts path strings containing path placeholders into regular path strings. path_name() and placeholder_paths() are converting regular path strings or parts of it into path placeholders.

by calling the function add_common_storage_paths() all storage paths provided by the plyer package will be added to the path placeholders (the PATH_PLACEHOLDERS dict).

path helper functions

the function move_tree() is moving entire directory trees (including their files and sub-folders), whereas move_files() are only moving the files from a directory tree to another location.

with path_files() you can easily determine the files within a folder structure that are matching the specified wildcards and path part placeholders (provided by PATH_PLACEHOLDERS).

file/folder collection and classification

more specific path examinations on the files and sub-folders of a file path can be done with the Collector and FilesRegister classes.

collecting files

Collector is scanning multiple paths for file names. the following example is using Collector to collect the files with the name xxx.cfg in the current working directory, in the folder above the application data folder, and in a folder with the name of the main application underneath the user data folder:

coll = Collector()
coll.collect("{cwd}", "{app}/..", "{usr}/{main_app_name}", append="xxx" + CFG_EXT)
found_files = coll.files

to add or overwrite the generic path placeholder parts values of the main application name ({main_app_name}) and the application data path ({app}) you simply specify them in the construction of the Collector instance:

coll = Collector(main_app_name=..., app=...)

additionally you can specify any other path placeholders that will be automatically used and replaced by the Collector instance:

coll = Collector(any_other_placeholder=...)

all found files are provided by the Collector.files instance attribute. found folders will be separately collected within the Collector instance attribute paths.

by default only the found file(s)/folder(s) of the first combination will be collected. to collect all files instead, pass an empty tuple to the only_first_of():

coll.collect(..., only_first_of=())

add one of the strings ‘prefix’, ‘append’ or ‘select’ to the only_first_of tuple argument to collect only the files/folders of the first combination of the specified prefixes, append-suffixes and select-suffixes.

the following example determines all folders underneath the current working directory with a name that contains the string ‘xxx’ or is starting with ‘yyy’ or is ending with ‘zzz’:

coll = Collector()
coll.collect('{cwd}', append=('*xxx*', 'yyy*', '*zzz'))
folders = coll.paths

by using the select argument the found files and folders will additionally be collected in the Collector instance attribute selected.

the combinations compiled via the select argument that are not existing will be counted. the results are provided by the instance attributes failed, prefix_failed and suffix_failed.

the select argument can also be passed together with the append argument.

multiple calls of collect() are accumulating found files and folders to the respective instance attributes:

coll = Collector()
coll.collect(...)
coll.collect(...)
...
files = coll.files
folders = coll.paths
items = coll.selected

files register

a files register is an instance of the FilesRegister, providing property based file collection and selection, which is e.g. used by the ae.gui_app ae namespace portion to find and select resource files like icon/image or sound files.

files can be collected from various places by a single instance of the class FilesRegister:

from ae.files import FilesRegister

fr = FilesRegister('first/path/to/collect')
fr.add_paths('second/path/to/collect/files/from')

registered_file = fr.find_file('file_name')

in this example the FilesRegister instance collects all files that are existing in any sub-folders underneath the two provided paths. then the find_file() method will return a file object of type RegisteredFile of the last found file with the base name (or stem) file_name.

several files with the same base name can be collected and registered e.g. with different formats, to be selected by the app by their different properties. assuming your application is providing an icon image in two sizes, provided within the following directory structure:

resources/
    size_72/
        app_icon.jpg
    size_150/
        app_icon.png

first create an instance of FilesRegister to collect both image files from the resources folder:

fr = FilesRegister('resources')

the resulting object fr behaves like a dict object, where the item key is the file base name without extension (‘app_icon’) and the item value is a list of instances of RegisteredFile. both files in the resources folder are provided as one dict item:

assert 'app_icon' in fr
assert len(fr) == 1
assert len(fr['app_icon']) == 2
assert isinstance(fr['app_icon'][0], RegisteredFile)

to select the appropriate image file you can use the find_file() method:

app_icon_image_path = fr.find_file('app_icon', dict(size=current_size))

as a shortcut you can alternatively call the object directly (leaving .find_file away):

app_icon_image_path = fr('app_icon', dict(size=current_size))

if the current_size variable contains the integer 150, then app_icon_image_path will result in “resources/size_150/app_icon.png”. in contrary if the current_size variable contains the integer 72, then app_icon_image_path will result in “resources/size_72/app_icon.png”.

for more complex selections you can use callables passed into the property_matcher and file_sorter arguments of find_file().

Module Attributes

APPEND_TO_END_OF_FILE_LIST

special flag default value for the first_index argument of the add_* methods of FilesRegister to append new file objects to the end of the name’s register file object list.

INSERT_AT_BEGIN_OF_FILE_LIST

special flag default value for the first_index argument of the add_* methods of FilesRegister to insert new file objects always at the begin of the name’s register file object list.

PATH_PLACEHOLDERS

placeholders of user-, os- and app-specific system paths and file name parts

Functions

add_common_storage_paths()

add common storage paths to PATH_PLACEHOLDERS depending on the operating system (OS).

app_data_path()

determine the os-specific absolute path of the {app} directory where user app data can be stored.

app_docs_path()

determine the os-specific absolute path of the {ado} directory where user documents app are stored.

copy_files(src_folder, dst_folder[, …])

copy files from src_folder into optionally created dst_folder, optionally overwriting destination files.

move_files(src_folder, dst_folder[, overwrite])

move files from src_folder into optionally created dst_folder, optionally overwriting destination files.

norm_path(path)

normalize/transform path replacing PATH_PLACEHOLDERS and the tilde character (for home folder).

path_files(file_mask[, recursive, file_class])

determine existing file(s) underneath the folder specified by path.

path_folders(folder_mask[, recursive, …])

determine existing folder(s) underneath the folder specified by path.

path_items(item_mask[, recursive, selector, …])

determine existing file/folder item(s) underneath the folder specified by path.

path_name(path)

determine placeholder key name of the specified path.

placeholder_key(path)

determine PATH_PLACEHOLDERS key of specified path.

placeholder_path(path)

replace begin of path string with the longest prefix found in PATH_PLACEHOLDERS.

series_file_name(file_path[, digits, …])

determine non-existent series file name with an unique series index.

user_data_path()

determine the os-specific absolute path of the {usr} directory where user data can be stored.

user_docs_path()

determine the os-specific absolute path of the {doc} directory where the user is storing the personal documents.

Classes

Collector([path_scanner])

file/folder collector

FilesRegister(*add_path_args[, …])

file register catalog - see also files register examples.

APPEND_TO_END_OF_FILE_LIST = 9223372036854775807

special flag default value for the first_index argument of the add_* methods of FilesRegister to append new file objects to the end of the name’s register file object list.

INSERT_AT_BEGIN_OF_FILE_LIST = -9223372036854775807

special flag default value for the first_index argument of the add_* methods of FilesRegister to insert new file objects always at the begin of the name’s register file object list.

add_common_storage_paths()[source]

add common storage paths to PATH_PLACEHOLDERS depending on the operating system (OS).

the following storage paths are provided by the plyer PyPi package (not all of them are available in each OS):

  • application: user application directory.

  • documents: user documents directory.

  • downloads: user downloads directory.

  • external_storage: external storage root directory.

  • home: user home directory.

  • music: user music directory.

  • pictures: user pictures directory.

  • root: root directory of the operating system partition.

  • sdcard: SD card root directory (only available in Android if sdcard is inserted).

  • videos: user videos directory.

additionally storage paths that are only available on certain OS (inspired by the method get_drives, implemented in https://github.com/kivy-garden/filebrowser/blob/master/kivy_garden/filebrowser/__init__.py):

  • Linux: external storage devices/media mounted underneath the system partition root in /mnt or /media.

  • Apple Mac OsX or iOS: external storage devices/media mounted underneath the system partition root in /Volume.

  • MS Windows: additional drives mapped as the drive partition name.

app_data_path()[source]

determine the os-specific absolute path of the {app} directory where user app data can be stored.

Hint

use app_docs_path() instead to get a more public path to the user.

Return type

str

Returns

path string of the user app data folder.

app_docs_path()[source]

determine the os-specific absolute path of the {ado} directory where user documents app are stored.

Hint

use app_data_path() instead to get a more hidden path to the user.

Return type

str

Returns

path string of the user documents app folder.

copy_file(src, dst, *, follow_symlinks=True)

alias for shutil.copy2() (compatible to shutil.copy(), shutil.copyfile() and ae.files.copy_bytes().

copy_tree(src, dst, symlinks=False, ignore=None, copy_function=<function copy2>, ignore_dangling_symlinks=False)

alias for shutil.copytree().

move_file(src, dst, copy_function=<function copy2>)

alias for shutil.move() (see also move_tree()).

copy_files(src_folder, dst_folder, overwrite=False, copier=<function copy2>)[source]

copy files from src_folder into optionally created dst_folder, optionally overwriting destination files.

Parameters
  • src_folder (str) – path to source folder/directory where the files get copied from. placeholders in PATH_PLACEHOLDERS will be recognized and substituted.

  • dst_folder (str) – path to destination folder/directory where the files get copied to. all placeholders in PATH_PLACEHOLDERS are recognized and will be substituted.

  • overwrite (bool) – pass True to overwrite existing files in the destination folder/directory. on False the files will only get copied if they not exist in the destination.

  • copier (Callable) – copy/move function with src_file and dst_file parameters, returning file path/name.

Return type

List[str]

Returns

list of copied files, with their destination path.

move_files(src_folder, dst_folder, overwrite=False)[source]

move files from src_folder into optionally created dst_folder, optionally overwriting destination files.

Parameters
  • src_folder (str) – path to source folder/directory where the files get moved from. placeholders in PATH_PLACEHOLDERS will be recognized and substituted. please note that the source folders itself will neither be moved nor removed (but will be empty after the operation finished).

  • dst_folder (str) – path to destination folder/directory where the files get moved to. all placeholders in PATH_PLACEHOLDERS are recognized and will be substituted.

  • overwrite (bool) – pass True to overwrite existing files in the destination folder/directory. on False the files will only get moved if they not exist in the destination.

Return type

List[str]

Returns

list of moved files, with their destination path.

move_tree(src, dst, copy_function=<function copy2>)

another alias for shutil.move() (see also move_file()).

norm_path(path)[source]

normalize/transform path replacing PATH_PLACEHOLDERS and the tilde character (for home folder).

Parameters

path (str) – path string to normalize/transform.

Return type

str

Returns

normalized path string.

path_files(file_mask, recursive=True, file_class=<class 'str'>, **file_kwargs)[source]

determine existing file(s) underneath the folder specified by path.

Parameters
  • file_mask (str) – glob file mask (with optional glob wildcards and PATH_PLACEHOLDERS) specifying the files to collect (by default including the sub-folders).

  • recursive (bool) – pass False to only collect the given folder (ignoring sub-folders).

  • file_class (Union[Type[Any], Callable]) – factory used for the returned list items (see path_items.creator).

  • file_kwargs – additional/optional kwargs apart from file name passed onto the used item_class.

Return type

List[Any]

Returns

list of files of the class specified by item_class.

path_folders(folder_mask, recursive=True, folder_class=<class 'str'>, **folder_kwargs)[source]

determine existing folder(s) underneath the folder specified by path.

Parameters
  • folder_mask (str) – glob folder mask (with optional glob wildcards and PATH_PLACEHOLDERS) specifying the folders to collect (by default including the sub-folders).

  • recursive (bool) – pass False to only collect the given folder (ignoring sub-folders).

  • folder_class (Union[Type[Any], Callable]) – class or factory used for the returned list items (see creator).

  • folder_kwargs – additional/optional kwargs apart from file name passed onto the used item_class.

Return type

List[Any]

Returns

list of folders of the class specified by item_class.

path_items(item_mask, recursive=True, selector=<function exists>, creator=<class 'str'>, **creator_kwargs)[source]

determine existing file/folder item(s) underneath the folder specified by path.

Parameters
  • item_mask (str) – file path mask (with optional glob wildcards and PATH_PLACEHOLDERS) specifying the files/folders to collect (by default including the sub-folders).

  • recursive (bool) – pass False to only collect within the specified folder (ignoring sub-folders).

  • selector (Callable[[str], bool]) – called with each found file/folder name to check if it has to be added to the returned list.

  • creator (Union[Type[Any], Callable]) – each found file/folder will be passed as argument to this class/callable and the instance/return-value will be appended as an item to the returned item list. if not passed then the str class will be used, which means that the items of the returned list will be strings of the file/folder path and name. if a class, like e.g. ae.files.CachedFile, ae.files.CachedFile or pathlib.Path, get passed then the items will be instances of this class. alternatively you can pass a callable which will be called on each found file/folder. in this case the return value of the callable will be inserted in the related item of the returned list.

  • creator_kwargs – additional/optional kwargs passed onto the used item_class apart from the item name.

Return type

List[Any]

Returns

list of found and selected items of the item class (item_class).

path_name(path)[source]

determine placeholder key name of the specified path.

Parameters

path (str) – path string to determine name of (can contain placeholders).

Return type

str

Returns

name (respectively dict key in PATH_PLACEHOLDERS) of the found path or empty string if not found.

placeholder_key(path)[source]

determine PATH_PLACEHOLDERS key of specified path.

Parameters

path (str) – path string starting with a PATH_PLACEHOLDERS path prefix.

Return type

str

Returns

placeholder key (if found as path prefix), else empty string.

placeholder_path(path)[source]

replace begin of path string with the longest prefix found in PATH_PLACEHOLDERS.

Parameters

path (str) – path string (optionally including sub-folders and file name).

Return type

str

Returns

path string with replaced placeholder prefix (if found).

series_file_name(file_path, digits=2, marker=' ', create=False)[source]

determine non-existent series file name with an unique series index.

Parameters
  • file_path (str) – file path and name (optional with extension).

  • digits (int) – number of digits used for the series index.

  • marker (str) – marker that will be put at the end of the file name and before the series index.

  • create (bool) – pass True to create the file (to reserve the series index).

Return type

str

Returns

file path extended with unique/new series index.

user_data_path()[source]

determine the os-specific absolute path of the {usr} directory where user data can be stored.

Hint

this path is not accessible on Android devices, use user_docs_path() instead to get a more public path to the user.

Return type

str

Returns

path string of the user data folder.

user_docs_path()[source]

determine the os-specific absolute path of the {doc} directory where the user is storing the personal documents.

Hint

use user_data_path() instead to get a more hidden user data.

Return type

str

Returns

path string of the user documents folder.

PATH_PLACEHOLDERS = {'ado': '/home/docs/Documents/docs', 'app': '/home/docs/.config/docs', 'app_name': 'docs', 'cwd': '/home/docs/checkouts/readthedocs.org/user_builds/ae/checkouts/stable/docs', 'doc': '/home/docs/Documents', 'usr': '/home/docs/.config'}

placeholders of user-, os- and app-specific system paths and file name parts

class Collector(path_scanner=<function path_files>, **placeholders)[source]

Bases: object

file/folder collector

__init__(path_scanner=<function path_files>, **placeholders)[source]

create new file/folder/item collector instance with individual (extended or overriding) placeholders.

Parameters
  • path_scanner (Callable[[str], Iterable]) – callable to determine the item type to collect. the default is the path_files() function. pass e.g. path_folders() to collect only folders or path_items() to collect both (files and folders).

  • placeholdersformat kwargs where keys are the placeholders and the values the replacements. the placeholders provided by PATH_PLACEHOLDERS are available too (but will be overwritten by these arguments).

paths: List[str]

list of found/collected folder names

files: List[str]

list of found/collected file names

selected: List[str]

list of found/collected file/folder item names

failed

number of not found select-combinations

prefix_failed: Dict[str, int]

number of not found select-combinations for each prefix

suffix_failed: Dict[str, int]

number of not found select-combinations for each suffix

placeholders

path part placeholders of this Collector instance

check_add(name, select=False)[source]

check if name file or folder and if yes append accordingly to instance lists else do nothing.

Parameters
  • name (str) – file/folder name, optionally including wildcards in the glob.glob format.

  • select (bool) – pass True to add found files/folders into selected.

Return type

bool

Returns

True if at least one file/folder got found/added, else False.

_collect_appends(prefix, appends, only_first_of)[source]
_collect_selects(prefix, selects, only_first_of)[source]
collect(*prefixes, append=(), select=(), only_first_of=('append', 'prefix', 'select'))[source]

collect additional files/folders by combining the given prefixes with all the given append/select suffixes.

Parameters
  • prefixes (str) – tuple of file/folder paths to be used as prefix.

  • append (Union[str, Tuple[str, …]]) – tuple of file/folder names to be used as append suffix.

  • select (Union[str, Tuple[str, …]]) – tuple of file/folder names to be used as select suffix.

  • only_first_of (Union[str, Tuple[str, …]]) – tuple with the strings ‘prefix’, ‘append’ or ‘select’ or one of these strings. if it contains the string ‘prefix’ then only the files/folders of the first combination will be collected. if it contains ‘append’ then only the files/folders of the first combination done with the suffixes passed into the append argument will be collected. if it contains ‘select’ then only the files/folders of the first combination done with the suffixes passed into the select argument will be collected. pass empty tuple to collect all combinations.

each of the passed prefixes will be combined with the suffixes specified in append and in select. the resulting file/folder paths that are exist, will then be added to the appropriate instance attribute, either files for a file or paths for a folder.

additionally the existing file/folder paths from the combinations of prefixes and select will be added in the selected list attribute.

all arguments of this method can either be passed either as tuples or as a single string value.

Hint

more details and some examples are available in the doc string of this module.

class FilesRegister(*add_path_args, property_matcher=None, file_sorter=None, **add_path_kwargs)[source]

Bases: dict

file register catalog - see also files register examples.

__init__(*add_path_args, property_matcher=None, file_sorter=None, **add_path_kwargs)[source]

create files register instance.

this method gets redirected with add_path_args and add_path_kwargs arguments to add_paths().

Parameters
__call__(*find_args, **find_kwargs)[source]

add_path_args and kwargs will be completely redirected to find_file().

Return type

Union[str, RegisteredFile, CachedFile, Path, PurePath, Any, None]

add_file(file_obj, first_index=9223372036854775807)[source]

add a single file to the list of this dict mapped by the file-name/stem as dict key.

Parameters
  • file_obj (Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]) – either file path string or any object with a stem attribute.

  • first_index (int) – pass list index -n-1..n-1 to insert file_obj in the name’s list. values greater than n (==len(file_list)) will append the file_obj to the end of the file object list and values less than n-1 will insert the file_obj to the begin.

add_files(files, first_index=9223372036854775807)[source]

add files from another FilesRegister instance.

Parameters
  • files (Iterable[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]]) – iterable with file objects to be added.

  • first_index (int) – pass list index -n-1..n-1 to insert the first file_obj in each name’s register list. values greater than n (==len(file_list)) will append the file_obj to the end of the file object list. the order of the added items will be unchanged if this value is greater or equal to zero. negative values will add the items from files in reversed order and after the item specified by this index value (so passing -1 will append the items to the end in reversed order, while passing -(n+1) will insert them at the begin in reversed order).

Return type

List[str]

Returns

list of paths of the added files.

add_paths(*file_path_masks, recursive=True, first_index=9223372036854775807, file_class=<class 'ae.files.RegisteredFile'>, **init_kwargs)[source]

add files found in the folder(s) specified by the file_path_masks args.

Parameters
  • file_path_masks (str) – file path masks (with optional wildcards and PATH_PLACEHOLDERS) specifying the files to collect (by default including the sub-folders).

  • recursive (bool) – pass False to only collect the given folder (ignoring sub-folders).

  • first_index (int) – pass list index -n-1..n-1 to insert the first file_obj in each name’s register list. values greater than n (==len(file_list)) will append the file_obj to the end of the file object list. the order of the added items will be unchanged if this value is greater or equal to zero. negative values will add the found items in reversed order and after the item specified by this index value (so passing -1 will append the items to the end in reversed order, while passing -(n+1) will insert them at the begin in reversed order).

  • file_class (Type[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]]) – the used file object class (see FileObject). each found file object will passed to the class constructor (callable) and added to the list which is a item of this dict.

  • init_kwargs – additional/optional kwargs passed onto the used file_class. pass e.g. the object_loader to use, if file_class is CachedFile (instead of the default: RegisteredFile).

Return type

List[str]

Returns

list of paths of the added files.

add_register(files_register, first_index=9223372036854775807)[source]

add files from another FilesRegister instance.

Parameters
  • files_register (FilesRegister) – files register instance containing the file_obj to be added.

  • first_index (int) – pass list index -n-1..n-1 to insert the first file_obj in each name’s register list. values greater than n (==len(file_list)) will append the file_obj to the end of the file object list. the order of the added items will be unchanged if this value is greater or equal to zero. negative values will add the found items in reversed order and after the item specified by this index value (so passing -1 will append the items to the end in reversed order, while passing -(n+1) will insert them at the begin in reversed order).

Return type

List[str]

Returns

list of paths of the added files.

find_file(name, properties=None, property_matcher=None, file_sorter=None)[source]

find file_obj in this register via properties, property matcher callables and/or file sorter.

Parameters
Return type

Union[str, RegisteredFile, CachedFile, Path, PurePath, Any, None]

Returns

registered/cached file object of the first found/correct file.

reclassify(file_class=<class 'ae.files.CachedFile'>, **init_kwargs)[source]

re-instantiate all name’s file registers items to instances of the class file_class.

Parameters
  • file_class (Type[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]]) – the new file object class (see FileObject). each found file object will passed to the class constructor (callable) and the return value will then replace the file object in the file list.

  • init_kwargs – additional/optional kwargs passed onto the used file_class. pass e.g. the object_loader to use, if file_class is CachedFile (the default file object class).