ae.paths

generic file path helpers

this pure python namespace portion is providing useful path helper functions as well as generic system paths for most platforms, like e.g.:

Android OS

iOS

Linux

Mac OS X

MS Windows

the only external hard dependencies of this module are the ae namespace portions ae.base and ae.files. optional dependencies are:

on android OS the PyPi package jnius, needed by the functions user_data_path() and user_docs_path().

the plyer PyPi package, needed by the function add_common_storage_paths().

path helper functions

the generator functions coll_files(), coll_folders() and coll_items() are building the fundament for most of the file and folder collection functionality, provided by this module.

the function path_files() is using these generators to determine the files within a folder structure that are matching the specified wildcards and path part placeholders. similarly the function path_folders() for folders, and path_items() to collect both, file and folder names.

use the functions copy_files() and move_files() to duplicate and move multiple files or entire file path trees. these two functions are based on copy_file() and move_file(). the functions copy_tree() and move_tree() provide an alternative way to copy or move entire directory trees.

the helper function normalize() converts path strings containing path placeholders into regular path strings, resolving symbolic links, or is converting a path string from absolute paths to relative paths and vice versa.

to determine if the path of a file or folder is matching a glob-like path pattern/mask with wildcards, the functions path_match() and paths_match() can be used. useful especially for cases where you don’t have direct access to the file system.

file paths for series of files, e.g., for logging, can be determined via the series_file_name() function.

the function skip_py_cache_files() can be used in path file collections to skip the files situated in the Python cache folder (PY_CACHE_FOLDER respectively __pycache__).

generic system paths

generic system paths are determined by the following helper functions:

app_data_path(): the application data path.
app_docs_path(): the application documents path.
user_data_path(): the user data path.
user_docs_path(): the user documents path.

these system paths together with additional generic paths like e.g., the current working directory, storage paths provided by the plyer package, or user and application paths, are provided as path placeholders, which get stored within the PATH_PLACEHOLDERS dict by calling the function add_common_storage_paths().

path_name() and placeholder_path() are converting regular path strings or parts of it into path placeholders.

file/folder collection and classification

more complex collections of files and folder paths, and the grouping of them, can be done with the classes Collector, described in the underneath section collecting files, and FilesRegister, described in the section file register.

use the Collector class for temporary quick file path searches on your local file systems as well as on remote servers/hosts. one implementation example is e.g., the method deployed_code_files() of the ae.pythonanywhere module.

the class FilesRegister helps you to create and cache file path registers permanently, to quickly find at any time the best fitting match for a requested purpose. for example, the gui portion is using it to dynamically select image/font/audio/… resource files depending on the current user preferences, hardware and/or software environment.

collecting files

to collect file names in the current working directory, create an instance of the Collector class and call its collect() method with a file or folder path, which can contain wildcards:

from ae.paths import Collector
coll = Collector()
coll.collect('*.png')
image_files_list = coll.files

after that a list containing the found file names can then be retrieved from the files attribute.

collect() can be called multiple times to accumulate and extend the files list:

coll = Collector()
coll.collect('*.png')
coll.collect('*.jpg')
image_files_list = coll.files

multiple calls of Collector.collect() can be joined into one code line, because it is returning its instance. the following statement is equivalent to the last example:

image_files_list = Collector().collect('*.png').collect('*.jpg').files

by specifying the ** wildcard entire folder trees can be scanned. the following example is a collection of all the files, including the hidden ones, in the folder tree under the current working directory:

test_files = Collector().collect('**/*').collect('**/.*').files

Hint

the second call of the Collector.collect() method in this example has to be done only if you are using Python’s glob.glob() as the searcher callback, which excludes hidden files (with a leading dot) from to match the * wildcard.

Collector is by default only returning files. to also collect folder paths in a deep folder tree, you have to pass the collecting generator function to the optional item_collector parameter of the Collector class. the accumulated files and folders can then be retrieved from their respective instance attributes files, paths and selected:

coll = Collector(item_collector=coll_items)
coll.collect(...)
...
files = coll.files
folders = coll.paths
file_and_folder_items = coll.selected

the found files are provided by the Collector.files instance attribute. found folders will be separately collected within the Collector instance attribute paths. the selected attribute contains all found files and folders in a single list.

collect from multiple locations

in a single call of collect(), providing the method parameters append or select, you can scan multiple combinations of path prefixes and suffixes, which can both contain wildcards and folder names, whereas the suffixes contain also parts of the file names to search for.

Hint

the wildcards *, ** and ? are allowed in the prefixes as well as in suffixes.

the resulting file paths are relative or absolute, depending on whether the specified prefix(es) contains absolute or relative paths.

in the following example determines the relative paths of all folders directly underneath the current working directory with a name that contains the string ‘xxx’ or is starting with ‘yyy’ or is ending with ‘zzz’:

coll = Collector(item_collector=coll_folders)
coll.collect('', append=('*xxx*', 'yyy*', '*zzz'))
folders = coll.paths

Hint

replace empty string in the first argument of collect() with ‘{cwd}’ to get absolute paths.

the following example is first collecting the absolute paths of files with the name xxx.cfg from all the found locations/folders. then it is starting to search in the current working directory ({cwd}), in the folder above the application data folder ({app}), and finally in a folder with the name of the main application underneath the user data folder ({usr}):

coll = Collector()
coll.collect("{cwd}", "{app}/..", "{usr}/{main_app_name}", append="xxx.cfg")
found_files = coll.files

to overwrite some of the generic path placeholder parts values only for a specific Collector instance, e.g., the main application name ({main_app_name}) and the application data path ({app}), you simply specify the changed values as kwargs in the construction of the Collector instance:

coll = Collector(main_app_name=..., app=...)

additionally, you can specify any other path placeholders that will be automatically used and replaced in the arguments of the collect() method for a so prepared Collector instance:

coll = Collector(any_other_placeholder=...)

by using the select argument, the found files and folders will additionally be collected in the Collector instance attribute selected.

not existing combinations collected via the select argument will be logged. the results are provided by the instance attributes failed, prefix_failed and suffix_failed.

file register

a file register is an instance of the FilesRegister, providing a property-based file collection and selection, which is e.g., used by the ae.gui ae namespace portion to find and select resource files like icon/image or sound files.

files can be collected from various places by a single instance of the class FilesRegister:

from ae.paths import FilesRegister

file_reg = FilesRegister('first/path/to/collect')
file_reg.add_paths('second/path/to/collect/files/from')

registered_file = file_reg.find_file('file_name')

in this example the FilesRegister instance collects all files that are existing in any subfolders underneath the two provided paths. then the find_file() method will return a RegisteredFile instance of the last collected file with the stem (base name w/o extension) ‘file_name’.

multiple files with the same stem can be collected and registered e.g., with different formats, to be selected by properties, which are specified in the folder names of the collected file paths.

for example, the following folder tree contains icon images in two sizes:

resources/
    size_72/
        app_icon.jpg
    size_150/
        app_icon.png

when you then create an instance of FilesRegister both image files underneath the resources folder will get registered, interpreting the subfolder names (size_*) as properties or attributes for the registered files, where size will result as the property name and the string after the underscore as the property value:

file_reg = FilesRegister('resources')

now, to retrieve the paths of the application image file with the size 72, call the find_file() method, specifying the property name(s) and value(s) as the second argument:

app_icon_image_path = file_reg.find_file('app_icon', dict(size=72))

the file path in app_icon_image_path will then result as “resources/size_72/app_icon.jpg”.

alternatively, and as a shortcut, you can call the instance object directly (leaving the explicit .find_file method away), like this:

app_icon_image_path = file_reg('app_icon', dict(size=150))

the resulting file path in app_icon_image_path of the last example will now result as “resources/size_150/app_icon.png”, because this time the size property value got specified as `150.

additionally, an instance of FilesRegister (file_reg) behaves like a dict object, where the item key is the file stem (‘app_icon’) and the item value is a list of instances of RegisteredFile. both files in the resources folder are provided as one dict item:

file_reg = FilesRegister('resources')
assert 'app_icon' in file_reg
assert len(file_reg) == 1
assert len(file_reg['app_icon']) == 2
assert isinstance(file_reg['app_icon'][0], RegisteredFile)

for more complex selections you can use callables passed into the property_matcher and file_sorter arguments of find_file().

Module Attributes

`APPEND_TO_END_OF_FILE_LIST`	special flag default value for the first_index argument of the add_* methods of `FilesRegister` to append new file objects to the end of the name's register file object list.
`INSERT_AT_BEGIN_OF_FILE_LIST`	special flag default value for the first_index argument of the add_* methods of `FilesRegister` to insert new file objects always at the begin of the name's register file object list.
`COLLECTED_FOLDER`	item type value for a folder\|directory\|node
`CollArgType`	argument type passed to most callable arguments of `coll_items()` except of `searcher`
`CollCreatorReturnType`	type of the return value of the `creator` callable, which will be returned to the caller
`CollYieldType`	type of collected item, which is either `COLLECTED_FOLDER` for a folder or the file extension for a file
`CollYieldItems`	type of collected item iterator yielding tuples of (CollYieldType, item[_path])
`SearcherRetType`	type of the return value of the callable `searcher` argument of `coll_items()`
`SearcherType`	type of the callable `searcher` argument of `coll_items()`
`copy_file`(src, dst, *[, follow_symlinks])	Copy data and metadata.
`copy_tree`(src, dst[, symlinks, ignore, ...])	Recursively copy a directory tree and return the destination directory.
`move_file`(src, dst[, copy_function])	Recursively move a file or directory to another location.
`move_tree`(src, dst[, copy_function])	Recursively move a file or directory to another location.
`PATH_PLACEHOLDERS`	placeholders dict of user-, os- and app-specific system paths and file name parts

Functions

`add_common_storage_paths`()	add common storage paths to `PATH_PLACEHOLDERS` depending on the operating system (OS).
`app_data_path`()	determine the os-specific absolute path of the {app} directory where user app data can be stored.
`app_docs_path`()	determine the os-specific absolute path of the {ado} directory where user documents app are stored.
`coll_files`(file_mask[, file_class])	determine existing file(s) underneath the folder specified by `file_mask`.
`coll_folders`(folder_mask[, folder_class])	determine existing folder(s) underneath the folder specified by `folder_mask`.
`coll_item_type`(item_path)	classify path item to be either file, folder or non-existent/excluded/skipped.
`coll_items`(item_mask[, searcher, selector, ...])	determine path-/file-like item(s) specified with optional wildcards by `item_mask`.
`copy_files`(src_folder, dst_folder[, ...])	copy files from src_folder into an optionally created dst_folder, optionally overwriting destination files.
`move_files`(src_folder, dst_folder[, overwrite])	move files from src_folder into an optionally created dst_folder, optionally overwriting destination files.
`normalize`(path[, make_absolute, ...])	normalize/transform a path replacing PATH_PLACEHOLDERS and the tilde character (for home folder).
`path_files`(file_mask[, file_class])	determine existing file(s) underneath the folder specified by `file_mask`.
`path_folders`(folder_mask[, folder_class])	determine existing folder(s) underneath the folder specified by `folder_mask`.
`path_items`(item_mask[, selector, creator])	determine existing file/folder item(s) underneath the folder specified by `item_mask`.
`path_join`(*parts)	join path parts preventing trailing path separator if last part is empty string.
`path_match`(path, mask)	return True if the specified path matches the specified path mask/pattern.
`path_name`(path)	determine the placeholder key name of the specified path.
`paths_match`(paths, masks)	filter the paths matching at least one of the specified glob-like wildcard masks.
`placeholder_key`(path)	determine `PATH_PLACEHOLDERS` key of the specified path.
`placeholder_path`(path)	replace the beginning of the specified path string with the longest prefix found in `PATH_PLACEHOLDERS`.
`relative_file_paths`(root_path, path_masks[, ...])	find all files underneath the specified root path, e.g. to collect the .py modules of a python package.
`series_file_name`(file_path[, digits, ...])	determine non-existent series file name with a unique series index.
`skip_py_cache_files`(file_path)	file exclusion callback for the files under Python's cache folders.
`user_data_path`()	determine the os-specific absolute path of the {usr} directory where user data can be stored.
`user_docs_path`()	determine the os-specific absolute path of the {doc} directory where the user is storing the personal documents.

Classes

`Collector`([item_collector])	file/folder collector class
`FilesRegister`(*add_path_args[, ...])	file register catalog - see also file register examples.

APPEND_TO_END_OF_FILE_LIST = 9223372036854775807: special flag default value for the first_index argument of the add_* methods of FilesRegister to append new file objects to the end of the name’s register file object list.

INSERT_AT_BEGIN_OF_FILE_LIST = -9223372036854775807: special flag default value for the first_index argument of the add_* methods of FilesRegister to insert new file objects always at the begin of the name’s register file object list.

COLLECTED_FOLDER = None: item type value for a folder|directory|node

CollArgType: argument type passed to most callable arguments of coll_items() except of searcher

CollCreatorReturnType: type of the return value of the creator callable, which will be returned to the caller

CollYieldType

type of collected item, which is either COLLECTED_FOLDER for a folder or the file extension for a file

alias of str | None

CollYieldItems

type of collected item iterator yielding tuples of (CollYieldType, item[_path])

alias of Iterable[tuple[str | None, Any]]

SearcherRetType

type of the return value of the callable searcher argument of coll_items()

alias of Iterable[Any]

SearcherType

type of the callable searcher argument of coll_items()

alias of Callable[[str], Iterable[Any]]

add_common_storage_paths()[source]

add common storage paths to PATH_PLACEHOLDERS depending on the operating system (OS).

the following storage paths are provided by the plyer PyPi package (not all of them are available in each OS):

{application}: user application directory.
{documents}: user documents directory.
{downloads}: user downloads directory.
{external_storage}: external storage root directory.
{home}: user home directory.
{music}: user music directory.
{pictures}: user pictures directory.
{root}: root directory of the operating system partition.
{sdcard}: SD card root directory (only available in Android if sdcard is inserted).
{videos}: user videos directory.

additionally, storage paths that are only available on certain OS (inspired by the method get_drives, implemented in https://github.com/kivy-garden/filebrowser/blob/master/kivy_garden/filebrowser/__init__.py):

Linux: external storage devices/media mounted underneath the system partition root in /mnt or /media.
Apple Mac OS X or iOS: external storage devices/media mounted underneath the system partition root in /Volume.
MS Windows: additional drives mapped as the drive partition name.

app_data_path()[source]

determine the os-specific absolute path of the {app} directory where user app data can be stored.

Hint

use app_docs_path() instead to get a more public path to the user.

Return type:: str
Returns:: path string of the user app data folder.

app_docs_path()[source]

determine the os-specific absolute path of the {ado} directory where user documents app are stored.

Hint

use app_data_path() instead to get a more hidden path to the user.

Return type:: str
Returns:: path string of the user documents app folder.

coll_files(file_mask, file_class=<class 'str'>, **file_kwargs)[source]

determine existing file(s) underneath the folder specified by file_mask.

Parameters:

file_mask¶ (str) – glob file mask (with optional glob wildcards and PATH_PLACEHOLDERS) specifying the files to collect (by default including the subfolders).
file_class¶ (Union[Type[Any], Callable]) – factory used for the returned list items (see coll_items.creator). silly mypy does not support Union[Type[Any], Callable[[str, KwArg()], Any]].
file_kwargs¶ – additional/optional kwargs apart from the file name passed onto the used item_class.

Return type:

Iterable[tuple[Optional[str], Any]]

Returns:

iterator/generator yielding a 2-item-tuple for each found/matching file. the first tuple-item is the file extension, and the second tuple-item is an instance of the specified file_class.

coll_folders(folder_mask, folder_class=<class 'str'>, **folder_kwargs)[source]

determine existing folder(s) underneath the folder specified by folder_mask.

Parameters:

folder_mask¶ (str) – glob folder mask (with optional glob wildcards and PATH_PLACEHOLDERS) specifying the folders to collect (by default including the subfolders).
folder_class¶ (Union[Type[Any], Callable]) – class or factory used for the returned list items (see coll_items.creator). silly mypy does not support Union[Type[Any], Callable[[str, KwArg()], Any]].
folder_kwargs¶ – additional/optional kwargs apart from the file name passed onto the used item_class.

Return type:

Iterable[tuple[Optional[str], Any]]

Returns:

iterator/generator yielding a 2-item-tuple for each found/matching folder/directory. the first tuple-item is COLLECTED_FOLDER (as long as folder_kwargs does not overwrite the coll_items.type_detector argument), and the second tuple-item is an instance of the specified folder_class.

coll_item_type(item_path)[source]

classify path item to be either file, folder or non-existent/excluded/skipped.

Parameters:: item_path¶ (str) – file/folder path string.
Return type:: Optional[str]
Returns:: COLLECTED_FOLDER for folders, the file extension for files or None if not found.

coll_items(item_mask, searcher=functools.partial(<function glob>, recursive=True), selector=<class 'str'>, type_detector=<function coll_item_type>, creator=<class 'str'>, **creator_kwargs)[source]

determine path-/file-like item(s) specified with optional wildcards by item_mask.

Parameters:

item_mask¶ (str) – file path mask with optional glob() wildcards, the “~” shortcut for the home folder path and any path placeholders, which is specifying the files/folders to collect. use the ‘**’ glob wildcard in a path to include also items from subfolders deeper than one level.
searcher¶ (Callable[[str], Iterable[Any]]) – callable to convert|resolve a path with optional wildcards, specified in item_mask, into multiple item file/folder path strings.
type_detector¶ (Callable[[Any], Optional[str]]) – callable to typify/classify a found item to be stored in the first tuple items of the returned list. if not passed, then coll_item_type() will be used.
selector¶ (Callable[[Any], Union[bool, Any]]) – called with each found file/folder name to check if it has to be added to the returned list. the default argument (str) results in returning every file/folder found by glob().
creator¶ (Callable[[Any], Any]) – each found file/folder will be passed as an argument to this class/callable, and the instance/return-value will be appended as an item to the returned item list. if not passed, then the str class will be used, which means that the items of the returned list will be strings of the file/folder path and name. passing a class, like e.g., ae.files.CachedFile, ae.files.CachedFile or pathlib.Path, will create instances of this class. alternatively, you can pass a callable which will be called on each found file/folder. in this case the return value of the callable will be inserted in the related item of the returned list. silly mypy does not support Union[Type[Any], Callable[[str, KwArg()], Any]].
creator_kwargs¶ – additional/optional kwargs passed onto the used item_class apart from the item name.

Return type:

Iterable[tuple[Optional[str], Any]]

Returns:

iterator/generator yielding a 2-item-tuple for each found/matching file system item. the first tuple-item is the file/folder type returned by the specified type_detector argument, and the second tuple-item is an instance of the item creator class (specified by the creator argument).

copy_file(src, dst, *, follow_symlinks=True): alias for shutil.copy2() (compatible to shutil.copy(), shutil.copyfile() and ae.files.copy_bytes().

copy_tree(src, dst, symlinks=False, ignore=None, copy_function=<function copy2>, ignore_dangling_symlinks=False, dirs_exist_ok=False): alias for shutil.copytree().

move_file(src, dst, copy_function=<function copy2>): alias for shutil.move() (see also move_tree()).

move_tree(src, dst, copy_function=<function copy2>): another alias for shutil.move() (see also move_file()).

copy_files(src_folder, dst_folder, overwrite=False, copier=<function copy2>)[source]

copy files from src_folder into an optionally created dst_folder, optionally overwriting destination files.

Parameters:

src_folder¶ (str) – path to the source folder / directory where the files get copied from. only the placeholders mapped in PATH_PLACEHOLDERS will be recognized and substituted.
dst_folder¶ (str) – path to the destination folder / directory where the files get copied to. all placeholders in PATH_PLACEHOLDERS are recognized and will be substituted.
overwrite¶ (bool) – pass True to overwrite existing files in the destination folder/directory. on False the files will only get copied if they do not exist in the destination.
copier¶ (Callable) – the copy/move function with src_file and dst_file parameters, returning file path/name.

Return type:

list[str]

Returns:

list of copied files, with their destination path.

move_files(src_folder, dst_folder, overwrite=False)[source]

move files from src_folder into an optionally created dst_folder, optionally overwriting destination files.

Parameters:

src_folder¶ (str) – path to the source folder / directory where the files get moved from. placeholders in PATH_PLACEHOLDERS will be recognized and substituted. please note that the source folder itself will neither be moved nor removed (but will be empty after the operation is finished).
dst_folder¶ (str) – path to the destination folder / directory where the files get moved to. all placeholders in PATH_PLACEHOLDERS are recognized and will be substituted.
overwrite¶ (bool) – pass True to overwrite existing files in the destination folder/directory. on False the files will only get moved if they do not exist in the destination.

Return type:

list[str]

Returns:

list of moved files, with their destination path.

normalize(path, make_absolute=True, remove_base_path='', remove_dots=True, resolve_sym_links=True)[source]

normalize/transform a path replacing PATH_PLACEHOLDERS and the tilde character (for home folder).

Parameters:

path¶ (str) – path string to normalize/transform.
make_absolute¶ (bool) – pass False to not convert the specified path to an absolute path.
remove_base_path¶ (str) – pass a valid base path to return a relative path, even if the argument values of make_absolute or resolve_sym_links are True.
remove_dots¶ (bool) – pass False to not replace/remove the . and .. placeholders.
resolve_sym_links¶ (bool) – pass False to not resolve symbolic links, passing True implies a True value also for the make_absolute argument.

Return type:

str

Returns:

normalized path string: absolute if remove_base_path is empty and either make_absolute or resolve_sym_links is True; relative if remove_base_path is a base path of path or if path got passed as a relative path and neither make_absolute nor resolve_sym_links is True.

path_files(file_mask, file_class=<class 'str'>, **file_kwargs)[source]

determine existing file(s) underneath the folder specified by file_mask.

Parameters:

file_mask¶ (str) – glob file mask (with optional glob wildcards and PATH_PLACEHOLDERS) specifying the files to collect (by default including the subfolders).
file_class¶ (Union[Type[Any], Callable]) – factory used for the returned list items (see path_items.creator). silly mypy does not support Union[Type[Any], Callable[[str, KwArg()], Any]].
file_kwargs¶ – additional/optional kwargs apart from the file name passed onto the used item_class.

Return type:

list[Any]

Returns:

list of files of the class specified by file_mask.

path_folders(folder_mask, folder_class=<class 'str'>, **folder_kwargs)[source]

determine existing folder(s) underneath the folder specified by folder_mask.

Parameters:

folder_mask¶ (str) – glob folder mask (with optional glob wildcards and PATH_PLACEHOLDERS) specifying the folders to collect (by default including the subfolders).
folder_class¶ (Union[Type[Any], Callable]) – class or factory used for the returned list items (see path_items.creator). silly mypy does not support Union[Type[Any], Callable[[str, KwArg()], Any]].
folder_kwargs¶ – additional/optional kwargs apart from the file name passed onto the used item_class.

Return type:

list[Any]

Returns:

list of folders of the class specified by folder_mask.

path_items(item_mask, selector=<class 'str'>, creator=<class 'str'>, **creator_kwargs)[source]

determine existing file/folder item(s) underneath the folder specified by item_mask.

Parameters:

item_mask¶ (str) – file path mask (with optional glob wildcards and PATH_PLACEHOLDERS) specifying the files/folders to collect.
selector¶ (Callable[[str], Any]) – called with each found file/folder name to check if it has to be added to the returned list. the default argument (str) results in returning every file/folder found by glob().
creator¶ (Union[Type[Any], Callable]) – each found file/folder will be passed as an argument to this class/callable, and the instance/return-value will be appended as an item to the returned item list. if not passed, then the str class will be used, which means that the items of the returned list will be strings of the file/folder path and name. passing a class, like e.g., ae.files.CachedFile, ae.files.CachedFile or pathlib.Path, will create instances of this class. alternatively, you can pass a callable which will be called on each found file/folder. in this case the return value of the callable will be inserted in the related item of the returned list. silly mypy does not support Union[Type[Any], Callable[[str, KwArg()], Any]].
creator_kwargs¶ – additional/optional kwargs passed onto the used item_class apart from the item name.

Return type:

list[Any]

Returns:

list of found and selected items of the item class (item_mask).

path_join(*parts)[source]

join path parts preventing trailing path separator if last part is empty string.

Parameters:: parts¶ (str) – path parts to join.
Return type:: str
Returns:: joined path string.

Hint

although os.path.join() is implemented in C, this function is faster:

import os
import timeit
from ae.paths import path_join
paths_secs = timeit.timeit('path_join("test", "sub_test", "sub_sub_test")', globals=globals())
os_secs = timeit.timeit('os.path.join("test", "sub_test", "sub_sub_test")', globals=globals())
assert paths_secs < os_secs

even if you import os.path.join() without the namespace prefixes, like this:

from os.path import join as path_join
os_secs = timeit.timeit('path_join("test", "sub_test", "sub_sub_test")', globals=globals())
assert paths_secs < os_secs

_path_match_replacement = re.compile('(/\\\\\\*\\\\\\*|\\\\\\*\\\\\\*/|\\\\\\*|\\\\\\?|\\\\\\[\\\\\\*\\\\\\]|\\\\\\[\\\\\\?\\\\\\]|\\\\\\[!|\\\\\\[|\\\\\\])'): pre-compiled regular expression for path_match(), inspired by the great SO answer of Pugsley (see https://stackoverflow.com/questions/27726545/63212852#63212852)

path_match(path, mask)[source]

return True if the specified path matches the specified path mask/pattern.

Parameters:

path¶ (str) – path string to match.
mask¶ (str) – path mask/pattern including glob-like wildcards.

Return type:

bool

Returns:

True if the path specified by path matches the mask/pattern specified by the mask argument.

path_name(path)[source]

determine the placeholder key name of the specified path.

Parameters:: path¶ (str) – path string to determine its placeholder key name of (can contain placeholders).
Return type:: str
Returns:: name (respectively dict key in PATH_PLACEHOLDERS) of the found path or empty string if not found.

paths_match(paths, masks)[source]

filter the paths matching at least one of the specified glob-like wildcard masks.

Parameters:

paths¶ (Iterable[str]) – iterable of path strings to be checked if they match at least one pattern/mask, specified by the masks argument.
masks¶ (Iterable[str]) – iterable of path masks/pattern with glob-like wildcards.

Return type:

Iterable[str]

Returns:

iterator, yielding the paths specified by paths that are matching at least one mask, specified by the masks argument.

placeholder_key(path)[source]

determine PATH_PLACEHOLDERS key of the specified path.

Parameters:: path¶ (str) – path string starting with a PATH_PLACEHOLDERS path prefix.
Return type:: str
Returns:: placeholder key (if found as path prefix), else empty string.

placeholder_path(path)[source]

replace the beginning of the specified path string with the longest prefix found in PATH_PLACEHOLDERS.

Parameters:: path¶ (str) – path string (optionally including subfolders and file name).
Return type:: str
Returns:: path string with replaced placeholder prefix (if found).

relative_file_paths(root_path, path_masks, skip_file_path=<function <lambda>>)[source]

find all files underneath the specified root path, e.g. to collect the .py modules of a python package.

Parameters:

root_path¶ (str) – path of the root folder. pass empty string to use the current working directory.
path_masks¶ (list[str]) – list of folder or subpackage path masks with glob-wildcards, relative to the root path.
skip_file_path¶ (Callable[[str], bool]) – called for each found file with their file path (relative to project root folder in root_path) as argument, returning True to exclude/skip the specified file.

Return type:

set[str]

Returns:

set of file paths relative to the root folder specified by the argument root_path.

series_file_name(file_path, digits=2, marker=' ', create=False)[source]

determine non-existent series file name with a unique series index.

Parameters:

file_path¶ (str) – file path and name (optional with extension).
digits¶ (int) – number of digits used for the series index.
marker¶ (str) – marker that will be put at the end of the file name and before the series index.
create¶ (bool) – pass True to create the file (to reserve the series index).

Return type:

str

Returns:

the file path extended with a unique / new series index.

skip_py_cache_files(file_path)[source]

file exclusion callback for the files under Python’s cache folders.

Parameters:: file_path¶ (str) – path to file to check for exclusion, relative to the project root folder.
Return type:: bool
Returns:: True if the file specified in file_path has to be excluded, else False.

user_data_path()[source]

determine the os-specific absolute path of the {usr} directory where user data can be stored.

Hint

this path is not accessible on Android devices, use user_docs_path() instead to get a more public path to the user.

Return type:: str
Returns:: path string of the user data folder.

user_docs_path()[source]

determine the os-specific absolute path of the {doc} directory where the user is storing the personal documents.

Hint

use user_data_path() instead to get the more hidden user data.

Return type:: str
Returns:: path string of the user documents folder.

PATH_PLACEHOLDERS = {'ado': '/home/docs/Documents/docs', 'app': '/home/docs/.config/docs', 'app_name': 'docs', 'cwd': '/home/docs/checkouts/readthedocs.org/user_builds/ae/checkouts/latest/docs', 'doc': '/home/docs/Documents', 'usr': '/home/docs/.config'}: placeholders dict of user-, os- and app-specific system paths and file name parts

class Collector(item_collector=<function coll_files>, **placeholders)[source]

Bases: object

file/folder collector class

__init__(item_collector=<function coll_files>, **placeholders)[source]

create a new file / folder / item collector instance with individual (extended or overriding) placeholders.

Parameters:

item_collector¶ (Callable[[str], Iterable[tuple[Optional[str], Any]]]) – callable to determine the item type to collect. the default is the coll_files() function. pass e.g. coll_folders() to collect only folders or coll_items() to collect both (files and folders). overload the arguments of these functions (with partial) to adapt/change their default arguments.
placeholders¶ – all other kwargs are placeholders with their names:replacements as keys:values. the placeholders provided by PATH_PLACEHOLDERS are available too (but will be overwritten by these arguments).

paths: list[Any]: list of found/collected folders

files: list[Any]: list of found/collected files

selected: list[Any]: list of found/collected files/folders item instances

failed: number of not found select-combinations

prefix_failed: dict[str, int]: not found select-combinations count for each prefix

suffix_failed: dict[str, int]: not found select-combinations count for each suffix

placeholders: path part placeholders of this Collector instance

check_add(item_mask, select=False)[source]

check if the item mask matches file/folder(s) and if yes, appends accordingly to collecting instance lists.

Parameters:

item_mask¶ (str) – file/folder mask, optionally including wildcards in the glob.glob format.
select¶ (bool) – pass True to additionally add found files/folders into selected.

Return type:

bool

Returns:

True if at least one file/folder got found/added, else False.

_collect_appends(prefix, appends)[source]

_collect_selects(prefix, selects)[source]

collect(*prefixes, append=(), select=())[source]

collect additional files/folders by combining the given prefixes with all the given append/select suffixes.

Note

all arguments of this method can either be passed either as tuples or for a single value as string.

Parameters:

prefixes¶ (str) – tuple of file/folder paths to be used as prefixes.
append¶ (Union[str, tuple[str, ...]]) – tuple of file/folder names to be used as suffixes.
select¶ (Union[str, tuple[str, ...]]) – tuple of file/folder names to be used as suffixes. this argument is, in contrary to append, also logging any not found prefixes combinations in the instance attributes failed, prefix_failed and suffix_failed.

Return type:

Collector

each of the passed prefixes will be combined with the suffixes specified in append and in select. all the matching file/folder paths will be added to the appropriate instance attribute, either files for a file or paths for a folder.

additionally, the existing file/folder paths from the combinations of prefixes and select will be added in the selected list attribute.

Hint

more details and some examples are available in the doc string of this module.

property error_message: str

returns an error message if an error occurred.

Returns:: error message string if collection failure/error occurred, else an empty string.

class FilesRegister(*add_path_args, property_matcher=None, file_sorter=None, **add_path_kwargs)[source]

Bases: dict

file register catalog - see also file register examples.

__init__(*add_path_args, property_matcher=None, file_sorter=None, **add_path_kwargs)[source]

create a file register instance.

this method gets redirected with add_path_args and add_path_kwargs arguments to add_paths().

Parameters:

add_path_args¶ – if passed, then add_paths() will be called with this args tuple.
property_matcher¶ (Optional[Callable[[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]], bool]]) – used as the default by find_file() if not specified there.
file_sorter¶ (Optional[Callable[[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]], Any]]) – used as the default value by find_file() if not specified there.
add_path_kwargs¶ – passed onto call of add_paths() if the add_path_args got provided by the caller.

__call__(*find_args, **find_kwargs)[source]

add_path_args and kwargs will be completely redirected to find_file().

Return type:: Union[str, RegisteredFile, CachedFile, Path, PurePath, Any, None]

add_file(file_obj, first_index=9223372036854775807)[source]

add a single file to the list of this dict mapped by the file-name/stem as a dict key.

Parameters:

file_obj¶ (Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]) – either file path string or any object with a stem attribute.
first_index¶ (int) – pass list index -n-1..n-1 to insert file_obj in the name’s list. values greater than n (==len(file_list)) will append the file_obj to the end of the file object list, and values less than n-1 will insert the file_obj to the start of the file.

add_files(files, first_index=9223372036854775807)[source]

add files from another FilesRegister instance.

Parameters:

files¶ (Iterable[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]]) – iterable with file objects to be added.
first_index¶ (int) – pass list index -n-1..n-1 to insert the first file_obj in each name’s register list. values greater than n (==len(file_list)) will append the file_obj to the end of the file object list. the order of the added items will be unchanged if this value is greater or equal to zero. negative values will add the items from files in reversed order, and after the item specified by this index value (so passing -1 will append the items to the end in reversed order, while passing -(n+1) will insert them at the beginning in reversed order).

Return type:

list[str]

Returns:

list of paths of the added files.

add_paths(*file_path_masks, first_index=9223372036854775807, file_class=<class 'ae.files.RegisteredFile'>, **init_kwargs)[source]

add files found in the folder(s) specified by the file_path_masks args.

Parameters:

file_path_masks¶ (str) – file path masks (with optional wildcards and PATH_PLACEHOLDERS) specifying the files to collect (by default including the subfolders).
first_index¶ (int) – pass list index -n-1..n-1 to insert the first file_obj in each name’s register list. values greater than n (==len(file_list)) will append the file_obj to the end of the file object list. the order of the added items will be unchanged if this value is greater or equal to zero. negative values will add the found items in reversed order, and after the item specified by this index value. so passing -1 will append the items to the end in reversed order, while passing -(n+1) will insert them at the beginning in reversed order.
file_class¶ (Type[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]]) – the used file object class (see FileObject). each found file object will be passed to the class constructor (callable) and added to the list, which is an item of this dict.
init_kwargs¶ – additional/optional kwargs passed onto the used file_class. pass e.g., the object_loader to use, if file_class is CachedFile (instead of the default: RegisteredFile).

Return type:

list[str]

Returns:

list of paths of the added files.

add_register(files_register, first_index=9223372036854775807)[source]

add files from another FilesRegister instance.

Parameters:

files_register¶ (FilesRegister) – the FilesRegister instance containing the file_obj to be added.
first_index¶ (int) – pass list index -n-1..n-1 to insert the first file_obj in each name’s register list. values greater than n (==len(file_list)) will append the file_obj to the end of the file object list. the order of the added items will be unchanged if this value is greater or equal to zero. negative values will add the found items in reversed order, and after the item specified by this index value. so passing -1 will append the items to the end in reversed order, while passing -(n+1) will insert them at the beginning in reversed order.

Return type:

list[str]

Returns:

list of paths of the added files.

find_file(name, properties=None, property_matcher=None, file_sorter=None)[source]

find file_obj in this register via properties, property matcher callables and/or file sorter.

Parameters:

name¶ (str) – file name (stem without extension) to find.
properties¶ (Optional[Dict[str, Union[int, float, str]]]) – properties to select the correct file.
property_matcher¶ (Optional[Callable[[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]], bool]]) – callable to match the correct file.
file_sorter¶ (Optional[Callable[[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]], Any]]) – callable to sort resulting match results.

Return type:

Union[str, RegisteredFile, CachedFile, Path, PurePath, Any, None]

Returns:

registered/cached file object of the first found/correct file.

reclassify(file_class=<class 'ae.files.CachedFile'>, **init_kwargs)[source]

re-instantiate all name’s file registers items to instances of the class file_class.

Parameters:

file_class¶ (Type[Union[str, RegisteredFile, CachedFile, Path, PurePath, Any]]) – the new file object class (see FileObject). each found file object will be passed to the class constructor (callable) and the return value will then replace the file object in the file list.
init_kwargs¶ – additional/optional kwargs passed onto the used file_class. pass e.g., the object_loader to use, if file_class is CachedFile (the default file object class).