alphatims.utils

This module provides generic utilities. These utilities primarily focus on:

• logging

• compilation

• parallelization

• generic io

Classes:

Global_Stack(all_available_options)	A stack that holds multiple option stacks.
Option_Stack(option_name, option_initial_value)	A stack with the option to redo and undo.

Functions:

check_github_version([silent])	Checks and logs the current version of AlphaTims.
create_dict_from_hdf_group(hdf_group[, ...])	Convert the contents of an HDF group and return as normal Python dict.
create_hdf_group_from_dict(hdf_group, ...[, ...])	Save a dict to an open hdf group.
load_parameters(parameter_file_name)	Load a parameter dict from a file.
njit([_func])	A wrapper for the numba.njit decorator.
pjit([_func, thread_count, ...])	A decorator that parallelizes the numba.njit decorator with threads.
progress_callback(iterable[, ...])	A generator that adds progress callback to an iterable.
save_parameters(parameter_file_name, paramaters)	Save parameters to a parameter file.
set_logger(*[, log_file_name, stream, ...])	Set the log stream and file.
set_progress_callback(progress_callback)	Set the global progress callback.
set_threads(threads[, set_global])	Parse and set the (global) number of threads.
show_platform_info()	Log all platform information.
show_python_info()	Log all Python information.
threadpool([_func, thread_count, ...])	A decorator that parallelizes a function with threads and callback.

class alphatims.utils.Global_Stack(all_available_options: dict)

Bases: object

A stack that holds multiple option stacks.

The current value of each option stack can be retrieved by indexing, i.e. option_value = self[option_key].

Methods:

__init__(all_available_options)	Create a global stack.
lock()	A context manager to lock this stack and prevent modification.
redo()	Increase the stack pointer with 1.
trim()	Remove all elements above of the current stack pointer
undo()	Reduce the stack pointer with 1.
update(option_key, option_value)	Update an option stack with a value.

Attributes:

current_values	A dict with (option_key: option_value) mapping.
is_locked	A flag to check if this stack is modifiable
size	The size of this stack without the initial value.

__init__(all_available_options: dict)

Create a global stack.

Parameters

all_available_options (dict) – A dictionary whose items are (str, type), which can be used to create an Option_Stack.

property current_values: dict

A dict with (option_key: option_value) mapping.

Type

dict

property is_locked

A flag to check if this stack is modifiable

Type

bool

lock()

A context manager to lock this stack and prevent modification.

redo() → tuple

Increase the stack pointer with 1.

Returns

(“”, None) if the pointer was already at the maximum. Otherwise (option_name, new_value) if the pointer was increased.

Return type

tuple

property size

The size of this stack without the initial value.

Type

int

trim() → bool

Remove all elements above of the current stack pointer

Returns

True if something was removed, i.e. if stack pointer was not at the top. False if nothing could be deleted, i.e. the stack pointer was at the top.

Return type

bool

undo() → tuple

Reduce the stack pointer with 1.

Returns

(“”, None) if the pointer was already at the maximum. Otherwise (option_name, new_value) if the pointer was reduced.

Return type

tuple

update(option_key: str, option_value) → tuple

Update an option stack with a value.

Parameters

• option_key (str) – The name of the option stack to update.

• option_value (type) – An value to add to this stack. Can be any object that supports the “!=” operator.

Returns

(“”, None) if the pointer was not updated, i.e. the latest update was equal to the current update. Otherwise (option_name, new_value).

Return type

tuple

class alphatims.utils.Option_Stack(option_name: str, option_initial_value)

Bases: object

A stack with the option to redo and undo.

Methods:

__init__(option_name, option_initial_value)	Create an option stack.
redo()	Increase the stack pointer with 1.
trim()	Remove all elements above of the current stack pointer
undo()	Reduce the stack pointer with 1.
update(option_value)	Update this stack with the value.

Attributes:

current_value	The current value of this stack.
option_name	The name of this stack.
size	The size of this stack without the initial value.

__init__(option_name: str, option_initial_value)

Create an option stack.

Parameters

• option_name (str) – The name of this option.

• option_initial_value (type) – The initial value of this stack. Can be any object that supports the “!=” operator.

property current_value

The current value of this stack.

Type

type

property option_name: str

The name of this stack.

Type

str

redo()

Increase the stack pointer with 1.

Returns

None if the pointer was already at the maximum. Otherwise the new value if the pointer was increased.

Return type

type

property size: int

The size of this stack without the initial value.

Type

int

trim() → bool

Remove all elements above of the current stack pointer

Returns

True if something was removed, i.e. if stack pointer was not at the top. False if nothing could be deleted, i.e. the stack pointer was at the top.

Return type

bool

undo()

Reduce the stack pointer with 1.

Returns

None if the pointer was already at the maximum. Otherwise the new value if the pointer was reduced.

Return type

type

update(option_value) → bool

Update this stack with the value.

Parameters

option_value (type) – An value to add to this stack. Can be any object that supports the “!=” operator.

Returns

True if the stack was updated. False if the provided value equald the current value of this stack.

Return type

bool

alphatims.utils.check_github_version(silent=False) → str

Checks and logs the current version of AlphaTims.

Check if the local version equals the AlphaTims GitHub master branch. This is only possible with an active internet connection and if no credentials are required for GitHub.

Parameters

silent (str) – Use the logger to display the obtained conclusion. Default is False.

Returns

The version on the AlphaTims GitHub master branch. “” if no version can be found on GitHub

Return type

str

alphatims.utils.create_dict_from_hdf_group(hdf_group, mmap_arrays=None, parent_file_name: Optional[str] = None) → dict

Convert the contents of an HDF group and return as normal Python dict.

Parameters

• hdf_group (h5py.File.group) – An open and readable HDF group.

• mmap_arrays (iterable) – These array will be mmapped instead of pre-loaded. Default is None

• parent_file_name (str) – The parent_file_name. This is required when mmap_arrays is not None. Default is None.

Returns

A Python dict. Keys of the dict are names of arrays, attrs and subgroups. Values are corresponding arrays and attrs. Subgroups are converted to subdicts. If a subgroup has an “is_pd_dataframe=True” attr, it is automatically converted to a pd.dataFrame.

Return type

dict

Raises

ValueError – When an attr value in the HDF group is not an int, float, str or bool.

alphatims.utils.create_hdf_group_from_dict(hdf_group, data_dict: dict, *, overwrite: bool = False, compress: bool = False, recursed: bool = False, chunked: bool = False) → None

Save a dict to an open hdf group.

Parameters

• hdf_group (h5py.File.group) – An open and writable HDF group.

• data_dict (dict) –

  A dict that needs to be written to HDF. Keys always need to be strings. Values are stored as follows:

  • subdicts -> subgroups.

  • np.array -> array

  • pd.dataframes -> subdicts with “is_pd_dataframe: True” attribute.

  • bool, int, float and str -> attrs.

  • None values are skipped and not stored explicitly.

• overwrite (bool) – If True, existing subgroups, arrays and attrs are fully truncated/overwritten. If False, the existing value in HDF remains unchanged. Default is False.

• compress (bool) – If True, all arrays are compressed with binary shuffle and “lzf” compression. If False, arrays are saved as provided. On average, compression halves file sizes, at the cost of 2-10 time longer accession times. Default is False.

• recursed (bool) – If False, the default progress callback is added while itereating over the keys of the data_dict. If True, no callback is added, allowing subdicts to not trigger callback. Default is False.

• chunked (bool) – If True, all arrays are chunked. If False, arrays are saved as provided. Default is False.

Raises

• ValueError – When a value of data_dict cannot be converted to an HDF value (see data_dict).

• KeyError – When a key of data_dict is not a string.

alphatims.utils.load_parameters(parameter_file_name: str) → dict

Load a parameter dict from a file.

Parameters

parameter_file_name (str) – A file name that contains parameters in .json format.

Returns

A dict with parameters.

Return type

dict

alphatims.utils.njit(_func=None, *args, **kwargs)

A wrapper for the numba.njit decorator.

The “cache” option is set to True by default. This can be overriden with kwargs.

Parameters

• _func (callable, None) – The function to decorate.

• *args – See numba.njit decorator.

• **kwargs – See numba.njit decorator.

Returns

A numba.njit decorated function.

Return type

function

alphatims.utils.pjit(_func=None, *, thread_count=None, include_progress_callback: bool = True, cache: bool = True, **kwargs)

A decorator that parallelizes the numba.njit decorator with threads.

The first argument of the decorated function need to be an iterable. A range-object will be most performant as iterable. The original function should accept a single element of this iterable as its first argument. The original function cannot return values, instead it should store results in e.g. one if its input arrays that acts as a buffer array. The original function needs to be numba.njit compatible. Numba argument “nogil” is always set to True.

Parameters

• _func (callable, None) – The function to decorate.

• thread_count (int, None) – The number of threads to use. This is always parsed with alphatims.utils.set_threads. Not possible as positional arguments, it always needs to be an explicit keyword argument. Default is None.

• include_progress_callback (bool) – If True, the default progress callback will be used as callback. (See “progress_callback” function.) If False, no callback is added. See set_progress_callback for callback styles. Default is True.

• cache (bool) – See numba.njit decorator. Default is True (in contrast to numba).

Returns

A parallelized numba.njit decorated function.

Return type

function

alphatims.utils.progress_callback(iterable, include_progress_callback: bool = True, total: int = - 1)

A generator that adds progress callback to an iterable.

Parameters

• iterable – An iterable.

• include_progress_callback (bool) – If True, the default progress callback will be used as callback. If False, no callback is added. See set_progress_callback for callback styles. Default is True.

• total (int) – The length of the iterable. If -1, this will be read as len(iterable), if __len__ is implemented. Default is -1.

Returns

A generator over the iterable with added callback.

Return type

iterable

alphatims.utils.save_parameters(parameter_file_name: str, paramaters: dict) → None

Save parameters to a parameter file.

IMPORTANT NOTE: This overwrites any existing file.

Parameters

• parameter_file_name (str) – The file name to where the parameters are written.

• paramaters (dict) – A dictionary with parameters.

alphatims.utils.set_logger(*, log_file_name='', stream: bool = True, log_level: int = 20, overwrite: bool = False) → str

Set the log stream and file.

All previously set handlers will be disabled with this command.

Parameters

• log_file_name (str, None) – The file name to where the log is written. Folders are automatically created if needed. This is relative to the current path. When an empty string is provided, a log is written to the AlphaTims “logs” folder with the name “log_yymmddhhmmss” (reversed timestamp year to seconds). If None, no log file is saved. Default is “”.

• stream (bool) – If False, no log data is sent to stream. If True, all logging can be tracked with stdout stream. Default is True.

• log_level (int) – The logging level. Usable values are defined in Python’s “logging” module. Default is logging.INFO.

• overwrite (bool) – If True, overwrite the log_file if one exists. If False, append to this log file. Default is False.

Returns

The file name to where the log is written.

Return type

str

alphatims.utils.set_progress_callback(progress_callback)

Set the global progress callback.

Parameters

progress_callback –

The new global progress callback. Options are:

• None, no progress callback will be used

• True, a textual progress callback (tqdm) will be enabled

• Any object that supports a max and value variable.

alphatims.utils.set_threads(threads: int, set_global: bool = True) → int

Parse and set the (global) number of threads.

Parameters

• threads (int) – The number of threads. If larger than available cores, it is trimmed to the available maximum. If 0, it is set to the maximum cores available. If negative, it indicates how many cores NOT to use.

• set_global (bool) – If False, the number of threads is only parsed to a valid value. If True, the number of threads is saved as a global variable. Default is True.

Returns

The number of threads.

Return type

int

alphatims.utils.show_platform_info() → None

Log all platform information.

This is done in the following format:

• [timestamp]> Platform information:

• [timestamp]> system - […]

• [timestamp]> release - […]

• [timestamp]> version - […]

• [timestamp]> machine - […]

• [timestamp]> processor - […]

• [timestamp]> cpu count - […]

• [timestamp]> cpu frequency - […]

• [timestamp]> ram - […]/[…] Gb (available/total)

alphatims.utils.show_python_info() → None

Log all Python information.

This is done in the following format:

• [timestamp]> Python information:

• [timestamp]> alphatims - [current_version]

• [timestamp]> [required package] - [current_version]

• …

• [timestamp]> [required package] - [current_version]

alphatims.utils.threadpool(_func=None, *, thread_count=None, include_progress_callback: bool = True, return_results: bool = False) → None

A decorator that parallelizes a function with threads and callback.

The original function should accept a single element as its first argument. If the caller function provides an iterable as first argument, the function is applied to each element of this iterable in parallel.

Parameters

• _func (callable, None) – The function to decorate.

• thread_count (int, None) – The number of threads to use. This is always parsed with alphatims.utils.set_threads. Not possible as positional arguments, it always needs to be an explicit keyword argument. Default is None.

• include_progress_callback (bool) – If True, the default progress callback will be used as callback. (See “progress_callback” function.) If False, no callback is added. See set_progress_callback for callback styles. Default is True.

• return_results (bool) – If True, it returns the results in the same order as the iterable. This can be much slower than not returning results. Iti is better to store them in a buffer results array instead (be carefull to avoid race conditions). If the iterable is not an iterable but a single index, a result is always returned. Default is False.

Returns

A parallelized decorated function.

Return type

function