composer.utils.iter_helpers#

Utilities for iterating over collections.

Functions

ensure_tuple

Converts x into a tuple.

iterate_with_callback

Invoke callback after each chunk is yielded from iterator.

map_collection

Applies map_fn on each element in collection.

Classes

IteratorFileStream

Class used to convert iterator of bytes into a file-like binary stream object.
class composer.utils.iter_helpers.IteratorFileStream(iterator)[source]#

Bases: io.RawIOBase

Class used to convert iterator of bytes into a file-like binary stream object.

Original implementation found here.

Parameters

iterator โ€“ An iterator over bytes objects

composer.utils.iter_helpers.ensure_tuple(x)[source]#

Converts x into a tuple.

  • If x is None, then tuple() is returned.

  • If x is a tuple, then x is returned as-is.

  • If x is a list, then tuple(x) is returned.

  • If x is a dict, then tuple(v for v in x.values()) is returned.

Otherwise, a single element tuple of (x,) is returned.

Parameters

x (Any) โ€“ The input to convert into a tuple.

Returns

tuple โ€“ A tuple of x.

composer.utils.iter_helpers.iterate_with_callback(iterator, total_len, callback=None)[source]#

Invoke callback after each chunk is yielded from iterator.

Parameters

  • iterator (Iterator) โ€“ The iterator, which should yield chunks of data.

  • total_len (int) โ€“ The total length of the iterator.

  • callback (Callable[[int, int], None], optional) โ€“

    The callback to invoke after each chunk of data is yielded back to the caller. Defaults to None, for no callback.

    It is called with the cumulative size of all chunks yielded thus far and the total_len.

composer.utils.iter_helpers.map_collection(collection, map_fn)[source]#

Applies map_fn on each element in collection.

  • If collection is a tuple or list of elements, map_fn is applied on each element, and a tuple or list, respectively, containing mapped values is returned.

  • If collection is a dictionary, map_fn is applied on each value, and a dictionary containing the mapped values is returned.

  • If collection is None, None is returned.

  • If collection is a single element, the result of applying map_fn on it is returned.

Parameters

  • collection โ€“ The element, or a tuple of elements.

  • map_fn โ€“ A function to invoke on each element.

Returns

  • Collection โ€“ The result of applying map_fn on each element of collection.

  • The type of ``collection`` is preserved.