composer.loggers.logger_destination#

Base class for logger callback.

Classes

LoggerDestination

Base class for logger destination.
class composer.loggers.logger_destination.LoggerDestination[source]#

Bases: composer.core.callback.Callback, abc.ABC

Base class for logger destination.

As this class extends Callback, logger destinations can run on any training loop Event. For example, it may be helpful to run on EPOCH_END to perform any flushing at the end of every epoch.

Example

>>> from composer.loggers import LoggerDestination
>>> class MyLogger(LoggerDestination):
...     def log_data(self, state, log_level, data):
...         print(f'Batch {int(state.timer.batch)}: {log_level} {data}')
>>> logger = MyLogger()
>>> trainer = Trainer(
...     ...,
...     loggers=[logger]
... )
log_data(state, log_level, data)[source]#

Log data.

Subclasses should implement this method to store logged data (e.g. write it to a file, send it to a server, etcโ€ฆ). However, not all loggers need to implement this method.

Note

This method will block the training loop. For optimal performance, it is recommended to deepcopy the data (e.g. copy.deepcopy(data)), and store the copied data in queue. Then, either:

  • Use background thread(s) or process(s) to read from this queue to perform any I/O.

  • Batch the data together and flush periodically on events, such as BATCH_END or EPOCH_END.

    See also

    FileLogger as an example.

Parameters

  • state (State) โ€“ The training state.

  • log_level (LogLevel) โ€“ The log level.

  • data (Dict[str, Any]) โ€“ The data to log.

log_file_artifact(state, log_level, artifact_name, file_path, *, overwrite)[source]#

Handle logging of a file artifact stored at file_path to an artifact named artifact_name.

Subclasses should implement this method to store logged files (e.g. copy it to another folder or upload it to an object store), then it should implement this method. However, not all loggers need to implement this method. For example, the TQDMLogger does not implement this method, as it cannot handle file artifacts.

Note

  • This method will block the training loop. For optimal performance, it is recommended that this method copy the file to a temporary directory, enqueue the copied file for processing, and return. Then, use a background thread(s) or process(s) to read from this queue to perform any I/O.

  • After this method returns, training can resume, and the contents of file_path may change (or be may deleted). Thus, if processing the file in the background (as is recommended), it is necessary to first copy the file to a temporary directory. Otherwise, the original file may no longer exist, or the logged artifact can be corrupted (e.g., if the logger destination is reading from file while the training loop is writing to it).

Parameters

  • state (State) โ€“ The training state.

  • log_level (Union[str, LogLevel]) โ€“ A LogLevel.

  • artifact_name (str) โ€“ The name of the artifact.

  • file_path (Path) โ€“ The file path.

  • overwrite (bool, optional) โ€“ Whether to overwrite an existing artifact with the same artifact_name. (default: False)

Handle creating a symlink of a file artifact stored at existing_artifact_name to an artifact named symlink_artifact_name.

Subclasses should implement this method to symlink logged files. However, not all loggers need to implement this method. For example, the TQDMLogger does not implement this method, as it cannot handle file artifacts and thus does not need to do any special symlinking.

Note

  • This method will block the training loop. For optimal performance, it is recommended that this method enqueue creating the symlink in the background and return immediately. Then, use a background thread(s) or process(s) to read from this queue to perform any I/O.

Parameters

  • state (State) โ€“ The training state.

  • log_level (Union[str, LogLevel]) โ€“ A LogLevel.

  • existing_artifact_name (str) โ€“ The name of symlinked artifact.

  • symlink_artifact_name (str) โ€“ The symlink name of artifact.

  • overwrite (bool, optional) โ€“ Whether to overwrite an existing artifact with the same symlink_artifact_name. (default: False)