logger_destination#
Base class for logger callback.
Classes
Base class for logger destination. |
- class composer.loggers.logger_destination.LoggerDestination(*args, **kwargs)[source]#
Bases:
composer.core.callback.Callback
,abc.ABC
Base class for logger destination.
As this class extends
Callback
, logger destinations can run on any training loopEvent
. For example, it may be helpful to run onEvent.EPOCH_END
to perform any flushing at the end of every epoch.Example
>>> from composer.loggers import LoggerDestination >>> class MyLogger(LoggerDestination): ... def log_data(self, state, log_level, data): ... print(f'Batch {int(state.timestamp.batch)}: {data}') >>> logger = MyLogger() >>> trainer = Trainer( ... ..., ... loggers=[logger] ... ) Batch 0: {'rank_zero_seed': ...}
- can_log_file_artifacts()[source]#
Indicates whether LoggerDestination can log file artifacts.
Defaults to false, should return True for derived logger classes that implement log_file_artifact().
- Returns
bool โ Whether the class supports logging file artifacts.
- get_file_artifact(artifact_name, destination, overwrite=False, progress_bar=True)[source]#
Handle downloading an artifact named
artifact_name
todestination
.- Parameters
artifact_name (str) โ The name of the artifact.
destination (str) โ The destination filepath.
overwrite (bool) โ Whether to overwrite an existing file at
destination
. Defaults toFalse
.progress_bar (bool, optional) โ Whether to show a progress bar. Ignored if
path
is a local file. (default:True
)
- log_data(state, log_level, data)[source]#
Log data.
Subclasses should implement this method to store logged data (e.g. write it to a file, send it to a server, etcโฆ). However, not all loggers need to implement this method.
Note
This method will block the training loop. For optimal performance, it is recommended to deepcopy the
data
(e.g.copy.deepcopy(data)
), and store the copied data in queue. Then, either:Use background thread(s) or process(s) to read from this queue to perform any I/O.
Batch the data together and flush periodically on events, such as
Event.BATCH_END
orEvent.EPOCH_END
.See also
FileLogger
as an example.
- log_file_artifact(state, log_level, artifact_name, file_path, *, overwrite)[source]#
Handle logging of a file artifact stored at
file_path
to an artifact namedartifact_name
.Subclasses should implement this method to store logged files (e.g. copy it to another folder or upload it to an object store), then it should implement this method. However, not all loggers need to implement this method. For example, the
TQDMLogger
does not implement this method, as it cannot handle file artifacts.Note
This method will block the training loop. For optimal performance, it is recommended that this method copy the file to a temporary directory, enqueue the copied file for processing, and return. Then, use a background thread(s) or process(s) to read from this queue to perform any I/O.
After this method returns, training can resume, and the contents of
file_path
may change (or be may deleted). Thus, if processing the file in the background (as is recommended), it is necessary to first copy the file to a temporary directory. Otherwise, the original file may no longer exist, or the logged artifact can be corrupted (e.g., if the logger destination is reading from file while the training loop is writing to it).
See also
Artifact Logging for notes for file artifact logging.
- Parameters