composer.callbacks.memory_monitor#

Log memory usage during training.

Classes

MemoryMonitor

Logs the memory usage of the model.

class composer.callbacks.memory_monitor.MemoryMonitor[source]#

Bases: composer.core.callback.Callback

Logs the memory usage of the model.

This callback calls the torch memory stats API for cuda (see torch.cuda.memory_stats()) on the AFTER_TRAIN_BATCH and reports different memory statistics.

Example

>>> from composer.callbacks import MemoryMonitor
>>> # constructing trainer object with this callback
>>> trainer = Trainer(
...     model=model,
...     train_dataloader=train_dataloader,
...     eval_dataloader=eval_dataloader,
...     optimizers=optimizer,
...     max_duration="1ep",
...     callbacks=[MemoryMonitor()],
... )

The memory statistics are logged by the Logger to the following keys as described below.

Key	Logged data
`memory/{statistic}`	Several memory usage statistics are logged on `AFTER_TRAIN_BATCH` event.

The following statistics are recorded:

Statistic	Description
alloc_requests	Number of memory allocation requests received by the memory allocator.
free_requests	Number of memory free requests received by the memory allocator.
allocated_mem	Amount of allocated memory in bytes.
active_mem	Amount of active memory in bytes at the time of recording.
inactive_mem	Amount of inactive, non-releaseable memory in bytes at the time of recording.
reserved_mem	Amount of reserved memory in bytes at the time of recording.
alloc_retries	Number of failed cudaMalloc calls that result in a cache flush and retry.

Note

Memory usage monitoring is only supported for the GPU devices.