composer.utils.run_directory#

The run directory stores artifacts created during training.

Callbacks, loggers, the checkpointer, and the profiler can write files to and read files from the run directory. For example,

The CheckpointSaver can save checkpoints inside the run directory.
The FileLogger can save log files inside the run directory.
The Profiler can save profiling traces inside the run directory.
The RunDirectoryUploader backs up files written to the run directory to cloud object storage.
The WandBLogger can store files in the run directory as artifacts.

Run Directory Location#

If using the composer launcher for distributed training, the run directory can be specified using the

--run_directory command line flag. For example:

composer --run_directory /path/to/run/directory -n 8 path/to/training/script.py

Otherwise, if launching python directly, the run directory can be specified via the COMPOSER_RUN_DIRECTORY environment variable. (The composer launcher sets this variable for you.)

If not specified, then the run directory defaults to ./runs/{timestamp}, relative to the current working directory (CWD).

This folder is partitioned into subfolders by each rank. For example:

>>> import os
>>> os.environ['COMPOSER_RUN_DIRECTORY'] = "./path/to/my_run_directory"
>>> from composer.utils import run_directory
>>> os.path.relpath(run_directory.get_run_directory())
'path/to/my_run_directory/rank_0'

Functions

`get_modified_files`	Returns a list of files (recursively) in the run directory that have been modified since `modified_since_timestamp`.
`get_node_run_directory`	Returns the run directory for the node.
`get_run_directory`	Returns the run directory for the current rank.
`get_run_directory_timestamp`	Returns the current timestamp on the run directory filesystem.

composer.utils.run_directory.get_modified_files(modified_since_timestamp, *, ignore_hidden=True)[source]#

Returns a list of files (recursively) in the run directory that have been modified since modified_since_timestamp.

Parameters

modified_since_timestamp (datetime) – The list of returned files must have a last modified timestamp at least as recent as this value.
ignore_hidden (bool, optional) – Whether to ignore hidden files and folders. (default: True)

Returns

List[str] – List of filepaths that have been modified since modified_since_timestamp.

composer.utils.run_directory.get_node_run_directory()[source]#

Returns the run directory for the node. This folder is shared by all ranks on the node.

Returns: str – The node run directory.

composer.utils.run_directory.get_run_directory()[source]#

Returns the run directory for the current rank.

Returns: str – The run directory.

composer.utils.run_directory.get_run_directory_timestamp()[source]#

Returns the current timestamp on the run directory filesystem.

Note

The disk time can differ from system time (e.g. when using network filesystems). As such, the result from this function should be used as the modified_since_timestamp parameter for calls to get_modified_files().

Returns: float – The current timestamp on the run directory filesystem.