Progress bar and status logging in python with tqdm

Photo by Chris Ried on Unsplash

When you write write code, especially in machine learning, there will be inevitably the situation where you loop over a larger amount of data, for example when you train your CNN. Progress bars are helpful in showing you how much work has been completed and how much longer you have to wait. My favorite progress bar for python is tqdm, which lets you create simple progress bars with just a few lines of code. The github page gives a very good introduction into the basic use of tqdm. Here a simple example:

import tqdm

for ii in tqdm.trange(100):
    pass
100%|█████████████████████████████████████| 100/100 [00:00<?, ?it/s]

The problem with tqdm I had was that I want a progress bar for my neural network training but also log some statistics of the last iteration to the console. tqdm does not work nicely with print() , but tqdm offers a couple of ways to work around that.

A) Add information to the progress bar description

Each progress bar has a description field in the beginning of the line where you can add a short text, for example Training or Epochs. But since the line length is limited and the actual progress tracking with remaining time and iterations per second needs some space, too.

import tqdm
outer = tqdm.tqdm(total=100, desc='Epoch', position=0)
for ii in range(100):
    inner = tqdm.tqdm(total=50, desc='Batch', position=1)
    for jj in range(50):
        #do your training
        inner.update(1)
    outer.update(1)
Epoch: 100%|██████████████████████████████| 100/100 [00:00<?, ?it/s]
Batch: 100%|████████████████████████████████| 50/50 [00:00<?, ?it/s]

B) Use write()

For longer texts or text not related to a specific progress bar tqdm offers write() which logs text safely to the console. To be precise, it adds the text above the progress bars, so each line you ever logged using write() will be visible.

import tqdm
pbar = tqdm.tqdm(total=5)
for ii in range(5):
    pbar.write(f'Hello {ii}')
    pbar.update(1)

Hello 0
Hello 1
Hello 2
Hello 3
Hello 4
100%|████████████████████████████████████████| 5/5 [00:00<?, ?it/s]

This is nice and all but not what I want from a compact progress and status logging. Fortunately tqdm can be hacked to display text in fixed positions.

C) Convert progress bars to status bars

The idea behind this “hack” is to use progress bars for the status logging by removing everything but the description field. This way we can specify the exact location of the status bar using the position argument when we create the progress bar. To remove all fields except the description field bar_format='{desc}' needs to be set. Now the progress bar shows only the description. To set the value of the description we need to use set_description_str() instead of set_description() as the latter adds a colon after the description text. Below is a toy example of a video to image converter that reads all videos from a folder and converts them to images. It uses progress bars to log the filename of video and frame as well as some data about the video file.

There you get a nice and compact overview on your status and progress:

python tqdm_log.py
Files:  25%|█████████                | 1/4 [03:05<09:15, 185.05s/it]
Current file: /tmp/videos/07.09.37.mp4
Frames:   6%|██                    | 26/440 [00:11<02:55,  2.36it/s]
440 frames of 480x480 pixels
00000328.png saved

tl;dr

• Use position=n to give progress bars a fixed position
• Use bar_format='{desc}' to remove all but the description from progress bars you want to use for logging
• Set the logging content using set_description_str() to avoid the colon after the text