Multiprocess logging

Python source

Demonstrates how Rerun can work with the Python multiprocessing library.

Used Rerun types used-rerun-types

Boxes2D, TextLog

Logging and visualizing with Rerun logging-and-visualizing-with-rerun

This example demonstrates how to use the Rerun SDK with multiprocessing to log data from multiple processes to the same Rerun viewer. It starts with the definition of the function for logging, the task, followed by typical usage of Python's multiprocessing library.

The function task is decorated with @rr.shutdown_at_exit. This decorator ensures that data is flushed when the task completes, even if the normal atexit-handlers are not called at the termination of a multiprocessing process.

@rr.shutdown_at_exit
def task(child_index: int) -> None:
    rr.init("rerun_example_multiprocessing")

    rr.connect_tcp()

    title = f"task_{child_index}"
    rr.log(
        "log",
        rr.TextLog(
            f"Logging from pid={os.getpid()}, thread={threading.get_ident()} using the Rerun recording id {rr.get_recording_id()}"
        )
    )
    if child_index == 0:
        rr.log(title, rr.Boxes2D(array=[5, 5, 80, 80], array_format=rr.Box2DFormat.XYWH, labels=title))
    else:
        rr.log(
            title,
            rr.Boxes2D(
                array=[10 + child_index * 10, 20 + child_index * 5, 30, 40],
                array_format=rr.Box2DFormat.XYWH,
                labels=title,
            ),
        )

The main function initializes Rerun with a specific application ID and manages the multiprocessing processes for logging data to the Rerun viewer.

Caution: Ensure that the recording id specified in the main function matches the one used in the logging functions

def main() -> None:
   # … existing code …

   rr.init("rerun_example_multiprocessing")
   rr.spawn(connect=False)  # this is the Viewer that each child process will connect to

   task(0)

   for i in [1, 2, 3]:
       p = multiprocessing.Process(target=task, args=(i,))
       p.start()
       p.join()

Run the code run-the-code

To run this example, make sure you have the Rerun repository checked out and the latest SDK installed:

pip install --upgrade rerun-sdk  # install the latest Rerun SDK
git clone git@github.com:rerun-io/rerun.git  # Clone the repository
cd rerun
git checkout latest  # Check out the commit matching the latest SDK release

Install the necessary libraries specified in the requirements file:

pip install -e examples/python/multiprocessing

To experiment with the provided example, simply execute the main Python script:

python -m multiprocessing # run the example

If you wish to customize it, explore additional features, or save it use the CLI with the --help option for guidance:

python -m multiprocessing --help