Logging

The subnet template framework uses multiple processes from the multiprocessing library. Due to this, the logging package does not use process-shared locks, so it is possible (depending on the handler type) for messages from different processes to get mixed up. This means that if multiple processes write to the same log file or stream at the same time, their log messages may interleave or become mixed together depending on the handler type.

How to use logging with multiprocessing

You may initialize loggers both in the parent process and inside child processes. However, they must not share handlers that write to the same output unless those handlers are explicitly made multiprocessing-safe (e.g., by using QueueHandler and QueueListener).

In the subnet template:

• The parent process uses the top-level logger normally.

• Child processes should create or retrieve their own logger instance, typically exposed as self.logger inside a multiprocessing class.

Each process effectively gets a separate logging context, preventing cross-process interference.

Minimal Example

Below is a minimal viable example that shows how to set up logging in both the parent process and a child process running under multiprocessing. The child process initializes its own logger inside run() and logs independently of the parent.

import multiprocessing as mp
from mesh.utils.logging import configure_subprocess_logging, get_logger

logger = get_logger(__name__)

class MVP(mp.Process):
    def __init__():
        super().__init__()
        self.stop = mp.Event()

    def run(self):
        configure_subprocess_logging()
        self.logger = get_logger(__name__)
        
    def example_function(self):
        """Child functions use the child process logging"""
        self.logger.info("Logging example")
        
    def shutdown(self, timeout: float = 5.0):
        """Shutdown uses the top-level logging"""
        if not self.stop.is_set():
            self.stop.set()

        if self.is_alive():
            self.join(3)
            if self.is_alive():
                logger.warning("Consensus did not shut down within the grace period; terminating it the hard way")
                self.terminate()
        else:
            logger.warning("Consensus shutdown had no effect, the process is already dead")