We Keep Coding

Logging configuration with subprocesses

I have a python script with logging configuration that is called with subprocess.call from a main script. I would like the subprocess to create the log file, and write to it, as it is configured in the subprocess's code. In addition, I would like to see the logs also displayed in the main process' console output.
The following works on mac, but doesn't work on a linux box with python 3.10.4
Main Code
import subprocess
import logging
log_format = '%(asctime)s :: %(name)s :: %(levelname)-8s :: %(message)s'
log_streamhandler = logging.StreamHandler()
logging.basicConfig(format=log_format, level=logging.INFO,
handlers=[log_streamhandler])
logger = logging.getLogger(__name__)
if __name__ == '__main__':
code = subprocess.call(["python3", "my_subprocess.py"])
logger.info(f'Subprocesses completed with code {code}')
my_subprocess.py
import sys
import logging
if __name__ == '__main__':
log_format = '%(asctime)s :: %(name)s :: %(levelname)-8s :: %(message)s'
log_filehandler = logging.FileHandler('my_subprocess.log')
log_streamhandler = logging.StreamHandler(stream=sys.stdout)
logging.basicConfig(format=log_format, level=logging.INFO,
handlers=[log_filehandler, log_streamhandler])
logger = logging.getLogger(__name__)
logger.info('Starting subprocess.')
logger.info('Completed subprocess.')
The reason why I create the log file from the subprocess, is because it also needs to be able to run on its own, and create the log file. Several such subprocesses are called by the main process. How can I restructure to keep these features and run on linux as expected?

unformated logging prints on the console (it only prints formated to a file) / Python

logger configuration to log to file and print to stdout would not work with me so:
I want to both print logging details in prog_log.txt AND in the console, I have:
# Debug Settings
Import logging
#logging.disable() #when program ready, un-indent this line to remove all logging messages
logger = logging.getLogger('')
logging.basicConfig(level=logging.DEBUG,
filename = 'prog_log.txt',
format='%(asctime)s - %(levelname)s - %(message)s',
filemode = 'w')
logger.addHandler(logging.StreamHandler())
logging.debug('Start of Program')
The above does print into the prog_log.txt file the logging details properly formated but it prints unformated logging messages multiple times as I re-run the program in the console like below..
In [3]: runfile('/Volumes/GoogleDrive/Mon Drive/MAC_test.py', wdir='/Volumes/GoogleDrive/Mon Drive/')
Start of Program
Start of Program
Start of Program #after third running
Any help welcome ;)

some of the logs are missing with TimedRotatingFileHandler

I need your help in solving this problem. So I am using Python's timedrotatingfilehandler lib to append logs for 24 hours in a log file and then rotate the log file. This works as expected but every time I run my script, timedrotatingfilehandler only logs till certain point. For example: If I have 10 log statement, each run only logs till 5 and then nothing gets logged in the log file. Appending happens correctly though. Not sure whats happening. Here is the python code:
#Configuring logger
LOG = logging.getLogger(__name__)
def config_Logs():
# format the log entries
global LOG
formatter = logging.Formatter('[%(asctime)s] : {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s')
handler = TimedRotatingFileHandler('collecting_logs.log',
when='h', interval=24, backupCount=0)
handler.setFormatter(formatter)
LOG.addHandler(handler)
LOG.setLevel(logging.DEBUG)
I am calling config_Logs() at the beginning of the script.

Python logger error

Hi I am trying a sample program using logger in python
import logging
import time,sys
import os
logger = logging.getLogger('myapp')
hdlr = logging.FileHandler('myapp1234.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logging.getLogger().setLevel(logging.DEBUG)
logger.error('We have a problem')
logger.info('While this is just chatty')
logger.debug("Sample")
hdlr.flush()
time.sleep(10)
logger.error('We have a problem')
logger.info('While this is just chatty')
logger.debug("Sample")
hdlr.close()
This code is not dynamically printing. I tried even handler.flush, sys.exit(0), sys.stdout.
When I try to open a file even by killing I am getting following error. Log is only printing after 120-200 seconds (sometimes it's taking even more).
How can I print immediately (at least by end of program)?
Did I miss any Handel closing.

Try removing the following statement.
time.sleep(10)

Python logging doesn't log when running via cron

If I run my script manually I see runlog.log populated with messages. When i run the script via cron, there's no output in my log file BUT i do see it in /var/spool/mail/root so I know the script works.
file_handler = logging.FileHandler("runlog.log", "a")
file_handler = logging.StreamHandler()
file_handler.setFormatter(default_formatter)
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logger.addHandler(file_handler)
...
logger.info("No new files")