We Keep Coding

Python - improving logging to file and console using multiprocessing

I am trying to download a file on CAN bus using python-can. It involves sending data very quickly (in the order of 2-3 messages per millisecond). I am trying to log to file these messages without impacting the speed of sending. Doing the file I/O slows down the sending due to the logging overhead. I tried various methods to improve this (including using queues and reading the queue from another thread but this was not much better - possibly due to GIL). Most of these tests started with using the Python logging module and trying various handlers (QueueHandler/QueueListener, MemoryHandler, etc).
I've managed to make some significant improvements by moving the file I/O into a separate process. I initially ran into an issue with the overhead of sending data from one process to another - so I now buffer it. Now, instead of taking 150% longer with direct file I/O in the main process, I see ~20% increase in time.
I thought that, since this is running in another process, I could also print() the data to console (which I know is relative expensive) but I see a huge increase in the file download time.
What is happening that means the print() affects the main process even though it is running in a child process?
Code below:
file_logger_mp() is called from the main process and it starts the child process that does the logging. The main process then uses the log_hdl function to add a message to the buffer. When the buffer reaches a certain size (100) it is sent to the child process for logging to file or printing to console.
Device: rpi4. And the main process uses asyncio, in case that affects it.
def file_logger_mp(logger_name: str, log_file_pth: str):
conn_rec, conn_send = multiprocessing.Pipe()
log_hdl_c = MyLogger(conn_send)
log_hdl = log_hdl_c.log_hdl # This is used by main code to provide log messages to child process
listener = MyProcess(conn_rec, log_file_pth)
atexit.register(log_hdl_c.final_flush, listener)
listener.start() # Start the child process
return log_hdl, listener
class MyLogger():
def __init__(self, conn_send) -> None:
self.buffer = []
self.conn_send = conn_send
def log_hdl(self, msg):
self.buffer.append(msg)
if len(self.buffer) > 100:
self.conn_send.send(self.buffer)
self.buffer.clear()
def final_flush(self, listener):
self.conn_send.send(self.buffer)
listener.terminate()
class MyProcess(multiprocessing.Process):
def __init__(self, queue, f_hdl):
multiprocessing.Process.__init__(self)
self.exit = multiprocessing.Event()
self.queue = queue
self.f_hdl = f_hdl
def run(self):
f = open(self.f_hdl, "w+")
while not self.exit.is_set():
try:
record = self.queue.recv()
for msg in record:
output = str(msg)
f.write(output+'\n')
print(output) # This `print()` causes large delays to main process?!
record.clear()
except Exception:
import sys, traceback
print('Whoops! Problem:', file=sys.stderr)
traceback.print_exc(file=sys.stderr)
for msg in record: # Flush any pending records before finishing
f.write(str(msg)+'\n')
f.close()
def terminate(self):
self.exit.set()

Stop systemd service running a python program

I created a service to run a Python program and added some lines of code to create a lock to avoid launching it twice.
Unfortunately I don't know how to configure the service to stop the running program correctly. When running the stop command it doesn't delete the lock, then I can't start the service anymore. If I execute the program myself via CLI and exit with a Ctrl+C, the lock is deleted.
I've read the manual about KillMode, ExecStop and Signal. My understanding is that the default configuration was the one I needed.
Any help please?
Main program
if __name__ == '__main__':
#Creating lock to avoid launching program twice
lock = pathlib.Path("program.lock")
if not lock.exists():
lock_acquired_on = datetime.now()
with open('program.lock', 'w') as lock:
lock.write(f'Lock acquired on {lock_acquired_on}')
logger.info('Added lock file to avoid running the program twice.')
try:
while True:
#Doing stuff here
except KeyboardInterrupt:
close_program() #Close other threads
#Removing the lock file
os.remove(pathlib.Path("program.lock"))
else:
with open('program.lock', 'r') as lock:
lock_acquisition_time = str(lock.readlines()[0])
logger.info('Programme Maquette Status is already running.')
logger.info(lock_acquisition_time)
Service
[Unit]
Description=Programme Maquette IoT End-to-End
After=multi-user.target
Conflicts=getty#tty1.service
[Service]
WorkingDirectory=/home/pi/Documents/ProductionMaquette
Type=simple
ExecStart=/usr/local/bin/python3.8 /home/pi/Documents/ProductionMaquette/Lo_main.py
StandardInput=tty-force
[Install]
WantedBy=multi-user.target

Systemd sends the SIGTERM to the process - so you need to handle that.
So following little example uses the a signal handler for SIGTERMto clean up a file. Actually it uses atexit to clean up the file, as that handles standard exit conditions as well and a signal handler to initiate in "normal" closing down of the process on receiving the SIGTERM signal
import atexit
import signal
import os
locking_file = "/var/lock/my_service.lock"
if __name__ == '__main__':
def clean_lock():
# atexit handler to clean up a file
os.remove(locking_file)
def signal_term_handler(sigNum, frame):
# on receiving a signal initiate a normal exit
raise SystemExit('terminating')
with open("test_file.lock", "w") as lock:
while True:
lock.write("x")
time.sleep(10)
# register the cleanup handler
atexit.register(clean_lock)
# register the signal handler
signal.signal(signal.SIGTERM, signal_term_handler)
As a note: there is a file locking library you might want to look at:https://pypi.org/project/filelock/ as that should handle that use case as well.
It is not only testing for presents of a file but uses the os-file locking mechanism. I.e. not only the existence of the file is tested - but if it can be locked as well. In effect that means even if the file still exists but the previous process died it is not a problem, as the file is no longer locked.

Allow process to finish rather than be interrupted when SIGTERM is used in Python 3

I am developing some code which I need to gracefully shutdown when a sigterm signal is sent from the command line in unix. I found this example https://stackoverflow.com/a/31464349/7019148 which works great, but there's one problem with it.
Code:
import signal
import time
class GracefulKiller:
def __init__(self):
signal.signal(signal.SIGTERM, self.exit_gracefully)
self.kill_now = False
def exit_gracefully(self, signum, frame):
self.kill_now = True
def run_something(self):
print("starting")
time.sleep(5)
print("ending")
if __name__ == '__main__':
killer = GracefulKiller()
print(os.getpid())
while True:
killer.run_something()
if killer.kill_now:
break
print("End of the program. I was killed gracefully :)")
When you pass the kill command kill -15 <pid>, the run_something method is interrupted and the process killed, gracefully. However, is there a way to do this so that the run_something method can complete before the process is killed? I.e. prevent the interruption?
Desired output:
>>> starting
*kill executed during the middle sleep*
>>> ending
>>> End of the program. I was killed gracefully :)
My use case is that this will be turned into a download script and if I want to terminate the process, I would like the process to finish downloading before terminating...

thread.join() waits till the thread finishes even if an exit signal was caught.
import threading
import Queue
import time
def download_for(seconds=5):
for i in range(seconds):
print("downloading...")
time.sleep(1)
print("finished download")
download_thread = threading.Thread(target=download_for, args=(3,))
download_thread.start()
# this waits till the thread finishes even if an exit signal was received
download_thread.join()
# this would just stop the download midway
# download_for(seconds=5)

The answer is in the original question. I am just leaving this here for future Google searchers.
I never had an issue in the first place, my terminal was just having a problem printing 'ending' following the kill command.

Multiprocessing Process locking up forever in PyCharm

I've been having an odd issue with PyCharm and subprocesses created by the multiprocessing library locking up forever. I'm using Windows with Python 3.5. What I'm trying to do is:
Start a background thread to block on stdin (waiting for input)
Have the main thread check occasionally for input from stdin and then delegate the work to Python processes created using multiprocessing
However, I've found that newly created multiprocessing Processes lock up forever if and only if the following conditions are met:
I'm running the code via Pycharm (both the latest and older versions)
The background thread is blocking on stdin
Here's the simplest example I can create that reproduces the problem:
import multiprocessing
import threading
import sys
def noop():
pass
def consume():
while True:
sys.stdin.readline()
if __name__ == '__main__':
# create a daemon thread to block on stdin
thread = threading.Thread(target=consume, daemon=True)
thread.start()
# create a background process
process = multiprocessing.Process(target=noop)
process.start()
I've Googled various combinations of "PyCharm stdin multiprocessing hang ..." and had no luck at finding an explanation, and I can't figure out why a thread of the main process blocking on stdin should ever cause a subprocess to also block/hang, let alone why it would only happen when running the script in PyCharm. The only think I can guess is that there might be some monkey-patching of either stdin or the multiprocessing library going on.
Has anyone else encountered this problem? Can anyone explain to me why this only occurs in PyCharm, and how I can make it work regardless of the Python editor I'm using?

I faced the same problem when I was trying to do multiple API calls to fetch data from a remote server. I replaced multiprocessing dummy with ThreadPoolExecutor. It works in the same way as dummy.
Following is a short snippet of a running code to write the response to a json file:
uids = [] # an array of the requisite parameters used in requests
with open('flight_config.json', 'w') as f:
futures = []
for i in range(chunk_index, len(uids)):
print('For uid[{}], fetching started:'.format(i))
chunk_index += 1
auth_token = get_header()
with ThreadPoolExecutor(max_workers=7) as executor:
future_to_url = {executor.submit(fetch_response_from_api, uid=uid, auth_token=auth_token): uid for uid in
uids[i]}
for future in concurrent.futures.as_completed(future_to_url):
result = future_to_url[future]
try:
data = future.result()
print(data)
except Exception as exc:
print('%r generated an exception: %s' % (result, exc))
else:
print('%r page is %d bytes' % (result, len(data)))

Run a script with web server and maintain data to be used later

Note: I want to implement this without using any framework.
I have to create an web application using python. The application should maintain a running average of the CPU usage for each process over the past 60 seconds. It should should act as a web server and when it gets a request, it should return the current average for each process. Following are the scripts I've written. record_usage.py is a script which I want to run as soon as the server.py is run. So that it runs and maintain the cpu usage data, which I intend to read whenever I get an XHR request and send it back to the client.
So, my problem is how do I invoke this requirement? I tried running record_usage.py using subprocess.POPEN after starting the server. record_usage.py starts running in background as well. But when I try accessing the data created by it, the class object I create is not the one it uses but a new one. How to complete this link?
Kindly ask things that I could not make clear.
Latest changes in server.py
if __name__ == '__main__':
RU_OBJ = RU(settings.SAMPLING_FREQ, settings.AVG_INTERVAL)
RU_LOCK = RLock()
# Record CPU usage in a thread.
ru_thread = Thread(target=RU_OBJ.record, args=(RU_LOCK,))
ru_thread.daemon = True
ru_thread.start()
# Run server.
run()
Latest change in record_usage.py
def record(self, lock):
while True:
with lock:
self.add_processes()
time.sleep(self.sampling_freq)
Is this a proper way of applying locks? A similar lock is being applied when am reading the processes information. Would it work?
Added the functions:
def add_processes(self,):
for _process in psutil.process_iter():
try:
new_proc = _process.as_dict(attrs=['cpu_times', 'name', 'pid',
'status'])
except psutil.NoSuchProcess:
continue
pid, (user, _sys) = new_proc['pid'], new_proc.pop('cpu_times')
# Get or create details object for the process.
existing = self.processes.setdefault(pid, new_proc)
# Get or create queue object for the CPU times of the process.
queue_dict = self.process_queue.setdefault(pid, dict())
# User CPU time.
user_q = queue_dict.setdefault('user_q', PekableQueue(self.avg_interval))
user_q.enqueue(user)
user_avg = get_avg(user_q)
# System CPU time.
sys_q = queue_dict.setdefault('sys_q', PekableQueue(self.avg_interval))
sys_q.enqueue(_sys)
sys_avg = get_avg(sys_q)
# Update the details object for the process.
existing.update(user_avg=user_avg, sys_avg=sys_avg, **new_proc)
def get_curr_processes(self):
return [self.processes[pid] for pid in psutil.get_pid_list()
if pid in self.processes]

To collect statistics in another thread:
if __name__ == '__main__':
from threading import Thread, Lock
import record_usage
lock = Lock()
t = Thread(target=record_usage.record, args=[lock])
t.daemon = True
t.start()
run(lock)
If you change some shared data in one thread and read it in another then you could protect the places where you access/change the value with a lock:
#...
with self.lock:
existing = self.processes.setdefault(pid, new_proc)
#...
with self.lock:
existing.update(user_avg=user_avg, sys_avg=sys_avg, **new_proc)
#...
def get_curr_processes(self):
with self.lock:
return [self.processes[pid] for pid in psutil.get_pid_list()
if pid in self.processes]
It is essential that self.lock is the same object in all threads. If self.processes is a dict then you don't need to use a lock in CPython. The methods are implemented in C and the interpreter doesn't release GIL (global lock) while calling them i.e., only one thread at a time accesses the dict.