We Keep Coding

How can I overwrite a text after it's printed a few lines before?

I want to make a task/time manager. I want the list of tasks to be at the top and a question to add a new task if the user wants, but the problem is that I can't keep my to-do list in the top and I can't make it updated. What should I do?
This is my code, this is the form I want and the code I've written:
TASKS:(I want this to be updated)
1)--------
2)----------
3)----------
|
v
do you want to add a task(YES/NO):
when do you want to start the task (HOUR:MINUTE):
task_num = 0
k = False
print("YOUR TASKS")
if k == True:
print("\r", tasks)
print("tasks" + ":" + str(task_num), "\n\n")
active = True
while active:
add_task = input("do you want to add a task(YES/NO):",)
if add_task != 'YES':
active = False
else:
k = True
start_time = input("when do you want to start the task (HOUR:MINUTE):")
end_time = input("when do you want to complete it:")
task_name = input("what do you want to call it:")
start_h, start_m = start_time.split(":")
end_h, end_m = end_time.split(":")
duration = str(abs(int(end_h) - int(start_h))) + ":" + str(abs(int(end_m) - int(start_m)))
task = {"task name": task_name, "start time": start_time, "end time": end_time, "duration": duration}
task_num += 1
tasks[task_num] = task
for key in tasks.keys():
task_key = tasks[key]
print(key,end=")")
for key1 in task_key.keys():
print(key1, ":", task_key[key1], end='|',)
print('\n')

On POSIX systems clear -x command works like a charm. It preserves the current scroll buffer and produces almost zero flickering. Combine that with hiding the cursor, ah it is marvelous.
import sys
import subprocess
try:
sys.stdout.write('\033[?25l') # hide cursor
subprocess.run(['clear', '-x'])
finally:
sys.stdout.write('\033[?25h') # show cursor
sys.stdout.flush()
On Windows, on the other hand, you'll probably need colorama package installed and initialized.See https://github.com/tartley/colorama
On Windows cmd:
import sys
import shutil
height = shutil.get_terminal_size().lines
h = height * '\n'
# move cursor down and up.
sys.stdout.write(f'{h}\033[{height}A')
sys.stdout.flush()
On Windows Terminal and other modern terminal emulators:
import sys
import shutil
height = shutil.get_terminal_size().lines
# clear the screen with the ANSI sequence.
sys.stdout.write((height - 1) * '\n' + '\033[2J')
sys.stdout.flush()
Play with them and see what works for you.

is there an easy way to add a progress bar/counter where I can add a line to increment it every so often - Not Timed

I have a script that is basically complete. I'd like to add some sort of a progress bar to it, instead of printing out each step as it passes by
is there anything that will let me do this.
setup a progress widget/counter/loop
give it a command function to increment
do some script
add the code to advance/increment the progress bar
do some more script
add the code to advance/increment the progress bar
do some more script
add the code to advance/increment the progress bar
do some more script
add the code to advance/increment the progress bar
also, can you please give me an example of some sort
I've looked at 3 or 4 different "progress bar" type libraries, and none give an example of doing it this way
all of the examples I seem to find want to do it by time or by byte size for downloading files

There is a number of progress bars in PIP, I recommend ezprogress if you run python3.
from ezprogress.progressbar import ProgressBar
import time
# Number of steps in your total script
steps_needed = 100
current_step = 0
# setup progress bar
pb = ProgressBar(steps_needed, bar_length=100)
pb.start()
# Do what your script wants
...
# Increment counter
current_step += 1
pb.update(current_step)
# Do what your script wants
...
# When you are done you can force the progress bar to finish
PB.finished()
The progress bar did not support turning off time estimation, however it is now possible in the newest version, just upgrade from PIP.
To turn off time estimation the progress bar just needs to be started with the parameter no_time=True like in the code below:
pb = ProgressBar(steps_needed, bar_length=100, no_time=True)

create your progressbar.py module
import sys
import copy
currentProgressCnt = 0
progressCntMax = 0 #
progressBarWidth = 50 # in chars
scaleFctr = 0.0
tasksToDo = []
class ProgressIndicator:
def showProgress(self):
global progressCntMax
global currentProgressCnt
cr = "\r"
progressChar = '#'
fillChar = '.'
progressBarDone = currentProgressCnt*progressChar*scaleFctr
progressBarRemain = fillChar*(progressCntMax - currentProgressCnt)*scaleFctr
percent = str(int((float(currentProgressCnt)/float(progressCntMax))*100)) + " % completed "
taskId = '(' + tasksToDo[currentProgressCnt - 1] + ') '
quote = str(currentProgressCnt) + '/' + str(progressCntMax) + ' '
sys.stdout.write(cr + progressBarDone + progressBarRemain + ' ' + percent + taskId + quote)
sys.stdout.flush()
if currentProgressCnt == progressCntMax:
print
def incProgress(self):
global currentProgressCnt
currentProgressCnt += 1
def setLastStep(self, size):
global progressCntMax, scaleFctr
progressCntMax = size
scaleFctr = progressBarWidth / progressCntMax
def setTaskList(self, taskList):
global tasksToDo
tasksToDo = copy.copy(taskList)
self.setLastStep(len(tasksToDo))
in main, use the ProgressIndicator class like this:
from progressbar import ProgressIndicator
import time
import datetime
#########################################
### MAIN ###
### SIMULATION ###
#########################################
# your procedure list you have to run
toDoList = ['proc1', 'proc2', 'proc3', 'proc1', 'proc4', 'proc5',
'proc6', 'proc7', 'proc21', 'proc32', 'proc43', 'proc51',
'proc4', 'proc65', 'proc76', 'proc87']
progressLine = ProgressIndicator() # create your indicator
progressLine.setTaskList(toDoList) # set params
# your main work
i = 0; lastTask = len(toDoList)
# log the start
startTime = str(datetime.datetime.now())
print ( startTime + " main started")
while i < lastTask:
# run your task list here
time.sleep(1) # simulating your toDoList[i]() run
i += 1
progressLine.incProgress() # use when task done, incrase progress
progressLine.showProgress() # use for update display
# work is done, log the end
endTime = str(datetime.datetime.now())
print ( endTime + " main finished")

Multiprocessing function not writing to file or printing

I'm working on a Raspberry Pi (3 B+) making a data collection device and I'm
trying to spawn a process to record the data coming in and write it to a file. I have a function for the writing that works fine when I call it directly.
When I call it using the multiprocess approach however, nothing seems to happen. I can see in task monitors in Linux that the process does in fact get spawned but no file gets written, and when I try to pass a flag to it to shut down it doesn't work, meaning I end up terminating the process and nothing seems to have happened.
I've been over this every which way and can't see what I'm doing wrong; does anyone else? In case it's relevant, these are functions inside a parent class, and one of the functions is meant to spawn another as a thread.
Code I'm using:
from datetime import datetime, timedelta
import csv
from drivers.IMU_SEN0 import IMU_SEN0
import multiprocessing, os
class IMU_data_logger:
_output_filename = ''
_csv_headers = []
_accelerometer_headers = ['Accelerometer X','Accelerometer Y','Accelerometer Z']
_gyroscope_headers = ['Gyroscope X','Gyroscope Y','Gyroscope Z']
_magnetometer_headers = ['Bearing']
_log_accelerometer = False
_log_gyroscope= False
_log_magnetometer = False
IMU = None
_writer=[]
_run_underway = False
_process=[]
_stop_value = 0
def __init__(self,output_filename='/home/pi/blah.csv',log_accelerometer = True,log_gyroscope= True,log_magnetometer = True):
"""data logging device
NOTE! Multiple instances of this class should not use the same IMU devices simultaneously!"""
self._output_filename = output_filename
self._log_accelerometer = log_accelerometer
self._log_gyroscope = log_gyroscope
self._log_magnetometer = log_magnetometer
def __del__(self):
# TODO Update this
if self._run_underway: # If there's still a run underway, end it first
self.end_recording()
def _set_up(self):
self.IMU = IMU_SEN0(self._log_accelerometer,self._log_gyroscope,self._log_magnetometer)
self._set_up_headers()
def _set_up_headers(self):
"""Set up the headers of the CSV file based on the header substrings at top and the input flags on what will be measured"""
self._csv_headers = []
if self._log_accelerometer is not None:
self._csv_headers+= self._accelerometer_headers
if self._log_gyroscope is not None:
self._csv_headers+= self._gyroscope_headers
if self._log_magnetometer is not None:
self._csv_headers+= self._magnetometer_headers
def _record_data(self,frequency,stop_value):
self._set_up() #Run setup in thread
"""Record data function, which takes a recording frequency, in herz, as an input"""
previous_read_time=datetime.now()-timedelta(1,0,0)
self._run_underway = True # Note that a run is now going
Period = 1/frequency # Period, in seconds, of a recording based on the input frequency
print("Writing output data to",self._output_filename)
with open(self._output_filename,'w',newline='') as outcsv:
self._writer = csv.writer(outcsv)
self._writer.writerow(self._csv_headers) # Write headers to file
while stop_value.value==0: # While a run continues
if datetime.now()-previous_read_time>=timedelta(0,1,0): # If we've waited a period, collect the data; otherwise keep looping
print("run underway value",self._run_underway)
if datetime.now()-previous_read_time>=timedelta(0,Period,0): # If we've waited a period, collect the data; otherwise keep looping
previous_read_time = datetime.now() # Update previous readtime
next_row = []
if self._log_accelerometer:
# Get values in m/s^2
axes = self.IMU.read_accelerometer_values()
next_row += [axes['x'],axes['y'],axes['z']]
if self._log_gyroscope:
# Read gyro values
gyro = self.IMU.read_gyroscope_values()
next_row += [gyro['x'],gyro['y'],gyro['z']]
if self._log_magnetometer:
# Read magnetometer value
b= self.IMU.read_magnetometer_bearing()
next_row += b
self._writer.writerow(next_row)
# Close the csv when done
outcsv.close()
def start_recording(self,frequency_in_hz):
# Create recording process
self._stop_value = multiprocessing.Value('i',0)
self._process = multiprocessing.Process(target=self._record_data,args=(frequency_in_hz,self._stop_value))
# Start recording process
self._process.start()
print(datetime.now().strftime("%H:%M:%S.%f"),"Data logging process spawned")
print("Logging Accelerometer:",self._log_accelerometer)
print("Logging Gyroscope:",self._log_gyroscope)
print("Logging Magnetometer:",self._log_magnetometer)
print("ID of data logging process: {}".format(self._process.pid))
def end_recording(self,terminate_wait = 2):
"""Function to end the recording multithread that's been spawned.
Args: terminate_wait: This is the time, in seconds, to wait after attempting to shut down the process before terminating it."""
# Get process id
id = self._process.pid
# Set stop event for process
self._stop_value.value = 1
self._process.join(terminate_wait) # Wait two seconds for the process to terminate
if self._process.is_alive(): # If it's still alive after waiting
self._process.terminate()
print(datetime.now().strftime("%H:%M:%S.%f"),"Process",id,"needed to be terminated.")
else:
print(datetime.now().strftime("%H:%M:%S.%f"),"Process",id,"successfully ended itself.")
====================================================================
ANSWER: For anyone following up here, it turns out the problem was my use of the VS Code debugger which apparently doesn't work with multiprocessing and was somehow preventing the success of the spawned process. Many thanks to Tomasz Swider below for helping me work through issues and, eventually, find my idiocy. The help was very deeply appreciated!!

I can see few thing wrong in your code:
First thing
stop_value == 0 will not work as the multiprocess.Value('i', 0) != 0, change that line to
while stop_value.value == 0
Second, you never update previous_read_time so it will write the readings as fast as it can, you will run out of disk quick
Third, try use time.sleep() the thing you are doing is called busy looping and it is bad, it is wasting CPU cycles needlessly.
Four, terminating with self._stop_value = 1 probably will not work there must be other way to set that value maybe self._stop_value.value = 1.
Well here is a pice of example code based on the code that you have provided that is working just fine:
import csv
import multiprocessing
import time
from datetime import datetime, timedelta
from random import randint
class IMU(object):
#staticmethod
def read_accelerometer_values():
return dict(x=randint(0, 100), y=randint(0, 100), z=randint(0, 10))
class Foo(object):
def __init__(self, output_filename):
self._output_filename = output_filename
self._csv_headers = ['xxxx','y','z']
self._log_accelerometer = True
self.IMU = IMU()
def _record_data(self, frequency, stop_value):
#self._set_up() # Run setup functions for the data collection device and store it in the self.IMU variable
"""Record data function, which takes a recording frequency, in herz, as an input"""
previous_read_time = datetime.now() - timedelta(1, 0, 0)
self._run_underway = True # Note that a run is now going
Period = 1 / frequency # Period, in seconds, of a recording based on the input frequency
print("Writing output data to", self._output_filename)
with open(self._output_filename, 'w', newline='') as outcsv:
self._writer = csv.writer(outcsv)
self._writer.writerow(self._csv_headers) # Write headers to file
while stop_value.value == 0: # While a run continues
if datetime.now() - previous_read_time >= timedelta(0, 1,
0): # If we've waited a period, collect the data; otherwise keep looping
print("run underway value", self._run_underway)
if datetime.now() - previous_read_time >= timedelta(0, Period,
0): # If we've waited a period, collect the data; otherwise keep looping
next_row = []
if self._log_accelerometer:
# Get values in m/s^2
axes = self.IMU.read_accelerometer_values()
next_row += [axes['x'], axes['y'], axes['z']]
previous_read_time = datetime.now()
self._writer.writerow(next_row)
# Close the csv when done
outcsv.close()
def start_recording(self, frequency_in_hz):
# Create recording process
self._stop_value = multiprocessing.Value('i', 0)
self._process = multiprocessing.Process(target=self._record_data, args=(frequency_in_hz, self._stop_value))
# Start recording process
self._process.start()
print(datetime.now().strftime("%H:%M:%S.%f"), "Data logging process spawned")
print("ID of data logging process: {}".format(self._process.pid))
def end_recording(self, terminate_wait=2):
"""Function to end the recording multithread that's been spawned.
Args: terminate_wait: This is the time, in seconds, to wait after attempting to shut down the process before terminating it."""
# Get process id
id = self._process.pid
# Set stop event for process
self._stop_value.value = 1
self._process.join(terminate_wait) # Wait two seconds for the process to terminate
if self._process.is_alive(): # If it's still alive after waiting
self._process.terminate()
print(datetime.now().strftime("%H:%M:%S.%f"), "Process", id, "needed to be terminated.")
else:
print(datetime.now().strftime("%H:%M:%S.%f"), "Process", id, "successfully ended itself.")
if __name__ == '__main__':
foo = Foo('/tmp/foometer.csv')
foo.start_recording(20)
time.sleep(5)
print('Ending recording')
foo.end_recording()

Profiling Serial communication using timeit

I need to communicate with an embedded system over RS232. For this I want to profile the time it takes to send a response to each command.
I've tested this code using two methods: datetime.now() and timeit()
Method #1
def resp_time(n,msg):
"""Given number of tries - n and bytearray list"""
msg = bytearray(msg)
cnt = 0
timer = 0
while cnt < n:
time.sleep(INTERVAL)
a = datetime.datetime.now()
ser.flush()
ser.write(msg)
line = []
for count in ser.read():
line.append(count)
if count == '\xFF':
# print line
break
b = datetime.datetime.now()
c = b-a
# print c.total_seconds()*1000
timer = timer + c.total_seconds()*1000
cnt = cnt + 1
return timer/n
ser = serial.Serial(COMPORT,BAUDRATE,serial.EIGHTBITS, serial.PARITY_NONE, serial.STOPBITS_ONE, timeout=16)
if ser.isOpen():
print "Serial port opened at: Baud:",COMPORT,BAUDRATE
cmd = read_file()
# returns a list of commands [msg1,msg2....]
n = 100
for index in cmd:
timer = resp_time(n,index)
print "Time in msecs over %d runs: %f " % (n,timer)
Method #2
def com_loop(msg):
msg = bytearray(msg)
time.sleep(INTERVAL)
ser.flush()
ser.write(msg)
line = []
for count in ser.read():
line.append(count)
if count == '\xFF':
break
if __name__ == '__main__':
import timeit
ser = serial.Serial(COMPORT,BAUDRATE,serial.EIGHTBITS, serial.PARITY_NONE, serial.STOPBITS_ONE, timeout=16)
if ser.isOpen():
print "Serial port opened at: Baud:",COMPORT,BAUDRATE
cmd = read_file()
# returns a list of commands [msg1,msg2....]
n = 100
for index in cmd:
t = timeit.timeit("com_loop(index)","from __main__ import com_loop;index=%s;" % index,number = n)
print t/100
With datetime I get 2 milli-sec to execute a command & with timeit I get 200 milli-sec for the same command.
I suspect I'm not calling timeit() properly, can someone point me in the right direction?

I'd assume 200µs is closer to the truth, considering your comport will have something like 115200baud; assuming messages are 8 bytes long, transmitting one message would take about 9/115200 s ~= 10/100000 = 1/10,000 = 100µs on the serial line alone. Being faster than that will be pretty impossible.
Python is definitely not the language of choice to do timing characterization at these scales. You will need to get a logic analyzer, or work very close to the serial controller (which I hope is directly attached to your PC's IO controller and not some USB devices, because that will introduce latencies in the same order of magnitude, at least). If you're talking about microseconds, the limiting factor in measurement is usually the random time it takes for your PC to react to an interrupt, the OS to run the interrupt service routine, the scheduler to continue your userland process, and then starts python with its levels and levels of indirection. You're basically measuring the size of single grains of sand by holding a banana next to them.

Use only the most recent value read from an Arduino in a python script, not the values stored in the buffer?

I have been working on a project using Python to read values from an arduino and then control video cameras. The Arduino controls two ultrasonic sensors and reports distance in cm. The python script then reads the distances from the Arduino using ser.readline(). When the script reads values outside the range everything works fine. However if it goes into the loop for the distance inside the required range it works correctly once and then reads old values from the Arduino instead of current "live" values which causes it to continue the record loop instead of exiting the loop. What can I do to get rid of the old values in the buffer and only read the most current value? I have found several methods and tested them but so far no luck.
Here is the code I am using (i know its not well written but its my first try using python and writing code outside of matlab)
import sys, time
import serial
import cv
import os
from time import strftime
#Create window for Camera 0
cv.NamedWindow("Camera 0", cv.CV_WINDOW_AUTOSIZE)
capture0 = cv.CreateCameraCapture(2)
cv.ResizeWindow("Camera 1", 640, 480)
cv.MoveWindow("Camera 0", 0, 0)
#Create window for Camera 1
cv.NamedWindow("Camera 1", cv.CV_WINDOW_AUTOSIZE)
capture1 = cv.CreateCameraCapture(1)
cv.MoveWindow("Camera 1", 150, 150)
#Initialize connection to Arduino
arduino = serial.Serial('COM12', 9600)
connected = False
#Confirm that Arduino is connected and software is able to read inputs
while not connected:
serin = arduino.readline()
connected = True
f = 'Sensor Connected'
print f
'''#Dummy variables for testing
value1 = 145
value2 = 30'''
#Initialize video record on as false (0)
vid = 0
#Initialize counter
counter_vid = 0
counter = 0
Accel = 1
def Camera0():
frame0=cv.QueryFrame(capture0)
cv.WriteFrame(writer0,frame0)
cv.ShowImage("Camera 0", frame0)
def Camera1():
frame1=cv.QueryFrame(capture1)
cv.WriteFrame(writer1,frame1)
cv.ShowImage("Camera 1", frame1)
while True:
status = arduino.readline()
value1=int((status[6:10]))-1000
value2=int((status[17:21]))-1000
print(value1)
print(value2)
if value1>180 and value2>180 and vid==0:
vid = 0
elif value1>180 and value2>180 and vid==1:
vid = 0
elif value1<180 and vid==0 or value2<180 and vid==0:
filename0 = strftime("OUTPUT\%Y_%m_%d %H_%M_%S") + " camera0.avi"
writer0=cv.CreateVideoWriter(filename0, 1, 15.0, (640,480), is_color=1)
filename1 = strftime("OUTPUT\%Y_%m_%d %H_%M_%S") + " camera1.avi"
writer1=cv.CreateVideoWriter(filename1, 1, 15.0, (640,480), is_color=1)
vid=1
while counter_vid<25 and vid==1:
Camera0()
Camera1()
counter_vid += 1
print(counter_vid)
cv.WaitKey(10)
else:
while counter_vid<25 and vid==1:
Camera0()
Camera1()
counter_vid += 1
print(counter_vid)
cv.WaitKey(10)
cv.WaitKey(25)
counter_vid = 0
counter += 1
print('End of Loop Counter')
print(counter)

You're right about the buffer filling up. You need a way to always get the most recent value out of the buffer.
I would suggest replacing this:
status = arduino.readline()
with this:
status = getLatestStatus()
and then further up towards the top, by your camera functions:
def getLatestStatus():
while arduino.inWaiting() > 0:
status = arduino.readline()
return status
This function getLatestStatus will go through the entire buffer every time it is called and only return the latest status, disregarding all the statuses returned in the meantime.
Your other option is to modify the "firmware" for your arduino to return a distance sensor value every time it receives a command, (say "M\n") so that way you don't have to worry about buffer problems. That's what I did for an arduino-powered ultrasonic distance device and I felt it was cleaner than the "read through the entire buffer" solution. It will introduce a bit more latency into your distance measurement though.