We Keep Coding

Micro Switch with pyserial RS232 starts/stops a timer in a tkinter thread but continues to run even when stopped

I have been using a micro switch connected to an RS232/USB serial converter cable on my windows PC to start stop and reset a timer.
The program runs smoothly most of the time but every so often updating the timer widget gets stuck running and the timer will not stop.
With the serial protocol i want to receive 1 byte b'\x00' for off and anything that's not b'\x00' should signify on.
I have replaced the micro switch with button widgets to simulate the switch and don't get the same error or i just have not kept at it for long enough.
It could be an issue with the RS232 causing an error i cannot see but my knowledge on this is sketchy and have exhausted all avenues looking online for any information on this.
import time
import sys
import serial
import threading
from tkinter import *
from tkinter import ttk
class Process(Frame):
def __init__(self, root, parent=None, **kw):
Frame.__init__(self, parent, kw)
self.root = root
self._cycStart = 0.0
self._cycTimeElapsed = 0.0
self._cycRunning = 0.0
self.cycTimeStr = StringVar()
self.cycTime_label_widget()
self.ser = serial.Serial(
port='COM4',
baudrate=1200,
timeout=0
)
self.t1 = threading.Thread(target=self.start_stop, name='t1')
self.t1.start()
def initUI(self):
root.focus_force()
root.title("")
root.bind('<Escape>', lambda e: root.destroy())
def cycTime_label_widget(self):
# Make the time label
cycTimeLabel = Label(root, textvariable=self.cycTimeStr, font=
("Ariel 12"))
self._cycleSetTime(self._cycTimeElapsed)
cycTimeLabel.place(x=1250, y=200)
cycTimeLabel_2 = Label(root, text="Cycle Timer:", font=("Ariel
12"))
cycTimeLabel_2.place(x=1150, y=200)
def _cycleUpdate(self):
""" Update the label with elapsed time. """
self._cycTimeElapsed = time.time() - self._cycStart
self._cycleSetTime(self._cycTimeElapsed)
self._cycTimer = self.after(50, self._cycleUpdate)
def _cycleSetTime(self, elap):
""" Set the time string to Minutes:Seconds:Hundreths """
minutes = int(elap/60)
seconds = int(elap - minutes*60.0)
hseconds = int((elap - minutes*60.0 - seconds)*100)
self.cycTimeStr.set('%02d:%02d:%02d' % (minutes, seconds,
hseconds))
return
def cycleStart(self):
""" Start the stopwatch, ignore if running. """
if not self._cycRunning:
self._cycStart = time.time() - self._cycTimeElapsed
self._cycleUpdate()
self._cycRunning = 1
else:
self.cycleReset()
def cycleStop(self):
""" Stop the stopwatch, ignore if stopped. """
if self._cycRunning:
self.after_cancel(self._cycTimer)
self._cycTimeElapsed = time.time() - self._cycStart
self._cycleSetTime(self._cycTimeElapsed)
self._cycRunning = 0
self._cycTimeElapsed = round(self._cycTimeElapsed, 1)
self.cycleTimeLabel = Label(root, text=(self._cycTimeElapsed,
"seconds"), font=("Ariel 35"))
self.cycleTimeLabel.place(x=900, y=285)
self.cycleReset()
def cycleReset(self):
""" Reset the stopwatch. """
self._cycStart = time.time()
self._cycTimeElapsed = 0
self._cycleSetTime(self._cycTimeElapsed)
def start_stop(self):
while True :
try:
data_to_read = self.ser.inWaiting()
if data_to_read != 0: # read if there is new data
data = self.ser.read(size=1).strip()
if data == bytes(b'\x00'):
self.cycleStop()
print("Off")
elif data is not bytes(b'\x00'):
self.cycleStart()
print("On")
except serial.SerialException as e:
print("Error")
if __name__ == '__main__':
root = Tk()
application = Process(root)
root.mainloop()
I expect the timer to start running when the micro switch is pressed. when depressed it should stop and reset back to zero and wait for the next press

With a better understanding of what you're trying to do better solutions come to mind.
As it turns out, you're not using your serial port to send or receive serial data. What you're actually doing is wiring a switch to its RX line and toggling it manually with a mechanical switch, feeding a high or low level depending on the position of the switch.
So what you're trying to do is emulating a digital input line with the RX line of your serial port. If you take a look a how a serial port works you'll see that when you send a byte the TX line toggles from low to high at the baud rate, but on top of the data you have to consider the start and stop bits. So, why your solution works (at least sometimes): that's easy to see when you look at a scope picture:
This is a screenshot of the TX line sending the \x00 byte, measured between pins 3 (TX) and 5 (GND) with no parity bit. As you can see the step only lasts for 7.5 ms (with a 1200 baud rate). What you are doing with your switch is something similar but ideally infinitely long (or until you toggle your switch back, which will be way after 7.5 ms no matter how fast you do it). I don't have a switch to try but if I open a terminal on my port and use a cable to shortcircuit the RX line to pin 4 (on a SUB-D9 connector) sometimes I do get a 0x00 byte, but mostly it's something else. You can try this experiment yourself with PuTTy or RealTerm and your switch, I guess you'll get better results but still not always the byte you expect because of the contacts bouncing.
Another approach: I'm sure there might be ways to improve on what you have, maybe reducing the baud rate to 300 or 150 bps, checking for a break in the line or other creative ideas.
But what you're trying to do is more akin to reading a GPIO line, and actually, the serial port has several digital lines intended (in the old days) for flow control.
To use these lines you should connect the common pole on your switch to the DSR line (pin 6 on a SUB-D9) and the NO and NC poles to lines DTR (pin 4) and RTS (pin 7).
The software side would be actually simpler than reading bytes: you just have to activate hardware flow control :
self.ser = serial.Serial()
self.ser.port='COM4'
self.ser.baudrate=1200 #Baud rate does not matter now
self.ser.timeout=0
self.ser.rtscts=True
self.ser.dsrdtr=True
self.ser.open()
Define the logical levels for your switch:
self.ser.setDTR(False) # We use DTR for low level state
self.ser.setRTS(True) # We use RTS for high level state
self.ser.open() # Open port after setting everything up, to avoid unkwnown states
And use ser.getDSR() to check the logical level of the DSR line in your loop:
def start_stop(self):
while True :
try:
switch_state = self.ser.getDSR()
if switch_state == False and self._cycRunning == True:
self.cycleStop()
print("Off")
elif switch_state == True and self._cycRunning == False:
self.cycleStart()
print("On")
except serial.SerialException as e:
print("Error")
I defined your self._cycRunning variable as boolean (in your initialization code you had defined it as float, but that was probably a typo).
This code works with no glitches at all even using a stripped wire as a switch.

You don't explain very well how your protocol works (I mean what is your switch supposed to be sending, or if it's sending a state change only once or several times or continuously).
But there are some red flags on your code anyway:
-With data = self.ser.read(size=1).strip() you read 1 byte but immediately you check if you have received 2 bytes. Is there a reason to do that?
-Your timer stop condition works comparing with the NULL character. That should not be a problem, but depending on your particular configuration it might (in some configurations the NULL character is read as something else, so it's wise to make sure you're really receiving it correctly).
-Your timer start condition seems too loose. Whatever you receive on the port, if it's one byte, you start your timer. Again, I don't know if that's the way your protocol works but it seems prone to trouble.
-When you replace your hardware switch with a software emulation it works as intended, but that is not surprising since you're probably imposing the condition. When you read from the serial port you have to deal with real world issues like noise, communication errors or the switch bouncing back and forth from ON to OFF. Maybe for a very simple protocol you don't need to use any error checking method, but it seems wise to at least check for parity errors. I'm not completely sure it would be straight-forward to do that with pyserial; on a quick glance I found this issue that's been open for a while.
-Again, the lack of info on your protocol: should you be using XON-XOFF flow control and two stop bits? I guess you have a reason to do it, but you should be very aware of why and how you're using those.
EDIT: With the comments below I can try to improve a bit my answer. This is just an idea for you to develop: instead of making the stop condition comparing exactly with 0x00 you can count the number of bits set to 1 and stop the counter if it's less or equal to 2. That way you can account for bits that are not received correctly.
You can do the same with the start condition but I don't know what hex value you send.
Credits for the bit counting function go to this question.
...
def numberOfSetBits(i):
i = i - ((i >> 1) & 0x55555555)
i = (i & 0x33333333) + ((i >> 2) & 0x33333333)
return (((i + (i >> 4) & 0xF0F0F0F) * 0x1010101) & 0xffffffff) >> 24
def start_stop(self):
while True :
try:
data_to_read = self.ser.inWaiting()
if data_to_read != 0: # read if there is new data
data = self.ser.read(size=1).strip()
if numberOfSetBits(int.from_bytes(data, "big")) <= 2:
self.cycleStop()
print("Off")
elif numberOfSetBits(int.from_bytes(data, "big")) >= 3: #change the condition here according to your protocol
self.cycleStart()
print("On")
except serial.SerialException as e:
print("Error")

How to stop a script within a script

I'm looking to control a script via Zigbee/XBee using X-CTU. I've created a script named zb_control.py. Now I'm trying to start and stop another script within this script. A script adxl345test.py is used to collect data from an attached accelerometer on my Raspberry Pi.
The idea behind the zb_control.py script is that I run it and then if I type "run" in X-CTU the script will start running adxl345test.py and collect data.
I'm trying to create a script within a script that can also be stopped again and then still have the zb_control.py running ready to recieve new input from X-CTU.
As you can tell I've tried different things:
import serial, time, sys, os, subprocess
from subprocess import check_call
from subprocess import call
while True:
ser=serial.Serial('/dev/ttyUSB0',9600,timeout=2)
inc=ser.readline().strip()
if inc=='run':
print("---------------")
print("Collecting data")
print("---------------")
p = subprocess.Popen("/home/pi/adxl345test.py", stdout=subprocess.PIPE, shell=True)
elif inc=='stop':
# check_call(["pkill", "-9", "-f", adxl345test.py])
# serial.write('\x03')
# os.system("pkill –f adxl345test.py")
# call(["killall", "adxl345test.py"])
p.kill()
print("-----------------------")
print("Script has been stopped")
print("-----------------------")
I got it to run and it's now collecting data properly. However now the problem is stopping the adxl345test.py again. As you can tell from the script from above I'm using p.kill() but the script doesn't stop collecting data. When I type "stop" in XCTU my zb_control.py does print the print-commands but the p.kill() isn't being executed. Any suggestions?
I've tried using p.terminate() alone and together with p.kill() aswell as the commands by themselves however it doesn't stop the adxl345test.py script. I can tell that the .csv-file is still increasing in size and therefore the script must still be collecting data.

Here is the adxl345test.py script for those interested:
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Example on how to read the ADXL345 accelerometer.
# Kim H. Rasmussen, 2014
import sys, math, os, spidev, datetime, ftplib
# Setup SPI
spi = spidev.SpiDev()
#spi.mode = 3 <-- Important: Do not do this! Or SPI won't work as intended, or even at all.
spi.open(0,0)
spi.mode = 3
# Read the Device ID (should be xe5)
id = spi.xfer2([128,0])
print 'Device ID (Should be 0xe5):\n'+str(hex(id[1])) + '\n'
# Read the offsets
xoffset = spi.xfer2([30 | 128,0])
yoffset = spi.xfer2([31 | 128,0])
zoffset = spi.xfer2([32 | 128,0])
accres = 2
accrate = 13
print 'Offsets: '
print xoffset[1]
print yoffset[1]
# print str(zoffset[1]) + "\n\nRead the ADXL345 every half second:"
# Initialize the ADXL345
def initadxl345():
# Enter power saving state
spi.xfer2([45, 0])
# Set data rate to 100 Hz. 15=3200, 14=1600, 13=800, 12=400, 11=200, 10=100 etc.
spi.xfer2([44, accrate])
# Enable full range (10 bits resolution) and +/- 16g 4 LSB
spi.xfer2([49, accres])
# Enable measurement
spi.xfer2([45, 8])
# Read the ADXL x-y-z axia
def readadxl345():
rx = spi.xfer2([242,0,0,0,0,0,0])
#
out = [rx[1] | (rx[2] << 8),rx[3] | (rx[4] << 8),rx[5] | (rx[6] << 8)]
# Format x-axis
if (out[0] & (1<<16 - 1 )):
out[0] = out[0] - (1<<16)
# out[0] = out[0] * 0.004 * 9.82
# Format y-axis
if (out[1] & (1<<16 - 1 )):
out[1] = out[1] - (1<<16)
# out[1] = out[1] * 0.004 * 9.82
# Format z-axis
if (out[2] & (1<<16 - 1 )):
out[2] = out[2] - (1<<16)
# out[2] = out[2] * 0.004 * 9.82
return out
# Initialize the ADXL345 accelerometer
initadxl345()
# Read the ADXL345 every half second
timetosend = 60
while(1):
with open('/proc/uptime','r') as f: # get uptime
uptime_start = float(f.readline().split()[0])
uptime_last = uptime_start
active_file_first = "S3-" + str(pow(2,accrate)*25/256) + "hz10bit" + str(accres) + 'g' + str(datetime.datetime.utcnow().strftime('%y%m%d%H%M')) $
active_file = active_file_first.replace(":", ".")
wStream = open('/var/log/sensor/' + active_file,'wb')
finalcount = 0
print "Creating " + active_file
while uptime_last < uptime_start + timetosend:
finalcount += 1
time1 = str(datetime.datetime.now().strftime('%S.%f'))
time2 = str(datetime.datetime.now().strftime('%M'))
time3 = str(datetime.datetime.now().strftime('%H'))
time4 = str(datetime.datetime.now().strftime('%d'))
time5 = str(datetime.datetime.now().strftime('%m'))
time6 = str(datetime.datetime.now().strftime('%Y'))
axia = readadxl345()
wStream.write(str(round(float(axia[0])/1024,3))+','+str(round(float(axia[1])/1024,3))+','+str(round(float(axia[2])/1024,3))+','+time1+','+ti$
# Print the reading
# print axia[0]
# print axia[1]
# print str(axia[2]) + '\n'
# elapsed = time.clock()
# current = 0
# while(current < timeout):
# current = time.clock() - elapsed
with open('/proc/uptime', 'r') as f:
uptime_last = float(f.readline().split()[0])
wStream.close()
def doftp(the_active_file):
session = ftplib.FTP('192.0.3.6','sensor3','L!ghtSp33d')
session.cwd("//datalogger//")
file = open('/var/log/sensor/' + active_file, 'rb') # file to send
session.storbinary('STOR' + active_file, file) # send the file
file.close()
session.quit

My suggestions:
If you're doing something at a specified interval, you're probably better off using threading.Timer rather than checking the time yourself.
As I said in the comment, you can check for an exit condition instead of brutally killing the process. This also allows to properly clean up what you need.
Those time1...time6 really don't look nice, how about a list? Also, time2, time3, time4, time5, time6 are not used.
You don't actually need strftime to get hour, day, month, year, etc. They're already there as attributes.
You can do something like:
cur_time = datetime.datetime.now()
cur_hour = cur_time.hour
cur_minute = cur_time.minute
...And so on, which is a bit better. In this specific case it won't matter, but if you start measuring milliseconds, the time will be slightly different after a few lines of code, so you should store it and use it from the variable.
As for the rest, if you want an example, here I check that a file exists to determine whether to stop or not. It's very crude but it should give you a starting point:
from threading import *
from os.path import exists
def hello():
print('TEST') # Instead of this, do what you need
if (exists('stop_file.txt')):
return
t = Timer(0.5, hello)
t.start()
hello()
Then in the other create you create the stop file when you want it to stop (don't forget to add a line to remove it before starting it again).

if statement for a subprocess python not working

I've tried to create a little app that plays a sound when you lose connectivity for an extended period and plays another when the connection is established. Useful for wireless connections.
I'm still new to Python :) trying little projects to improve my knowledge. If you do answer I will be very grateful if you could include any information about how to use subprocess.
I've defined the subprocess but I'm not sure how to word my if statement so it loops from one function to the other. IE Function 1 = IF ping loss > 15 pings play sound and move on to function 2... If function 2 ping success > 15 pings play sound and move back to function 1. So on.
I've yet to wrap the program in a loop, at this point I'm just trying to get the ping to work with the if statement.
So right now the application just continuously loop pings.
import os
import subprocess
import winsound
import time
def NetFail():
winsound.Beep(2000 , 180), winsound.Beep(1400 , 180)
def NetSucc():
winsound.Beep(1400 , 250), winsound.Beep(2000 , 250),
ips=[]
n = 1
NetSuccess = 10
NetFailure = 10
PinSuc = 0
PinFail = 0
x = '8.8.8.8'
ips.append(x)
for ping in range(0,n):
ipd=ips[ping]
def PingFailure():
while PinFail < NetSuccess:
res = subprocess.call(['ping', '-n', '10', ipd])
if ipd in str(res):
PingSuccess()
else:
print ("ping to", ipd, "failed!"), NetFail()
def PingSuccess():
while PinFail < NetFailure: # This needs to be cleaned up so it doesn't interfere with the other function
res = subprocess.call(['ping', '-n', '10', ipd])
if ipd in str(res):
PingFail()
else:
print ("ping to", ipd, "successful!"), NetSucc()

As you use the command ping -n 10 ip, I assume that you are using a Windows system, as on Linux (or other Unix-like) it would be ping -c 10 ip.
Unfortunately, on Windows ping always return 0, so you cannot use the return value to know whether peer was reached. And even the output is not very clear...
So you should:
run in a cmd console the command ping -n 1 ip with an accessible and inaccessible ip, note the output and identify the differences. On my (french) system, it writes Impossible, I suppose that you should get Unable or the equivalent in your locale
start the ping from Python with subprocess.Popen redirecting the output to a pipe
get the output (and error output) from the command with communicate
search for the Unable word in output.
Code could be like:
errWord = 'Unable' # replace with what your locale defines...
p = subprocess.Popen([ 'ping', '-n', '1', ipd],
stdout = subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if errWord in out:
# process network disconnected
else:
# process network connected
Alternatively, you could search pypi for a pure Python implementation of ping such as py-ping ...
Anyway, I would not use two functions in flip-flop because it will be harder if you later wanted to test connectivity to multiple IPs. I would rather use an class
class IP(object):
UNABLE = "Unable" # word indicating unreachable host
MAX = 15 # number of success/failure to record new state
def __init__(self, ip, failfunc, succfunc, initial = True):
self.ip = ip
self.failfunc = failfunc # to warn of a disconnection
self.succfunc = succfunc # to warn of a connection
self.connected = initial # start by default in connected state
self.curr = 0 # number of successive alternate states
def test(self):
p = subprocess.Popen([ 'ping', '-n', '1', self.ip],
stdout = subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if self.UNABLE in out:
if self.connected:
self.curr += 1
else:
self.curr = 0 # reset count
else:
if not self.connected:
self.curr += 1
else:
self.curr = 0 # reset count
if self.curr >= self.MAX: # state has changed
self.connected = not self.connected
self.curr = 0
if self.connected: # warn for new state
self.succfunc(self)
else:
self.failfunc(self)
Then you can iterate over a list of IP objects, repeatedly calling ip.test(), and you will be warned for state changes

Not quite sure, what you want to achieve, but your if statement has to be part of the while loop if you want it to be executed each time ping is called via subprocess is called.
Also:
Here is the documentation for subprocess: https://docs.python.org/3/library/subprocess.html
For viewing the output of a process you have to call it via subprocess.call_output:
ls_output = subprocess.check_output(['ls'])
For further information have a look at this: http://sharats.me/the-ever-useful-and-neat-subprocess-module.html#a-simple-usage

tail -f over ssh with Paramiko has an increasing delay

I'm trying to check for errors in a log file of a running embedded system.
I already have implemented paramiko in my scripts, as I've been told this is the best way to use ssh in python.
Now when I tail the log file I see that there is a big delay build up. Which increases with about 30 seconds per minute.
I already used a grep to decrease the number of lines which is printed as I thought I was receiving too much input but that isn't the case.
How can I decrease this delay or stop the delay from increasing during runtime. I want to tail for hours...
def mkssh_conn(addr):
"""returns an sshconnection"""
paramiko.util.logging.getLogger('paramiko').setLevel(logging.WARN)
sshcon = paramiko.SSHClient()
sshcon.set_missing_host_key_policy(paramiko.AutoAddPolicy())
sshcon.connect(addr , username, password)
return sshcon
while True:
BUF_SIZE = 1024
client = mkssh_conn() #returns a paramiko.SSHClient()
transport = client.get_transport()
transport.set_keepalive(1)
channel = transport.open_session()
channel.settimeout(delta)
channel.exec_command( 'killall tail')
channel = transport.open_session()
channel.settimeout(delta)
cmd = "tail -f /log/log.log | grep -E 'error|statistics'"
channel.exec_command(cmd)
while transport.is_active():
print "transport is active"
rl, wl, xl = select.select([channel], [], [], 0.0)
if len(rl) > 0:
buf = channel.recv(BUF_SIZE)
if len(buf) > 0:
lines_to_process = LeftOver + buf
EOL = lines_to_process.rfind("\n")
if EOL != len(lines_to_process)-1:
LeftOver = lines_to_process[EOL+1:]
lines_to_process = lines_to_process[:EOL]
else:
LeftOver = ""
for line in lines_to_process.splitlines():
if "error" in line:
report_error(line)
print line
client.close()

I've found a solution:
It seems that if I lower BUF_SIZE to 256 the delay decreases. Obviously.
I need to recheck if the delay still increases during runtime or not.

BUFFER_SIZE should be on the higher end to reduce the cpu cycles ( and in turn to reduce the overall delay due to network latency ) in case if you are working with high throughput tailed pipe.
Further making the BUFFER_SIZE to higher number, should not reduce the performance. ( if paramiko doesn't wail till the buffer fills out in low throughput pipe )
Contradiction between #studioj 's answer and this, probably due to the upgrade of paramiko ( fixed now )

python run a scapy function as a background process

I would like to know in any way that i can run a function as a background process.
I have gone through reading threading but i am still unclear on how to implement it.
In my system , the scapy detection script run when i click on the Button.But then the system will be hang as the scapy function is running.
What i want to achieve is that when i click on the button that initiate the scapy detection script , i would like to be able to do other function in the same system.(general idea was to prevent the system from hang)
def startMonitor(self,event):
selectedInterface = self.interfaces_cblist.GetValue()
#selectInterfaceStr = str(selectedInterface)
if len(selectedInterface) == 0:
noInterfaceSelected = wx.MessageDialog(None,"Please select an interface","",wx.ICON_ERROR)
noInterfaceSelected.ShowModal()
else:
#confirmMonitor = wx.MessageDialog(None,"Monitoring Start on %s interface"%selectedInterface,"",wx.OK)
#confirmMonitor.ShowModal()
x = selectedInterface
thread.start_new_thread(self.camtableDetection(x))
def camtableDetection(self,a):
global interface
interface = str(a)
THRESH=(254/4)
START = 5
def monitorPackets(p):
if p.haslayer(IP):
hwSrc = p.getlayer(Ether).src
if hwSrc not in hwList:
hwList.append(hwSrc)
delta = datetime.datetime.now() - start
if((delta.seconds > START) and ((len(hwList)/delta.seconds) > THRESH)):
camAttackDetected = wx.MessageDialog(None,"Cam Attack Detected","",wx.ICON_ERROR)
camAttackDetected.ShowModal()
hwList = []
start = datetime.datetime.now()
sniff(iface=interface,prn=monitorPackets)