I'm trying to check for errors in a log file of a running embedded system.
I already have implemented paramiko in my scripts, as I've been told this is the best way to use ssh in python.
Now when I tail the log file I see that there is a big delay build up. Which increases with about 30 seconds per minute.
I already used a grep to decrease the number of lines which is printed as I thought I was receiving too much input but that isn't the case.
How can I decrease this delay or stop the delay from increasing during runtime. I want to tail for hours...
def mkssh_conn(addr):
"""returns an sshconnection"""
paramiko.util.logging.getLogger('paramiko').setLevel(logging.WARN)
sshcon = paramiko.SSHClient()
sshcon.set_missing_host_key_policy(paramiko.AutoAddPolicy())
sshcon.connect(addr , username, password)
return sshcon
while True:
BUF_SIZE = 1024
client = mkssh_conn() #returns a paramiko.SSHClient()
transport = client.get_transport()
transport.set_keepalive(1)
channel = transport.open_session()
channel.settimeout(delta)
channel.exec_command( 'killall tail')
channel = transport.open_session()
channel.settimeout(delta)
cmd = "tail -f /log/log.log | grep -E 'error|statistics'"
channel.exec_command(cmd)
while transport.is_active():
print "transport is active"
rl, wl, xl = select.select([channel], [], [], 0.0)
if len(rl) > 0:
buf = channel.recv(BUF_SIZE)
if len(buf) > 0:
lines_to_process = LeftOver + buf
EOL = lines_to_process.rfind("\n")
if EOL != len(lines_to_process)-1:
LeftOver = lines_to_process[EOL+1:]
lines_to_process = lines_to_process[:EOL]
else:
LeftOver = ""
for line in lines_to_process.splitlines():
if "error" in line:
report_error(line)
print line
client.close()
I've found a solution:
It seems that if I lower BUF_SIZE to 256 the delay decreases. Obviously.
I need to recheck if the delay still increases during runtime or not.
BUFFER_SIZE should be on the higher end to reduce the cpu cycles ( and in turn to reduce the overall delay due to network latency ) in case if you are working with high throughput tailed pipe.
Further making the BUFFER_SIZE to higher number, should not reduce the performance. ( if paramiko doesn't wail till the buffer fills out in low throughput pipe )
Contradiction between #studioj 's answer and this, probably due to the upgrade of paramiko ( fixed now )
Related
I wrote a small pyserial interface to read the data from the COM port after issuing a command. For eg : in my case my system has a lot of network interface so i need to validate whether all the interfaces are up using ifconfig command. But when i gave this command , the output of the command is getting truncated at the last few lines. The approximate size of the output in bytes would be 6500-7000 bytes but i am receiving only around 6000-6150 bytes all the time. Please find my code below
'''
import serial
import time
com_serial = serial.Serial("COM6", 115200, timeout = 10)
com_serial.reset_input_buffer()
com_serial.write(b"ifconfig\n")
data_all = b" "
time.sleep(5)
while True:
bytetoread = com_serial.inWaiting()
time.sleep(2)
print ("Bytetoread: " , bytetoread)
data = com_serial.read(bytetoread)
data_all += data
if bytetoread < 1:
break
print ("Data:", data_all)
com_serial.close()
'''
**Output:
Bytetoread: 3967
Bytetoread: 179
Bytetoread: 2049
Bytetoread: 0
**
Data: *********with missing data at the end.
I am not sure why the logs are missing?
I have tried another approach.
'''
import serial
import time
com_serial = serial.Serial("COM6", 115200, timeout = 10)
com_serial.reset_input_buffer()
com_serial.write(b"ifconfig\n")
time.sleep(5)
data_all = b" "
data_all = com_serial.read(100000000)
print (data_all)
com_serial.close()
'''
Here also the last few logs are getting truncated.
The root cause seems to be inadequate buffer size of the Tx and Rx serial buffer. By increasing the buffer size using .set_buffer_size() resolved the issue.
'''
import serial
import time
com_serial = serial.Serial("COM6", 115200, timeout = 10)
com_serial.set_buffer_size(rx_size = 12800, tx_size = 12800)
com_serial.reset_input_buffer()
com_serial.write(b"ifconfig\n")
data_all = b" "
data_all = com_serial.read(100000000)
print (data_all)
com_serial.close()
'''
I have an FPGA that streams data on the USB bus through an FT2232H and I have observed that about 10% of the data has to be thrown away because some bytes in the frame are missing. Here are the technical details:
FPGA is an Artix 7. A batch of 4002 byte is ready every 9 ms. So that works out to 444,667 byte/s of data.
My laptop runs python 3.7 (from anaconda) on Ubuntu 18.04LTS
The FPGA/FT2232H is opened via the following initialization lines:
SYNCFF = 0x40
SIO_RTS_CTS_HS = (0x1 << 8)
self.device = pylibftdi.Device(mode='t', interface_select=pylibftdi.INTERFACE_A, encoding='latin1')
self.device.ftdi_fn.ftdi_set_bitmode(0xff, SYNCFF)
self.device.ftdi_fn.ftdi_read_data_set_chunksize(0x10000)
self.device.ftdi_fn.ftdi_write_data_set_chunksize(0x10000)
self.device.ftdi_fn.ftdi_setflowctrl(SIO_RTS_CTS_HS)
self.device.flush()
Then the data is read via this simple line:
raw_usb_data = my_fpga.device.read(0x10000)
I have observed the following:
I always get 0x10000 of data per batch, which is what I expect.
Reading 2**16 = 65,536 byte at once using device.read should take 147.4 ms given that a batch is ready every 9 ms. But timing that line gives a mean of 143 ms with a std deviation of 6.6 ms.
My first guess is that there is no buffer/a tiny buffer somewhere and that some information is lost because the OS (priority issue?) or python (garbage collection?) does something else at some point for too long.
How can I reduce the amount of bytes lost while reading the device?
The FT2232H has internal FIFO buffers with a capacity of ~4 kbits. Chances are that you are limited by them. Not sure how pylibftdi deals with them but maybe using an alternative approach might work if you can use the VCP driver. This allows you to address the FT2232H as standard comport e.g. via pyserial.
Some excerpts from one of my projects which actually works for baud rates >12 Mbps (UART is limited to 12 Mbps but e.g. fast opto can reach ~25 Mbps):
import traceback
import serial
import serial.tools.list_ports
import multiprocessing
import multiprocessing.connection
def IO_proc(cntr_pipe, data_pipe):
try:
search_str="USB VID:PID=0403:6010 SER="
ports = [x.device for x in serial.tools.list_ports.comports() if search_str in x.hwid]
baud_rate = 12000000 #only matters for uart and not for fast opto or fifo mode
ser = serial.Serial(port, baud_rate)
while not cntr_pipe.closed:
time.sleep(0)
in_data = ser.read(ser.inWaiting())
[...do some pattern matching, package identification etc...]
data_pipe.send_bytes(in_data)
except EOFError:
ret_code = 2
except Exception as e:
cntr_pipe.send(traceback.format_exc())
cntr_pipe.close()
ret_code = 4
finally:
cntr_pipe.close()
ser.close()
multiprocessing.connection.BUFSIZE = 2 ** 20 #only required for windows
child_cntr, parent_cntr = multiprocessing.Pipe()
child_data, parent_data = multiprocessing.Pipe()
process = multiprocessing.Process(target = IO_proc, args=(child_cntr, child_data))
#called frequently
def update():
if child_cntr.poll():
raise Exception("error",child_cntr.recv())
buf = bytes()
while parent_data.poll():
buf += parent_data.recv_bytes()
[...do something fancy...]
I tried to c&p a minimum example. It is untested so please forgive me if it is not working out of the box. To get this working one actually needs to make sure that the VCP and not the D2XX driver is loaded.
P.S: Actually while scanning through my files I realized that the pylibftdi way should work as well as I use a "decorator" class in case the D2XX driver is loaded:
try: import pylibftdi
except: pylibftdi = None
class pylibftdi_device:
def __init__(self,speed):
self.dev = pylibftdi.Device(interface_select=2)
self.dev.baudrate = speed
self.buf = b''
def write(self, data):
self.dev.write(data)
def read(self, bytecount):
while bytecount > len(self.buf):
self._read()
ret = self.buf[:bytecount]
self.buf = self.buf[bytecount:]
return ret
def flushInput(self):
self.dev.flush_input()#FT_PURGE_RX
self.buf = b''
def _read(self):
self.buf += self.dev.read(2048)
#property
def in_waiting(self):
self._read()
return len(self.buf)
def close(self):
self.dev.close()
def find_device_UART(baudrate=12000000,index=1, search_string="USB VID:PID=0403:6010 SER="):
if pylibftdi:
return pylibftdi_device(baudrate),"pylibftdi_device"
try:
ports = [x.device for x in serial.tools.list_ports.comports() if search_string in x.hwid]
module_logger.info(str(ports))
if len(ports) == 0:
return None,"no device found"
else:
ser = serial.Serial(ports[index],baudrate)
return ser,"found device %s %d"%(ser.name,ser.baudrate)
except serial.SerialException as e:
return None,"error during device detection - \n"+str(e)
So main difference to your example is that the recv buffer is read more frequently and put into a buffer which is then searched for the packets later on. And maybe this all is a complete overkill for your application and you just need to make smaller read calls to ensure the buffers never overflow.
I have been using a micro switch connected to an RS232/USB serial converter cable on my windows PC to start stop and reset a timer.
The program runs smoothly most of the time but every so often updating the timer widget gets stuck running and the timer will not stop.
With the serial protocol i want to receive 1 byte b'\x00' for off and anything that's not b'\x00' should signify on.
I have replaced the micro switch with button widgets to simulate the switch and don't get the same error or i just have not kept at it for long enough.
It could be an issue with the RS232 causing an error i cannot see but my knowledge on this is sketchy and have exhausted all avenues looking online for any information on this.
import time
import sys
import serial
import threading
from tkinter import *
from tkinter import ttk
class Process(Frame):
def __init__(self, root, parent=None, **kw):
Frame.__init__(self, parent, kw)
self.root = root
self._cycStart = 0.0
self._cycTimeElapsed = 0.0
self._cycRunning = 0.0
self.cycTimeStr = StringVar()
self.cycTime_label_widget()
self.ser = serial.Serial(
port='COM4',
baudrate=1200,
timeout=0
)
self.t1 = threading.Thread(target=self.start_stop, name='t1')
self.t1.start()
def initUI(self):
root.focus_force()
root.title("")
root.bind('<Escape>', lambda e: root.destroy())
def cycTime_label_widget(self):
# Make the time label
cycTimeLabel = Label(root, textvariable=self.cycTimeStr, font=
("Ariel 12"))
self._cycleSetTime(self._cycTimeElapsed)
cycTimeLabel.place(x=1250, y=200)
cycTimeLabel_2 = Label(root, text="Cycle Timer:", font=("Ariel
12"))
cycTimeLabel_2.place(x=1150, y=200)
def _cycleUpdate(self):
""" Update the label with elapsed time. """
self._cycTimeElapsed = time.time() - self._cycStart
self._cycleSetTime(self._cycTimeElapsed)
self._cycTimer = self.after(50, self._cycleUpdate)
def _cycleSetTime(self, elap):
""" Set the time string to Minutes:Seconds:Hundreths """
minutes = int(elap/60)
seconds = int(elap - minutes*60.0)
hseconds = int((elap - minutes*60.0 - seconds)*100)
self.cycTimeStr.set('%02d:%02d:%02d' % (minutes, seconds,
hseconds))
return
def cycleStart(self):
""" Start the stopwatch, ignore if running. """
if not self._cycRunning:
self._cycStart = time.time() - self._cycTimeElapsed
self._cycleUpdate()
self._cycRunning = 1
else:
self.cycleReset()
def cycleStop(self):
""" Stop the stopwatch, ignore if stopped. """
if self._cycRunning:
self.after_cancel(self._cycTimer)
self._cycTimeElapsed = time.time() - self._cycStart
self._cycleSetTime(self._cycTimeElapsed)
self._cycRunning = 0
self._cycTimeElapsed = round(self._cycTimeElapsed, 1)
self.cycleTimeLabel = Label(root, text=(self._cycTimeElapsed,
"seconds"), font=("Ariel 35"))
self.cycleTimeLabel.place(x=900, y=285)
self.cycleReset()
def cycleReset(self):
""" Reset the stopwatch. """
self._cycStart = time.time()
self._cycTimeElapsed = 0
self._cycleSetTime(self._cycTimeElapsed)
def start_stop(self):
while True :
try:
data_to_read = self.ser.inWaiting()
if data_to_read != 0: # read if there is new data
data = self.ser.read(size=1).strip()
if data == bytes(b'\x00'):
self.cycleStop()
print("Off")
elif data is not bytes(b'\x00'):
self.cycleStart()
print("On")
except serial.SerialException as e:
print("Error")
if __name__ == '__main__':
root = Tk()
application = Process(root)
root.mainloop()
I expect the timer to start running when the micro switch is pressed. when depressed it should stop and reset back to zero and wait for the next press
With a better understanding of what you're trying to do better solutions come to mind.
As it turns out, you're not using your serial port to send or receive serial data. What you're actually doing is wiring a switch to its RX line and toggling it manually with a mechanical switch, feeding a high or low level depending on the position of the switch.
So what you're trying to do is emulating a digital input line with the RX line of your serial port. If you take a look a how a serial port works you'll see that when you send a byte the TX line toggles from low to high at the baud rate, but on top of the data you have to consider the start and stop bits. So, why your solution works (at least sometimes): that's easy to see when you look at a scope picture:
This is a screenshot of the TX line sending the \x00 byte, measured between pins 3 (TX) and 5 (GND) with no parity bit. As you can see the step only lasts for 7.5 ms (with a 1200 baud rate). What you are doing with your switch is something similar but ideally infinitely long (or until you toggle your switch back, which will be way after 7.5 ms no matter how fast you do it). I don't have a switch to try but if I open a terminal on my port and use a cable to shortcircuit the RX line to pin 4 (on a SUB-D9 connector) sometimes I do get a 0x00 byte, but mostly it's something else. You can try this experiment yourself with PuTTy or RealTerm and your switch, I guess you'll get better results but still not always the byte you expect because of the contacts bouncing.
Another approach: I'm sure there might be ways to improve on what you have, maybe reducing the baud rate to 300 or 150 bps, checking for a break in the line or other creative ideas.
But what you're trying to do is more akin to reading a GPIO line, and actually, the serial port has several digital lines intended (in the old days) for flow control.
To use these lines you should connect the common pole on your switch to the DSR line (pin 6 on a SUB-D9) and the NO and NC poles to lines DTR (pin 4) and RTS (pin 7).
The software side would be actually simpler than reading bytes: you just have to activate hardware flow control :
self.ser = serial.Serial()
self.ser.port='COM4'
self.ser.baudrate=1200 #Baud rate does not matter now
self.ser.timeout=0
self.ser.rtscts=True
self.ser.dsrdtr=True
self.ser.open()
Define the logical levels for your switch:
self.ser.setDTR(False) # We use DTR for low level state
self.ser.setRTS(True) # We use RTS for high level state
self.ser.open() # Open port after setting everything up, to avoid unkwnown states
And use ser.getDSR() to check the logical level of the DSR line in your loop:
def start_stop(self):
while True :
try:
switch_state = self.ser.getDSR()
if switch_state == False and self._cycRunning == True:
self.cycleStop()
print("Off")
elif switch_state == True and self._cycRunning == False:
self.cycleStart()
print("On")
except serial.SerialException as e:
print("Error")
I defined your self._cycRunning variable as boolean (in your initialization code you had defined it as float, but that was probably a typo).
This code works with no glitches at all even using a stripped wire as a switch.
You don't explain very well how your protocol works (I mean what is your switch supposed to be sending, or if it's sending a state change only once or several times or continuously).
But there are some red flags on your code anyway:
-With data = self.ser.read(size=1).strip() you read 1 byte but immediately you check if you have received 2 bytes. Is there a reason to do that?
-Your timer stop condition works comparing with the NULL character. That should not be a problem, but depending on your particular configuration it might (in some configurations the NULL character is read as something else, so it's wise to make sure you're really receiving it correctly).
-Your timer start condition seems too loose. Whatever you receive on the port, if it's one byte, you start your timer. Again, I don't know if that's the way your protocol works but it seems prone to trouble.
-When you replace your hardware switch with a software emulation it works as intended, but that is not surprising since you're probably imposing the condition. When you read from the serial port you have to deal with real world issues like noise, communication errors or the switch bouncing back and forth from ON to OFF. Maybe for a very simple protocol you don't need to use any error checking method, but it seems wise to at least check for parity errors. I'm not completely sure it would be straight-forward to do that with pyserial; on a quick glance I found this issue that's been open for a while.
-Again, the lack of info on your protocol: should you be using XON-XOFF flow control and two stop bits? I guess you have a reason to do it, but you should be very aware of why and how you're using those.
EDIT: With the comments below I can try to improve a bit my answer. This is just an idea for you to develop: instead of making the stop condition comparing exactly with 0x00 you can count the number of bits set to 1 and stop the counter if it's less or equal to 2. That way you can account for bits that are not received correctly.
You can do the same with the start condition but I don't know what hex value you send.
Credits for the bit counting function go to this question.
...
def numberOfSetBits(i):
i = i - ((i >> 1) & 0x55555555)
i = (i & 0x33333333) + ((i >> 2) & 0x33333333)
return (((i + (i >> 4) & 0xF0F0F0F) * 0x1010101) & 0xffffffff) >> 24
def start_stop(self):
while True :
try:
data_to_read = self.ser.inWaiting()
if data_to_read != 0: # read if there is new data
data = self.ser.read(size=1).strip()
if numberOfSetBits(int.from_bytes(data, "big")) <= 2:
self.cycleStop()
print("Off")
elif numberOfSetBits(int.from_bytes(data, "big")) >= 3: #change the condition here according to your protocol
self.cycleStart()
print("On")
except serial.SerialException as e:
print("Error")
I've tried to create a little app that plays a sound when you lose connectivity for an extended period and plays another when the connection is established. Useful for wireless connections.
I'm still new to Python :) trying little projects to improve my knowledge. If you do answer I will be very grateful if you could include any information about how to use subprocess.
I've defined the subprocess but I'm not sure how to word my if statement so it loops from one function to the other. IE Function 1 = IF ping loss > 15 pings play sound and move on to function 2... If function 2 ping success > 15 pings play sound and move back to function 1. So on.
I've yet to wrap the program in a loop, at this point I'm just trying to get the ping to work with the if statement.
So right now the application just continuously loop pings.
import os
import subprocess
import winsound
import time
def NetFail():
winsound.Beep(2000 , 180), winsound.Beep(1400 , 180)
def NetSucc():
winsound.Beep(1400 , 250), winsound.Beep(2000 , 250),
ips=[]
n = 1
NetSuccess = 10
NetFailure = 10
PinSuc = 0
PinFail = 0
x = '8.8.8.8'
ips.append(x)
for ping in range(0,n):
ipd=ips[ping]
def PingFailure():
while PinFail < NetSuccess:
res = subprocess.call(['ping', '-n', '10', ipd])
if ipd in str(res):
PingSuccess()
else:
print ("ping to", ipd, "failed!"), NetFail()
def PingSuccess():
while PinFail < NetFailure: # This needs to be cleaned up so it doesn't interfere with the other function
res = subprocess.call(['ping', '-n', '10', ipd])
if ipd in str(res):
PingFail()
else:
print ("ping to", ipd, "successful!"), NetSucc()
As you use the command ping -n 10 ip, I assume that you are using a Windows system, as on Linux (or other Unix-like) it would be ping -c 10 ip.
Unfortunately, on Windows ping always return 0, so you cannot use the return value to know whether peer was reached. And even the output is not very clear...
So you should:
run in a cmd console the command ping -n 1 ip with an accessible and inaccessible ip, note the output and identify the differences. On my (french) system, it writes Impossible, I suppose that you should get Unable or the equivalent in your locale
start the ping from Python with subprocess.Popen redirecting the output to a pipe
get the output (and error output) from the command with communicate
search for the Unable word in output.
Code could be like:
errWord = 'Unable' # replace with what your locale defines...
p = subprocess.Popen([ 'ping', '-n', '1', ipd],
stdout = subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if errWord in out:
# process network disconnected
else:
# process network connected
Alternatively, you could search pypi for a pure Python implementation of ping such as py-ping ...
Anyway, I would not use two functions in flip-flop because it will be harder if you later wanted to test connectivity to multiple IPs. I would rather use an class
class IP(object):
UNABLE = "Unable" # word indicating unreachable host
MAX = 15 # number of success/failure to record new state
def __init__(self, ip, failfunc, succfunc, initial = True):
self.ip = ip
self.failfunc = failfunc # to warn of a disconnection
self.succfunc = succfunc # to warn of a connection
self.connected = initial # start by default in connected state
self.curr = 0 # number of successive alternate states
def test(self):
p = subprocess.Popen([ 'ping', '-n', '1', self.ip],
stdout = subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if self.UNABLE in out:
if self.connected:
self.curr += 1
else:
self.curr = 0 # reset count
else:
if not self.connected:
self.curr += 1
else:
self.curr = 0 # reset count
if self.curr >= self.MAX: # state has changed
self.connected = not self.connected
self.curr = 0
if self.connected: # warn for new state
self.succfunc(self)
else:
self.failfunc(self)
Then you can iterate over a list of IP objects, repeatedly calling ip.test(), and you will be warned for state changes
Not quite sure, what you want to achieve, but your if statement has to be part of the while loop if you want it to be executed each time ping is called via subprocess is called.
Also:
Here is the documentation for subprocess: https://docs.python.org/3/library/subprocess.html
For viewing the output of a process you have to call it via subprocess.call_output:
ls_output = subprocess.check_output(['ls'])
For further information have a look at this: http://sharats.me/the-ever-useful-and-neat-subprocess-module.html#a-simple-usage
I need to communicate with an embedded system over RS232. For this I want to profile the time it takes to send a response to each command.
I've tested this code using two methods: datetime.now() and timeit()
Method #1
def resp_time(n,msg):
"""Given number of tries - n and bytearray list"""
msg = bytearray(msg)
cnt = 0
timer = 0
while cnt < n:
time.sleep(INTERVAL)
a = datetime.datetime.now()
ser.flush()
ser.write(msg)
line = []
for count in ser.read():
line.append(count)
if count == '\xFF':
# print line
break
b = datetime.datetime.now()
c = b-a
# print c.total_seconds()*1000
timer = timer + c.total_seconds()*1000
cnt = cnt + 1
return timer/n
ser = serial.Serial(COMPORT,BAUDRATE,serial.EIGHTBITS, serial.PARITY_NONE, serial.STOPBITS_ONE, timeout=16)
if ser.isOpen():
print "Serial port opened at: Baud:",COMPORT,BAUDRATE
cmd = read_file()
# returns a list of commands [msg1,msg2....]
n = 100
for index in cmd:
timer = resp_time(n,index)
print "Time in msecs over %d runs: %f " % (n,timer)
Method #2
def com_loop(msg):
msg = bytearray(msg)
time.sleep(INTERVAL)
ser.flush()
ser.write(msg)
line = []
for count in ser.read():
line.append(count)
if count == '\xFF':
break
if __name__ == '__main__':
import timeit
ser = serial.Serial(COMPORT,BAUDRATE,serial.EIGHTBITS, serial.PARITY_NONE, serial.STOPBITS_ONE, timeout=16)
if ser.isOpen():
print "Serial port opened at: Baud:",COMPORT,BAUDRATE
cmd = read_file()
# returns a list of commands [msg1,msg2....]
n = 100
for index in cmd:
t = timeit.timeit("com_loop(index)","from __main__ import com_loop;index=%s;" % index,number = n)
print t/100
With datetime I get 2 milli-sec to execute a command & with timeit I get 200 milli-sec for the same command.
I suspect I'm not calling timeit() properly, can someone point me in the right direction?
I'd assume 200µs is closer to the truth, considering your comport will have something like 115200baud; assuming messages are 8 bytes long, transmitting one message would take about 9/115200 s ~= 10/100000 = 1/10,000 = 100µs on the serial line alone. Being faster than that will be pretty impossible.
Python is definitely not the language of choice to do timing characterization at these scales. You will need to get a logic analyzer, or work very close to the serial controller (which I hope is directly attached to your PC's IO controller and not some USB devices, because that will introduce latencies in the same order of magnitude, at least). If you're talking about microseconds, the limiting factor in measurement is usually the random time it takes for your PC to react to an interrupt, the OS to run the interrupt service routine, the scheduler to continue your userland process, and then starts python with its levels and levels of indirection. You're basically measuring the size of single grains of sand by holding a banana next to them.