We Keep Coding

Conditioning reading from serial port

I'm writing this code to read and fill a .txt from serial communication. I'm using a counter that increments after each reading, but it is not enough. I want the reading to stop when the last line is either " Test completed and passed " or " Test completed and failed ". Below is the part of code that I wrote:
** timeout = 0
#open log file
f = open(Test_name+'_log.txt','a')
while (timeout < 90000):####
# Read line from uart
data=ser.readline().decode('utf-8', 'ignore')
#convert data to string
data=str(data.lower())
# Remove \r from data
data = data.replace('\r','')
print(data)
# Write data inside log file
f.write(data)
# Increment timer
timeout=timeout+1
print(timeout)
if data.lower().startswith(' Test completed and passed ',' Test completed and failed '):
break
f.close()
ser.close()
print('out',timeout)
# Close log file
f.close()
# Close serial port
ser.close()

python socket file transfer verified with sha256 not working, but only sometimes?

Client side:
def send_file_to_hashed(data, tcpsock):
time.sleep(1)
f = data
flag = 0
i=0
tcpsock.send(hashlib.sha256(f.read()).hexdigest())
f.seek(0)
time.sleep(1)
l = f.read(BUFFER_SIZE-64)
while True:
while (l):
tcpsock.send(hashlib.sha256(l).hexdigest() + l)
time.sleep(1)
hashok = tcpsock.recv(6)
if hashok == "HASHOK":
l = f.read(BUFFER_SIZE-64)
flag = 1
if hashok == "BROKEN":
flag = 0
if not l:
time.sleep(1)
tcpsock.send("DONE")
break
return (tcpsock,flag)
def upload(filename):
flag = 0
while(flag == 0):
with open(os.getcwd()+'\\data\\'+ filename +'.csv', 'rU') as UL:
tuplol = send_file_to_hashed(UL ,send_to_sock(filename +".csv",send_to("upload",TCP_IP,TCP_PORT)))
(sock,flagn) = tuplol
flag = flagn
time.sleep(2)
sock.close()
Server Side:
elif(message == "upload"):
message = rec_OK(self.sock)
fis = os.getcwd()+'/data/'+ time.strftime("%H:%M_%d_%m_%Y") + "_" + message
f = open(fis , 'w')
latest = open(os.getcwd()+'/data/' + message , 'w')
time.sleep(1)
filehash = rec_OK(self.sock)
print("filehash:" + filehash)
while True:
time.sleep(1)
rawdata = self.sock.recv(BUFFER_SIZE)
log.write("rawdata :" + rawdata + "\n")
data = rawdata[64:]
dhash = rawdata[:64]
log.write("chash: " + dhash + "\n")
log.write("shash: " + hashlib.sha256(data).hexdigest() + "\n")
if dhash == hashlib.sha256(data).hexdigest():
f.write(data)
latest.write(data)
self.sock.send("HASHOK")
log.write("HASHOK\n" )
print"HASHOK"
else:
self.sock.send("HASHNO")
print "HASHNO"
log.write("HASHNO\n")
if rawdata == "DONE":
f.close()
f = open(fis , 'r')
if (hashlib.sha256(f.read()).hexdigest() == filehash):
print "ULDONE"
log.write("ULDONE")
f.close()
latest.close()
break
else:
self.sock.send("BROKEN")
print hashlib.sha256(f.read()).hexdigest()
log.write("BROKEN")
print filehash
print "BROKEN UL"
f.close()
So the data upload is working fine in all tests that i ran from my computer, even worked fine while uploading data over my mobile connection and still sometimes people say it takes a long time and they kill it after a few minutes. the data is there on their computers but not on the server. I don't know what is happening please help!

First of all: this is unrelated to sha.
Streaming over the network is unpredictable. This line
rawdata = self.sock.recv(BUFFER_SIZE)
doesn't guarantee that you read BUFFER_SIZE bytes. You may have read only 1 byte in the worst case scenario. Therefore your server side is completely broken because of the assumption that rawdata contains whole message. It is even worse. If the client sends command and hash fast you may get e.g. rawdata == 'DONEa2daf78c44(...) which is a mixed output.
The "hanging" part just follows from that. Trace your code and see what happens when the server receives partial/broken messages ( I already did that in my imagination :P ).
Streaming over the network is almost never as easy as calling sock.send on one side and sock.recv on the other side. You need some buffering/framing protocol. For example you can implement this simple protocol: always interpret first two bytes as the size of incoming message, like this:
client (pseudocode)
# convert len of msg into two-byte array
# I am assuming the max size of msg is 65536
buf = bytearray([len(msg) & 255, len(msg) >> 8])
sock.sendall(buf)
sock.sendall(msg)
server (pseudocode)
size = to_int(sock.recv(1))
size += to_int(sock.recv(1)) << 8
# You need two calls to recv since recv(2) can return 1 byte.
# (well, you can try recv(2) with `if` here to avoid additional
# syscall, not sure if worth it)
buffer = bytearray()
while size > 0:
tmp = sock.recv(size)
buffer += tmp
size -= len(tmp)
Now you have properly read data in buffer variable which you can work with.
WARNING: the pseudocode for the server is simplified. For example you need to check for empty recv() result everywhere (including where size is calculated). This is the case when the client disconnects.
So unfortunately there's a lot of work in front of you. You have to rewrite whole sending and receving code.

Why is my 'for' loop not looping?

The following python script should do following:
wait for a key press, then
send X1650 Y0 Z0 to an embedded device, then
fill the variable line byte by byte with the response
Altough print (ser.in_waiting) claims that the input buffer is properly filled, the for is not iterating over it.
Code:
import serial
import time
# configure the serial connections
ser = serial.Serial(
port='COM3',
baudrate=9600,
)
while 1 :
# Wait until user presses a key
eingabe = input("PROMPT >> ")
# Send text string to embedded device
destination_position = 'X1650 Y0 Z0\r\n'
ser.write(destination_position.encode('ascii'))
# Wait until embedded device responds
while ser.in_waiting == 0:
time.sleep(0.1)
# How long is the response?
print ('The response is: ')
print (ser.in_waiting)
print (' bytes long')
# Traverse through the queue
line = []
for c in ser.read():
line.append(chr(c))
print(line)
Output:
D:\7-Thema\Programmieren\projects\robot\remote-control-scripts>python test.py
PROMPT >> GO!
The response is:
39
bytes long
['X']
PROMPT >>

You must specify the number of bytes to be read. But I don't know the return type so when you print a line, you will see :) and then you can convert accordingly
line = ser.read(ser.in_waiting)
print("%r"%line)
Here is a docs link

Unindent does not match any outer indentation level

On line 51/52, I get the error:
Unindent does not match any outer indentation level.
I understand this has something to do with tabs and spaces.
Note I did not write this code, I found it online and plan to modify it.
Full code (also at http://pastebin.com/n7ML6Rpz)
import os
import re
import socket
import sys
import time
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # Create socket
server_socket.bind(("", 9020)) #Bind server to this socket
server_socket.listen(4) #Max number of queued connections
# Welcome message
print ("TCP chat server now awaiting client connection on port 9020...")
chat_log = [] #Contains chat log
time = time.strftime('%l:%M %p %Z on %b %d, %Y') #Server start time formatted nicely
start_time = str(time) #Convert server start time to string
username = "ChatUser" #Default server username if user does not provide one
# Support ~2^x client connections, where x is the number of process forks
os.fork()
os.fork()
os.fork()
# This variable contains the help documentation for the "help" command
chatHelp = ("The chat server accepts the following commands:\n"
+ "adios Closes the program\n"
+ "connection Shows client connection info (IP, port)\n"
+ "get Returns complete chat log\n"
+ "getrange <#> <#> Get chat log entries from <#> to <#> (starts at 1)\n"
+ "help Lists valid commands\n"
+ "name: <text> Sets your username to <text>\n"
+ "test: <text> Echo data back to you <text>\n"
+ "time Shows time when server was initiated\n"
+ "push: <text> Add <text> to chat log\n"
+ "save Save chat log to file\n")
while 1:
# Accept connection
client_socket, address = server_socket.accept()
# Print connection info from client for server log
print ("Received connection from client at"), address
# Used in the connection command function (client request) below
connection = str(address)
# Send welcome string to client
client_socket.send("Welcome to Nigel's chat room! You are logged in as ChatUser.\n Type help for a list of valid commands.\n")
# Loop indefinitely while server running
while 1:
data = client_socket.recv(2048) #Receive client data into buffer
process_data = data.lower() #Lowercase received data for processing
print ("Data received from client>>"), process_data #Print data received from client for log reference
# Functions for the received commands (I use the find library to reduce compatibility errors with other languages)
# ---"adios" command function---
if (process_data.find("adios") == 0):
client_socket.close() #Close socket connection
print ("<Ctrl+C to exit.>>")
break;
# ---"connection:" command function---
elif(process_data.find("connection") == 0):
client_socket.send("Client connection info: " + connection + "\n")
print "User requested connection information"
# ---"getrange" command function w/ regular expression filtering (must be BEFORE "get" command function)---
elif(re.match(r'getrange\s+(\d+)\s+(\d+)',process_data)): # Regex to find correct match with dynamic numbers input
match = re.match(r'getrange\s+(\d+)\s+(\d+)',process_data)
getValue = "Chat log from range "+ match.group(1) + " and " + match.group(2) + ":\n" # Grab first and second range number provided by client
if(len(chat_log) >= int(match.group(1)) and len(chat_log) >= int(match.group(2))): # Check to see if chat log extends to given range
count = int(match.group(1)) - 1
while(count < int(match.group(2))):
getValue += chat_log[count] + "\n"
count += 1
else:
getValue += "<>\n" #No data in range provided by client
client_socket.send(getValue) #Send results to client
# ---"get" command function---
elif(process_data.find("get") == 0):
log = "Chat log: \n"
for item in chat_log:
log += item+" \n"
client_socket.send(log)
# ---"help:" command function---
elif(process_data.find("help") == 0):
client_socket.send(chatHelp + "\n")
print "User requested help"
# ---"name:" command function---
elif(process_data.find("name:") == 0):
username = data[5:].strip() #Only grab the value client set (not "name:")
client_socket.send("Username set to: " + data[5:] + "\n")
# ---"test:" command function---
elif(process_data.find("test:") == 0):
client_socket.send(data[5:]+"\n") #Echo last 5 elements to client
print data
# ---"time" command function---
elif(process_data.find("time") == 0):
client_socket.send("Chat server was started at: " + start_time + "\n")
print "User requested server start time"
# ---"save" command function---
elif(process_data.find("save") == 0):
print "(Saving chat log to file)"
client_socket.send("Saving chat log to file..." + "\n")
filename = "chat.log"
file = open(filename,"w") #Create file
for item in chat_log: #Loop through elements in chat_log
file.write("%s\n" % item) #Write elements one by one on a new line
file.close() #Close/write file
# ---"push" command function---
elif(process_data.find("push:") == 0):
print "(Pushing data to chat log)"
if(username != ""):
chat_log.append(username + ": " + data[5:].strip()) #Save actual chat text to log (not "push:")
else:
chat_log.append(data[5:].strip())
client_socket.send("OK\n")
else:
print "<<Unknown Data Received>>",data #Server log
try:
client_socket.send("Unrecognized command: " + data + "") #Advise client of invalid command
except socket.error, e:
print "<<Ctrl+C to exit>>" #Server log
break;

Python code is sensitive to the indent level you use. Your code reads:
if (process_data.find("adios") == 0):
client_socket.close() #Close socket connection
print ("<Ctrl+C to exit.>>")
break;
Inside the if block, the statements must all line up. Notice how client_socket.close() and the following print statement have different indent levels. You need to change this so that they both have the same indent, like this:
if (process_data.find("adios") == 0):
client_socket.close() #Close socket connection
print ("<Ctrl+C to exit.>>")
break;

The code presently reads:
if (process_data.find("adios") == 0):
client_socket.close() #Close socket connection
print ("<Ctrl+C to exit.>>")
break;
The first statement in the body of the if is indented 6 spaces, while the last two statements are only indented by 1 space. The indentation level ought to be same and consistent throughout the program, something like this:
if (process_data.find("adios") == 0):
client_socket.close() #Close socket connection
print ("<Ctrl+C to exit.>>")
break;
The indentation in the code doesn't seem very standard (or consistent). For instance the body of the while loop in 41/42 is indented more than other bodies of similar statements, e.g., lines 31-33, that's trouble in a language like Python where whitespace matters.
Finally, note it's not a good idea to mix tabs and spaces. PEP 8 -- Style Guide for Python Code recommends the use of spaces over tabs and an indentation of 4 spaces per level.

pyserial readline() : SerialException

I'm writing a code used to send order to an avr. I send several information but between each write, I have to wait for an answer (I have to wait for the robot to reach a point on the coordinate system). As I read in the documentation, readline() should at least read until the timeout but as soon as I send the first coordinate, the readline() automatically return :
SerialException: device reports readiness to read but returned no data (device disconnected?)
When I put a sleep() between each write() in the for loop, everything works fine. I tried to use inWaiting() but it still does not work. Here is an example of how I used it:
for i in chemin_python:
self.serieInstance.ecrire("goto\n" + str(float(i.x)) + '\n' + str(float(-i.y)) + '\n')
while self.serieInstance.inWaiting():
pass
lu = self.serieInstance.readline()
lu = lu.split("\r\n")[0]
reponse = self.serieInstance.file_attente.get(lu)
if reponse != "FIN_GOTO":
log.logger.debug("Erreur asservissement (goto) : " + reponse)

Here an snipet how to use serial in python
s.write(command);
st = ''
initTime = time.time()
while True:
st += s.readline()
if timeout and (time.time() - initTime > t) : return TIMEOUT
if st != ERROR: return OK
else: return ERROR

This method allows you to separately control the timeout for gathering all the data for each line, and a different timeout for waiting on additional lines.
def serial_com(self, cmd):
'''Serial communications: send a command; get a response'''
# open serial port
try:
serial_port = serial.Serial(com_port, baudrate=115200, timeout=1)
except serial.SerialException as e:
print("could not open serial port '{}': {}".format(com_port, e))
# write to serial port
cmd += '\r'
serial_port.write(cmd.encode('utf-8'))
# read response from serial port
lines = []
while True:
line = serial_port.readline()
lines.append(line.decode('utf-8').rstrip())
# wait for new data after each line
timeout = time.time() + 0.1
while not serial_port.inWaiting() and timeout > time.time():
pass
if not serial_port.inWaiting():
break
#close the serial port
serial_port.close()
return lines