I got GPS module who returns NMEA data.
When I'm trying to print all data it returns using following code, I'm getting this.
while True:
try:
rcv = port.read()
print rcv
Then, I've made some modification that will read NMEA data cleaner. It looks like this:
port = serial.Serial("/dev/ttyAMA0", baudrate=9600, timeout=10.0)
line = []
print("connected to: " + port.portstr)
while True:
try:
rcv = port.read()
except:
rcv = ''
line.append(rcv)
if rcv == '\n':
line = "".join(line)
print line
line = []
Output looks like that:
$GPGGA,183345.000,5023.3424,N,01857.3817,E,1,7,1.25,313.3,M,42.1,M,,*53
$GPGSA,A,3,09,26,28,08,15,18,17,,,,,,1.52,1.25,0.88*06
$GPRMC,183345.000,A,5023.3424,N,01857.3817,E,0.40,55.07,050214,,,A*54
$GPVTG,55.07,T,,M,0.40,N,0.74,K,A*0D
$GPGGA,183346.000,5023.3423,N,01857.3817,E,1,7,1.25,313.3,M,42.1,M,,*57
$GPGSA,A,3,09,26,28,08,15,18,17,,,,,,1.52,1.25,0.88*06
The problem is that sometimes it misses some commas or other data, and NMEA parser is reading it wrong. Is there any better and cleaner way to read whole NMEA frames via serial?
you can use readline instead of read, That will continue to read characters until an EOL is received.
Related
I am trying to write data in the serial port and then read the same data. But it only returns a blank page when I try to read the data. How can I fix this issue?
My code
ser = serial.Serial('/dev/cu.usbmodem14302', 115200, timeout = 1)
print ("Writing: ")
ser.write(b'somedata\r\n')
time.sleep(2)
read_val = ser.readline().decode()
print(read_val) # returns a blank line
The expected result: "somedata"
The actual result: " "
I have a pcapng file that contains a little bit of traffic. One of the packets I am trying to print out is containing JSON data. If I open the packet up in Wireshark, I am able to see the values in the JSON. But when using scapy to read the file and print I don't see it.
from scapy.all import IP, sniff
from scapy.layers import http
def process_tcp_packet(packet):
if packet.haslayer(http.HTTPRequest):
http_layer = packet.getlayer(http.HTTPRequest)
ip_layer = packet.getlayer(IP)
#print('\n{0[src]} just requested a {1[Method]} {1[Host]}{1[Path]}'.format(ip_layer.fields, http_layer.fields))
#print(ip_layer.fields)
#print(http_layer.fields)
#packet.show()
print('Packet: ' + str(packet))
print('\n\n')
# Start sniffing the network.
sniff(offline='test.pcapng', prn=process_tcp_packet, count=2)
Here is the JSON content Wireshark is showing me:
And this is the output I am getting for that packet using the code above..
Packet: b'\x18\x0fv\xef0\x8a\xc4\x98\\\xe7=\x18\x08\x00E\x00\x01&&S#\x00#\x06}\n\xc0\xa8\x89\x94#\xa7(\x91\x9b\xd0\x00P\x16-/\x9e\xb1\xa1\xe8V\x80\x18\x01K\x97\xaf\x00\x00\x01\x01\x08\n\x00\x00\t\xd5\xfb\xc3b\x89POST /v1/identify HTTP/1.1\r\nHost: api.segment.io\r\nUser-Agent: Roku/DVP-9.10 (489.10E04121A)\r\nAccept: application/json\r\nAuthorization: Basic: NHJmY3AzUEJmTUhPVlJsWVZZNTZKRDZ0N1JuMUNoaVY=\r\nContent-Type: application/json\r\nContent-Length: 704\r\n\r\n'
I was reading on how to print the entire content of the packet and thats where I came across both packet.show() and print(packet) however both of them are still missing the JSON data.
I want to get the JSON data because I want to be able to manually parse it. I don't like how Wireshark has all the JSON nested into arrows that I have to drop down to see.
This is the output of show:
And I am using the latest version of scapy.
It's an old question, but for future people who search for an answer, here is how I did it:
packet_dict = {}
for line in packet.show2(dump=True).split('\n'):
if '###' in line:
layer = line.strip('#[] ')
packet_dict[layer] = {}
elif '=' in line:
key, val = line.split('=', 1)
packet_dict[layer][key.strip()] = val.strip()
print(json.dumps(packet_dict))
If it can be useful to someone, starting from Yechiel's code I made some improvements:
Key values are returned in the correct format instead of all as a string
Sublayers are parsed
def pkt2dict(pkt):
packet_dict = {}
for line in pkt.show2(dump=True).split('\n'):
if '###' in line:
if '|###' in line:
sublayer = line.strip('|#[] ')
packet_dict[layer][sublayer] = {}
else:
layer = line.strip('#[] ')
packet_dict[layer] = {}
elif '=' in line:
if '|' in line and 'sublayer' in locals():
key, val = line.strip('| ').split('=', 1)
packet_dict[layer][sublayer][key.strip()] = val.strip('\' ')
else:
key, val = line.split('=', 1)
val = val.strip('\' ')
if(val):
try:
packet_dict[layer][key.strip()] = eval(val)
except:
packet_dict[layer][key.strip()] = val
else:
log.debug("pkt2dict packet not decoded: " + line)
return packet_dict
To check if it works on all types of layers returned by scapy.
What is the best approach to process a socket connection where I need var data to end with a line break \n?
I'm using the code below but sometimes the tcp packets get chunked and it takes a long time to match data.endswith("\n").
I've also tried other approaches, like saving the last line if it doesn't end with \n and append it to dataon the next loop. but this also doesn't work because multiple packets get chunked and the 1st and 2nd part don't match.
I've no control over the other end, it basically sends multiple lines that end in \r\n.
Any suggestion will be welcome, as I don't have much knowledge on socket connections.
def receive_bar_updates(s):
global all_bars
data = ''
buffer_size = 4096
while True:
data += s.recv(buffer_size)
if not data.endswith("\n"):
continue
lines = data.split("\n")
lines = filter(None, lines)
for line in lines:
if line.startswith("BH") or line.startswith("BC"):
symbol = str(line.split(",")[1])
all_bars[symbol].append(line)
y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
y.start()
data = ""
Example of "normal" data:
line1\r\n
line2\r\n
line3\r\n
Example of chunked data:
line1\r\n
line2\r\n
lin
If you have a raw input that you want to process as line, the io module is your friend because it will do the low level assembling of packets in lines.
You could use:
class SocketIO(io.RawIOBase):
def __init__(self, sock):
self.sock = sock
def read(self, sz=-1):
if (sz == -1): sz=0x7FFFFFFF
return self.sock.recv(sz)
def seekable(self):
return False
It is more robust than endswith('\n') because if one packet contains an embedded newline ('ab\ncd'), the io module will correctly process it. Your code could become:
def receive_bar_updates(s):
global all_bars
data = ''
buffer_size = 4096
fd = SocketIO(s) # fd can be used as an input file object
for line in fd:
if should_be_rejected_by_filter(line): continue # do not know what filter does...
if line.startswith("BH") or line.startswith("BC"):
symbol = str(line.split(",")[1])
all_bars[symbol].append(line)
y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
y.start()
Use socket.socket.makefile() to wrap the socket in a class that implenents Text I/O. It handles buffering, converting between bytes and strings, and lets you iterate over lines. Remember to flush any writes.
Example:
#!/usr/bin/env python3
import socket, threading, time
def client(addr):
with socket.create_connection(addr) as conn:
conn.sendall(b'aaa')
time.sleep(1)
conn.sendall(b'bbb\n')
time.sleep(1)
conn.sendall(b'cccddd\n')
time.sleep(1)
conn.sendall(b'eeefff')
time.sleep(1)
conn.sendall(b'\n')
conn.shutdown(socket.SHUT_WR)
response = conn.recv(1024)
print('client got %r' % (response,))
def main():
with socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) as listen_socket:
listen_socket.bind(('localhost', 0))
listen_socket.listen(1)
addr = listen_socket.getsockname()
threading.Thread(target=client, args=(addr,)).start()
conn, _addr = listen_socket.accept()
conn_file = conn.makefile(mode='rw', encoding='utf-8')
for request in conn_file:
print('server got %r' % (request,))
conn_file.write('response1\n')
conn_file.flush()
if __name__ == '__main__':
main()
$ ./example.py
server got 'aaabbb\n'
server got 'cccddd\n'
server got 'eeefff\n'
client got b'response1\n'
$
Are you accepting different connections? Or is it one stream of data, split up by \r\n's?
When accepting multiple connections you'd wait for a connection with s.accept() and then process all its data. When you have all of the packet, process its data, and wait for the next connection.
What you do then depends on what the structure of each packet would be.
(Example: https://wiki.python.org/moin/TcpCommunication)
If instead you are consuming a stream of data, you should probably process each 'line' you find in a separate thread, while you keep consuming on another.
Edit:
So, if I have your situation correct; one connection, the data being a string broken up by \r\n, ending with a \n. The data however does not correspond to what you are expecting, instead looping infinitely waiting for a \n.
The socket interface, as I understand it, ends with an empty data result. So the last buffer might have ended with a \n, but then just continued getting None objects, trying to find another \n.
Instead, try adding this:
if not data:
break
Full code:
def receive_bar_updates(s):
global all_bars
data = ''
buffer_size = 4096
while True:
data += s.recv(buffer_size)
if not data:
break
if not data.endswith("\n"):
continue
lines = data.split("\n")
lines = filter(None, lines)
for line in lines:
if line.startswith("BH") or line.startswith("BC"):
symbol = str(line.split(",")[1])
all_bars[symbol].append(line)
y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
y.start()
data = ""
Edit2: Oops, wrong code
I have not tested this code, but it should work:
def receive_bar_updates(s):
global all_bars
data = ''
buf = ''
buffer_size = 4096
while True:
if not "\r\n" in data: # skip recv if we already have another line buffered.
data += s.recv(buffer_size)
if not "\r\n" in data:
continue
i = data.rfind("\r\n")
data, buf = data[:i+2], data[i+2:]
lines = data.split("\r\n")
lines = filter(None, lines)
for line in lines:
if line.startswith("BH") or line.startswith("BC"):
symbol = str(line.split(",")[1])
all_bars[symbol].append(line)
y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
y.start()
data = buf
Edit: Forgot to mention, i only modified the code for receiving the data, i have no idea what the rest of the function (starting with lines = data.split("\n")) is for.
Edit 2: Now uses "\r\n" for linebreaks instead of "\n".
Edit 3: Fixed an issue.
You basically seem to want to read lines from the socket. Maybe you're better off not using low level recv calls but just use sock.makefile() and treat the result as a regular file where you can read lines from: from line in sfile: ...
That leaves the delay/chunk issue. This is likely to be caused by Nagle's algorithm on the sending side. Try disabling that:
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
I'm using python version 2.7.9 and i try to send png file.
But something strange happens..i using sockets and sends a post request(or kind of).
I send the request to the server from the client,then i prints the length of the request received on the server, for example, the length is:1051.
Then I do a regex to take the png file data, and then prints the length, and the length is 2632, that he larger than the response?!
I think the problem is that it's actually write the content, but not the right of representation, I tried different things but they did not work, so I ask here how to solve this problem.
Server source code:
import socket
import re
server = socket.socket()
server.bind(('0.0.0.0',8080))
while True:
server.listen(2)
(client, client_addr) = server.accept()
print 'IP :',client_addr
res = client.recv(0xfffffff)
print len(res)
#get file name
file_name = res.split('&')[0]
file_name = str(file_name.split('=')[1])
print repr(res)
#get the data of the file
raw_img = str(re.findall("&photo_data=(.*)" ,res ,re.DOTALL))
print "File name:" + file_name
print "Size:" + str(len(raw_img))
with open(file_name, 'wb') as f:
f.write(raw_img)
print "Done"
Client source code:
import socket
client = socket.socket()
client.connect(('127.0.0.1',8080))
raw_data = open('test.png', 'rb').read()
save_file_name = raw_input("Enter the file name:")
print len(raw_data)
output = 'POST /upload HTTP/1.1\r\n'
output += 'Content-Length:' + str(len(raw_data)) + str(len(save_file_name)) + '\r\n\r\n'
output += 'file_name=' + save_file_name + '&'
output += 'photo_data=' + raw_data
print len(output)
client.send(output)
client.close()
First, you should use while True to receive the full data:
res = ''
while True:
data = client.recv(1024)
if not data:
break
res += data
print len(res)
Then, re.findall actually returns an array, not a string. So you should do this:
r = re.findall("&photo_data=(.*)" ,res ,re.DOTALL)
raw_img = str(r[0])
Now it works fine.
Why doesn't the code before work? Let's say we have a list:
r = ['\x45']
The data in raw_img part is basically like this. If we brutely convert this list to a str, we have:
print len(str[r])) # ['E'], 5
Actually, what we need is r[0]:
print len(str[r[0])) # 1
That's why the size of the file became larger.
I have a stupid python problem.
I'm trying to read a line from a file everytime i get a 'READY' message from a serial connection so i wrote this :
import serial
from time import sleep
port = "/dev/tty.usbserial-A400fYTT"
speed = 57600
polarfile = 'polarfile.pg'
f = open(polarfile, 'r')
ser = serial.Serial(port, speed, timeout=0)
while True:
data = ser.read(9999)
if len(data) > 0:
if(data == 'READY'):
f.readline()
else:
sleep(0.5)
sleep(1)
ser.close()
But it doesn't work, however if i replace the if(data == 'READY' block by print data. I get the READY message.
Also i can read my file with f.readline()...
Thanks to give advice to a py newbie
--
edit :
Important info, the serial doesn't receive only "READY" message, but a bunch of other, but i want just to react when the "READY" messsage is received.
I just replace
data = ser.read(9999)
by
data = ser.readline(9999) which gives me the message line by line instead of second by second of input data and then replace
if( data == 'READY' ):by
if (data.startswith('READY')):
and now it works :)