Related
Explanation
I'm currently trying to control a smart power strip using a python script. To accomplish this, I'm using a TCP connection with the socket module. Around 75% of the time, I get the response/data I was looking for and everything works perfectly. However, around 25% of the time, the response is cut off at the exact same length, 1024 bytes. This doesn't make any sense to me, as my buffer size is actually set to 2048 bytes. The speed at which I wait in between using recv() doesn't seem to effect/cause this either. Altough TCP is a stream of bytes, is it still possible that this could have to do with packet fragmentation?
Code
Main Code
ip='192.168.0.62'
port=9999
sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock_tcp.connect((ip, port))
sock_tcp.send(encrypt('{"system":{"get_sysinfo":{}}}'))
data = sock_tcp.recv(2048)
sock_tcp.close()
print len(data) #On succesful runs output is 1221, on unsuccesful runs it is 1024
rec = decrypt(data[4:])
print str(rec) #See output below
Encrypt Function
def encrypt(string):
key = 171
result = pack('>I', len(string))
for i in string:
a = key ^ ord(i)
key = a
result += chr(a)
return result
Decrypt Function
def decrypt(string):
key = 171
result = ""
for i in string:
a = key ^ ord(i)
key = ord(i)
result += chr(a)
return result
Output
The string itself that I recieve. It's most likeley not relevant, but I thought I would include it anyway. This is value of the variable rec.
Desired and regular output
Full desired output
{"system":{"get_sysinfo":{"sw_ver":"1.0.6 Build 180627
Rel.081000","hw_ver":"1.0","model":"HS300(US)","deviceId":"80067B24A755F99C4D6C1807455E09F91AB7B2AA","oemId":"5C9E6254BEBAED63B2B6102966D24C17","hwId":"34C41AA028022D0CCEA5E678E8547C54","rssi":-60,"longitude_i":-1222955,"latitude_i":379078,"alias":"TP-LINK_Power
Strip_4F01","mic_type":"IOT.SMARTPLUGSWITCH","feature":"TIM:ENE","mac":"B0:BE:76:12:4F:01","updating":0,"led_off":0,"children":[{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA00","state":0,"alias":"CezHeat","on_time":0,"next_action":{"type":-1}},{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA01","state":1,"alias":"CezUVB","on_time":191208,"next_action":{"type":-1}},{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA02","state":1,"alias":"CyanHeat","on_time":191208,"next_action":{"type":-1}},{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA03","state":1,"alias":"ZanderHeat","on_time":191208,"next_action":{"type":-1}},{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA04","state":1,"alias":"CairoHeat","on_time":191208,"next_action":{"type":-1}},{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA05","state":1,"alias":"KodaMister","on_time":191208,"next_action":{"type":-1}}],"child_num":6,"err_code":0}}}
Abnormal and rarer output
Cut off output
{"system":{"get_sysinfo":{"sw_ver":"1.0.6 Build 180627
Rel.081000","hw_ver":"1.0","model":"HS300(US)","deviceId":"80067B24A755F99C4D6C1807455E09F91AB7B2AA","oemId":"5C9E6254BEBAED63B2B6102966D24C17","hwId":"34C41AA028022D0CCEA5E678E8547C54","rssi":-59,"longitude_i":-1222955,"latitude_i":379078,"alias":"TP-LINK_Power
Strip_4F01","mic_type":"IOT.SMARTPLUGSWITCH","feature":"TIM:ENE","mac":"B0:BE:76:12:4F:01","updating":0,"led_off":0,"children":[{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA00","state":0,"alias":"CezHeat","on_time":0,"next_action":{"type":-1}},{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA01","state":1,"alias":"CezUVB","on_time":191207,"next_action":{"type":-1}},{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA02","state":1,"alias":"CyanHeat","on_time":191207,"next_action":{"type":-1}},{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA03","state":1,"alias":"ZanderHeat","on_time":191207,"next_action":{"type":-1}},{"id":"80067B24A755F99C4D6C1807455E09F91AB7B2AA04","state":1,"alias":"CairoHeat","on
Conclusion
If anyone could provide me with a solution or explanation as to why the output/stream gets cut off, it would be much appreciated. I used a lot of the code from this open source module. I'm also looking to understand more of how this all works, so if you could explain a bit more I would really appreciate it.
As per the documentation, the bufsize argument only specifies the maximum amount of data to be read:
socket.recv(bufsize[, flags])
Receive data from the socket. The return
value is a bytes object representing the data received. The maximum
amount of data to be received at once is specified by bufsize. See the
Unix manual page recv(2) for the meaning of the optional argument
flags; it defaults to zero.
To ensure full data transfer a function like this can be used, which waits for the end of the socket connection (indicated by and empty string returned from recv):
def recv_all(connection):
"""
Function for all data
:param connection: socket connection
:return: received data
"""
data = list()
while True:
data.append(connection.recv(2048))
if not data[-1]:
return b''.join(data)
Another example that might fit your application better could be to wait for a fixed message size (1221 as indicated by your question):
def recv_message(connection):
data = list()
transferred_bytes= 0
while transferred_bytes < 1221:
data.append(connection.recv(min(1221-transferred_bytes, 2048)))
if not data[-1]:
raise RuntimeError("socket connection broken")
transferred_bytes += len(data[-1])
return b''.join(data)
This is only a complement to SimonF's answer. The cause of the problem is indeed that TCP is a stream protocol, so packets can be fragmented or re-assembled at any state: sender TCP/IP stack, network equipments, receiver TCP/IP stack - I include the user layer library in the TCP/IP stack here for simplification.
That is the reason why, you should always use a higher level protocol above TCP to be able to split the stream in sensible messages. Here you could note that the end of a message is '}}}', so you could concatenate the input in a buffer until you find that pattern:
def recv_until(c, guard):
"""Receive data from a socket until guard if found on input"""
guard_sz = len(guard) - 1
data = b''
sz = 0
while True:
buffer = c.recv(1024) # read by chuncks of size 1024 (change value to your needs)
got = len(buffer)
data += buffer # concatenate in buffer
ix = data.find(guard, sz - guard_sz if sz > guard_sz else 0) # is guard found?
if ix != -1:
return (data[:ix + guard_sz + 1], # return the message, and what could be behind it
data[ix + guard_sz + 1:])
sz += got
The trick is to considere guard_sz byte from the last chunk, in the case where the guard could be split in two chunks.
Marco, please use recv_into(buffer[, nbytes[, flags]]) method for the socket.
My example for TCP-microserver:
import socket
import struct
def readReliably(s,n):
buf = bytearray(n)
view = memoryview(buf)
sz = 0
while sz < n:
k = s.recv_into(view[sz:],n-sz)
sz += k
# print 'readReliably()',sz
return sz,buf
def writeReliably(s,buf,n):
sz = 0
while sz < n:
k = s.send(buf[sz:],n-sz)
sz += k
# obj = s.makefile(mode='w')
# obj.flush()
# print 'writeReliably()',sz
return sz
# Client
host = "127.0.0.1"
port = 23456
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(10)
s.connect((host,port))
# Request
buf = struct.pack("4B",*[0x01,0x02,0x03,0x04])
io.writeReliably(s,buf,4)
# Response
sz,buf = io.readReliably(s,4)
a = struct.unpack("4B",buf)
print repr(a)
# Server
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
#s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
#s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
#s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
s.bind((host,port))
s.listen(10) # unaccepted connections
while True:
sk,skfrom = s.accept()
sz,buf = io.readReliably(sk,4)
a = struct.unpack("4B",buf)
print repr(a)
# ...
io.writeReliably(sk,struct.pack("4B",*[0x01,0x02,0x03,0x04]))
What is the best approach to process a socket connection where I need var data to end with a line break \n?
I'm using the code below but sometimes the tcp packets get chunked and it takes a long time to match data.endswith("\n").
I've also tried other approaches, like saving the last line if it doesn't end with \n and append it to dataon the next loop. but this also doesn't work because multiple packets get chunked and the 1st and 2nd part don't match.
I've no control over the other end, it basically sends multiple lines that end in \r\n.
Any suggestion will be welcome, as I don't have much knowledge on socket connections.
def receive_bar_updates(s):
global all_bars
data = ''
buffer_size = 4096
while True:
data += s.recv(buffer_size)
if not data.endswith("\n"):
continue
lines = data.split("\n")
lines = filter(None, lines)
for line in lines:
if line.startswith("BH") or line.startswith("BC"):
symbol = str(line.split(",")[1])
all_bars[symbol].append(line)
y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
y.start()
data = ""
Example of "normal" data:
line1\r\n
line2\r\n
line3\r\n
Example of chunked data:
line1\r\n
line2\r\n
lin
If you have a raw input that you want to process as line, the io module is your friend because it will do the low level assembling of packets in lines.
You could use:
class SocketIO(io.RawIOBase):
def __init__(self, sock):
self.sock = sock
def read(self, sz=-1):
if (sz == -1): sz=0x7FFFFFFF
return self.sock.recv(sz)
def seekable(self):
return False
It is more robust than endswith('\n') because if one packet contains an embedded newline ('ab\ncd'), the io module will correctly process it. Your code could become:
def receive_bar_updates(s):
global all_bars
data = ''
buffer_size = 4096
fd = SocketIO(s) # fd can be used as an input file object
for line in fd:
if should_be_rejected_by_filter(line): continue # do not know what filter does...
if line.startswith("BH") or line.startswith("BC"):
symbol = str(line.split(",")[1])
all_bars[symbol].append(line)
y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
y.start()
Use socket.socket.makefile() to wrap the socket in a class that implenents Text I/O. It handles buffering, converting between bytes and strings, and lets you iterate over lines. Remember to flush any writes.
Example:
#!/usr/bin/env python3
import socket, threading, time
def client(addr):
with socket.create_connection(addr) as conn:
conn.sendall(b'aaa')
time.sleep(1)
conn.sendall(b'bbb\n')
time.sleep(1)
conn.sendall(b'cccddd\n')
time.sleep(1)
conn.sendall(b'eeefff')
time.sleep(1)
conn.sendall(b'\n')
conn.shutdown(socket.SHUT_WR)
response = conn.recv(1024)
print('client got %r' % (response,))
def main():
with socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) as listen_socket:
listen_socket.bind(('localhost', 0))
listen_socket.listen(1)
addr = listen_socket.getsockname()
threading.Thread(target=client, args=(addr,)).start()
conn, _addr = listen_socket.accept()
conn_file = conn.makefile(mode='rw', encoding='utf-8')
for request in conn_file:
print('server got %r' % (request,))
conn_file.write('response1\n')
conn_file.flush()
if __name__ == '__main__':
main()
$ ./example.py
server got 'aaabbb\n'
server got 'cccddd\n'
server got 'eeefff\n'
client got b'response1\n'
$
Are you accepting different connections? Or is it one stream of data, split up by \r\n's?
When accepting multiple connections you'd wait for a connection with s.accept() and then process all its data. When you have all of the packet, process its data, and wait for the next connection.
What you do then depends on what the structure of each packet would be.
(Example: https://wiki.python.org/moin/TcpCommunication)
If instead you are consuming a stream of data, you should probably process each 'line' you find in a separate thread, while you keep consuming on another.
Edit:
So, if I have your situation correct; one connection, the data being a string broken up by \r\n, ending with a \n. The data however does not correspond to what you are expecting, instead looping infinitely waiting for a \n.
The socket interface, as I understand it, ends with an empty data result. So the last buffer might have ended with a \n, but then just continued getting None objects, trying to find another \n.
Instead, try adding this:
if not data:
break
Full code:
def receive_bar_updates(s):
global all_bars
data = ''
buffer_size = 4096
while True:
data += s.recv(buffer_size)
if not data:
break
if not data.endswith("\n"):
continue
lines = data.split("\n")
lines = filter(None, lines)
for line in lines:
if line.startswith("BH") or line.startswith("BC"):
symbol = str(line.split(",")[1])
all_bars[symbol].append(line)
y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
y.start()
data = ""
Edit2: Oops, wrong code
I have not tested this code, but it should work:
def receive_bar_updates(s):
global all_bars
data = ''
buf = ''
buffer_size = 4096
while True:
if not "\r\n" in data: # skip recv if we already have another line buffered.
data += s.recv(buffer_size)
if not "\r\n" in data:
continue
i = data.rfind("\r\n")
data, buf = data[:i+2], data[i+2:]
lines = data.split("\r\n")
lines = filter(None, lines)
for line in lines:
if line.startswith("BH") or line.startswith("BC"):
symbol = str(line.split(",")[1])
all_bars[symbol].append(line)
y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
y.start()
data = buf
Edit: Forgot to mention, i only modified the code for receiving the data, i have no idea what the rest of the function (starting with lines = data.split("\n")) is for.
Edit 2: Now uses "\r\n" for linebreaks instead of "\n".
Edit 3: Fixed an issue.
You basically seem to want to read lines from the socket. Maybe you're better off not using low level recv calls but just use sock.makefile() and treat the result as a regular file where you can read lines from: from line in sfile: ...
That leaves the delay/chunk issue. This is likely to be caused by Nagle's algorithm on the sending side. Try disabling that:
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
How can I find with scapy wireless networks around? If I do sniff() and if pkt.haslayer(Dot11) and then if pkt.info then I collect them but very slow, for example my Android phone do it in seconds and this script in minutes or even more...
The reason for the difference is that your phone is actively looking for WiFi points by sending out requests to any access points nearby - sniff is listening for any passing traffic.
You might find is a lot quicker to:
Specifically select your network adapter - so you are not sniffing all adapters.
Do some digging to find out how to actively query for wifi networks and use sr with such packets, read the IEEE 802.11 specification to find out more, I would especially look for "Probe request frame".
The example on how to send WiFi packets from packet header may well help, (not my code and not tested by me):
#!/usr/bin/env python
"""
802.11 Scapy Packet Example
Author: Joff Thyer, 2014
"""
# if we set logging to ERROR level, it supresses the warning message
# from Scapy about ipv6 routing
# WARNING: No route found for IPv6 destination :: (no default route?)
import logging
logging.getLogger("scapy.runtime").setLevel(logging.ERROR)
from scapy.all import *
class Scapy80211():
def __init__(self,intf='wlan0',ssid='test',\
source='00:00:de:ad:be:ef',\
bssid='00:11:22:33:44:55',srcip='10.10.10.10'):
self.rates = "\x03\x12\x96\x18\x24\x30\x48\x60"
self.ssid = ssid
self.source = source
self.srcip = srcip
self.bssid = bssid
self.intf = intf
self.intfmon = intf + 'mon'
# set Scapy conf.iface
conf.iface = self.intfmon
# create monitor interface using iw
cmd = '/sbin/iw dev %s interface add %s type monitor >/dev/null 2>&1' \
% (self.intf, self.intfmon)
try:
os.system(cmd)
except:
raise
def Beacon(self,count=10,ssid='',dst='ff:ff:ff:ff:ff:ff'):
if not ssid: ssid=self.ssid
beacon = Dot11Beacon(cap=0x2104)
essid = Dot11Elt(ID='SSID',info=ssid)
rates = Dot11Elt(ID='Rates',info=self.rates)
dsset = Dot11Elt(ID='DSset',info='\x01')
tim = Dot11Elt(ID='TIM',info='\x00\x01\x00\x00')
pkt = RadioTap()\
/Dot11(type=0,subtype=8,addr1=dst,addr2=self.source,addr3=self.bssid)\
/beacon/essid/rates/dsset/tim
print '[*] 802.11 Beacon: SSID=[%s], count=%d' % (ssid,count)
try:
sendp(pkt,iface=self.intfmon,count=count,inter=0.1,verbose=0)
except:
raise
def ProbeReq(self,count=10,ssid='',dst='ff:ff:ff:ff:ff:ff'):
if not ssid: ssid=self.ssid
param = Dot11ProbeReq()
essid = Dot11Elt(ID='SSID',info=ssid)
rates = Dot11Elt(ID='Rates',info=self.rates)
dsset = Dot11Elt(ID='DSset',info='\x01')
pkt = RadioTap()\
/Dot11(type=0,subtype=4,addr1=dst,addr2=self.source,addr3=self.bssid)\
/param/essid/rates/dsset
print '[*] 802.11 Probe Request: SSID=[%s], count=%d' % (ssid,count)
try:
sendp(pkt,count=count,inter=0.1,verbose=0)
except:
raise
def ARP(self,targetip,count=1,toDS=False):
if not targetip: return
arp = LLC()/SNAP()/ARP(op='who-has',psrc=self.srcip,pdst=targetip,hwsrc=self.source)
if toDS:
pkt = RadioTap()\
/Dot11(type=2,subtype=32,FCfield='to-DS',\
addr1=self.bssid,addr2=self.source,addr3='ff:ff:ff:ff:ff:ff')\
/arp
else:
pkt = RadioTap()\
/Dot11(type=2,subtype=32,\
addr1='ff:ff:ff:ff:ff:ff',addr2=self.source,addr3=self.bssid)\
/arp
print '[*] ARP Req: who-has %s' % (targetip)
try:
sendp(pkt,inter=0.1,verbose=0,count=count)
except:
raise
ans = sniff(lfilter = lambda x: x.haslayer(ARP) and x.op == 2,
store=1,count=1,timeout=1)
if len(ans) > 0:
return ans[0][ARP].hwsrc
else:
return None
def DNSQuery(self,query='www.google.com',qtype='A',ns=None,count=1,toDS=False):
if ns == None: return
dstmac = self.ARP(ns)
dns = LLC()/SNAP()/IP(src=self.srcip,dst=ns)/\
UDP(sport=random.randint(49152,65535),dport=53)/\
DNS(qd=DNSQR(qname=query,qtype=qtype))
if toDS:
pkt = RadioTap()\
/Dot11(type=2,subtype=32,FCfield='to-DS',\
addr1=self.bssid,addr2=self.source,addr3=dstmac)/dns
else:
pkt = RadioTap()\
/Dot11(type=2,subtype=32,\
addr1=dstmac,addr2=self.source,addr3=self.bssid)/dns
print '[*] DNS query %s (%s) -> %s?' % (query,qtype,ns)
try:
sendp(pkt,count=count,verbose=0)
except:
raise
# main routine
if __name__ == "__main__":
print """
[*] 802.11 Scapy Packet Crafting Example
[*] Assumes 'wlan0' is your wireless NIC!
[*] Author: Joff Thyer, 2014
"""
sdot11 = Scapy80211(intf='wlan0')
sdot11.Beacon()
sdot11.ProbeReq()
sdot11.DNSQuery(ns='10.10.10.2')
I once wrote a script that could scan wireless network .
Its simple to use :
python rs.py mon0
Here mon0 is our interface. There are comments in the code to understand it properly.
#Implementation of a wireless scanner using Scapy library
#!/usr/bin/env python
# rs.py - Wireless AP scanner
#author rahil sharma
# date 15/3/2013 #rs
#usage python rs.py mon0
#where mon0 is your monitoring interface
#used this using my alfa card in bactrack
import sys, os, signal
from multiprocessing import Process
from scapy.all import *
interface='' # monitor interface
aps = {} # dictionary to store unique APs
# process unique sniffed Beacons and ProbeResponses.
#haslayer packet has Dot11 layer present
#ord() string to integer ex ord('a) will give 97
def sniffAP(p):
if ( (p.haslayer(Dot11Beacon))):
ssid = p[Dot11Elt].info
bssid = p[Dot11].addr3
channel = int( ord(p[Dot11Elt:3].info))
capability = p.sprintf("{Dot11Beacon:%Dot11Beacon.cap%}\
{Dot11ProbeResp:%Dot11ProbeResp.cap%}")
# Check for encrypted networks
#now we put Dot11Beacon.cap info in capability and using regular expression search inbuilt function in python we search for privacy if it is present then the network is encrypted
#output of the above cap file is somewhat like this short-slot+DSSS-OFDM+res15+ESS
if re.search("privacy", capability): enc = 'Y'
else: enc = 'N'
# Save discovered AP
aps[p[Dot11].addr3] = enc
# Display discovered AP
print "%02d %s %s %s" % (int(channel), enc, bssid, ssid)
# Channel hopper - we are making a channel hopper because we want to scan the whole wireless spectrum.
#first choose a random channel using randrange function
#use system to run the shell command iw dev wlan0 set channel 1
#exit when a keyboard interrupt is given CTrl+c
def channel_hopper():
while True:
try:
channel = random.randrange(1,15)
os.system("iw dev %s set channel %d" % (interface, channel))
time.sleep(1)
except KeyboardInterrupt:
break
# Capture interrupt signal and cleanup before exiting
#terminate is used to end the child process
#before exiting the program we will be displaying number of aps found etc.
#here Cntrl+c is used to
#signal_handler used to do clean up before the program exits
def signal_handler(signal, frame):
p.terminate()
p.join()
print "\n-=-=-=-=-= STATISTICS =-=-=-=-=-=-"
print "Total APs found: %d" % len(aps)
print "Encrypted APs : %d" % len([ap for ap in aps if aps[ap] =='Y'])
print "Unencrypted APs: %d" % len([ap for ap in aps if aps[ap] =='N'])
sys.exit(0)
#use this for command line variables
#for checking the number of command line variables and if they are in right order
if __name__ == "__main__":
if len(sys.argv) != 2:
print "Usage %s monitor_interface" % sys.argv[0]
sys.exit(1)
interface = sys.argv[1]
#take mon0 as interface given in the fist command line variable
# Print the program header
print "-=-=-=-=-=-= rs_scan.py =-=-=-=-=-=-"
print "CH ENC BSSID SSID"
# Start the channel hopper
#In multiprocessing, processes are spawned by creating a Process object and then calling its start() method
p = Process(target = channel_hopper)
p.start()
# Capture CTRL-C
#this will call the signal handler CTRL+C comes under the SIGINT
signal.signal(signal.SIGINT, signal_handler)
# Start the sniffer
sniff(iface=interface,prn=sniffAP)
#inbuit scapy function to start sniffing calls a function which defines the criteria and we need to give the interface`enter code here`
I'm working on a server, and all of the data is line based. I want to be able to raise an exception when a line exceeds a given length without reading any more data than I have to. For example, client X sends a line that's 16KB long even though the line-length limit is 1024 bytes. After reading more than 1024 bytes, I want to stop reading additional data, close the socket and raise an exception. I've looked through the docs and some of the source code, and I don't see a way to do this without rewriting the _readline method. Is there an easier way that I'm overlooking?
EDIT: Comments made me realize I need to add more information. I know I could write the logic to do this without much work, but I was hoping to use builtins to take advantage of efficient buffering with memoryview rather than implementing it myself again or going with the naive approach of reading chunks, joing and splitting as needed without a memoryview.
I don't really like accepting answers that don't really answer the question, so here's the approach I actually ended up taking, and I'll just mark it community wiki or unanswered later if no one has a better solution:
#!/usr/bin/env python3
class TheThing(object):
def __init__(self, connection, maxlinelen=8192):
self.connection = connection
self.lines = self._iterlines()
self.maxlinelen = maxlinelen
def _iterlines(self):
"""
Yield lines from class member socket object.
"""
buffered = b''
while True:
received = self.connection.recv(4096)
if not received:
if buffered:
raise Exception("Unexpected EOF.")
yield received
continue
elif buffered:
received = buffered + received
if b'\n' in received:
for line in received.splitlines(True):
if line.endswith(b'\n'):
if len(line) > self.maxlinelen:
raise LineTooLong("Line size: %i" % len(line))
yield line
else:
buffered = line
else:
buffered += received
if len(buffered) > self.maxlinelen:
raise LineTooLong("Too much data in internal buffer.")
def _readline(self):
"""
Return next available line from member socket object.
"""
return next(self.lines)
I haven't bothered comparing the code to be certain, but I'm doing fewer concatenations and splits, so I think mine may be more efficient.
I realize that your edit is clarifying that what you want is a builtin approach to achieving your goal. But I am not aware of anything existing that will help you in that fine grained control over the readline approach. But I thought I might just include an example that does do a coded approach with a generator and a split... Just for fun.
Reference this other question/answer for a nice generator that reads lines:
https://stackoverflow.com/a/822788/496445
Based on that reader:
server.py
import socket
MAXLINE = 100
def linesplit(sock, maxline=0):
buf = sock.recv(16)
done = False
while not done:
# mid line check
if maxline and len(buf) > maxline:
yield buf, True
if "\n" in buf:
(line, buf) = buf.split("\n", 1)
err = maxline and len(line) > maxline
yield line+"\n", err
else:
more = sock.recv(16)
if not more:
done = True
else:
buf = buf+more
if buf:
err = maxline and len(buf) > maxline
yield buf, err
HOST = ''
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
for line, err in linesplit(conn, MAXLINE):
if err:
print "Error: Line greater than allowed length %d (got %d)" \
% (MAXLINE, len(line))
break
else:
print "Received data:", line.strip()
conn.close()
client.py
import socket
import time
import random
HOST = ''
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
while True:
val = 'x'*random.randint(1, 50)
if random.random() > .5:
val += "\n"
s.sendall(val)
time.sleep(.1)
s.close()
output
Connected by ('127.0.0.1', 57912)
Received data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Received data: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Received data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
...
Received data: xxxxxxxxxxx
Received data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Error: Line greater than allowed length 100 (got 102)
The server reads over the data it receives and constantly checks the length of the line once it assembles one. If at any time the line exceeds the amount specified, it returns an error code. I threw this together kind of fast so I am sure the checks could be cleaned up a bit more, and the read buffer amount can be changed to address how quickly you want to detect the long lines before consuming too much data. In the output example above, I only got 2 more bytes than is allowed, and it stopped.
The client just sends random length data, with a 50/50 change of a newline.
I'm writing a code used to send order to an avr. I send several information but between each write, I have to wait for an answer (I have to wait for the robot to reach a point on the coordinate system). As I read in the documentation, readline() should at least read until the timeout but as soon as I send the first coordinate, the readline() automatically return :
SerialException: device reports readiness to read but returned no data (device disconnected?)
When I put a sleep() between each write() in the for loop, everything works fine. I tried to use inWaiting() but it still does not work. Here is an example of how I used it:
for i in chemin_python:
self.serieInstance.ecrire("goto\n" + str(float(i.x)) + '\n' + str(float(-i.y)) + '\n')
while self.serieInstance.inWaiting():
pass
lu = self.serieInstance.readline()
lu = lu.split("\r\n")[0]
reponse = self.serieInstance.file_attente.get(lu)
if reponse != "FIN_GOTO":
log.logger.debug("Erreur asservissement (goto) : " + reponse)
Here an snipet how to use serial in python
s.write(command);
st = ''
initTime = time.time()
while True:
st += s.readline()
if timeout and (time.time() - initTime > t) : return TIMEOUT
if st != ERROR: return OK
else: return ERROR
This method allows you to separately control the timeout for gathering all the data for each line, and a different timeout for waiting on additional lines.
def serial_com(self, cmd):
'''Serial communications: send a command; get a response'''
# open serial port
try:
serial_port = serial.Serial(com_port, baudrate=115200, timeout=1)
except serial.SerialException as e:
print("could not open serial port '{}': {}".format(com_port, e))
# write to serial port
cmd += '\r'
serial_port.write(cmd.encode('utf-8'))
# read response from serial port
lines = []
while True:
line = serial_port.readline()
lines.append(line.decode('utf-8').rstrip())
# wait for new data after each line
timeout = time.time() + 0.1
while not serial_port.inWaiting() and timeout > time.time():
pass
if not serial_port.inWaiting():
break
#close the serial port
serial_port.close()
return lines