Socket module has a socket.recv_into method, so it can use user-defined bytebuffer (like bytearray) for zero-copy. But perhaps BaseEventLoop has no method like that. Is there a way to use method like socket.recv_into in asyncio?
The low-level socket operations defined for BaseEventLoop require a socket.socket object to be passed in, e.g. BaseEventLoop.sock_recv(sock, nbytes). So, given that you have a socket.socket, you could call sock.recv_into(). Whether it is a good idea to do that is another question.
You may implement own asyncio transport which utilizes .recv_into() function but yes, for now asyncio has not a way to use .recv_into() out-the-box.
Personally I doubt in very big speedup: when you develop with C the zero-copy is extremely important but for high-level languages like Python benefits are much lesser.
Update: Starting with Python 3.7.0, which is in alpha release as I write this, the standard library's asyncio module documents AbstractEventLoop.sock_recv_into().
Edit: expanding answer as requested...
A call to asyncio's sock_recv_into() typically looks like:
byte_count = await loop.sock_recv_into(sock, buff)
The buff is a mutable object that implements Python's buffer protocol, examples of which include a bytearray and a memoryview on a bytearray. The code below demonstrates receiving into a bytearray using a memoryview.
Working demo code for asyncio sockets necessarily includes a bunch of scaffolding to set up both sides of the connections and run the event loop. The point here is the use of asyncio's sock_recv_into() in the sock_read_exactly() co-routine below.
#!/usr/bin/env python3
"""Demo the asyncio module's sock_recv_into() facility."""
import sys
assert sys.version_info[:2] >= (3, 7), (
'asyncio sock_recv_into() new in Python 3.7')
import socket
import asyncio
def local_echo_server(port=0):
"""Trivial treaded echo server with sleep delay."""
import threading
import time
import random
ssock = socket.socket()
ssock.bind(('127.0.0.1', port))
_, port = ssock.getsockname()
ssock.listen(5)
def echo(csock):
while True:
data = csock.recv(8192)
if not data:
break
time.sleep(random.random())
csock.sendall(data)
csock.shutdown(1)
def serve():
while True:
csock, client_addr = ssock.accept()
tclient = threading.Thread(target=echo, args=(csock,), daemon=True)
tclient.start()
tserve = threading.Thread(target=serve, daemon=True)
tserve.start()
return port
N_COROS = 100
nrunning = 0
async def sock_read_exactly(sock, size, loop=None):
"Read and return size bytes from sock in event-loop loop."
if loop is None: loop = asyncio.get_event_loop()
bytebuff = bytearray(size)
sofar = 0
while sofar < size:
memview = memoryview(bytebuff)[sofar:]
nread = await loop.sock_recv_into(sock, memview)
print('socket', sock.getsockname(), 'read %d bytes' % nread)
if not nread:
raise RuntimeError('Unexpected socket shutdown.')
sofar += nread
return bytebuff
async def echo_client(port):
"Send random data to echo server and test that we get back the same."
from os import urandom
global nrunning
loop = asyncio.get_event_loop()
sock = socket.socket()
sock.setblocking(False)
await loop.sock_connect(sock, ('127.0.0.1', port))
for size in [1, 64, 1024, 55555]:
sending = urandom(size)
await loop.sock_sendall(sock, sending)
received = await sock_read_exactly(sock, size)
assert received == sending
nrunning -= 1
if not nrunning:
loop.stop()
if __name__ == '__main__':
port = local_echo_server()
print('port is', port)
loop = asyncio.get_event_loop()
for _ in range(N_COROS):
loop.create_task(echo_client(port))
nrunning += 1
print('Start loop.')
loop.run_forever()
Related
I try to connect two M5StickC to PC via BLE to upload their sensor data.
I wrote a data acquisition python script using bleak library.
The data acquisition rate is very slow when I connect 2 devices.
How to improve data acquisition rate on my script ?
I hope to get 20 data per second.
import asyncio
from bleak import BleakClient
address1 = "D8:A0:1D:55:EE:8A"
UUID1 = "beb5483e-36e1-4688-b7f5-ea07361b26a8"
address2 = "94:B9:7E:93:21:76"
UUID2 = "beb5483e-36e1-4688-b7f5-ea07361b26a2"
async def main():
client1 = BleakClient(address1)
client2 = BleakClient(address2)
print(client1.address)
print(client2.address)
await client1.connect()
# await client2.connect()
while(True): # very slow when comment out client2
print(await client1.read_gatt_char(UUID1))
# print(await client2.read_gatt_char(UUID2))
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
edit:
Thank you for your comments.
I have updated my script following ukBaz's link.
Data upload rate is much improved, but I could get almost only single M5StickC data, the data from another one available sparsely...
I will ask this question to bleak maintainer.
from bleak import BleakClient
import asyncio
address1 = "D8:A0:1D:55:EE:8A"
UUID1 = "beb5483e-36e1-4688-b7f5-ea07361b26a8"
address2 = "94:B9:7E:93:21:76"
UUID2 = "beb5483e-36e1-4688-b7f5-ea07361b26a2"
def callback(sender, data):
print(sender, data)
def run(addresses, UUIDs):
loop = asyncio.get_event_loop()
tasks = asyncio.gather(*(connect_to_device(address, UUID) for address, UUID in zip(addresses, UUIDs)))
loop.run_until_complete(tasks)
async def connect_to_device(address, UUID):
print("starting", address, "loop")
async with BleakClient(address, timeout=5.0) as client:
print("connect to", address)
while(True):
try:
print(await client.read_gatt_char(UUID))
except Exception as e:
print(e)
if __name__ == "__main__":
run([address1, address2], [UUID1, UUID2])
I have an FPGA that streams data on the USB bus through an FT2232H and I have observed that about 10% of the data has to be thrown away because some bytes in the frame are missing. Here are the technical details:
FPGA is an Artix 7. A batch of 4002 byte is ready every 9 ms. So that works out to 444,667 byte/s of data.
My laptop runs python 3.7 (from anaconda) on Ubuntu 18.04LTS
The FPGA/FT2232H is opened via the following initialization lines:
SYNCFF = 0x40
SIO_RTS_CTS_HS = (0x1 << 8)
self.device = pylibftdi.Device(mode='t', interface_select=pylibftdi.INTERFACE_A, encoding='latin1')
self.device.ftdi_fn.ftdi_set_bitmode(0xff, SYNCFF)
self.device.ftdi_fn.ftdi_read_data_set_chunksize(0x10000)
self.device.ftdi_fn.ftdi_write_data_set_chunksize(0x10000)
self.device.ftdi_fn.ftdi_setflowctrl(SIO_RTS_CTS_HS)
self.device.flush()
Then the data is read via this simple line:
raw_usb_data = my_fpga.device.read(0x10000)
I have observed the following:
I always get 0x10000 of data per batch, which is what I expect.
Reading 2**16 = 65,536 byte at once using device.read should take 147.4 ms given that a batch is ready every 9 ms. But timing that line gives a mean of 143 ms with a std deviation of 6.6 ms.
My first guess is that there is no buffer/a tiny buffer somewhere and that some information is lost because the OS (priority issue?) or python (garbage collection?) does something else at some point for too long.
How can I reduce the amount of bytes lost while reading the device?
The FT2232H has internal FIFO buffers with a capacity of ~4 kbits. Chances are that you are limited by them. Not sure how pylibftdi deals with them but maybe using an alternative approach might work if you can use the VCP driver. This allows you to address the FT2232H as standard comport e.g. via pyserial.
Some excerpts from one of my projects which actually works for baud rates >12 Mbps (UART is limited to 12 Mbps but e.g. fast opto can reach ~25 Mbps):
import traceback
import serial
import serial.tools.list_ports
import multiprocessing
import multiprocessing.connection
def IO_proc(cntr_pipe, data_pipe):
try:
search_str="USB VID:PID=0403:6010 SER="
ports = [x.device for x in serial.tools.list_ports.comports() if search_str in x.hwid]
baud_rate = 12000000 #only matters for uart and not for fast opto or fifo mode
ser = serial.Serial(port, baud_rate)
while not cntr_pipe.closed:
time.sleep(0)
in_data = ser.read(ser.inWaiting())
[...do some pattern matching, package identification etc...]
data_pipe.send_bytes(in_data)
except EOFError:
ret_code = 2
except Exception as e:
cntr_pipe.send(traceback.format_exc())
cntr_pipe.close()
ret_code = 4
finally:
cntr_pipe.close()
ser.close()
multiprocessing.connection.BUFSIZE = 2 ** 20 #only required for windows
child_cntr, parent_cntr = multiprocessing.Pipe()
child_data, parent_data = multiprocessing.Pipe()
process = multiprocessing.Process(target = IO_proc, args=(child_cntr, child_data))
#called frequently
def update():
if child_cntr.poll():
raise Exception("error",child_cntr.recv())
buf = bytes()
while parent_data.poll():
buf += parent_data.recv_bytes()
[...do something fancy...]
I tried to c&p a minimum example. It is untested so please forgive me if it is not working out of the box. To get this working one actually needs to make sure that the VCP and not the D2XX driver is loaded.
P.S: Actually while scanning through my files I realized that the pylibftdi way should work as well as I use a "decorator" class in case the D2XX driver is loaded:
try: import pylibftdi
except: pylibftdi = None
class pylibftdi_device:
def __init__(self,speed):
self.dev = pylibftdi.Device(interface_select=2)
self.dev.baudrate = speed
self.buf = b''
def write(self, data):
self.dev.write(data)
def read(self, bytecount):
while bytecount > len(self.buf):
self._read()
ret = self.buf[:bytecount]
self.buf = self.buf[bytecount:]
return ret
def flushInput(self):
self.dev.flush_input()#FT_PURGE_RX
self.buf = b''
def _read(self):
self.buf += self.dev.read(2048)
#property
def in_waiting(self):
self._read()
return len(self.buf)
def close(self):
self.dev.close()
def find_device_UART(baudrate=12000000,index=1, search_string="USB VID:PID=0403:6010 SER="):
if pylibftdi:
return pylibftdi_device(baudrate),"pylibftdi_device"
try:
ports = [x.device for x in serial.tools.list_ports.comports() if search_string in x.hwid]
module_logger.info(str(ports))
if len(ports) == 0:
return None,"no device found"
else:
ser = serial.Serial(ports[index],baudrate)
return ser,"found device %s %d"%(ser.name,ser.baudrate)
except serial.SerialException as e:
return None,"error during device detection - \n"+str(e)
So main difference to your example is that the recv buffer is read more frequently and put into a buffer which is then searched for the packets later on. And maybe this all is a complete overkill for your application and you just need to make smaller read calls to ensure the buffers never overflow.
For part 1 of the project, you will implement a simple go-back-N protocol similar to TCP. This protocol is called the 352RDPv1.
(unfortunately my python knowledge is not that strong, I am being forced to code it in python)
I must implement: init(udp_port1, udpport2)
socket()
connect(address)
I am given the following pseudo code:
import binascii
import socket as syssock
import struct
import sys
# this init function is global to the class and
# defines the UDP ports all messages are sent
# and received from.
def init(UDPportTx,UDPportRx): # initialize your UDP socket here
# create a UDP/datagram socket
# bind the port to the Rx (receive) port number
pass
class socket:
def __init__(self): # fill in your code here
# create any lists/arrays/hashes you need
return
def connect(self,address): # fill in your code here
global UDPportTx # example using a variable global to the Python module
# create a new sequence number
# create a new packet header with the SYN bit set in the flags (use the Struct.pack method)
# also set the other fields (e.g sequence #)
# add the packet to the outbound queue
# set the timeout
# wait for the return SYN
# if there was a timeout, retransmit the SYN packet
# set the outbound and inbound sequence numbers
return
I have given a shot so far at a few methods but I know I have errors and my program does not work.
import binascii
import socket as syssock
import struct
from collections import namedtuple
import sys
import select
version = 0x1
header_len = 7
payload_len = 0
flags = 0
SOCK352_SYN = 0x01
SOCK352_FIN = 0x02
SOCK352_ACK = 0x04
SOCK352_RESET = 0x08
SOCK352_HAS_OPT = 0xA0
sequence_no = 0
ack_no = 0
timeout = 0.2
#given these values to set them to
def init(UDPportTx,UDPportRx):
global sock
global useRx
global useTx
useTx = int(UDPportTx)
useRx = int(UDPportRx)
sock = syssock.syssock(syssock.AF_INET, syssock.SOCK_DGRAM)
sock.bind(('', useRx))
print "Listening on port :", useRx
pass
class socket:
def __init__(self):
global pkt
pkt = namedtuple("pkt",["version", "flags", "sequence_no", "ack_no", "payload_len"])
return
def connect(self, address):
global header_raw
udpPkt = struct.Struct('!BLBBB')
header_raw = udpPkt.pack(version, SOCK352_SYN, sequence_no, ack_no, payload_len)
sock.sendto(header_raw, ('', useTx))
return
I believe I am having errors in these first few methods and before I move onto the others I need to figure in I want to see if anyone is able to help me understand how to handle these few to begin.
I am trying to write some simple loops to control objects in Pygazebo, but alas it only ever calls the method once and then the loops appears to block.
# -*- coding: utf-8 -*-
"""
Created on Thu Jul 2 12:52:50 2015
#author: skylion
"""
import trollius #NOTE: Trollius requires protobuffer from Google
from trollius import From
import pygazebo
import pygazebo.msg.joint_cmd_pb2
import time
def apply_joint_force(world_name, robot_name, joint_name, force, duration=-1):
#trollius.coroutine
def joint_force_loop():
manager = yield From(pygazebo.connect())
print("connected")
publisher = yield From(
manager.advertise('/gazebo/' + world_name + '/' + robot_name + '/joint_cmd',
'gazebo.msgs.JointCmd'))
message = pygazebo.msg.joint_cmd_pb2.JointCmd()
message.name = robot_name + '::' + joint_name #format should be: name_of_robot + '::name_of_joint'
message.force = force
#t_end = time.time() + duration # The time that you want the controller to stop
while True: #time.time() < t_end or duration == -1:
try:
yield From(publisher.publish(message))
yield From(trollius.sleep(1.0))
except:
pass
#Nothing
print("Connection closed")
wait_net_service('localhost',11345)
loop = trollius.new_event_loop()
loop.run_until_complete(joint_force_loop())
raise
def wait_net_service(server, port, timeout=None):
""" Wait for network service to appear
#param timeout: in seconds, if None or 0 wait forever
#return: True of False, if timeout is None may return only True or
throw unhandled network exception
"""
import socket
import errno
s = socket.socket()
if timeout:
from time import time as now
# time module is needed to calc timeout shared between two exceptions
end = now() + timeout
while True:
try:
if timeout:
next_timeout = end - now()
if next_timeout < 0:
return False
else:
s.settimeout(next_timeout)
s.connect((server, port))
time.sleep(1)
except socket.timeout, err:
# this exception occurs only if timeout is set
if timeout:
return False
except socket.error, err:
# catch timeout exception from underlying network library
# this one is different from socket.timeout
if type(err.args) != tuple or (err[0] != errno.ETIMEDOUT and err[0] != errno.ECONNREFUSED):
raise err
else:
s.close()
return True
I thought #coroutines were suppose to be wrapped asynchronously? Do I just misunderstand the use this code? Or am I doing something else wrong? This is my first time with concurrency in Python btw.
Also this is how I am calling that function:
counter = 0
for joint_def in self.all_joint_props:
print("each joint_def")
apply_joint_force(world_name, robot_name, "hingejoint" + str(counter), joint_def[2])
#print("Appliing joint force")
Any idea why it keep blocking the thread? Should I be using a different method to this? Any help would be appreciated
So, the answer is quite simple really. You have to queue up the multiple Trollius.Tasks you want to run as a list before starting the object and combine that with Trollius.wait() to achieve this. To ensure the thread is non-blocking you then use the following method
Here is my code so far:
tasks = []
for joint_name in joint_names:
tasks.append(trollius.Task(joint_force_loop(world_name, robot_name, joint_name, force, duration))
loop = trollius.get_event_loop()
loop.run_until_complete(trollius.wait(tasks))
I need to built a python chat and I'm stacked in the very final step. I've built the server and the client and I have the following problem while running the code:
server.py 127.0.0.1
-in a separate window client.py 127.0.0.1
-another client
-type the nicknames to chat for both clients and get the correct answer 'yuppie' meaning you are connected
a client try to speak
message is not read by the other client until it doesn't print something, after printing it get the message printed on its screen correctly.
I'd like to get the message without being obliged to print something, it's pretty unrealistic!!! Code of client and server are below in 2 different classes. Thank you!
#! /usr/bin/env python
import socket,sys,select,re
PORT=1060
class Server():
def __init__(self,host):
#building listen_sock
self.listen_sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.listen_sock.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
self.listen_sock.bind((host,PORT))
self.listen_sock.listen(20)
#building dict for socket and socket state
self.sockets={self.listen_sock.fileno(): self.listen_sock}
self.socket_state={self.listen_sock.fileno():''}
#building poll object
self.poll=select.poll()
self.poll.register(self.listen_sock,select.POLLIN)
#users' list
self.users_list={}
#DON'T LOOK HERE
#initialize the sender
#self.sender=0
# self.users=re.compile("\s*\$(get users connected)$\s*",re.IGNORECASE)
# self.nick=re.compile("\s*\$\$(\w*\d*)\$\$\s*",re.IGNORECASE)
# self.quit=re.compile("\s*\$(quit)\$\s*",re.IGNORECASE)
#self.commands=[self.users,self.nick,self.quit]
#funcion to receive message from client (work well)
def recv_until(self,fd,suffix):
self.message=''
#checking the end of the message
while not self.message.endswith(suffix):
data=self.sockets[fd].recv(16)
if not data:
raise EOFError('socket closed before we saw %r' % suffix)
self.message+=data
self.message=self.message[:-1]
#delete client (work well)
def del_client(self,fd):
del self.users_list[fd]
del self.socket_state[fd]
self.poll.unregister(fd)
#print the remaining active connections
if not len(self.users_list):
print 'Anyone is connected, waiting for new connection'
else:
print self.users_list
#add new client and change the of the file descriptor for that client (work well)
def new_client(self,fd):
newsock, sockname = self.listen_sock.accept()
print 'new connection from ', newsock.getpeername()
newsock.setblocking(False)
#recording the new connection
fd=newsock.fileno()
self.sockets[fd]=newsock
self.poll.register(fd,select.POLLOUT)
self.socket_state[fd]='ask nick'
#DON'T LOOK HERE
# def handle_query(self,fd):
# for n,command in enumerate(self.commands):
# match=command.search(self.message)
# if n==1 and match:
# self.users_list[self.sockets[fd].getpeername()]=match.group(1)
# print self.users_list
# for value in self.users_list.values():
# self.sockets[fd].sendall(value+'\n')
#starting the main function of the class
def chat(self):
while True:
#here il where the code hangs up waitng and waiting (WORKS BAD)
#return a tuple, identify where (fd) the event (event) is happening
for fd,event in self.poll.poll():
#print the state of each socket and the poll object
print self.socket_state
print self.poll.poll()
#starting the state machine
#remove closed sockets
if event & (select.POLLHUP | select.POLLERR |
select.POLLNVAL):
#deleting the socket closed at fd
self.del_client(fd)
#if the socket referred to is our listen_sock and we have a new connection request
elif self.sockets[fd] is self.listen_sock:
#recording the new entry!
self.new_client(fd)
#managing all the situation where it is necessary to answer to a client
#and changing the state of the socket and that of the sockets[fd]
elif event & select.POLLOUT:
if self.socket_state[fd]=='ask nick':
self.sockets[fd].sendall('identify\n')
self.poll.modify(self.sockets[fd],select.POLLIN)
self.socket_state[fd]='get user'
if self.socket_state[fd]=='invalid nick':
self.sockets[fd].sendall('invalid nick\n')
for value in self.users_list.values():
self.sockets[fd].sendall('\n'+value+'\n')
self.socket_state[fd]='ask nick'
if self.socket_state[fd]=='connected':
print '3'
self.sockets[fd].sendall('yuppie\n')
self.poll.modify(self.sockets[fd],select.POLLIN)
self.socket_state[fd]='ready to communicate'
if self.socket_state[fd]=='ready to receive':
self.sockets[fd].sendall(self.message)
print '4'
self.poll.modify(self.sockets[fd],select.POLLIN)
self.socket_state[fd]='ready to communicate'
#managing all the situation where it is necessary to get values from clients
elif event & select.POLLIN:
if self.socket_state[fd]=='get user':
self.recv_until(fd,'\n')
if self.message not in self.users_list.values():
self.users_list[fd]=self.message
self.poll.modify(self.sockets[fd],select.POLLOUT)
self.socket_state[fd]='connected'
else:
self.poll.modify(self.sockets[fd],select.POLLOUT)
self.socket_state[fd]='invalid nick'
if self.socket_state[fd]=='ready to communicate':
self.recv_until(fd,'\n')
print '5'
for i in self.users_list.keys():
if i!=fd:
self.poll.modify(self.sockets[i],select.POLLOUT)
self.socket_state[i]='ready to receive'
if __name__ == '__main__':
se=Server(sys.argv[1])
se.chat()
#! /usr/bin/env python
import sys,socket,select,threading,time
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
HOST=sys.argv.pop()
PORT=1060
class Client():
def setup(self):
server_address=(HOST,PORT)
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect(server_address)
def chat(self):
while True:
time.sleep(1)
text=raw_input('>>> ')
self.sock.sendall(text+'\n')
def rec(self):
while True:
mess=self.sock.recv(16)
if mess:
print '$$$ ', mess,
def start(self):
l=threading.Thread(target=self.rec)
t=threading.Thread(target=self.chat)
t.start()
l.start()
if __name__=='__main__':
cl=Client()
cl.setup()
cl.start()
Next time take a look at http://www.zeromq.org/, it has a nice python binding http://zeromq.github.com/pyzmq/. It's perfect for this kind of stuff.