I have inherited python/twisted code written by a former employee.
The code that I have (and it works) opens a serial port and receives data for 5 seconds, then writes it back in reverse order. Here is the code:
from twisted.internet import reactor
from twisted.internet.protocol import Protocol
from twisted.internet.serialport import SerialPort
import serial
class ReverseEchoProtocol(Protocol):
"""Wait for specific amount of data.
Regardless of success, closes connection timeout seconds after opening.
"""
def __init__(self, port, timeout, logger):
self._logger = logger
self._timeout = timeout
def connectionMade(self):
self._logger.info('RS485 connection made.')
reactor.callLater(self._timeout, self.transport.loseConnection, 'timeout')
def connectionLost(self, reason):
self._logger.info('RS485 connection lost. ' + str(reason))
def dataReceived(self, data):
self._logger.info('RS485 received data. ' + repr(data))
self.transport.write(data[::-1])
self.transport.flushOutput()
And from inside a python function the above code is initiated with this call:
protocol = ReverseEchoProtocol(port, 5 self._logger)
try:
port.open(protocol)
except serial.SerialException as err:
# print/log err.message here
return False
return True
This call to port.open returns immediately after successfully opening the port (well before the 5 seconds complete)
Here is what I am trying to write. From inside a python function, I need to initiate a serial transaction. It needs to wait for the transaction to either complete, fail or timeout.
Here is what the serial transaction needs to do:
The transaction is passed in a string and a timeout value.
The serial port is opened. Failure to open results in an error being returned
The string is written to the serial port. Failure to write results in an error being returned
If write is successful, the same port is then continually read for "timeout" seconds. As data is read (could be multiple reads), it is appended to a string.
After "timeout" seconds, the string of all data read from the port during that time is returned (or the empty string if nothing is read).
Here is my question....trying to adapt the code I already have, I can write a new protocol. In connectionMade, it can do the write, initiate the read and then setup a timeout by calling reactor.callLater. Inside dataReceived I can append the read data to a string. And inside connectionLost I can return the string read.
But how do I make the python function calling port.open wait until the transaction completes? (Is there something like a reactor.wait function or a join function?) Also, if there is an error (or exception), how do I pass that up (a try block?) How do I pass the string back up to the python function?
I think the code I inherited gets me close...I just need those couple of questions answered for me to be able to complete the task.
To answer your first two questions, you are looking for reactor.run() to run the twisted mainloop but it sounds like and looks like you are expecting a blocking api and twisted is event driven which could possibly mean you are forcing the use of twisted. You could just use the serial module directly without twisted to get what you want done. If you do want to be event driven to be non-blocking then you will have to ask more specific questions about that.
Related
I'm having an issue with Python's socket module that I haven't been able to find anywhere else.
I'm building a simple TCP chat client, and while it successfully connects to the server initially, the script hangs endlessly on sock.recv() despite the fact that I explicitly set a timeout length.
I've tried using different timeout values and including setblocking(False) but no matter what I do it keeps acting like the socket is in blocking mode.
Here are the relevant parts of my code:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(1)
def listen_to_server():
global connected
while connected:
ready_to_read, ready_to_write, in_error = select.select([sock], [], [])
if ready_to_read:
try:
data = sock.recv(1024)
except socket.timeout:
print('TIMEOUT')
if not data:
append_to_log('Disconnected from server.\n')
connected = False
else:
append_to_log(str(data))
Any suggestions would be helpful, I'm at a total loss here.
You've mixed two things the socket timeout and the select.
When you set socket timeout then you are telling to that socket: if I try do some operation (e.g. recv()) and it won't be finished until my limit then raise timeout exception.
The select takes file descriptors (on Windows only sockets) and start checking if the rlist (the first parameter) contains any socket ready to read (aka some data have arrived). If any data arrived then the program continues.
Now your code do this:
Set timeout for socket operations
Select start waiting for data (if you don't send them then they never arrives)
and that's it. You stuck at the select.
You should just call the recv() without select. Than your timeout should be applied.
If you need manage multiple sockets at once than you have to use select and set the 4th parameter timeout.
I'm trying to use hardware serial port devices with Python, but I'm having timing issues. If I send an interrogation command to the device, it should respond with data. If I try to read the incoming data too quickly, it receives nothing.
import serial
device = serial.Serial("/dev/ttyUSB0", 9600, timeout=0)
device.flushInput()
device.write("command")
response = device.readline()
print response
''
The readline() command isn't blocking and waiting for a new line as it should. Is there a simple workaround?
readline() uses the same timeout value you passed to serial.Serial().
If you want readline to be blocking, just delete the timeout argument, the default value is None.
You could also set it to None before calling readline(), if you want to have a timeout for openening the device:
import serial
try:
device = serial.Serial("/dev/ttyUSB0", 9600, timeout=0.5)
except:
#Exception handeling
device.flushInput()
device.write("command")
device.timeout=None
response = device.readline()
print response
I couldn't add a commend so I will just add this as an answer. You can reference this stackoverflow thread. Someone attempted something similar to your question.
Seems they put their data reading in a loop and continuously looped over it while data came in. You have to ask yourself one thing if you will take this approach, when will you stop collecting data and jump out of the loop? You can try and continue to read data, when you are already collecting, if nothing has come in for a few milliseconds, jump out and take that data and do what you want with it.
You can also try something like:
While True:
serial.flushInput()
serial.write(command)
incommingBYTES = serial.inWaiting()
serial.read(incommingBYTES)
#rest of the code down here
Im new to twisted. I have written a client which connects to a server on two ports 8037 and 8038. I understand that the factory creates two connection objects. Now when i press Ctrl-C, it says
Connection Lost Connection to the other side was lost in a non clean fashion.
Connection Lost Connection to the other side was lost in a non clean fashion.
Below is the code:
from twisted.internet import protocol,reactor
class TestClient(protocol.Protocol):
def __init__(self):
pass
def connectionMade(self):
print "Connected "
self.sayHello()
def connectionLost(self,reason):
self.transport.loseConnection()
def sayHello(self):
self.transport.write("Hello")
def dataReceived(self,data):
print "Received data ",data
class TestClientFactory(protocol.ClientFactory):
def buildProtocol(self,addr):
return TestClient()
def clientConnectionFailed(self,connectory,reason):
print "Connection Failed ",reason.getErrorMessage()
def clientConnectionLost(self,connector,reason):
print "Connection Lost ",reason.getErrorMessage()
reactor.connectTCP("<server_ip>",8037,TestClientFactory())
reactor.connectTCP("<server_ip>",8038,TestClientFactory())
reactor.run()
How can i make the client close both tcp connections cleanly ?.
How to call the sayHello() method for only one connection ?
Im new to twisted, so an example would be helpful.
Thanks
When you are connected, if you want to call sayHello, you can use the thought of rpc.
For example, you send a message like 'sayHello_args', parse msg and call sayhello by args.
If you don't want to send any msg. When you connected, d.addCallback(sayHello) to call.
d = defer.succeed(0)
d.addCallback(lambda _ : self.sayHello())
And if you want to close connection, to use reactor.stop()
Unclean connection shutdown is really nothing to worry about. Getting a clean exit would potentially make your shutdown process slower and buggier because it requires a bunch of additional code, and you have to be able to deal with abnormal network connection termination no matter what. In fact calling it "clean" is maybe even a bit misleading: "simultaneously confirmed" might be closer to what it's actually telling you about how the connection was closed.
As far as how to call sayHello, I don't fully understand your question, but if you use AMP, calling a method on the opposite side of the connection is pretty easy.
I coded a basic socket system with "select". I want get the list of connected clients instantly.
When the timeout of "select" has passed and several clients come after, it's the drama..
Example - Concerns:
I have 3 clients with one that connects before the timeout, 2 others are connected after the timeout, so I'm going to refresh my list if it took into account two other clients after the timeout.
1st result: I display my variable "list", I see the first socket that is connected before the timeout + one of the other socket who is connected after the timeout. Total: 2 of 3 clients
2nd result: I still re-display my variable "list", and the three clients are there ....
But I want the list without having to re-display the list every time for every customer you can imagine I have 10 clients and I have to show my liste10 times
So I thought to use the asyncore module who is more fluid, what do you think? Do you have a solution for me (easier)? Should I use the multi-threading or stayed on asyncore or select module?
EDIT CODE SOURCE:
import socket, select
hote = ''
port = 81
mainConnection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mainConnection.bind((hote, port))
mainConnection.listen(5)
print("Listen to {}".format(port))
client_online = []
while True:
connection_access, wlist, xlist = select.select([mainConnection], [], [], 10)
for connexion in connection_access:
connection_client, infos_connexion = connexion.accept()
client_online.append(connection_client)
refresh = input(">>> ")
while True:
try:
refresh = int(refresh)
except ValueError:
print("Not allowed")
refresh = int(refresh)
else:
break
if refresh == 1:
print("List client : {}".format(client_online))
There are three major problems with your code:
You call input in your loop. This function will block until ENTER is pressed.
If a non-integer is input from the console, you will get an exception. You handle that exception, but you handle it wrongly. Instead or asking for input again, you simply try to perform the same operation that caused the exception again.
You only check for incoming connection in your select call. You never check if any of the connected sockets have sent anything.
The major problem here for you is the call to input as it will completely stop your program until input from the console is entered.
Your post is very unclear but I can tell you that the problem is that you aren't understanding how to use select.
The code you posted only calls select one time. The program gets to the select() call and waits for mainConnection to be readable (or for the timeout). If mainConnection becomes readable before the timeout, select() returns with one readable file descriptor which you then process in your for loop. But that's it. select is never called again and so your program never checks for any more incoming connections.
In almost every application select should be in a loop. Each time through the loop the program waits in the select() call until one or more sockets is ready for reading or writing. When that happens, select gives you the file descriptors that are ready and it's your job to have other code actually do something. For example, if select returns a socket's file descriptor as readable it's your job to call .recv() on that socket.
You can certainly use asyncore. In fact, I think you should study the source code for asyncore to learn how to properly use select.
I have a client that connects to an HTTP stream and logs the text data it consumes.
I send the streaming server an HTTP GET request... The server replies and continuously publishes data... It will either publish text or send a ping (text) message regularly... and will never close the connection.
I need to read and log the data it consumes in a non-blocking manner.
I am doing something like this:
import urllib2
req = urllib2.urlopen(url)
for dat in req:
with open('out.txt', 'a') as f:
f.write(dat)
My questions are:
will this ever block when the stream is continuous?
how much data is read in each chunk and can it be specified/tuned?
is this the best way to read/log an http stream?
Hey, that's three questions in one! ;-)
It could block sometimes - even if your server is generating data quite quickly, network bottlenecks could in theory cause your reads to block.
Reading the URL data using "for dat in req" will mean reading a line at a time - not really useful if you're reading binary data such as an image. You get better control if you use
chunk = req.read(size)
which can of course block.
Whether it's the best way depends on specifics not available in your question. For example, if you need to run with no blocking calls whatever, you'll need to consider a framework like Twisted. If you don't want blocking to hold you up and don't want to use Twisted (which is a whole new paradigm compared to the blocking way of doing things), then you can spin up a thread to do the reading and writing to file, while your main thread goes on its merry way:
def func(req):
#code the read from URL stream and write to file here
...
t = threading.Thread(target=func)
t.start() # will execute func in a separate thread
...
t.join() # will wait for spawned thread to die
Obviously, I've omitted error checking/exception handling etc. but hopefully it's enough to give you the picture.
You're using too high-level an interface to have good control about such issues as blocking and buffering block sizes. If you're not willing to go all the way to an async interface (in which case twisted, already suggested, is hard to beat!), why not httplib, which is after all in the standard library? HTTPResponse instance .read(amount) method is more likely to block for no longer than needed to read amount bytes, than the similar method on the object returned by urlopen (although admittedly there are no documented specs about that on either module, hmmm...).
Another option is to use the socket module directly. Establish a connection, send the HTTP request, set the socket to non-blocking mode, and then read the data with socket.recv() handling 'Resource temporarily unavailable' exceptions (which means that there is nothing to read). A very rough example is this:
import socket, time
BUFSIZE = 1024
s = socket.socket()
s.connect(('localhost', 1234))
s.send('GET /path HTTP/1.0\n\n')
s.setblocking(False)
running = True
while running:
try:
print "Attempting to read from socket..."
while True:
data = s.recv(BUFSIZE)
if len(data) == 0: # remote end closed
print "Remote end closed"
running = False
break
print "Received %d bytes: %r" % (len(data), data)
except socket.error, e:
if e[0] != 11: # Resource temporarily unavailable
print e
raise
# perform other program tasks
print "Sleeping..."
time.sleep(1)
However, urllib.urlopen() has some benefits if the web server redirects, you need URL based basic authentication etc. You could make use of the select module which will tell you when there is data to read.
Yes when you catch up with the server it will block until the server produces more data
Each dat will be one line including the newline on the end
twisted is a good option
I would swap the with and for around in your example, do you really want to open and close the file for every line that arrives?