Application: Python 2.7 + Pymodbus 1.4.0 (asynchronous client)
Problem: I want to handle a situation when user wants to read from the wrong address (Illegal Address) in asynchronous version, but my solution does not work.
Simplified code - asynchronous version:
#!/usr/bin/env python
from __future__ import print_function
from twisted.internet import reactor, protocol
from pymodbus.client.async import ModbusClientProtocol
from twisted.internet import serialport, reactor
def print_result(response, address):
try:
print("Response: ", response.registers[0])
except:
print("Response: ", response)
def start_measurements(client):
# 16408, 16409 addresses are ok
# 9015 is an Illegal Address
addresses = [16408, 9015, 16409]
for address in addresses:
rr = client.read_holding_registers(address, 1, unit=255)
# setting Callback/Errback function with an address argument
rr.addBoth(print_result, address)
reactor.callLater(10, client.transport.loseConnection)
reactor.callLater(11, reactor.stop)
def handler_tcp_connection_error(client):
print("Error while connecting")
reactor.callLater(0, reactor.stop)
if __name__ == "__main__":
defer = protocol.ClientCreator(reactor, ModbusClientProtocol).connectTCP("192.168.168.100", 502)
defer.addCallbacks(start_measurements, handler_tcp_connection_error)
reactor.run()
When I read from these registers in synchronous version I just don't receive a value from wrong address, but further registers are read anyway.
In asynchronous version (pasted above) when Illegal Address is read, I've got no response even from further registers and the output is:
Response: 15
Response: Exception Response(131, 3, IllegalAddress)
*waiting about 10s with no response*
Response: [Failure instance: Traceback (failure with no frames): <class 'pymodbus.exceptions.ConnectionException'>: Modbus Error: [Connection] Connection lost during request]
I'd like to get all correct values (not from Illegal Addresses) like in the synchronous version:
Response: 15
Exception Response(131, 3, IllegalAddress)
Response 7
How can I fix this, to skip IllegalAddreses and get values from all correct addreses?
Related
The code below is a simplified version of a Tornado based TCP server that is currently used to host a Videotex system. This code was derived from the Tornado documentation and the server has been running in a live environment for some time without issue, however, there is a feature I need to add.
The system currently blocks until a character is received from the client before returning the data via the stream.write. As the system typically runs at 1200 baud at the client end (via a telnet modem), this means that the user has to wait until all stream writes have completed before the next 'user entered' character is processed.
What I would like to do is find a way that would allow me to abandon writing data to stream.write if another character is received form the client.
I am new to Tornado and fairly new to Python, however, I have coded asynchronous functions and threaded solutions in the past using C#.
From the documentation the stream.write operation is asynchronous, I am assuming therefore that the call may return before the data is completely written, I am left thinking that I need a method to abandon/empty/advance the write buffer to stop the write operation if a new char is detected on the stream.read.
One option that would seem to give me what I need is to somehow perform the stream.writes on another thread , however, this approach seems inappropriate when using Tornado's IOLoop etc.
Is there a way to give me the facility I am after? I have full control of the code and am happy to restructure the app if needed.
import logging
import struct
import os
import traceback
from tornado import gen
from tornado.ioloop import IOLoop
from tornado.iostream import StreamClosedError
from tornado.tcpserver import TCPServer
# Configure logging.
logger = logging.getLogger(os.path.basename(__file__))
logger.setLevel(logging.INFO)
# Cache this struct definition; important optimization.
int_struct = struct.Struct("<i")
_UNPACK_INT = int_struct.unpack
_PACK_INT = int_struct.pack
class TornadoServer(TCPServer):
def start(self, port):
self.port = port
server.listen(port)
#gen.coroutine
def handle_stream(self, stream, address):
logging.info("[viewdata] Connection from client address {0}.".format(address))
try:
while True:
char = yield stream.read_bytes(1) # this call blocks
asc = ord(char)
logger.info('[viewdata] Byte Received {0} ({1})'.format(hex(asc), asc))
# Do some processing using the received char and return the appropriate page of data
stream.write('This is the data you asked for...'.encode())
except StreamClosedError as ex:
logger.info("[viewdata] {0} Disconnected: {1} Message: {2}".format(address, type(ex), str(ex)))
except Exception as ex:
logger.error("[viewdata] {0} Exception: {1} Message: {2}".format(address, type(ex), str(ex)))
logger.error(traceback.format_exc())
if __name__ == '__main__':
server = TornadoServer()
server.start(25232)
loop = IOLoop.current()
loop.start()
The main idea is that you move long processing into separate task.
When you receive some new data, you choose what to do (in case below I cancel current operation)
import logging
import os
import traceback
import threading
from tornado import gen
from tornado.ioloop import IOLoop
from tornado.iostream import StreamClosedError
from tornado.tcpserver import TCPServer
# Configure logging.
logger = logging.getLogger(os.path.basename(__file__))
logger.setLevel(logging.INFO)
class TornadoServer(TCPServer):
def start(self, port):
self.port = port
server.listen(port)
async def process_stream(self, stream, char, cancel_event):
asc = ord(char)
logger.info('[viewdata] Byte Received {0} ({1})'.format(hex(asc), asc))
N = 5
for i in range(N):
if cancel_event.is_set():
logger.info('[viewdata] Abort streaming')
break
# Do some processing using the received char and return the appropriate page of data
msg = 'This is the {0} data you asked for...'.format(i)
logger.info(msg)
await stream.write('This is the part {0} of {1} you asked for...'.format(i, N).encode())
await gen.sleep(1.0) # make this processing longer..
async def handle_stream(self, stream, address):
process_stream_future = None
cancel_event = None
logging.info("[viewdata] Connection from client address {0}.".format(address))
while True:
try:
char = await stream.read_bytes(1) # this call blocks
# when received client input, cancel running job
if process_stream_future:
process_stream_future.cancel()
if cancel_event:
cancel_event.set()
cancel_event = threading.Event()
process_stream_future = gen.convert_yielded(
self.process_stream(stream, char, cancel_event))
self.io_loop.add_future(process_stream_future, lambda f: f.result())
except StreamClosedError as ex:
logger.info("[viewdata] {0} Disconnected: {1} Message: {2}".format(address, type(ex), str(ex)))
except Exception as ex:
logger.error("[viewdata] {0} Exception: {1} Message: {2}".format(address, type(ex), str(ex)))
logger.error(traceback.format_exc())
if __name__ == '__main__':
server = TornadoServer()
server.listen(25232)
loop = IOLoop.current()
loop.start()
I am trying to make use of the recently introduced twisted.application.internet.ClientService class in a twisted application that does simple modbus-tcp polling using pymodbus. I feel my issues have nothing to do with the modbus Protocol that I am using, as I have created quite a few other working prototypes using the lower level twisted APIs; but this new ClientService looks like it fits my needs exactly, thus should reduce my code footprint and keep it neat if I can get it to work.
My tests show the ClientService handles reconnections just as it is expected to and I have easy access to the first connections Protocol. The problem that I am having is getting hold of subsequent Protocol objects for the reconnections. Here is a simplified version of the code I am having the issue with:
from twisted.application import internet, service
from twisted.internet.protocol import ClientFactory
from twisted.internet import reactor, endpoints
from pymodbus.client.async import ModbusClientProtocol
class ModbusPollingService(internet.ClientService):
def __init__(self, addrstr, numregs=5):
self.numregs=numregs
internet.ClientService.__init__(self,
endpoints.clientFromString(reactor, addrstr),
ClientFactory.forProtocol(ModbusClientProtocol))
def startService(self):
internet.ClientService.startService(self)
self._pollWhenConnected()
def _pollWhenConnected(self):
d = self.whenConnected()
d.addCallback(self._connected)
d.addErrback(self._connfail)
def _connected(self, p):
self._log.debug("connected: {p}", p=p)
self._mbp = p
self._poll()
return True
def _connfail(self, failstat):
self._log.failure('connection failure', failure=failstat)
self._mbp = None
self._pollWhenConnected()
def _poll(self):
self._log.debug("poll: {n}", n=self.numregs)
d = self._mbp.read_holding_registers(0, self.numregs)
d.addCallback(self._regs)
d.addErrback(self._connfail)
def _regs(self, res):
self._log.debug("regs: {r}", r=res.registers)
# Do real work of dealing storing registers here
reactor.callLater(1, self._poll)
return res
application = service.Application("ModBus Polling Test")
mbpollsvc = ModbusPollingService('tcp:127.0.0.1:502')
mbpollsvc.setServiceParent(application)
When the connection fails (for whatever reason) the errback of the deferred returned from read_holding_registers() gets called with the intention that my service can abandon that Protocol and go back into a state of waiting for a new connections Protocol to be returned by the whenConnected() callback... however what seems to be happening is that the ClientService does not yet realise the connection is dead and returns me the same disconnected Protocol, giving me a log full of:
2016-05-05 17:28:25-0400 [-] connected: <pymodbus.client.async.ModbusClientProtocol object at 0x000000000227b558>
2016-05-05 17:28:25-0400 [-] poll: 5
2016-05-05 17:28:25-0400 [-] connection failure
Traceback (most recent call last):
Failure: pymodbus.exceptions.ConnectionException: Modbus Error: [Connection] Client is not connected
2016-05-05 17:28:25-0400 [-] connected: <pymodbus.client.async.ModbusClientProtocol object at 0x000000000227b558>
2016-05-05 17:28:25-0400 [-] poll: 5
2016-05-05 17:28:25-0400 [-] connection failure
Traceback (most recent call last):
Failure: pymodbus.exceptions.ConnectionException: Modbus Error: [Connection] Client is not connected
or very similar, note repeated ModbusClientProtocol object address.
I'm pretty sure that I've probably just made a poor choice of pattern for this API, but I've iterated through a few different possibilities such as creating my own Protocol and Factory based on ModbusClientProtocol and handling the polling mechanism entirely within that class; but it felt a bit messy passing the persistent config and mechanism to store the polled data that way, it seems like handling this at or above the ClientService level is a cleaner approach but I can't work out the best way of keeping track of the currently connected Protocol. I guess what I'm really looking for is a best practice recommendation for usage of the ClientService class in extended polling situations.
This is an old question. But, hopefully, it will help somebody else.
The problem that I am having is getting hold of subsequent Protocol objects for the reconnections.
Supply prepareConnection callable to ClientService constructor. It will supply current connection.
In the example below MyService attaches itself to MyFactory. The main reason for this is so that MyFactory can let MyService know when ClientService disconnected. It's possible because ClientService calls Factory.stopFactory on disconnect.
Next time ClientService reconnects it will call its prepareConnection supplying current protocol instance.
(Reconnecting) ClientService:
# clientservice.py
# twistd -y clientservice.py
from twisted.application import service, internet
from twisted.internet.protocol import Factory
from twisted.internet import endpoints, reactor
from twisted.protocols import basic
from twisted.logger import Logger
class MyProtocol(basic.Int16StringReceiver):
_log = Logger()
def stringReceived(self, data):
self._log.info('Received data from {peer}, data={data}',
peer=self.transport.getPeer(),
data=data)
class MyFactory(Factory):
_log = Logger()
protocol = MyProtocol
def stopFactory(self):
# Let service know that its current connection is stale
self.service.on_connection_lost()
class MyService(internet.ClientService):
def __init__(self, endpoint, factory):
internet.ClientService.__init__(self,
endpoint,
factory,
prepareConnection=self.on_prepare_connection)
factory.service = self # Attach this service to factory
self.connection = None # Future protocol instance
def on_prepare_connection(self, connection):
self.connection = connection # Attach protocol to service
self._log.info('Connected to {peer}',
peer=self.connection.transport.getPeer())
self.send_message('Hello from prepare connection!')
def on_connection_lost(self):
if self.connection is None:
return
self._log.info('Disconnected from {peer}',
peer=self.connection.transport.getPeer())
self.connection = None
def send_message(self, message):
if self.connection is None:
raise Exception('Service is not available')
self.connection.sendString(bytes(message, 'utf-8'))
application = service.Application('MyApplication')
my_endpoint = endpoints.clientFromString(reactor, 'tcp:localhost:22222')
my_factory = MyFactory()
my_service = MyService(my_endpoint, my_factory)
my_service.setServiceParent(application)
Slightly modified echo server from twisted examples:
#!/usr/bin/env python
# echoserv.py
# python echoserv.py
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
from twisted.protocols import basic
### Protocol Implementation
# This is just about the simplest possible protocol
class Echo(basic.Int16StringReceiver):
def stringReceived(self, data):
"""
As soon as any data is received, write it back.
"""
print("Received:", data.decode('utf-8'))
self.sendString(data)
def main():
f = Factory()
f.protocol = Echo
reactor.listenTCP(22222, f)
reactor.run()
if __name__ == '__main__':
main()
You're not calling self.transport.loseConnection() anywhere that I can see in response to your polling, so as far as twisted can tell, you aren't actually disconnected. It may later, when you stop doing anything on the old transport, but by then you've lost track of things.
How to connect to the following url using twisted and get the response from the tracker.
udp://tracker.publicbt.com:80/announce?uploaded=0&downloaded=0&compact=1&event=started&peer_id=kovid_agarwal1235467&port=6881&info_hash=3389809f0c9096819294a680beb4adb96a738419&left=763922958
I tried the following code but I am getting connection Id mismatch error
from twisted.internet.protocol import DatagramProtocol
from MakeUrlRequest import encoded_Value
class SendAndReceiveUDP(DatagramProtocol):
def startProtocol(self):
self.transport.connect("31.172.124.3",80)
self.transport.write(encoded_Value())
def datagramReceived(self, datagram, addr):
print "Received %r" %datagram
from twisted.internet import reactor
udpclient=SendAndReceiveUDP()
reactor.listenUDP(6881,udpclient)
reactor.run()
Above the encoded_Value is nothing but the value as follows:
uploaded=0&downloaded=0&compact=1&event=started&peer_id=kovid_agarwal1235467&port=6881&info_hash=3389809f0c9096819294a680beb4adb96a738419&left=763922958
According to http://en.wikipedia.org/wiki/UDP_tracker, the UDP-based protocol for exchanging data with Bittorrent trackers uses "a custom binary format". It doesn't look like the data you're sending conforms to this format. So the error you receive probably indicates that you need to format your request differently. See http://www.bittorrent.org/beps/bep_0015.html for further details about that format.
I have a server where I have implemented a child of the NetstringReceiver protocol. I want it to perform an asynchronous operation (using txredisapi) based on the client's request and then respond with the results of the operation. A generalization of my code:
class MyProtocol(NetstringReceiver):
def stringReceived(self, request):
d = async_function_that_returns_deferred(request)
d.addCallback(self.respond)
# self.sendString(myString)
def respond(self, result_of_async_function):
self.sendString(result_of_async_function)
In the above code, the client connecting to my server does not get a response. However, it does get myString if I uncomment
# self.sendString(myString)
I also know that result_of_async_function is a non-empty string because I print it to stdout .
What can I do that will allow me to respond to the client with the result of the asynchronous function?
Update: Runnable source code
from twisted.internet import reactor, defer, protocol
from twisted.protocols.basic import NetstringReceiver
from twisted.internet.task import deferLater
def f():
return "RESPONSE"
class MyProtocol(NetstringReceiver):
def stringReceived(self, _):
d = deferLater(reactor, 5, f)
d.addCallback(self.reply)
# self.sendString(str(f())) # Note that this DOES send the string.
def reply(self, response):
self.sendString(str(response)) # Why does this not send the string and how to fix?
class MyFactory(protocol.ServerFactory):
protocol = MyProtocol
def main():
factory = MyFactory()
from twisted.internet import reactor
port = reactor.listenTCP(8888, factory, )
print 'Serving on %s' % port.getHost()
reactor.run()
if __name__ == "__main__":
main()
There's one specific feature about NetstringReceiver:
The connection is lost if an illegal message is received
Are you sure that your messages conform djb's Netstring protocol?
Obviously the client sends illegal string that could not be parsed, and connection is lost by protocol conditions. Everything else looks good in your code.
If you don't need that specific protcol, you'd better inherit LineReceiver instead of NetstringReceiver.
The reason you never get the response is because by the time it's sent the connection is closed. The reason the connection is closed is because the message you send with 'nc' is:
1:a,\n
Because you have to type a newline to get nc to send the message, but nc includes it as part of the message. That violates the NetString protocol...
I worked around it (with your code modified with some additional prints) by sending this message instead:
1:a,40:\n
blahblahblahDon't hit return here, just wait for the reply
8:RESPONSE,
Code:
from socket import *
sP = 14000
servSock = socket(AF_INET,SOCK_STREAM)
servSock.bind(('',sP))
servSock.listen(1)
while 1:
connSock, addr = servSock.accept()
connSock.send('HTTP/1.0 200 OK\nContent-Type:text/html\nConnection:close\n<html>...</html>')
connSock.close()
When I go to the browser and type in localhost:14000, I get an error 101- ERR_CONNECTION_RESET The connection was reset? Not sure why! What am I doing wrong
Several bugs, some more severe than others ... as #IanWetherbee already noted, you need an empty line before the body. You also should send \r\n not just \n. You should use sendall to avoid short sends. Last, you need to close the connection once you're done sending.
Here's a slightly modified version of the above:
from socket import *
sP = 14000
servSock = socket(AF_INET,SOCK_STREAM)
servSock.bind(('',sP))
servSock.listen(1)
while 1:
connSock, addr = servSock.accept()
connSock.sendall('HTTP/1.0 200 OK\r\nContent-Type:text/html\r\nConnection:close\r\n\r\n<html><head>foo</head></html>\r\n')
connSock.close()
Running your code, I have similar errors and am unsure on their origins too. However, rather than rolling your own HTTP server, have you considered a built in one? Check out the sample below. This can also support POST as well (have to add the do_POST method).
Simple HTTP Server
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
class customHTTPServer(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
self.wfile.write('<HTML><body>Hello World!</body></HTML>')
return
def main():
try:
server = HTTPServer(('',14000),customHTTPServer)
print 'server started at port 14000'
server.serve_forever()
except KeyboardInterrupt:
server.socket.close()
if __name__=='__main__':
main()