We Keep Coding

Why does pyro4 fail to locate nameserver under 127.0.1.1 but succeed with 127.0.0.1?

I am trying to run a pyro4 server with a custom event loop on a Raspberry Pi running Raspbian 8 (jessie). When I create a nameserver using the hostname obtained from socket.gethostname(), specifically 'raspberrypi', my client script cannot find the nameserver. When I use 'localhost' as the hostname, my client script is able to find the hostname. In /etc/hosts, 'raspberrypi' is bound to 127.0.1.1, while 'localhost' is obviously bound to 127.0.0.1. I had thought that both of these addresses were bound to the loopback interface, so I don't understand why one should work and not the other.
For what it's worth, after some digging in the pyro4 code, it looks like at l.463 of Pyro4.naming.py, the call to proxy.ping() fails with 127.0.1.1 but not with 127.0.0.1, and this is ultimately what triggers the failure with the former address. Not being an expert in Pyro, it isn't clear to be whether this behavior is expected. Any thoughts? I assume this must be a common problem because most (all?) flavors of Debian include separate lines in /etc/hosts for these two addresses.
I have attached code below that reproduces the problem. This is basically just a slightly modified version of the "eventloop" example that ships with pyro.
server.py:
import socket
import select
import sys
import Pyro4.core
import Pyro4.naming
import MotorControl
Pyro4.config.SERVERTYPE="thread"
hostname=socket.gethostname()
print("initializing services... servertype=%s" % Pyro4.config.SERVERTYPE)
# start a name server with broadcast server as well
nameserverUri, nameserverDaemon, broadcastServer = Pyro4.naming.startNS(host=hostname)
pyrodaemon=Pyro4.core.Daemon(host=hostname)
motorcontroller = MotorControl.MotorControl()
serveruri=pyrodaemon.register(motorcontroller)
nameserverDaemon.nameserver.register("example.embedded.server",serveruri)
# below is our custom event loop.
while True:
nameserverSockets = set(nameserverDaemon.sockets)
pyroSockets = set(pyrodaemon.sockets)
rs = []
rs.extend(nameserverSockets)
rs.extend(pyroSockets)
rs,_,_ = select.select(rs,[],[], 0.001)
eventsForNameserver=[]
eventsForDaemon=[]
for s in rs:
if s in nameserverSockets:
eventsForNameserver.append(s)
elif s in pyroSockets:
eventsForDaemon.append(s)
if eventsForNameserver:
nameserverDaemon.events(eventsForNameserver)
if eventsForDaemon:
pyrodaemon.events(eventsForDaemon)
motorcontroller.increment_count()
nameserverDaemon.close()
broadcastServer.close()
pyrodaemon.close()
client.py:
from __future__ import print_function
import Pyro4
proxy=Pyro4.core.Proxy("PYRONAME:example.embedded.server")
print("count = %d" % proxy.get_count())
MotorControl.py
class MotorControl(object):
def __init__(self):
self.switches = 0
def get_count(self):
return self.switches
def increment_count(self):
self.switches = self.switches + 1
error:
Traceback (most recent call last):
File "pyroclient.py", line 5, in <module>
print("count = %d" % proxy.get_count())
File "/usr/local/lib/python2.7/dist-packages/Pyro4/core.py", line 248, in __getattr__
self._pyroGetMetadata()
File "/usr/local/lib/python2.7/dist-packages/Pyro4/core.py", line 548, in _pyroGetMetadata
self.__pyroCreateConnection()
File "/usr/local/lib/python2.7/dist-packages/Pyro4/core.py", line 456, in __pyroCreateConnection
uri = resolve(self._pyroUri, self._pyroHmacKey)
File "/usr/local/lib/python2.7/dist-packages/Pyro4/naming.py", line 548, in resolve
nameserver = locateNS(uri.host, uri.port, hmac_key=hmac_key)
File "/usr/local/lib/python2.7/dist-packages/Pyro4/naming.py", line 528, in locateNS
raise e
Pyro4.errors.NamingError: Failed to locate the nameserver

Pyro's name server lookup relies on two things:
broadcast lookup
direct lookup by hostname/ip-address
The first is not available when you're using the loopback adapter to bind the name server on (loopback doesn't support broadcast sockets). So we're left with the second one.
The answer to your question is then simple: the direct lookup is done on the value of the NS_HOST config item, which is by default set to 'localhost'. As localhost resolves to 127.0.0.1 it will never connect to 127.0.1.1.
Suggestion: bind the name server on 0.0.0.0 or "" (empty hostname) and it should be able to start a broadcast responder as well. Then your clients won't have any problem locating it.
Alternatively, simply set NS_HOST to 127.0.1.1 (or the hostname of your box) for your clients and they should be able to locate the name server as well if it's bound on 127.0.1.1

createResolver() in twisted not working

I have a simple proxy script where I use installResolver function to choose where the script reads its hosts file:
from twisted.internet import reactor
from twisted.web import proxy, server
from twisted.python import log
from twisted.names import client
def createResolver(servers=None, resolvconf=None, hosts=None):
if platform.getType() == 'posix':
if resolvconf is None:
resolvconf = b'/etc/resolv.conf'
if hosts is None:
hosts = b'/root/example'
theResolver = Resolver(resolvconf, servers)
hostResolver = hostsModule.Resolver(hosts)
else:
if hosts is None:
hosts = r'c:\windows\hosts'
from twisted.internet import reactor
bootstrap = _ThreadedResolverImpl(reactor)
hostResolver = hostsModule.Resolver(hosts)
theResolver = root.bootstrap(bootstrap, resolverFactory=Resolver)
L = [hostResolver, cache.CacheResolver(), theResolver]
return resolve.ResolverChain(L)
site = server.Site(proxy.ReverseProxyResource('www.example.com', 80, ''.encode("utf-8")))
reactor.listenTCP(80, site)
reactor.run()
Whenever I run this script, It just times out and doesn't work. Could anybody tell me what it is I'm doing wrong? thanks!!

createResolver() in twisted not working
You need to pass in the hosts file you want it to read and you need to make sure that it's readable ie where it is /root/example means you need to be running as root to read it. You don't need to override the method. I'll install it tomorrow and try it.
Create a file /tmp/example with an entry in it ie I used
127.0.0.6 twistfoo.com
This is working code
from twisted.names import client
from twisted.internet import reactor
def do_lookup(domain):
hosts = "/tmp/example"
resolver = client.createResolver(servers=None, resolvconf=None, hosts="/tmp/example")
d = resolver.getHostByName(domain)
d.addBoth(lookup_done)
def lookup_done(result):
print(result)
reactor.stop()
domain = b'twistfoo.com'
reactor.callLater(0, do_lookup, domain)
reactor.run()
Run this on command line and you will see the output
127.0.0.6

How should I handle reconnections in twisted.application.internet.ClientService?

I am trying to make use of the recently introduced twisted.application.internet.ClientService class in a twisted application that does simple modbus-tcp polling using pymodbus. I feel my issues have nothing to do with the modbus Protocol that I am using, as I have created quite a few other working prototypes using the lower level twisted APIs; but this new ClientService looks like it fits my needs exactly, thus should reduce my code footprint and keep it neat if I can get it to work.
My tests show the ClientService handles reconnections just as it is expected to and I have easy access to the first connections Protocol. The problem that I am having is getting hold of subsequent Protocol objects for the reconnections. Here is a simplified version of the code I am having the issue with:
from twisted.application import internet, service
from twisted.internet.protocol import ClientFactory
from twisted.internet import reactor, endpoints
from pymodbus.client.async import ModbusClientProtocol
class ModbusPollingService(internet.ClientService):
def __init__(self, addrstr, numregs=5):
self.numregs=numregs
internet.ClientService.__init__(self,
endpoints.clientFromString(reactor, addrstr),
ClientFactory.forProtocol(ModbusClientProtocol))
def startService(self):
internet.ClientService.startService(self)
self._pollWhenConnected()
def _pollWhenConnected(self):
d = self.whenConnected()
d.addCallback(self._connected)
d.addErrback(self._connfail)
def _connected(self, p):
self._log.debug("connected: {p}", p=p)
self._mbp = p
self._poll()
return True
def _connfail(self, failstat):
self._log.failure('connection failure', failure=failstat)
self._mbp = None
self._pollWhenConnected()
def _poll(self):
self._log.debug("poll: {n}", n=self.numregs)
d = self._mbp.read_holding_registers(0, self.numregs)
d.addCallback(self._regs)
d.addErrback(self._connfail)
def _regs(self, res):
self._log.debug("regs: {r}", r=res.registers)
# Do real work of dealing storing registers here
reactor.callLater(1, self._poll)
return res
application = service.Application("ModBus Polling Test")
mbpollsvc = ModbusPollingService('tcp:127.0.0.1:502')
mbpollsvc.setServiceParent(application)
When the connection fails (for whatever reason) the errback of the deferred returned from read_holding_registers() gets called with the intention that my service can abandon that Protocol and go back into a state of waiting for a new connections Protocol to be returned by the whenConnected() callback... however what seems to be happening is that the ClientService does not yet realise the connection is dead and returns me the same disconnected Protocol, giving me a log full of:
2016-05-05 17:28:25-0400 [-] connected: <pymodbus.client.async.ModbusClientProtocol object at 0x000000000227b558>
2016-05-05 17:28:25-0400 [-] poll: 5
2016-05-05 17:28:25-0400 [-] connection failure
Traceback (most recent call last):
Failure: pymodbus.exceptions.ConnectionException: Modbus Error: [Connection] Client is not connected
2016-05-05 17:28:25-0400 [-] connected: <pymodbus.client.async.ModbusClientProtocol object at 0x000000000227b558>
2016-05-05 17:28:25-0400 [-] poll: 5
2016-05-05 17:28:25-0400 [-] connection failure
Traceback (most recent call last):
Failure: pymodbus.exceptions.ConnectionException: Modbus Error: [Connection] Client is not connected
or very similar, note repeated ModbusClientProtocol object address.
I'm pretty sure that I've probably just made a poor choice of pattern for this API, but I've iterated through a few different possibilities such as creating my own Protocol and Factory based on ModbusClientProtocol and handling the polling mechanism entirely within that class; but it felt a bit messy passing the persistent config and mechanism to store the polled data that way, it seems like handling this at or above the ClientService level is a cleaner approach but I can't work out the best way of keeping track of the currently connected Protocol. I guess what I'm really looking for is a best practice recommendation for usage of the ClientService class in extended polling situations.

This is an old question. But, hopefully, it will help somebody else.
The problem that I am having is getting hold of subsequent Protocol objects for the reconnections.
Supply prepareConnection callable to ClientService constructor. It will supply current connection.
In the example below MyService attaches itself to MyFactory. The main reason for this is so that MyFactory can let MyService know when ClientService disconnected. It's possible because ClientService calls Factory.stopFactory on disconnect.
Next time ClientService reconnects it will call its prepareConnection supplying current protocol instance.
(Reconnecting) ClientService:
# clientservice.py
# twistd -y clientservice.py
from twisted.application import service, internet
from twisted.internet.protocol import Factory
from twisted.internet import endpoints, reactor
from twisted.protocols import basic
from twisted.logger import Logger
class MyProtocol(basic.Int16StringReceiver):
_log = Logger()
def stringReceived(self, data):
self._log.info('Received data from {peer}, data={data}',
peer=self.transport.getPeer(),
data=data)
class MyFactory(Factory):
_log = Logger()
protocol = MyProtocol
def stopFactory(self):
# Let service know that its current connection is stale
self.service.on_connection_lost()
class MyService(internet.ClientService):
def __init__(self, endpoint, factory):
internet.ClientService.__init__(self,
endpoint,
factory,
prepareConnection=self.on_prepare_connection)
factory.service = self # Attach this service to factory
self.connection = None # Future protocol instance
def on_prepare_connection(self, connection):
self.connection = connection # Attach protocol to service
self._log.info('Connected to {peer}',
peer=self.connection.transport.getPeer())
self.send_message('Hello from prepare connection!')
def on_connection_lost(self):
if self.connection is None:
return
self._log.info('Disconnected from {peer}',
peer=self.connection.transport.getPeer())
self.connection = None
def send_message(self, message):
if self.connection is None:
raise Exception('Service is not available')
self.connection.sendString(bytes(message, 'utf-8'))
application = service.Application('MyApplication')
my_endpoint = endpoints.clientFromString(reactor, 'tcp:localhost:22222')
my_factory = MyFactory()
my_service = MyService(my_endpoint, my_factory)
my_service.setServiceParent(application)
Slightly modified echo server from twisted examples:
#!/usr/bin/env python
# echoserv.py
# python echoserv.py
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
from twisted.protocols import basic
### Protocol Implementation
# This is just about the simplest possible protocol
class Echo(basic.Int16StringReceiver):
def stringReceived(self, data):
"""
As soon as any data is received, write it back.
"""
print("Received:", data.decode('utf-8'))
self.sendString(data)
def main():
f = Factory()
f.protocol = Echo
reactor.listenTCP(22222, f)
reactor.run()
if __name__ == '__main__':
main()

You're not calling self.transport.loseConnection() anywhere that I can see in response to your polling, so as far as twisted can tell, you aren't actually disconnected. It may later, when you stop doing anything on the old transport, but by then you've lost track of things.

Python socket closed before all data have been consumed by remote

I am writing a Python module which is communicating with a go program through unix sockets. The client (the python module) write data to the socket and the server consume them.
# Simplified version of the code used
outputStream = socket.socket(socketfamily, sockettype, protocol)
outputStream.connect(socketaddress)
outputStream.setblocking(True)
outputStream.sendall(message)
....
outputStream.close()
My issue is that the Python client tends to finish and close the socket before the data have been effectively read by the server which leads to a "broken pipe, connection reset by peer" on the server side. Whatever I do, for the Python code everything has been sent and so the calls to send() sendall() select() are all successful...
Thanks in advance
EDIT: I can't use shutdown because of mac OS
EDIT2: I also tried to remove the timeout and call setblocking(True) but it doesn't change anything
EDIT3: After ready this issue http://bugs.python.org/issue6774 it seems that the documentation is unnecessary scary so I restored the shutdown but I still have the same issue:
# Simplified version of the code used
outputStream = socket.socket(socketfamily, sockettype, protocol)
outputStream.connect(socketaddress)
outputStream.settimeout(5)
outputStream.sendall(message)
....
outputStream.shutdown(socket.SHUT_WR)
outputStream.close()

IHMO this is best done with an Asynchornous I/O library/framework. Here's such a solution using circuits:
The server echos what it receives to stdout and the client opens a file and sends this to the server waiting for it to complete before closing the socket and terminating. This is done with a mixture of Async I/O and Coroutines.
server.py:
from circuits import Component
from circuits.net.sockets import UNIXServer
class Server(Component):
def init(self, path):
UNIXServer(path).register(self)
def read(self, sock, data):
print(data)
Server("/tmp/server.sock").run()
client.py:
import sys
from circuits import Component, Event
from circuits.net.sockets import UNIXClient
from circuits.net.events import connect, close, write
class done(Event):
"""done Event"""
class sendfile(Event):
"""sendfile Event"""
class Client(Component):
def init(self, path, filename, bufsize=8192):
self.path = path
self.filename = filename
self.bufsize = bufsize
UNIXClient().register(self)
def ready(self, *args):
self.fire(connect(self.path))
def connected(self, *args):
self.fire(sendfile(self.filename, bufsize=self.bufsize))
def done(self):
raise SystemExit(0)
def sendfile(self, filename, bufsize=8192):
with open(filename, "r") as f:
while True:
try:
yield self.call(write(f.read(bufsize)))
except EOFError:
break
finally:
self.fire(close())
self.fire(done())
Client(*sys.argv[1:]).run()
In my testing of this it behaves exactly as I expect it to with no
errors and the servers gets the complete file before the client clsoes
the socket and shuts down.

After a discussion with a colleague aware of the C sockets (in cpython the socket module is a wrapper for the C sockets) he spoke about this http://ia600609.us.archive.org/22/items/TheUltimateSo_lingerPageOrWhyIsMyTcpNotReliable/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable.html (that's how it is done in PHP internal for the record)
TL&DR: shutdown + quick poll + close or ioctl(SIOCOUTQ) on linux

twisted - get OS-chosen listen port

I am writing a twisted P2P client using the application framework. The listen port for incoming connections will be on a random (OS-determined) port. However, I need a way to determine what that port is after creating it:
import twisted... etc.
application = service.Application('vmesh')
peerservice = MyPeerService()
servicecollection = service.IServiceCollection(application)
factory = MyPeerFactory(peerservice)
server = internet.TCPServer(0, factory) # listen on random port
listen_port = server.getHost().port # ??? doesn't work...
server.setServiceParent(servicecollection)
I can't find anything in the docs about querying the port created by internet.TCPServer() or by reactor.listenTCP() which it forwards to. I can't simply wait for a connection to occur since the client has to announce its port in order for those connections to ever happen.

listenTCP returns an IListeningPort, which has a getHost() method that gives back an object with a port. For example:
>>> from twisted.internet import reactor
>>> from twisted.internet.protocol import Factory
>>> port = reactor.listenTCP(0, Factory())
>>> port.getHost().port
55791
However, TCPServer doesn't call listenTCP until it is started with privilegedStartService. Plus, the IListeningPort isn't actually exposed via a public API. So, you will need to write your own Service. Luckily, it's quite easy to do this; TCPServer doesn't do very much. You just need to write one that reports its port somewhere as soon as it starts listening. Here's an example:
from twisted.internet import reactor
from twisted.application.service import Service
class PortReporter(Service, object):
def __init__(self, factory, reportPort):
self.factory = factory
self.reportPort = reportPort
def privilegedStartService(self):
self.listeningPort = reactor.listenTCP(0, self.factory)
self.reportPort(self.listeningPort.getHost().port)
return super(PortReporter, self).privilegedStartService()
def stopService(self):
self.listeningPort.stopListening()
return super(PortReporter, self).stopService()
You can then use this in a tac file, like so:
from twisted.internet.protocol import Factory
from twisted.application.service import Application
application = Application("test")
def showPortNumber(n):
print("The port number is: %d" % (n,))
PortReporter(Factory(), showPortNumber).setServiceParent(application)

FWIW if you need to do this with endpoints here is my implementation with a slight tweak for my local setup (the callback option would work well here too):
class PortReporter(StreamServerEndpointService, object):
def __init__(self, endpoint, factory):
StreamServerEndpointService.__init__(self, endpoint, factory)
self._reportedPort = None
def privilegedStartService(self):
r = super(PortReporter, self).privilegedStartService()
self._waitingForPort.addCallback(self.port_cb)
return r
def port_cb(self, port):
self._reportedPort = port.getHost().port
return port
def getReportedPort(self):
return self._reportedPort

You can access the port bind to your server like so if you didn't start the server yet (didn't call startService yet):
>>> serv._getPort()._realPortNumber
Else you can also do:
>>> serv._port._realPortNumber