We Keep Coding

Socket disconnected after a message, (Connection refused by remote host), using python

I have uploaded my python socket, (cloud service project), on azure ,and when ever I connected to Hercules client side socket ....after a message or two, connection closed by remote host... forcefully...?
Server Code
from twisted.internet.protocol import Factory, Protocol
from twisted.internet import reactor
import SocketServer
class IphoneChat(Protocol):
def connectionMade(self):
self.factory.clients.append(self)
print('clients are'), self.factory.clients
def connectionLost(self, reason):
self.factory.clients.remove(self)
def dataReceived(self, data):
msg = ""
msg = data.strip()
for c in self.factory.clients:
c.message(msg)
def message(self, message):
self.transport.write(message + '\n')
print ('Iphone Chat server startedd')
factory = Factory()
factory.protocol = IphoneChat
factory.clients = []
reactor.listenTCP(9073, factory)
reactor.run()

I tried to reproduce your issue on my environment, but failed. Base on my experience, we often pay attention to 3 points if we use socket in azure worker role.
Open input TCP port. If we open a port as listener port, we'd better to add this port into the endpoints setting.
Considered the worker role status, I suggest we can code logic into the while True loop function, as following,
while True:
reactor.listenTCP(65534, f)
print "server run"
reactor.run()
sleep(1.0)
According to the error message, I guess the reason is that azure load balancer will kill the idle connection in 240s. I recommend you can refer to this blog to configure your idleTimeoutInMinutes value.
Please try to check your project and configuration. Any further findings, please let me know.

Echoing messages received through UDP back through a different TCP port

I use Python 2.7.x (2.7.8) and am trying to write a program using twisted python to function like this:
Program waits for a messaged received through TCP 7001 or UDP 7000.
Messages received through UDP 7000 are bridged over to TCP 7001 going out.
I couldn't figure out how to bridge UDP to TCP, so I looked up an example on this site, like this one, Twisted UDP to TCP Bridge but the problem is that the example is lying because it doesn't work. I added a "print datagram" on datagramReceived to see if UDP responds to receiving anything at all and it does not. This is totally frustrating.
Here's my current test code altered a little bit from that example:
from twisted.internet.protocol import Protocol, Factory, DatagramProtocol
from twisted.internet import reactor
class TCPServer(Protocol):
def connectionMade(self):
self.port = reactor.listenUDP(7000, UDPServer(self))
def connectionLost(self, reason):
self.port.stopListening()
def dataReceived(self, data):
print "Server said:", data
class UDPServer(DatagramProtocol):
def __init__(self, stream):
self.stream = stream
def datagramReceived(self, datagram, address):
print datagram
self.stream.transport.write(datagram)
def main():
f = Factory()
f.protocol = TCPServer
reactor.listenTCP(7001, f)
reactor.run()
if __name__ == '__main__':
main()
As you can see I changed the ports to comply with my test environment and added a print datagram to see if anything calls datagramReceived. I have no problems sending TCP things to this program, TCPServer works just fine because dataReceived can be called.

I ran the code from your question (with one slight modification: I enabled logging). I used telnet to connect to the TCP server on port 7001. I used a Python REPL to create a UDP socket and send some datagrams to port 7000.
Here is my REPL transcript:
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> s.sendto('hello', ('127.0.0.1', 7000))
5
>>> s.sendto('world', ('127.0.0.1', 7000))
5
>>>
Here is my server log (formatted to fit on your screen):
... Log opened.
... Factory starting on 7001
... Starting factory <twisted.internet.protocol.Factory instance at 0x2b9b128>
... UDPServer starting on 7000
... Starting protocol <__main__.UDPServer instance at 0x2e8f8c0>
... hello
... world
And here's my telnet session transcript:
$ telnet localhost 7001
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
helloworld
My interpretation of these results is that the program actually works as specified. I wonder what you did differently when you tried it that produced different, non-working results.

Python asyncore UDP server

I am writing server application in Python that listens for requests, processes them, and sends a response.
All req/resp are send from the same address and port to the server application. I need to recv/send messages simultaneously, and the server need to recieve/send messages from/to the same port and address. I found some tutorials for asynchore sockets, but there are only examples for TCP connections.
Unfortunately, I need UDP. When I change SOCK_STREAM to SOCK_DGRAM in the create method, I get an error.
return getattr(self._sock,name)(*args)
socket.error: [Errno 95] Operation not supported
I tried to use twisted, but I dont know how to write the sender part, which can bind to the same port as its listening. The last result was blocked port.
Is there any way how to use asyncore sockets with UDP or how to use twisted to send from the same port? Some examples will be higly appreciated.

You can pretty much just write the sending and receiving part of your code and they'll work together. Note that you can send and receive on a single listening UDP socket - you don't need one for each (particularly if you want to send from and receive on the same address).
from __future__ import print_function
from sys import stdout
from twisted.python.log import startLogging
from twisted.internet import reactor
from twisted.internet.protocol import DatagramProtocol
class SomeUDP(DatagramProtocol):
def datagramReceived(self, datagram, address):
print(u"Got a datagram of {} bytes.".format(len(datagram)))
def sendFoo(self, foo, ip, port):
self.transport.write(
(u"Foo datagram: {}".format(foo)).encode("utf-8"),
(ip, port))
class SomeSender(object):
def __init__(self, proto):
self.proto = proto
def start(self):
reactor.callLater(3, self._send)
def _send(self):
self.proto.sendFoo(u"Hello or whatever", b"127.0.0.1", 12345)
self.start()
startLogging(stdout)
proto = SomeUDP()
reactor.listenUDP(12345, proto)
SomeSender(proto).start()
reactor.run()

Python Sockets Peer to Peer

I'm trying to make a simple Peer to Peer Network in Python 2.7. The problem is, I can't seem to be able to create a connection between two machines in which they both act as a server and a client. I can get it to work when one is a server and the other is a client but not when they are both, both. Do I need to create 2 sockets? Also I'm using TCP to connect.
UPDATE:
import socket, sys # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.
s.bind((host, port)) # Bind to the port
if sys.argv[1] == "connect":
host = sys.argv[2]
s.connect((host, port))
s.close
else:
s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
c.send('Thank you for connecting')
c.close()
The codes not very good because for someone to connect as a client they have to use the argument "connect" followed by the hostname or IP of the second machine. I can't get the two to connect and serve to each other simultaneously.

Yes, two sockets are necessary. The listening socket should open on a constant port, and the client port should be opened on a different (potentially dynamic) port, usually higher in the port range. As an example:
Server sockets on port 1500, client sockets on port 1501.
Peer1: 192.168.1.101
Peer2: 192.168.1.102
When peer1 connects to peer2 it looks like this: 192.168.1.101:1501 -> 192.168.1.102:1500.
When peer2 connects to peer1 it looks like this: 192.168.1.102:1501 -> 192.168.1.101:1500.
Listening TCP sockets are also generally run on a separate thread since they are blocking.

Yes, you'll need to use two sockets, one for accepting connections (server) and one for initiating connections (client). However, you can bind both sockets to the same local port, using that port number as both the source and destination port and thereby ensuring that you'll end up with only a single connection between each pair of peers. If both peers try to connect simultaneously (e.g. because they discovered each other at the same time), one of the client connection attempts will fail (where the peer's server socket accepted the connection), you'll have to handle (ignore) that.
To bind two sockets on the same port, you'll need to set the SO_REUSEPORT/SO_REUSEADDR flags on both.
Here is an example program demonstrating this technique (using the excellent trio on Python 3):
from errno import EADDRNOTAVAIL
from functools import partial
from itertools import count
import trio
import socket
async def peer(SRC, DEST):
counter = count(start=1)
async def sender(stream, n):
print(f"sender{n}#{SRC}: started!")
while True:
data = bytes(f"Hello from {n}#{SRC}", "utf8")
print(f"sender{n}#{SRC}: sending {data!r}")
await stream.send_all(data)
await trio.sleep(1)
async def receiver(stream, n):
print(f"receiver{n}#{SRC}: started!")
async for data in stream:
print(f"receiver{n}#{SRC}: got data {data!r}")
print(f"receiver{n}#{SRC}: connection closed")
async with trio.open_nursery() as nursery:
async def run(connection: trio.SocketStream):
count = next(counter)
print(f"peer#{SRC} got connection{count} from {method}() with {connection.socket.getpeername()}")
async with connection:
async with trio.open_nursery() as nursery:
print(f"peer#{SRC}: spawning sender...")
nursery.start_soon(sender, connection, count)
print(f"peer#{SRC}: spawning receiver...")
nursery.start_soon(receiver, connection, count)
print(f"peer: listening at {SRC}")
servers = await trio.open_tcp_listeners(SRC[1], host=SRC[0])
servers[0].socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
servers[0].socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
await nursery.start(trio.serve_listeners, partial(run, "listen"), servers)
print(f"peer: connecting from {SRC} to {DEST}")
client = trio.socket.socket()
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
await client.bind(address=SRC)
try:
await client.connect(address=DEST)
except OSError as err:
if err.errno != EADDRNOTAVAIL:
raise
# the other client was faster than us
print(f"peer#{SRC}: {err.strerror}")
else:
await run('connect', trio.SocketStream(client))
async def main():
async with trio.open_nursery() as nursery:
a = ("127.0.0.1", 12345)
b = ("127.0.0.1", 54321)
nursery.start_soon(peer, a, b)
nursery.start_soon(peer, b, a)
trio.run(main)
In this small demonstration, the two peers run within the same program on different ports on the same host, but it works just the same with two programs using the same ports but different hosts. Notice that if you comment out the client.bind(address=SRC) bit, they will use ephemeral source ports, and create two separate connections not just a single one.

Python TCP socket doesn't close?

Maybe someone here will have a response for this thing which is just driving me insane.
To make it simple, I'm making a kind of proxy. Whenever it receives something, it forwards everything to a server, and sends back the response. So there is one socket always listening on port 4557 for clients, and for each incoming connection, there is a new socket created on a random port to connect to the server port 4556.
Clients <==> Proxy <==> Server
Also, there another socket which is instantiated and listening for requests coming from the server and to be forwarded to the corresponding client.
Here is an example:
Client A connects to proxy on port 4557
Proxy creates a socket to Server on port 4556
Along with that, it creates a socket listening on port 40100
Client sends stuff, forwarded to Server
Client disconnects. Close client connection and socket to server
Some time later, Server sends stuff to proxy on port 40100
Everything's forwarded to Client A (port 40100 corresponding to Client A)
And so on..
So far in my tests, I use a simple python script for sending a unique tcp packet to the proxy, along with a dump server showing received data and echoing back.
So the issue is that when a connection to the proxy is closed, the connection to the Server should also be closed with "sock.close()". However it just seems to be completely ignored. The socket remains as ESTABLISHED.
About the code now.
A few notes.
DTN and Node are respectively Server and Clients.
runCallback is called in a loop until thread dies.
finalCallback is called when the thread is dying.
Associations between remote hosts (Client), proxy ports (to Server) and proxies are kept in the dictionaries: TCPProxyHostRegister (RemoteHost => Proxy), TCPProxyPortRegister (Port => Proxy), TCPPortToHost (Port => RemoteHost).
The first class is TCPListenerThread.
It just listen on a specific port and instantiate proxies (one for each Client=>Server couple and Server=>Client couple) and forward them connections.
class TCPListenerThread(StoppableThread):
def __init__(self, tcp_port):
StoppableThread.__init__(self)
self.tcp_port = tcp_port
self.sock = socket.socket( socket.AF_INET, # Internet
socket.SOCK_STREAM ) # tcp
self.sock.bind( (LOCAL_ADDRESS, self.tcp_port) )
self.sock.listen(1)
def runCallback(self):
print "Listen on "+str(self.tcp_port)+".."
conn, addr = self.sock.accept()
if isFromDTN(addr):
tcpProxy = getProxyFromPort(tcp_port)
if not tcpProxy:
tcpProxy = TCPProxy(host, True)
else:
host = addr[0]
tcpProxy = getProxyFromHost(host)
if not tcpProxy:
tcpProxy = TCPProxy(host, False)
tcpProxy.handle(conn)
def finalCallback(self):
self.sock.close()
Now comes the TCP Proxy:
It associates a remote host (Client) with a port connecting to Server.
If it's a connection coming from a new Client, it will create a new listener (see above) for the Server and create a socket ready to forward everything to Server.
class TCPProxy():
def __init__(self, remote, isFromDTN):
#remote = port for Server or Remote host for Client
self.isFromDTN = isFromDTN
self.conn = None
#add itself to proxy registries
#If listening from a node
if not isFromDTN:
#Set node remote host
self.remoteHost = remote
TCPProxyHostRegister[self.remoteHost] = self
#Set port to DTN interface + listener
self.portToDTN = getNewTCPPort()
TCPPortToHost[self.portToDTN] = self.remoteHost
newTCPListenerThread(self.portToDTN)
#Or from DTN
else:
self.portToDTN = remote
TCPProxyPortRegister[self.portToDTN] = self
self.remoteHost = getRemoteHostFromPortTCP(self.portToDTN)
def handle(self, conn):
print "New connection!"
#shouldn't happen, but eh
if self.conn != None:
self.closeConnections()
self.conn = conn
#init socket with remote
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
if self.isFromDTN:
self.sock.connect((self.remoteHost, 4556)) #TODO: handle dynamic port..
else:
self.sock.connect((DTN_Address, DTN_TCPPort))
#handle connection in a thread
self.handlerThread = newTCPHandlerThread(self)
#handle reply in a therad
self.replyThread = newTCPReplyThread(self)
def closeConnections(self):
try:
if self.conn != None:
print "Close connections!"
self.sock.close()
self.conn.close()
self.conn = None
self.handlerThread.kill()
self.replyThread.kill()
except Exception, err:
print str(err)
#pass
def forward(self, data):
print "TCP forwarding data: "+data
self.sock.send(data)
def forwardBack(self, data):
print "TCP forwarding data back: "+data
self.conn.send(data)
In this proxy class, I instantiate two classes, TCPHandlerThread and TCPReplyThread. They are responsible for forwarding to Server, and forwarding back to Client, respectively.
class TCPHandlerThread(StoppableThread):
def __init__(self, proxy):
StoppableThread.__init__(self)
self.proxy = proxy
def runCallback(self):
test = False
while 1:
data = self.proxy.conn.recv(BUFFER_SIZE)
if test:
self.proxy.sock.close()
test = True
if not data:
break
print "TCP received data:", data
self.proxy.forward(data)
self.kill()
def finalCallback(self):
self.proxy.closeConnections()
class TCPReplyThread(StoppableThread):
def __init__(self, proxy):
StoppableThread.__init__(self)
self.proxy = proxy
def runCallback(self):
while 1:
data = self.proxy.sock.recv(BUFFER_SIZE)
if not data:
break
print "TCP received back data: "+data
self.proxy.forwardBack(data)
self.kill()
def finalCallback(self):
self.proxy.closeConnections()
You see that whenever a connection is closed, the thread dies and the other connection (Client/Server to proxy or Proxy to Server/Client) should be closed in Proxy.closeConnections()
I noticed that when closeConnections() is "data = self.proxy.conn.recv(BUFFER_SIZE)", it goes well, but when it's called even right after the latter statement, it goes wrong.
I wiresharked TCP, and the proxy doesn't send any "bye signal". The socket state doesn't go to TIME_WAIT or whatever, it just remains ESTABLISHED.
Also, I tested it on Windows and Ubuntu.
On Windows it goes exactly as I explained
On Ubuntu, it works well for usually (not always), 2 connections, and the third time I connect with the same client in exactly the same way to the proxy, it goes wrong again exactly as explained.
Here are the three files i'm using so that you can have a look at the whole code. I'm sorry the proxy file might not be really easy to read. Was SUPPOSED to be a quick dev.
http://hognerud.net/stackoverflow/
Thanks in advance..
It's surely something stupid. Please don't hit me too hard when you see it :(

First I'm sorry that I currently have not the time to actually run and test your code.
But the idea came to my mind, that your problem might actually have something todo with using blocking mode vs. non-blocking mode on the socket. In that case you should checkout the "socket" module help in the python documentation, especially socket.setblocking().
My guess is, that the proxy.conn.recv() function only returns, when actually BUFFER_SIZE bytes where received by the socket. Because of this the thread is blocked until enough data was received and therefore the socket doesn't get closed.
As I said first, this is currently just a guess, so please don't vote me down if it doesn't solve the problem...