We Keep Coding

Multi Port Network Application

I want to create a python network application that can run on multiple ports (ex: TCP:1234, TCP:5678, etc).
So I have lets say n number of Sockets, each listening for a client connection. I programmed a simple network application that listens to a range of ports but when I run the application it gets stuck at the listening phase of the first socket process!
How can I make my single python program when run to listen to N number of ports and each waiting for a client to connect to it. All sockets are running and listening at the same time.
Socket/Process #1: Listening on TCP Port 5000
Socket/Process #2: Listening on TCP Port 5001
Socket/Process #3: Listening on TCP Port 5002
...
Socket/Process #N: Listening on TCP Port 6000
Appreciate any ideas.
#!/usr/bin/env python
import socket
def getPortList():
ports=[]
nPort=int(raw_input("# how many ports you want? "))
j = 0
for i in range(0,nPort):
ports.append(int(raw_input("Enter port number: ")))
j+=1
return ports
def myTCPSocket(port=5000):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
s.bind(("", int(port)))
print ("\nWaiting for connections!\n")
s.listen(5)
(clientsock, clientaddr) = s.accept()
print(clientaddr)
data = "start"
while len(data):
clientsock.send("\nWelcome to Echo Server\n")
data = clientsock.recv(1024)
print ("Data sent is: ", data)
clientsock.send(data)
if data == "exit\r\n":
clientsock.close()
plst = getPortList()
for item in plst:
myTCPSocket(item)

Listening on multiple sockets is really no different from listening on a single socket.
You already need to handle the listener socket and all client connection sockets somehow. You can do this by:
Writing a loop around select.select (or poll, kqueue, epoll, etc.).
Using the standard-library reactor asyncore.
Using a third-party reactor or proactor like Twisted.
Using OS-specific functionality (e.g., using a Cocoa runloop and server via PyObjC).
Creating a thread for each new connection.
Creating a subprocess for each new connection.
Almost all of these schemes will also work for dealing with multiple listeners. The simplest thing to do is to combine the two into one (e.g., a single select loop that handles all of their listeners and all of their client sockets, or a separate thread for each listener and client socket).
For performance or debugging reasons, you might want to instead use a two-tier hybrid approach (e.g., a thread for each listener, each with a select loop for all of its client sockets, or a process for each listener, each with a thread for each client socket). But if you don't have any good reason to do that, don't add the complexity.
http://pastebin.com/QebZMKz3 shows a simple single-select implementation. Here's the output:
$ ./multiserve.py 22222 22223 &
(('127.0.0.1', 22222), ' listening')
(('127.0.0.1', 22223), ' listening')
$ echo 'abc' | nc localhost 22222
(('127.0.0.1', 22222), ' <- ', ('127.0.0.1', 64633))
(('127.0.0.1', 64633), ' <- ', 'abc\n')
(('127.0.0.1', 64633), ' EOF')
If you think you'll never actually need to handle two simultaneous clients… well, you're probably wrong, but… You can use most of the above techniques, and it may be marginally simpler. For example, you can select on the listeners, and then do the accept and client-socket communication synchronously before returning to the loop. Or you can create a process or thread for each listener but handle the accept and client-socket communication synchronously within each. And so on.
http://pastebin.com/wLVLT49i shows a simple example that seems to be what you were trying to do. Since it uses a process for each socket (via os.fork), it does allow simultaneous connections on different ports; since it doesn't do anything asynchronously within each process, it doesn't allow simultaneous connections to the same port. (And of course it's POSIX-specific because it uses fork.)
If you're looking to learn how to write asynchronous network servers in the first place, I'd suggest you do two different implementations: select and threads. They conceptually fundamental, and relatively easy to code.
First, for select, you have to get your head around the idea of an event loop—the events are each new incoming connection, each incoming network packet on an existing connection, even each time a pipe you were writing to gets unclogged. The tricky bit here is that, as with any event loop, you need to handle each event and return without blocking, and without spending too much CPU time. For example, for an echo server, you can't just do a write on the other sockets, because some of them might be busy. So instead, you have to stick the output in a write buffer for each socket, and they'll get it in some future run through the event loop, when thye're ready.
Meanwhile, for threads, a separate thread for each connection seems like it makes everything trivial, but what happens when you need to echo a message from one thread to another? You either need some form of inter-thread communication, or shared data with inter-thread synchronization. So, you might have a Queue for writes on each socket, so any other socket's thread can just push a message onto the queue.
Neither of these will be as good as what a well-turned reactor or proactor can do, but it'd worth learning the basics—especially since you're going to face both the blocking issue (from select) and the communication issue (from threads) with any solution, and they'll be much more mysterious and harder to debug when you're working at a higher level.

Handling Incoming Data from Multiple Sockets in Python

Background:
I have a current implementation that receives data from about 120 different socket connections in python. In my current implementation, I handle each of these separate socket connections with a dedicated thread for each. Each of these threads parse the data and eventually store it within a shared locked dictionary. These sockets DO NOT have uniform data rates, some sockets get more data than others.
Question:
Is this the best way to handle incoming data in python, or does python have a better way on handling multiple sockets per thread?

Using an asynchronous approach will make you much happier. For an example of a well-done implementation of this as a well-known application Tornado is perfect. You can easily use Tornado's ioloop for things other than web servers, too.
There are alternative libraries such as gevent; but I believe Tornado is a better place to look at first since it both provides the loop and a web server implemented on top of it as a great example of how to use the loop well.

If you're using threads, that's basically the way you'd go about it.
The alternative is to use one of the various asynchronous networking libraries out there, such as Twisted, Tornado, or GEvent.

As mentioned in Asynchronous UDP Socket Reading question from you, asyncoro can be used to process many asynchronous sockets efficiently. Another benefit with asyncoro in your problem is that you don't need to worry about locking shared dictionary, as with asyncoro at most one coroutine is executing at any time and there is no forced preemption.

How to implement threaded socket.recv() in python?

I have a number of devices from which i need to get status updates. A socket object is all I have, and socket.recv() is all I need to get the status. Put into a single threaded application, no problems occur:
class Device:
def receive(self):
log.debug("receive waiting: %r", self.device_id)
try:
packet = self.socket.recv(255)
except Exception as e:
self.report_socket_error(e)
self.reconnect()
log.debug("received response: %r", self.device_id)
d = Device()
d.connect()
while True:
d.receive()
However, the same code wrapped in a threading.Thread class causes deadlocks and funny behaviour. Wrapping it with locks didn't change anything. I traced the problem down to be the socket.recv() call...So, how to implement multiple threads where each thread owns one socket (1 thread owns exclusively 1 socket), which are able to wait for data simultaneously?
Thanks in advance

I know this does not answer your question on how to fix your deadlock problem, however it appears as your use of threads is overhead in your case:
You can just use one thread in which you use select() to find out which socket has available data and then handle the reported data. Unless the handling takes long or your protocol is more complicated select should be just fine and avoid all threading issues.
Have a look at http://docs.python.org/howto/sockets.html#non-blocking-sockets for more details.

How many different sockets do you have to read from?
If the answer is "just one", then use just one thread. Adding another helps you in no way and only complicates your life, as you found out.
If the answer is "several", than one way to organize this is indeed to have a thread per socket. recv is a blocking operation, which makes a thread an attractive option to organize code. Each thread owns a separate socket and reads from it at its leisure. You should have no problems and deadlocks with this.
Locks are unnecessary as long as no resources are shared. Even if you do share resources (logging, some data store, etc.) don't just use simple locks - Python has higher-level utilities for that like the Queue module.

python monitoring over serial port

Good afternoon,
I would ask some suggestion about the best way to monitor events over the serial port.
I'm using PySerial to write "commands" over the serial port towards some devices and
I would like to receive feedback about the status of this devices.
Wich is the best way: 1) fullfill a pipe and read into, 2) a new thread delegated to read only, or what?
Can I also ask for a simple code to implement the solution?

For general tips on working with pyserial, look at the search S.Lott suggested in the comment.
Regarding the best strategy to implement your application - it all depends on how your protocols are defined. Do the devices immediately respond to queries? Or do they continually send data that must be monitored? This is important to define, as it certainly affects the way you'll want to handle the communication.
Generally, I've found it simple and stable to have a separate thread reading everything from the serial port and just pumping the data into a Queue. The main application logic then can query this queue whenever it needs to and read the data.

The strategy choosen is to use python multiprocessing and queue
see:
http://www.ibm.com/developerworks/aix/library/au-threadingpython/index.html
and
http://www.ibm.com/developerworks/aix/library/au-multiprocessing/index.html?ca=dgr-lnxw9dPython-Multi&S_TACT=105AGX59&S_CMP=grsitelnxw9d
for reference

How can I do synchronous rpc calls

I'm building a program that has a class used locally, but I want the same class to be used the same way over the network. This means I need to be able to make synchronous calls to any of its public methods. The class reads and writes files, so I think XML-RPC is too much overhead. I created a basic rpc client/server using the examples from twisted, but I'm having trouble with the client.
c = ClientCreator(reactor, Greeter)
c.connectTCP(self.host, self.port).addCallback(request)
reactor.run()
This works for a single call, when the data is received I'm calling reactor.stop(), but if I make any more calls the reactor won't restart. Is there something else I should be using for this? maybe a different twisted module or another framework?
(I'm not including the details of how the protocol works, because the main point is that I only get one call out of this.)
Addendum & Clarification:
I shared a google doc with notes on what I'm doing. http://docs.google.com/Doc?id=ddv9rsfd_37ftshgpgz
I have a version written that uses fuse and can combine multiple local folders into the fuse mount point. The file access is already handled within a class, so I want to have servers that give me network access to the same class. After continuing to search, I suspect pyro (http://pyro.sourceforge.net/) might be what I'm really looking for (simply based on reading their home page right now) but I'm open to any suggestions.
I could achieve similar results by using an nfs mount and combining it with my local folder, but I want all of the peers to have access to the same combined filesystem, so that would require every computer to bee an nfs server with a number of nfs mounts equal to the number of computers in the network.
Conclusion:
I have decided to use rpyc as it gave me exactly what I was looking for. A server that keeps an instance of a class that I can manipulate as if it was local. If anyone is interested I put my project up on Launchpad (http://launchpad.net/dstorage).

If you're even considering Pyro, check out RPyC first, and re-consider XML-RPC.
Regarding Twisted: try leaving the reactor up instead of stopping it, and just ClientCreator(...).connectTCP(...) each time.
If you self.transport.loseConnection() in your Protocol you won't be leaving open connections.

For a synchronous client, Twisted probably isn't the right option. Instead, you might want to use the socket module directly.
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((self.host, self.port))
s.send(output)
data = s.recv(size)
s.close()
The recv() call might need to be repeated until you get an empty string, but this shows the basics.
Alternatively, you can rearrange your entire program to support asynchronous calls...

Why do you feel that it needs to be synchronous?
If you want to ensure that only one of these is happening at a time, invoke all of the calls through a DeferredSemaphore so you can rate limit the actual invocations (to any arbitrary value).
If you want to be able to run multiple streams of these at different times, but don't care about concurrency limits, then you should at least separate reactor startup and teardown from the invocations (the reactor should run throughout the entire lifetime of the process).
If you just can't figure out how to express your application's logic in a reactor pattern, you can use deferToThread and write a chunk of purely synchronous code -- although I would guess this would not be necessary.

If you are using Twisted you should probably know that:
You will not be making synchronous calls to any network service
The reactor can only ever be run once, so do not stop it (by calling reactor.stop()) until your application is ready to exit.
I hope this answers your question. I personally believe that Twisted is exactly the correct solution for your use case, but that you need to work around your synchronicity issue.
Addendum & Clarification:
Part of what I don't understand is
that when I call reactor.run() it
seems to go into a loop that just
watches for network activity. How do I
continue running the rest of my
program while it uses the network? if
I can get past that, then I can
probably work through the
synchronicity issue.
That is exactly what reactor.run() does. It runs a main loop which is an event reactor. It will not only wait for entwork events, but anything else you have scheduled to happen. With Twisted you will need to structure the rest of your application in a way to deal with its asynchronous nature. Perhaps if we knew what kind of application it is, we could advise.