I have a python program with many threads. I was thinking of creating a socket, bind it to localhost, and have the threads read/write to this central location. However I do not want this socket open to the rest of the network, just connections from 127.0.0.1 should be accepted. How would I do this (in Python)? And is this a suitable design? Or is there something a little more elegant?
Given a socket created with socket.socket(), you can use bind() before listening:
socket.bind(('127.0.0.1', 80))
Using the address 127.0.0.1 indicates that the socket should bind to the local interface only.
http://www.amk.ca/python/howto/sockets/
Shows some socket example. This tidbit is interesting to you I think
we used socket.gethostname() so that the socket would be visible to the outside world. If we had used s.bind(('', 80)) or s.bind(('localhost', 80)) or s.bind(('127.0.0.1', 80)) we would still have a "server" socket, but one that was only visible within the same machine.
I guess there is your answer (see below for correction)
As to the validity of using this method for thread communications. I'm not sure how well this handles multiple threads and reading/writing
EDIT
There seems to be a python recipe linked below that does some inter-thread communication
http://code.activestate.com/recipes/491281/
Have fun!
EDIT
The article is incorrect and as pointed out "s.bind(('', 80)) will bind to INADDR_ANY"
If you are running on a UNIX-based system, you might want to consider using UNIX Domain Sockets instead of Internet sockets. I think something like the following should work:
>>> # in one window/shell
>>> import socket
>>> sd = socket.socket(socket.AF_UNIX)
>>> sd.bind('/path/to/my/socket')
>>> sd.listen(5)
>>> (client,addr) = sd.accept()
>>> client.recv(1024)
'hello'
>>>
>>> # in a different shell
>>> import socket
>>> sd = socket.socket(socket.AF_UNIX)
>>> sd.connect('/path/to/my/socket')
>>> sd.send('hello')
You might want to use the queue module from the standard library instead. It's designed specifically to facilitate communication between threads. A quote from the docs:
The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics. It depends on the availability of thread support in Python; see the threading module.
notionOn TCP/IP networks 127.0.0.0/8 is a non-routeable network, so you should not be able to send an IP datagram destined to 127.0.0.1 across a routed infrastructure. The router will just discard the datagram. However, it is possible to construct and send datagrams with a destination address of 127.0.0.1, so a host on the same network (IP sense of network) as your host could possibly get the datagram to your host's TCP/IP stack. This is where your local firewal comes into play. Your local (host) firewall should have a rule that discards IP datagrams destined for 127.0.0.0/8 coming into any interface other than lo0 (or the equivalent loopback interface). If your host either 1) has such firewall rules in place or 2) exists on its own network (or shared with only completely trusted hosts) and behind a well configured router, you can safely just bind to 127.0.0.1 and be fairly certain any datagrams you receive on the socket came from the local machine. The prior answers address how to open and bind to 127.0.0.1.
If you do sock.bind((port,'127.0.0.1')) it will only listen on localhost, and not on other interfaces, so that's all you need.
Related
I'm trying to make a python function that scans a range of addresses. I started a socket and pass the socket as an argument to the function that connects to it:
def scan(socket, address, port):
c = socket.connect_ex((address, port))
print(c)
then I call scan for each address, each in its own thread. I'm getting Error 114: Operation already in progress..
Do I need to start a new socket for each connection? I'm trying to read about socket reusage, and I found that there exists flags like SO_ADDREUSE or something like that. I tried to insert but it didn't work.
I'm trying to think how a socket works. I think the moment I create one, it choses a tcp source port, and then when I create a connection, it sends to a destination port. I think I can't reuse the same socket because the source port would be the same for all destination ports, so the clients would answer to the same port and would cause confusion.
So do I need to create a new socket for each connection?
You can not connect stream socket multiple times.
One of the connect possible errors is EISCONN.
The socket is already connected.
This goes for stream sockets.
man bind also has this:
[EINVAL] The socket is already bound to an address, and the
protocol does not support binding to a new address; or
the socket has been shut down.
Again, this goes for stream sockets.
From the man connect:
Generally, stream sockets may successfully connect() only once; datagram sockets may use connect() multiple times to change their association.
I made emphasis on the important line.
stream sockets can not be connected multiple times. datagram sockets can be connected multiple times. Generally speaking, BSD sockets have multiple protocols, types, domains avaible. You shall read documentation for your particular case.
P.S Get yourself familiar with the readings that were suggested in the comment to your question. That will explain enough to manipulate socket family of functions.
Do I need to start a new socket for each connection?
Yes.
I'm trying to read about socket reusage
There is no such thing as 'socket reusage'. There is port reuse. Not the same thing. You cannot reconnect an existing socket once you've tried to connect it, even if the connect attempt failed.
I found that there exists flags like SO_ADDREUSE or something like that
SO_REUSEADDR means to reuse the port. Not the socket.
I'm trying to think how a socket works. I think the moment I create one, it choses a tcp source port,
Between creating a socket using the socket() system call and using it to create an outgoing connection with the connect() system call, there is an opportunity to optionally use the bind() system call to set source IP address and/or port if you want to. If you don't use bind(), the operating system will automatically bind the socket to the first available port in the appropriate range when you use the connect() system call. In this case, the source IP address is normally selected to match the network interface that provides the shortest route to the specified destination according to the routing table.
At least, that's how it works at the system call level. Some programming languages or libraries may choose to combine some of these operations into one.
To your actual question, man 7 ip says:
A TCP local socket address that has been bound is unavailable for some
time after closing, unless the SO_REUSEADDR flag has been set. Care
should be taken when using this flag as it makes TCP less reliable.
The idea is to delay the re-use of a port until any possible re-sent copies of packages that belonged to the closed connection have for sure expired on the network.
According to the bind() man page, trying to re-bind a socket that is already bound to an address will result in an EINVAL error. So "recycling" a socket using bind(socket, INADDR_ANY, 0) (after ending a connection that used SO_REUSEADDR) does not seem to be possible.
And even if that would be possible, when you're using multiple threads on a modern multi-core system, you end up (very probably) doing multiple things in parallel. A socket can be used for just one outgoing connection at a time. Each of your scan threads will need its own socket.
I am making a python server that should handle connection from remote hosts on port 9090 and connection only from localhost (a GUI sending data) on port 9091.
I think the problem is in socket.listen(backlog), because this blocks everything.
So there is a way to make socket1.listen(backlog) and socket2.listen(backlog) running at the same time?
Also, is it a good, and above all safe, idea to make a socket for receiving data from a GUI (written in Java)?
This can be done in C: Listen to multiple ports from one server
and C++: Create multiple listening sockets
so you should be able to do this in Python.
The only thing that would make this not safe is if your server performs a dangerous activity in response to the GUI, but you can probably add code to check and protect against hacks.
I am using python2.7.3, and as stated in my comment I couldn't reproduce the listen blocking issue.
Notes:
for the simplicity's sake I am only using IPv4 sockets
I included Python's prompter in the code to show that it didn't block
regarding eventual security concerns: the socket (s_local) listening on 9091, only accepts connections from localhost (from other hosts will fail)
Here's a snippet from a CentOS5 machine:
>>> import socket
>>> s_global = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s_global.bind(("", 9090))
>>> backlog = 0xFF
>>> s_global.listen(backlog)
>>> s_local = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s_local.bind(("127.0.0.1", 9091))
>>> s_local.listen(backlog)
>>>
and here's some output on the same machine:
netstat -an | grep 909
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9091 0.0.0.0:* LISTEN
As seen, both server sockets are listening for connections.
Now, if something is wrong in my environments (some reading revealed that by default sockets are blocking by default), after creating each socket, you could add (as specified in one comment) the following line(s):
for s_global
s_global.setblocking(0)
for s_local
s_local.setblocking(0)
Now in order to receive incoming connections from both sockets, you should use Python's select module (used for async IO) with no need for other threads; your server app should constantly listen for connections, something like:
TIMEOUT = 1 #seconds
terminate = False # variable to be modified externally to end the loop
listening_sockets = [s_global, s_local]
while (not terminate):
notified_socket_list = select.select(listening_sockets, [], [], TIMEOUT)[0]
for notified_socket in notified_socket_list:
incoming_socket, addr = notified_socket.accept()
handle(incoming_socket, addr)
Note: worth being mentioned: epoll(select.epoll) is performance-wise preferred to select(select.select) but i haven't tried it yet.
Now, regarding the connections handling function(handle): if your server is designed only for educative purposes, you can leave it as it is, but if your server is designed to accept lots of simultaneous connections and lots of data exchange from/to each client, you should handle each connection in a separate thread(this is a general advice that doesn't apply to Python because of its GIL, sometimes threads only slow things down)/process as done in SocketServer.py - part of Python's standard library.
#Edit1 - reply to 1st comment:
It's simple, all you have to do is modify a little bit the inner for loop:
for notified_socket in notified_socket_list:
incoming_socket, addr = notified_socket.accept()
if notified_socket == s_global:
handle_global(incoming_socket, addr)
else:
handle_local(incoming_socket, addr)
or you can use my original solution and do the differentiate the connections in handle: the addr argument contains the (local) port that the socket connected to; you can test it and depending on its value (9090 or 9091) handle the connection differently.
I am using the logging utility on my local network to get log data from one python script to another. The server and client scripts works on the same machine, but does not work from a different machine.
The client script snippet, running on IP "192.168.1.9" is-
import logging, logging.handlers
rootLogger = logging.getLogger('')
rootLogger.setLevel(logging.DEBUG)
socketHandler = logging.handlers.SocketHandler('192.168.1.10', 50005)
The server script snippet, running on IP "192.168.1.10" locally is -
def __init__(self, host='localhost', port=50005, handler=LogRecordStreamHandler):
socketserver.ThreadingTCPServer.__init__(self, (host, port), handler)
When I run this, both the client and server are unresponsive, as if no message was ever sent.
My iptables is empty (default), and there is nothing except a simple switch between the two machines on the network. I can remotely use MySQL just fine. Even a basic TCP socket connection at a random open port worked fine. So what could be going wrong here? Is it something to do with the logger code above, or could be an entirely different networking reason?
When you construct a socketserver.TCPServer, the (host, port) ultimately gets passed to socket.socket.bind.
The Socket Programming HOWTO explains a bit about what this means, but the short version is that the point of specifying a host is to tell the listener socket which interface(s) to listen to. It resolves the name to an address, asks your OS which interface owns that address, and listens only to that interface.
I'll use my Macintosh as an example here, but the details will be pretty much the same anywhere, except some slight differences in the names.
'localhost' resolves to '127.0.0.1', which belongs to an interface named 'lo0', the "loopback" interface, which listens to nothing but connections from the same machine on the localhost address.
'192.168.1.10' belongs to an interface named 'en0', an Ethernet adapter that listens to everything coming over my Ethernet port. So, that's why it works for you. But it's still not what you (probably) want.
'0.0.0.0' is a special address that belongs to every interface. So this may be what you want.
But notice that specifying an IPv4 address—even '0.0.0.0'—means, at least on some platforms, that you'll only listen for IPv4 connections. If you want to handle IPv6 as well, how do you do that? Well, you can't do it on all platforms, but on those you can, it's just ''.
(On some platforms, that still won't work for IPv6; you need to actually create IPv6 and IPv4 sockets, and bind them to the specific IPv6 and IPv4 "any" addresses separately. But on such platforms, Python still lets you use '' for both "any" addresses, and the default socket will be IPv4, so work-case scenario, this works just as well as '0.0.0.0'.)
So, most likely, you just want:
def __init__(self, host='', port=50005, handler=LogRecordStreamHandler):
I have this basic UDP forward script in Python 3.
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
class Forward(DatagramProtocol):
def __init__(self, targetTuples):
print ('in init, targetTuples are ', targetTuples)
self._targetTuples = targetTuples
def datagramReceived(self, data, hostAndPort):
print ('self._targetTuples is ', self._targetTuples)
for (targetHost, targetPort) in self._targetTuples:
self.transport.write(data, (targetHost, targetPort))
reactor.listenUDP(5005, Forward([('10.35.203.24', 5000), ('10.35.200.251', 5005)]))
reactor.run()
So I'm listening on port 5005 UDP, and forwarding those packets to the two IP addresses and different ports.
My question is this -
How do I preserve the original IP address that twisted gets while listening on port 5005?
Source IP (10.1.1.1) --> Twisted (10.30.1.1) --> Multiple Destinations
How can I get Multiple Destinations to see the packet source preserved from the Source IP of (10.1.1.1) ?
When sending UDP datagrams using the BSD socket API (around which, as a first approximation, Twisted is a wrapper), the source address is set to the address the socket is bound to. You can specify the IP of the bind address for a UDP socket in Twisted by passing a value for the interface argument to reactor.listenTCP. However, you are typically restricted in what addresses you are allowed to bind to. Typically the only values allowed are addresses which are assigned to a local network interface. If you are forwarding traffic for 10.1.1.1 but you are on host 10.30.1.1 then you probably cannot set the source address of the UDP packets you send to 10.1.1.1 because that address isn't assigned to a network interface on the host doing the forwarding. If you assigned it to one, routing on your network would probably break in weird ways because then two different hosts would have the same IP address.
This doesn't mean it's not possible to do what you want - but it does mean you probably cannot do it using Twisted's basic UDP support. There are a number of other approaches you could take. For example, you can rewrite source addresses using iptables on Linux. Or you can operate on the IP level and parse and generate full UDP datagrams yourself letting you specify any source address you want (you can do this with Twisted on Linux, too, using twisted.pair.tuntap). There are probably a number of other ways. The best solution for you may depend on which platforms you're targeting and the particular reasons you want to do this kind of forwarding.
I'm in the process of implementing a service -- written in Python with the Twisted framework, running on Debian GNU/Linux -- that checks the availability of SIP servers. For this I use the OPTIONS method (a SIP protocol feature), as this seems to be a commonplace practice. In order to construct correct and RFC compliant headers, I need to know the source IP address and the source port for the connection that is going to be established. [How] can this be done with Twisted?
This is what I tried:
I subclassed protocol.DatagramProtocol and within startProtocol(self) I used self.transport.getHost().host and self.transport.getHost().port. The latter is indeed the port that's going to be used, whereas the former only yields 0.0.0.0.
I guess that at this point Twisted doesn't [yet?] know which interface and as such which source IP address will be used. Does Twisted provide a facility that could help me with this or do I need to interface with the OS (routing) in a different way? Or did I just use self.transport.getHost().host incorrectly?
For the sake of completeness I answer my own question:
Make sure you use connect() on the transport before trying to determine the host's source IP address. The following excerpt shows the relevant part of a protocol implementation:
class FooBarProtocol(protocol.DatagramProtocol):
def startProtocol(self):
self.transport.getHost().host # => 0.0.0.0
self.transport.connect(self.dstHost, self.dstPort)
self.transport.getHost().host # => 192.168.1.102
If you are using UDP then the endpoint is determined by either:
calling bind() on the socket and explicitly giving it an address
sending a packet
If you want a few more details, check this response.
The problem is that I'm not that familiar with twisted. From what I can tell by a quick perusal of the source, it looks like you might want to use a reactor like t.i.d.SelectReactor instead. This is what t.n.d.DNSDatagramProtocol does under the hood.
If you take twisted out of the picture, then the following snippet shows what is going on:
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 0)
<socket._socketobject object at 0x10025d670>
>>> s.getsockname() # this is an unbound or unnamed socket
('0.0.0.0', 0)
>>> s.bind( ('0.0.0.0', 0) ) # 0.0.0.0 is INADDR_ANY, 0 means pick a port
>>> s.getsockname() # IP is still zero, but it has picked a port
('0.0.0.0', 56814)
Get the host name is a little trickier if you need to support multiple network interfaces or IPv4 and IPv6. If you can make the interface used configurable, then pass it in as the first member of the tuple to socket.bind() and you are set.
Now the hard part is doing this within the confines of the abstractions that twisted provides. Unfortunately, I can't help a whole lot there. I would recommend looking for examples on how you can get access to the underlying socket or find a way to pass the socket information into the framework.
Good luck.
Did you see if that you want to do is possible with the SIP implementation that is part of Twisted?
In any case, how you set the source address and port for UDP in Twisted is quite similar to how you set them without Twisted. In Twisted, reactor.listenUDP(port, protocol, interface) binds an UDP socket to a specific port and interface and handles the received datagrams to your protocol. Inside the protocol, self.transport.write(msg, addr) sends a datagram to addr using the address that the protocol is bound to as source address.
Reading your question again, I think the only part you were missing was passing interface to reactor.listenUDP(...).