I'm trying to connect a socket to an endpoint until the socket receives data from that endpoint. This is because the endpoint might not exist at that time.
Currently the connect stalls, i'm guessing because it can't resolve the hostname and that takes a while.
Is there any way to set a timeout on a socket connect?
import zmq
import time
endpoint = 'tcp://doesnt_exist:12345'
ctx = zmq.Context.instance()
s = ctx.socket(zmq.SUB)
t = time.time()
try:
s.connect(endpoint)
except Exception:
pass
print time.time() - t
If you provide a host name to connect, ZeroMQ uses synchronous DNS resolution via a call to getaddrinfo, which is why you see the connect call blocking.
If you really need to connect in controllable way, I suggest you do DNS resolve on your own, using one of the asynchronous DNS resolvers already available for Python (check this example based on pyuc/pycares).
Also see my reply to similar question.
The problem is not the connection, but the DNS lookup. The blocking is done at the OS level, on the gethostbyname call.
Since the timeout is controlled by the OS, working around it is hard (but feasible). My suggestion is that you simply hardcode the IP
Related
Giving the port forward object
from kubernetes.stream import stream, portforward
pf = portforward(
k8s_client_v1.connect_get_namespaced_pod_portforward,
pod_name,
name_space,
ports=port
)
pf.socket(port)
pf is a socket object from AF_UNIX family.
I need a better understating whether another AF_INET family socket has to be instantiated in oder to achieve the same functionality as kubectl client has, ex: kubectl port-forward $pod_name $port
I appreciate in advance if someone could share any snippets of this type of implementation.
At this moment creating port forward object was inspired from here
I am not an expert on sockets, but I managed to wrap the socket from the port forwarding with a http.client.HTTPConnection. I wanted to patch requests, but I could not make it work.
Anyway, this way the http requests can be implemented more conveniently in a high level fashion. Not sure if you were looking for that, but I was. To query pod names, ports and more you can check the api reference for kubernetes and look up the corresponding method in the python package.
from http.client import HTTPConnection
from kubernetes import client, config
from kubernetes.stream import portforward
from kubernetes.stream.ws_client import PortForward
class ForwardedKubernetesHTTPConnection(HTTPConnection):
def __init__(self, forwarding: PortForward, port: int):
super().__init__("127.0.0.1", port)
self.sock = forwarding.socket(port)
def connect(self) -> None:
pass
def close(self) -> None:
pass
if __name__ == "__main__":
config.load_kube_config()
client_v1 = client.CoreV1Api()
name_space = "my-namespace" # or get from api somehow
pod_name = "my-pod" # or get from api somehow
port = 80
pf = portforward(
client_v1.connect_get_namespaced_pod_portforward,
pod_name,
name_space,
ports=str(port)
)
conn = ForwardedKubernetesHTTPConnection(pf, port)
conn.request("GET", "/my/url") # will fail for other methods
resp = conn.getresponse()
print(resp.status)
print(resp.headers)
print(resp.read())
conn.request("GET", "/my/other/url")
# ...
Thanks to your code snipped, I could get this to work for me - which means that your code basically works: I have successfully used it to send TCP data on a custom port to a kubernetes services (haven't tried UDP, though). It seems to work quite similar but not exactly the same as kubectl port-forward.
The main difference is that it doesn't actually open an AF_INET socket on your system, but an AF_UNIX socket, which means, that it basically is addressed via a file name instead of a tuple of an IP address and port number. So it really depends on what you try to achieve.
If you just want to write to a socket from a Python script that is connected to a specific port of a service: This is the solution. It doesn't really make a difference for you that the kubernetes API provides you an AF_UNIX socket.
However, if you want to open a local port that connects to the kubernetes service port (e.g. use local tools to connect to a kubernetes service like a CLI based database management tool), this is only half of what you want to achieve. After some time playing around, including trying hacks like socket.fromfd(unix_sock.fileno(), socket.AF_INET, socket.SOCK_STREAM), I came to the conclusion that a straight forward approach to "convert" the unix socket to a internet socket isn't possible.
A solution, I would suggest is to implement a simple port forwarding in Python. That would mean running a thread that opens an internet socket on the local port you'd like to use, receive data on that and simply forward it to the unix socket. It doesn't feel very elegant and probably isn't very efficient, either, but it should do the job.
Another alternative would be using an external tool like socat if available to connect the two sockets.
Is there a way to detect whether a socket.socket instance passed to a function that already has the connection setup is a server-side socket (Executed socket.accept()) or is a client-side socket (Executed socket.connect().
Basically I'm looking for something to check about the socket to implement is_server_socket being used below:
def func(sock):
if is_server_socket(sock):
print("I'm a server!")
else:
print("I'm a client!")
A connected socket has a peer while a socket which is not connected does not have a peer. Thus you can use getpeername to distinguish between connected and not connected socket.
Note that not every socket which is not connected is a server socket. It might simply be a socket which is not connected yet (i.e. socket created but connect not called). But if you need to only distinguish between a socket which you can call accept on and a socket which you got as the result of accept than getpeername is enough. Still, it would be better to keep this information in a local variable instead of calling getpeername all the time since this is a system call and thus comparably expensive.
In MacOS, I used to go with the following solution:
import socket
import time
server = socket.socket()
server.bind(('127.0.0.1', 7777))
time.sleep(5)
server.listen(1)
If another socket tries to connect to this address, it will be hung up until server calls listen.
But this is not the case in Linux, trying to connect before the listen call results in Connection Refused immediately.
So is there a better way to emulate connect timeout when testing?
I don't know the specifics of what you're testing, but you perhaps could go ahead and do an accept, and then sleep - you'll appear to get a connection in the client, but it doesn't do anything useful (for a while). Or play with your firewall settings :)
You can use poll or select. It returns until n seconds , which it is specified by you, or when it is writable, which means connectable for clients.
http://docs.python.org/3/library/multiprocessing.html?highlight=poll#multiprocessing.Connection.poll
So I'm making a basic "ping" application using UDP for an assignment, and everything is working except the implementation of socket.settimeout().
I can't seem to figure out why, but it has to do with the bound socket. It might just be a quirk in Python but I'd like to confirm it before I document it.
I'm not looking for a functional code answer (that'd be cheating), but rather why what I have is broken. (e.g: some undocumented reason that Python doesn't like client/server on same machine etc)
Python Socket Timeout Details: http://docs.python.org/2/library/socket.html#socket.socket.settimeout
In the code represented below, the communication with server running on same machine is successful, but only when the client does not bind to the socket. But if it does not bind to the socket, the timeout system fails (this being tested by turning off the server, in which all ten timeouts get printed immediately and at once).
Note: Code is not ideal, but this is a networking theory class and not a programming class. It just has to work in the end. I could hand it in right now and get an A, but I want to understand why the timeout function does not work.
EDIT: To clarify an issue, the use of Bind in the client was after seeing the server code had it before I realized UDP doesn't need it, but it happened to make the timeout function work properly, but breaks the normal operation.
Could the socket.settimeout() declaration only work for TCP maybe?
Client Code (which has the timeout):
import socket
import time
import select
data = "Re-verify our range to target... one ping only. \n"
addrDest = ("127.0.0.1",8675)
addrLocal = ("127.0.0.1",12345)
totalTime = 0
averageTime = 0
totalPings = 0
#timeout_seconds = 1.0
UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
UDPSock.bind(addrLocal)
# adding bind here even though it's UDP makes timeout work,
# but breaks normal functionality
UDPSock.settimeout(1)
while (totalPings < 10):
totalPings = (totalPings + 1)
start = time.clock()
str_list = []
str_list.append(str(totalPings))
str_list.append(" ")
str_list.append(str(start))
str_list.append(" ")
str_list.append(data)
dataOut = ''.join(str_list)
UDPSock.sendto(dataOut,addrDest)
try:
dataIn,addrIn = UDPSock.recvfrom(1024)
print dataIn.strip(),"\n",addrIn
elapsed = ((time.clock() - start) * 1000)
print elapsed, " ms round trip"
except socket.error:
print "Connection timed out for Packet #", totalPings
Server Code:
import socket
UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
# (to all IP addresses on this system)
listen_addr = ("",8675)
UDPSock.bind(listen_addr)
# Report on all data packets received and
# where they came from in each case (as this is
# UDP, each may be from a different source and it's
# up to the server to sort this out!)
while True:
data,addr = UDPSock.recvfrom(1024)
print data.strip(),addr
UDPSock.sendto(data,addr)
Why do you need to bind to local address of the client? Will the client act as a server too at any point? If not there is no need to bind the client at all. You need a specific port only if you need your client to act as a server, if you don't call bind it will create a random port no ranging from (0 - 1023 are reserved) 1024 - 65535 (if I remember correctly) and that will be Source Port in the UDP Packet, Source Address is the Address where client runs.
According to Berkley Sockets
bind() assigns a socket to an address. When a socket is created using socket(),
it is only given a protocol family, but not assigned an address. This association with an address must be performed with the bind() system call before the socket can accept connections to other hosts
If this is a Networking class project and you are trying to implement Client-Server architecture then you should never call bind from within your client code because Client should never act as a Server and Client should connect to a listening Server not Server connecting to Client.
Update:
Bind may be required to be called from a TCP Client-Server design but not from a UDP Client-Server model because UDP is a send and forget design and doesn't have low level packet send success acknowledgement. A UDP packet will have Source Address and Port within itself.
I found the cause of the problem by removing the exception handling. There is a socket error when the server is turned off, specifically "socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host" when it tries to read from the socket.
This apparently ignores the timeout function when the socket is not bound in Python (which is why the timeout worked when I bound it).
If I run the server, but just have it not send any data (by commenting the last line), the program times out correctly when it does not receive its packet back.
I am also going to use a more specific exception handler
In the end it's just a quirk in Python and the fact that UDP is connection-less.
Also, someone mentioned the use of "select" to solve this problem. I looked into it, and I would end up with an if...else statement block which kinda works, but the native exceptions are preferred.
Thanks all.
-Jimmy
I have a HTTP client in Python which needs to use TLS. I need not only
to make encrypted connections but also to retrieve info from the
remote machine, such as the certificate issuer. I need to make
connection to many HTTP servers, often badly behaved, so I absolutely
need to have a timeout. With non-TLS connections,
mysocket.settimeout(5) does what I want.
Among the many TLS Python modules:
python-gnutls does not allow to use settimeout() on sockets because
it uses non-blocking sockets:
gnutls.errors.OperationWouldBlock: Function was interrupted.
python-openssl has a similar issue:
OpenSSL.SSL.WantReadError
The SSL module of the standard library does not work with Python
2.5.
Other libraries like TLSlite apparently does not give access to
the metadata of the certificate.
The program is threaded so I cannot use signals. I need detailed
control on the HTTP dialog so I cannot use a standard library like urllib2.
Background: this is
the survey project DNSwitness. Relevant SO threads: Timeout on a
Python function call and How to limit execution time of a function call in Python.
Although I've never used it for exactly this purpose, Twisted should do what you want. The only downside is that it's a rather large library, and you will also need to install PyOpenSSL (Twisted depends on it). If you've never used it before, Twisted's callback-based architecture can take some getting used to (you really want to read the tutorials before starting).
But aside from that, it's designed around the idea of managing a lot of connections, it of course lets you specify timeouts, reconnects, etc., and you can retrieve certificate info (see here).
I assume the problems you're having is the following, you're opening a connection using PyOpenSSL and you always get a WantReadError exception. And you can't distinguish between this error and a timeout. Consider the following example:
#!/usr/bin/python
import OpenSSL
import socket
import struct
context = OpenSSL.SSL.Context(OpenSSL.SSL.TLSv1_METHOD)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(5)
connection = OpenSSL.SSL.Connection(context,s)
connection.connect(("www.gmail.com",443))
# Put the socket in blocking mode
connection.setblocking(1)
# Set the timeout using the setsockopt
tv = struct.pack('ii', int(6), int(0))
connection.setsockopt(socket.SOL_SOCKET, socket.SO_RCVTIMEO, tv)
print "Connected to " , connection.getpeername()
print "Sate " , connection.state_string()
while True:
try:
connection.do_handshake()
break
except OpenSSL.SSL.WantReadError:
print "Exception"
pass
print "Sate " , connection.state_string()
print connection.send("koekoek\r\n")
while True:
try:
recvstr = connection.recv(1024)
break
except OpenSSL.SSL.WantReadError:
print "Exception"
pass
print recvstr
This will open an SSL connection to gmail, send an invalid string, read the response and print it. Note that:
* the connection is explicitely set to blocking-mode
* the recv timeout is explicitely set to in this case 6 seconds.
Now what will the behavior be, when the timeout occurs, the WantReadError exception will be thornw, in this case after waiting for 6 seconds. (You can remove the while True to avoid the retry, but in this case i added them for testing). The timeout set on the socket only appears to be effective in the connect() call.
An alternative would be when keeping the sockets in non-blocking mode which probably applies for the GNUTLS case as well is to perform the timekeeping yourself, you get the time when you launch the call, and in the while True, try: except WantReadError you perform the check every time yourself to see if you haven't been waiting for too long.
I would also recommend Twisted, and using M2Crypto for the TLS parts.
One simple solution could be to change the socket type depending on the operation. I tested this with gnutls and it worked:
Do settimeout() on the socket before doing connect() on the bare socket wrapped by gnutls, that way connect() is subject to the timeout as you wanted.
Make sure you remove the timeout with settimeout(None) or setblocking(1) BEFORE GnuTLS's handshake()