Giving the port forward object
from kubernetes.stream import stream, portforward
pf = portforward(
k8s_client_v1.connect_get_namespaced_pod_portforward,
pod_name,
name_space,
ports=port
)
pf.socket(port)
pf is a socket object from AF_UNIX family.
I need a better understating whether another AF_INET family socket has to be instantiated in oder to achieve the same functionality as kubectl client has, ex: kubectl port-forward $pod_name $port
I appreciate in advance if someone could share any snippets of this type of implementation.
At this moment creating port forward object was inspired from here
I am not an expert on sockets, but I managed to wrap the socket from the port forwarding with a http.client.HTTPConnection. I wanted to patch requests, but I could not make it work.
Anyway, this way the http requests can be implemented more conveniently in a high level fashion. Not sure if you were looking for that, but I was. To query pod names, ports and more you can check the api reference for kubernetes and look up the corresponding method in the python package.
from http.client import HTTPConnection
from kubernetes import client, config
from kubernetes.stream import portforward
from kubernetes.stream.ws_client import PortForward
class ForwardedKubernetesHTTPConnection(HTTPConnection):
def __init__(self, forwarding: PortForward, port: int):
super().__init__("127.0.0.1", port)
self.sock = forwarding.socket(port)
def connect(self) -> None:
pass
def close(self) -> None:
pass
if __name__ == "__main__":
config.load_kube_config()
client_v1 = client.CoreV1Api()
name_space = "my-namespace" # or get from api somehow
pod_name = "my-pod" # or get from api somehow
port = 80
pf = portforward(
client_v1.connect_get_namespaced_pod_portforward,
pod_name,
name_space,
ports=str(port)
)
conn = ForwardedKubernetesHTTPConnection(pf, port)
conn.request("GET", "/my/url") # will fail for other methods
resp = conn.getresponse()
print(resp.status)
print(resp.headers)
print(resp.read())
conn.request("GET", "/my/other/url")
# ...
Thanks to your code snipped, I could get this to work for me - which means that your code basically works: I have successfully used it to send TCP data on a custom port to a kubernetes services (haven't tried UDP, though). It seems to work quite similar but not exactly the same as kubectl port-forward.
The main difference is that it doesn't actually open an AF_INET socket on your system, but an AF_UNIX socket, which means, that it basically is addressed via a file name instead of a tuple of an IP address and port number. So it really depends on what you try to achieve.
If you just want to write to a socket from a Python script that is connected to a specific port of a service: This is the solution. It doesn't really make a difference for you that the kubernetes API provides you an AF_UNIX socket.
However, if you want to open a local port that connects to the kubernetes service port (e.g. use local tools to connect to a kubernetes service like a CLI based database management tool), this is only half of what you want to achieve. After some time playing around, including trying hacks like socket.fromfd(unix_sock.fileno(), socket.AF_INET, socket.SOCK_STREAM), I came to the conclusion that a straight forward approach to "convert" the unix socket to a internet socket isn't possible.
A solution, I would suggest is to implement a simple port forwarding in Python. That would mean running a thread that opens an internet socket on the local port you'd like to use, receive data on that and simply forward it to the unix socket. It doesn't feel very elegant and probably isn't very efficient, either, but it should do the job.
Another alternative would be using an external tool like socat if available to connect the two sockets.
Related
I am using the logging utility on my local network to get log data from one python script to another. The server and client scripts works on the same machine, but does not work from a different machine.
The client script snippet, running on IP "192.168.1.9" is-
import logging, logging.handlers
rootLogger = logging.getLogger('')
rootLogger.setLevel(logging.DEBUG)
socketHandler = logging.handlers.SocketHandler('192.168.1.10', 50005)
The server script snippet, running on IP "192.168.1.10" locally is -
def __init__(self, host='localhost', port=50005, handler=LogRecordStreamHandler):
socketserver.ThreadingTCPServer.__init__(self, (host, port), handler)
When I run this, both the client and server are unresponsive, as if no message was ever sent.
My iptables is empty (default), and there is nothing except a simple switch between the two machines on the network. I can remotely use MySQL just fine. Even a basic TCP socket connection at a random open port worked fine. So what could be going wrong here? Is it something to do with the logger code above, or could be an entirely different networking reason?
When you construct a socketserver.TCPServer, the (host, port) ultimately gets passed to socket.socket.bind.
The Socket Programming HOWTO explains a bit about what this means, but the short version is that the point of specifying a host is to tell the listener socket which interface(s) to listen to. It resolves the name to an address, asks your OS which interface owns that address, and listens only to that interface.
I'll use my Macintosh as an example here, but the details will be pretty much the same anywhere, except some slight differences in the names.
'localhost' resolves to '127.0.0.1', which belongs to an interface named 'lo0', the "loopback" interface, which listens to nothing but connections from the same machine on the localhost address.
'192.168.1.10' belongs to an interface named 'en0', an Ethernet adapter that listens to everything coming over my Ethernet port. So, that's why it works for you. But it's still not what you (probably) want.
'0.0.0.0' is a special address that belongs to every interface. So this may be what you want.
But notice that specifying an IPv4 address—even '0.0.0.0'—means, at least on some platforms, that you'll only listen for IPv4 connections. If you want to handle IPv6 as well, how do you do that? Well, you can't do it on all platforms, but on those you can, it's just ''.
(On some platforms, that still won't work for IPv6; you need to actually create IPv6 and IPv4 sockets, and bind them to the specific IPv6 and IPv4 "any" addresses separately. But on such platforms, Python still lets you use '' for both "any" addresses, and the default socket will be IPv4, so work-case scenario, this works just as well as '0.0.0.0'.)
So, most likely, you just want:
def __init__(self, host='', port=50005, handler=LogRecordStreamHandler):
I'm trying to connect a socket to an endpoint until the socket receives data from that endpoint. This is because the endpoint might not exist at that time.
Currently the connect stalls, i'm guessing because it can't resolve the hostname and that takes a while.
Is there any way to set a timeout on a socket connect?
import zmq
import time
endpoint = 'tcp://doesnt_exist:12345'
ctx = zmq.Context.instance()
s = ctx.socket(zmq.SUB)
t = time.time()
try:
s.connect(endpoint)
except Exception:
pass
print time.time() - t
If you provide a host name to connect, ZeroMQ uses synchronous DNS resolution via a call to getaddrinfo, which is why you see the connect call blocking.
If you really need to connect in controllable way, I suggest you do DNS resolve on your own, using one of the asynchronous DNS resolvers already available for Python (check this example based on pyuc/pycares).
Also see my reply to similar question.
The problem is not the connection, but the DNS lookup. The blocking is done at the OS level, on the gethostbyname call.
Since the timeout is controlled by the OS, working around it is hard (but feasible). My suggestion is that you simply hardcode the IP
I need to connect from python client to a tornado server using an url like ws://localhost:8006/user?id=666.
I have tried something like this:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect('ws://localhost:8006/user?id=666')
print s.recv(1024)
s.close
...and got the following error:
TypeError: getsockaddrarg: AF_INET address must be tuple, not str
Thanks
You don't connect to a remote socket with a namespace and parameters. You connect with a host and port. Period. Python's socket module is a thin wrapper around native sockets. It doesn't know anything about protocols like http, or ws.
You can connect to the host and port that your web socket uses. Then you can send any parameters you like...but it's not going to understand them, unless you use the websocket protocol.
Your best bet is to use a websocket client that someone else has written (see some answers here)
If you want to write it yourself, here is a minimal example.
You must convert string to tuple. Pratic example:
x = "(1,2,3)"
t = tuple(int(v) for v in re.findall("[0-9]+", x))
I have a python program with many threads. I was thinking of creating a socket, bind it to localhost, and have the threads read/write to this central location. However I do not want this socket open to the rest of the network, just connections from 127.0.0.1 should be accepted. How would I do this (in Python)? And is this a suitable design? Or is there something a little more elegant?
Given a socket created with socket.socket(), you can use bind() before listening:
socket.bind(('127.0.0.1', 80))
Using the address 127.0.0.1 indicates that the socket should bind to the local interface only.
http://www.amk.ca/python/howto/sockets/
Shows some socket example. This tidbit is interesting to you I think
we used socket.gethostname() so that the socket would be visible to the outside world. If we had used s.bind(('', 80)) or s.bind(('localhost', 80)) or s.bind(('127.0.0.1', 80)) we would still have a "server" socket, but one that was only visible within the same machine.
I guess there is your answer (see below for correction)
As to the validity of using this method for thread communications. I'm not sure how well this handles multiple threads and reading/writing
EDIT
There seems to be a python recipe linked below that does some inter-thread communication
http://code.activestate.com/recipes/491281/
Have fun!
EDIT
The article is incorrect and as pointed out "s.bind(('', 80)) will bind to INADDR_ANY"
If you are running on a UNIX-based system, you might want to consider using UNIX Domain Sockets instead of Internet sockets. I think something like the following should work:
>>> # in one window/shell
>>> import socket
>>> sd = socket.socket(socket.AF_UNIX)
>>> sd.bind('/path/to/my/socket')
>>> sd.listen(5)
>>> (client,addr) = sd.accept()
>>> client.recv(1024)
'hello'
>>>
>>> # in a different shell
>>> import socket
>>> sd = socket.socket(socket.AF_UNIX)
>>> sd.connect('/path/to/my/socket')
>>> sd.send('hello')
You might want to use the queue module from the standard library instead. It's designed specifically to facilitate communication between threads. A quote from the docs:
The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics. It depends on the availability of thread support in Python; see the threading module.
notionOn TCP/IP networks 127.0.0.0/8 is a non-routeable network, so you should not be able to send an IP datagram destined to 127.0.0.1 across a routed infrastructure. The router will just discard the datagram. However, it is possible to construct and send datagrams with a destination address of 127.0.0.1, so a host on the same network (IP sense of network) as your host could possibly get the datagram to your host's TCP/IP stack. This is where your local firewal comes into play. Your local (host) firewall should have a rule that discards IP datagrams destined for 127.0.0.0/8 coming into any interface other than lo0 (or the equivalent loopback interface). If your host either 1) has such firewall rules in place or 2) exists on its own network (or shared with only completely trusted hosts) and behind a well configured router, you can safely just bind to 127.0.0.1 and be fairly certain any datagrams you receive on the socket came from the local machine. The prior answers address how to open and bind to 127.0.0.1.
If you do sock.bind((port,'127.0.0.1')) it will only listen on localhost, and not on other interfaces, so that's all you need.
I'm in the process of implementing a service -- written in Python with the Twisted framework, running on Debian GNU/Linux -- that checks the availability of SIP servers. For this I use the OPTIONS method (a SIP protocol feature), as this seems to be a commonplace practice. In order to construct correct and RFC compliant headers, I need to know the source IP address and the source port for the connection that is going to be established. [How] can this be done with Twisted?
This is what I tried:
I subclassed protocol.DatagramProtocol and within startProtocol(self) I used self.transport.getHost().host and self.transport.getHost().port. The latter is indeed the port that's going to be used, whereas the former only yields 0.0.0.0.
I guess that at this point Twisted doesn't [yet?] know which interface and as such which source IP address will be used. Does Twisted provide a facility that could help me with this or do I need to interface with the OS (routing) in a different way? Or did I just use self.transport.getHost().host incorrectly?
For the sake of completeness I answer my own question:
Make sure you use connect() on the transport before trying to determine the host's source IP address. The following excerpt shows the relevant part of a protocol implementation:
class FooBarProtocol(protocol.DatagramProtocol):
def startProtocol(self):
self.transport.getHost().host # => 0.0.0.0
self.transport.connect(self.dstHost, self.dstPort)
self.transport.getHost().host # => 192.168.1.102
If you are using UDP then the endpoint is determined by either:
calling bind() on the socket and explicitly giving it an address
sending a packet
If you want a few more details, check this response.
The problem is that I'm not that familiar with twisted. From what I can tell by a quick perusal of the source, it looks like you might want to use a reactor like t.i.d.SelectReactor instead. This is what t.n.d.DNSDatagramProtocol does under the hood.
If you take twisted out of the picture, then the following snippet shows what is going on:
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 0)
<socket._socketobject object at 0x10025d670>
>>> s.getsockname() # this is an unbound or unnamed socket
('0.0.0.0', 0)
>>> s.bind( ('0.0.0.0', 0) ) # 0.0.0.0 is INADDR_ANY, 0 means pick a port
>>> s.getsockname() # IP is still zero, but it has picked a port
('0.0.0.0', 56814)
Get the host name is a little trickier if you need to support multiple network interfaces or IPv4 and IPv6. If you can make the interface used configurable, then pass it in as the first member of the tuple to socket.bind() and you are set.
Now the hard part is doing this within the confines of the abstractions that twisted provides. Unfortunately, I can't help a whole lot there. I would recommend looking for examples on how you can get access to the underlying socket or find a way to pass the socket information into the framework.
Good luck.
Did you see if that you want to do is possible with the SIP implementation that is part of Twisted?
In any case, how you set the source address and port for UDP in Twisted is quite similar to how you set them without Twisted. In Twisted, reactor.listenUDP(port, protocol, interface) binds an UDP socket to a specific port and interface and handles the received datagrams to your protocol. Inside the protocol, self.transport.write(msg, addr) sends a datagram to addr using the address that the protocol is bound to as source address.
Reading your question again, I think the only part you were missing was passing interface to reactor.listenUDP(...).