I have a python client that needs to talk to a remote server I manage. They communicate using zeromq. When I tested the client/server locally everything worked. But now I have the client and server deployed on the cloud, each using a different provider. My question is, what's the simplest way (that is safe) to make the connection? I'm assuming I can't pass the password over, and even if I could I'm guessing there are safer alternatives.
I know how to set an ssh connection without a password using ssh-keygen. Would that work? Would the client need to make an ssh connection with the server before sending the tcp req? If there's a python library that helps with this it'd be a big help.
Thanks!
Update:
So more than 24 hours passed and no one replied/answered. I think I'm getting closer to solve this, but not quite there yet. I added my client's key to .ssh/authorized_key on the server, and now I can ssh from the client to the server without a password. Next, I followed this post about "Tunneling PyZMQ Connections with SSH". Here's what I have in my client code:
1 context = zmq.Context()
2 socket = context.socket(zmq.REQ)
3 socket.connect("tcp://localhost:5555")
4 ssh.tunnel_connection(socket, "tcp://locahost:5555", "myuser#remote-server-ip:5555")
5 socket.send_string(some_string)
6 reply = socket.recv()
This doesn't work. I don't really understand lines 3 & 4 and I assume I do something wrong there. Also, my server (hosted on linode) has a "Default Gateway" IP and a "Public IP" -- in the tunnel connection I only specify the public ip, which is also the ip I use to ssh to the machine.
Indeed, ZMQ way is - tunnelling connection with the SSH. Your example is exactly what needs to be done, except that one should either use connect or tunnel_connection, not both.
Also, when specifying server to connect to, make sure to define the SSH port, not the ZMQ REP socket port. That is, instead of myuser#remote-server-ip:5555 you might try myuser#remote-server-ip or myuser#remote-server-ip:22.
import zmq
import zmq.ssh
context = zmq.Context()
socket = context.socket(zmq.REQ)
zmq.ssh.tunnel_connection(socket, "tcp://locahost:5555", "myuser#remote-server-ip")
socket.send(b"Hello")
reply = socket.recv()
Finally, make sure you've installed either pexpect or paramiko - they will do the tunnelling actually. Note that if you're using Windows, paramiko is the only solution which will work - pexpect openssh tunnelling won't work on Windows.
If you use paramiko instead of pexpect, make sure to set paramiko=True in the tunnel_connection arguments.
I have found ssh in Python to be iffy at best, even with paramiko and fabric libraries, so to debug, you might try setting up a tunnel separately, just to see if that's the issue with the broken connection.
For example:
ssh myuser#remote-server-ip -L 5050:localhost:5555 -N
This says: connect to myuser#remote-server-ip, and whenever I request a connection to localhost:5050 on my machine, forward it across the ssh connection so that the server at remote-server-ip thinks it's receiving a connection from localhost:5555.
-L constructs the tunnel, and -N means don't do anything else on the connection.
With that running in another shell, e.g., a different Terminal window, on your local development machine, try to connect to a zeromq server at localhost:5050, which will actually be the zeromq running on the remote server.
You could use 5555:localhost:5555 in the ssh command above, but I find that can be confusing and often conflicts with a local copy of the same service.
Related
I'll try be concise, but please let me know if I can provide any more helpful pieces of information.
I have client and server Python programs, and they work fine when ran on the same machine, and when the client connects to my machine's local IP (not 127.0.0.1, but the IP assigned to my machine). I have not been able to get this to work with my public IP.
I get a [Errno 61] Connection refused error when I try to get the client to connect to my router's public IP address. My server binds to all interfaces using bind(("0.0.0.0", 50000)), and I already set up port forwarding for my router. I verified that the program is listening on that port by running netstat -an | grep LISTEN and finding the following line:
tcp4 0 0 *.50000 *.* LISTEN
I can also seemingly reach the port through an online port checking tool, which shows that the port is open when I am running my program, and closed when I close that program. My program also registers the connection from this tool.
The fact that my program accepts the connection from the port checking tool gives me the impression that my client code is missing something, but I can't find any answers. It might be worth noting that I am still running my server and client code on the same machine, but I'm not sure why that would derail things. Here's the code I use to connect on the client side:
tcp_client = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)
tcp_client.connect(('my_public_ip', 50000))
Are there any diagnostic steps that I can follow to narrow down my issue?
Before you spend any more time on this, try connecting to your public ip from a computer outside your home network. Spend a couple of dollars on an AWS instance for an hour if you have to, or try connecting from a friend's machine, whatever. It will probably work just fine.
I suspect the problem is simply that you cannot, from inside your home network, connect to your router's public ip address. I tried the same thing with my local network and ran into the same behavior.
If you really need to your public ip during development, you can just assign that as an alias to one of your local interfaces (ip addr add 1.2.3.4/32 dev eth0)...but it's probably easier just to use your an address on your local network, or just arrange for regular access to a remote system for testing.
So for personal reasons, I want to connect to a socket I create via telnetlib, I can connect to it from netcat but when I try from python it refuses the connection.
tn.write(b"/usr/bin/nc -l -p 3333 -e /bin/sh\n")
print("netcat listening on 3333 on target, trying to connect")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((tn.host, 3333))
s.send('ls')
print(s.recv(1024))
s.close()
If I can connect to it via netcat (by putting for example and infinite loop after tn.write())
You have a classic race condition:
The packet containing the command to start an nc is sent at only nanoseconds before the connection request (TCP SYN packet) of s.connect. It can even happen that the SYN gets to the remote host before the command to start nc.
You need to add proper synchronization. From the code you have shown, there is really no need to use two channels in the first place, so why not send the ls via the existing telnet channel to the remote host?
If you absolutely must use a second channel, try one of these options:
Add import time and time.sleep(5) just before calling s.connect. That way, the connection attempt will be deferred by 5 seconds. However, in general, there's no guarantee that 5 seconds is enough.
Send an additional command to wait for the port to be taken, as a sign that nc is ready.
Retry multiple times (and wait in between).
Also note that your code has three different security vulnerabilities:
You are using telnetlib, which neither ensures confidentiality nor integrity of commands, output and passwords. Use ssh instead.
You are connecting to port 3333 in plain, which neither ensures confidentiality nor integrity of commands and output. Use ssh instead.
In between nc starting and your program connecting to it, anyone can connect to port 3333 and run arbitrary commands. Use ssh instead.
What's the easiest way to establish an emulated TCP connection over HTTP with python 2.7.x?
Server: a python program on pythonanywhere (or some analogue) free hosting, that doesn't provide a dedicated ip. Client: a python program on a Windows PC.
Connection is established via multiprocessing.BaseManager and works fine when testing both server and client on the same machine.
Is there a way to make this work over HTTP with minimal additions to the code?
P.S. I need this for a grid computing project.
P.P.S. I'm new to python & network & web programming, started studying it several days ago.
Found this: http://code.activestate.com/recipes/577643-transparent-http-tunnel-for-python-sockets-to-be-u/. Appears to be exactly what I need, though I don't understand how to invoke setup_http_proxy() on server/client side. Tried setup_http_proxy("my.proxy", 8080) on both sides, but it didn't work.
Also found this: http://docs.python.org/2/library/httplib.html. What does the HTTPConnection.set_tunnel method actually do? Can I use it to solve the problem in question?
Usage on the client:
setup_http_proxy("THE_ADRESS", THE_PORT_NUMBER) # address of the Proxy, port the Proxy is listening on
The code wraps sockets to perform an initial HTTP CONNECT request to the proxy setup to get an HTTP Proxy to proxy the TCP connection for you but for that you'll need a compliant proxy (most won't allow you to open TCP connections unless it's for HTTPS).
HTTPConnection.set_tunnel basically does the same thing.
For your use case, a program running on free hosting, this just won't work. Your free host probably will only allow you to handle http requests, not have long running processes listen for tcp connections(which the code assumes).
You should rethink your need to tunnel and organize your communication to post data (and poll for messages from the server, unless they're answers to the stuff you post). Or you can purchase a VPS hosting that will give you more control over what you can host remotely.
I am running a Graphite server to monitor instruments at remote locations. I have a "perpetual" ssh tunnel to the machines from my server (loving autossh) to map their local ports to my server's local port. This works well, data comes through with no hasstles. However we use a flaky satellite connection to the sites, which goes down rather regularly. I am running a "data crawler" on the instrument that is running python and using socket to send packets to the Graphite server. The problem is, if the link goes down temporarily (or the server gets rebooted, for testing mostly), I cannot re-establish the connection to the server. I trap the error, and then run socket.close(), and then re-open, but I just can't re-establish the connection. If I quit the python program and restart it, the connection comes up just fine. Any ideas how I can "refresh" my socket connection?
It's hard to answer this correctly without a code sample. However, it sounds like you might be trying to reuse a closed socket, which is not possible.
If the socket has been closed (or has experienced an error), you must re-create a new connection using a new socket object. For this to work, the remote server must be able to handle multiple client connections in its accept() loop.
For a class project I'm trying to do some socket programming Python but running into a very basic issue. I can't create a TCP connection from my laptop to a lab machine. (Which I'm hoping to use as the "server") Without even getting into the scripts I have written, I've been simply trying interpreter line commands with no success. On the lab machine (kh4250-39.cselabs.umn.edu) I type the following into Python:
from socket import *
sock = socket()
sock.bind(('', 8353))
sock.listen(5)
sock.accept()
And then on my laptop I type:
from socket import *
sock = socket()
sock.connect(('kh4250-39.cselabs.umn.edu', 8353))
At which point both machines block and don't do anything until the client times out or I send a SIGINT. This code is pretty much exactly copied from examples I've found online and from Mark Lutz's book Programming Python (using '' for the server host name apparently uses the OS default and is fairly common). If I run both ends in my computer and use 'localhost' for the hostname it works fine, so I suspect it's some problem with the hostnames I'm using on one or both ends. I'm really not sure what could be going wrong on such a simple example. Does anyone have an idea?
A good way to confirm whether it's a firewall issue or not is to perform a telnet from the command-line to the destination host in question:
% telnet kh4250-39.cselabs.umn.edu 8353
Trying 128.101.38.44...
And then sometime later:
telnet: connect to address 128.101.38.44: Connection timed out
If it just hangs there at Trying and then eventually times out, chances are the connection to the remote host on that specific port is being blocked by a firewall. It could either be at the network layer (e.g. a real firewall or a router access-list) or at the host, such as iptables or other host-based filtering mechanisms.
Access to this lab host might only be available from within the lab or the campus network. Talk with your professor or a network administrator or someone "in the know" on the network to find out for sure.
Try to bind the server to 'kh4250-39.cselabs.umn.edu' instead of '':
sock.bind(('kh4250-39.cselabs.umn.edu', 8353))
If this does not work: Another reason could be a firewall blocking the port 8353....