What's the easiest way to establish an emulated TCP connection over HTTP with python 2.7.x?
Server: a python program on pythonanywhere (or some analogue) free hosting, that doesn't provide a dedicated ip. Client: a python program on a Windows PC.
Connection is established via multiprocessing.BaseManager and works fine when testing both server and client on the same machine.
Is there a way to make this work over HTTP with minimal additions to the code?
P.S. I need this for a grid computing project.
P.P.S. I'm new to python & network & web programming, started studying it several days ago.
Found this: http://code.activestate.com/recipes/577643-transparent-http-tunnel-for-python-sockets-to-be-u/. Appears to be exactly what I need, though I don't understand how to invoke setup_http_proxy() on server/client side. Tried setup_http_proxy("my.proxy", 8080) on both sides, but it didn't work.
Also found this: http://docs.python.org/2/library/httplib.html. What does the HTTPConnection.set_tunnel method actually do? Can I use it to solve the problem in question?
Usage on the client:
setup_http_proxy("THE_ADRESS", THE_PORT_NUMBER) # address of the Proxy, port the Proxy is listening on
The code wraps sockets to perform an initial HTTP CONNECT request to the proxy setup to get an HTTP Proxy to proxy the TCP connection for you but for that you'll need a compliant proxy (most won't allow you to open TCP connections unless it's for HTTPS).
HTTPConnection.set_tunnel basically does the same thing.
For your use case, a program running on free hosting, this just won't work. Your free host probably will only allow you to handle http requests, not have long running processes listen for tcp connections(which the code assumes).
You should rethink your need to tunnel and organize your communication to post data (and poll for messages from the server, unless they're answers to the stuff you post). Or you can purchase a VPS hosting that will give you more control over what you can host remotely.
Related
I make a web server on sockets. I want to support persistent connections.
When I write in the address bar of the browser a request to the server (on the localhost) in the headers I see "Connection: keep-alive", but the browser displays the data sent only after the connection is closed. I even do a "flush" on the connection (in python you can create a connection file and make a "flush" on it). I guess I don’t quite understand how sockets should behave in the persistent connection.
Please, help me figure out. If it is possible with a Python code examples. Sorry for my bad English.
I guess I don’t quite understand how sockets should behave in the persistent connection
This seems to be the case. Persistent HTTP connection just means that the server is may keep the TCP connection open after sending the HTTP response in order to process another HTTP request and that the client may send another request on the same TCP connection if the TCP connection is still open by the server. Both server and client might decide to not send/receive another request and close the connection whenever it is idle (i.e. no outstanding HTTP response).
Persistent HTTP connection in no way change the semantics of HTTP from a request-response protocol to "anything sockets can do". This means the way you want to use persistence is wrong.
I am creating a colloabrative note-making app in python.
Here, one guy on computer running the app can create the server subseuqently the changes on the screen([color, pixel], where pixel=[x,y]) will be transmitted to others connected to the server.
I am using kivy for creating the app. My question is with respect to transmitting the data over the server.
I can create server using this:
import socket
ip_address=socket.gethostbyname(socket.gethostname())
execfile( "manage.py runserver "+ip_address+":8000" )
Now, how do others connect to the server and request the data(assuming the above code is correct). Also, how to send the data in django.
Well, Django is a framework that allows creating a site or API that is reachable through HTTP protocol. This has several consequences for you:
Server cannot send a message to client unless the client asks. HTTP is a "request-response" protocol. Client sends a request (for example, http://server.com/getUpdates?id=100500) and gets a response from server.
Creating clients that ask the server to give them updates all the time is a bad practice, probably leading to server DoS.
Although you can use WebSockets, using Django for such a task is really an overkill.
Summarizing, you need a reliable duplex channel for sending data in both directions. I'd start with TCP server, rather than HTTP. Fortunately, Python stdlib has a module you can start with - socketserver.
Additional reading
TCP
UDP (you will probably want this for broadcasting)
Berkeley sockets (a socket standard underlying socketserver module)
TCP vs. UDP
When deciding what protocol to use, following aspects should be considered:
TCP is reliable. Messages never disappear implicitly. If there was a network error, message will be resent. If there's no connection, explicit error will be raised. TCP uses several algorithms to fit into the network channel. It is an intelligent protocol.
UDP is unreliable. It possesses no feature TCP has. Packets can disappear, get reordered. But UDP messages are lightweight and in experienced hands they summon to life such systems as network action games and streaming video (lost and reordered messages aren't crucial here and TCP becomes too slow).
So I'd recommend to start with TCP. It's way more easier to get working fast and correct than UDP. Switch to UDP if you have some experience with TCP and there are a lot of people using you app and wanting to get the lowest latency possible.
I want to build a peer to peer chat engine that runs over the Internet. So far my code works on a local network but not further. This is due to the fact that listening on sockets using python sockets does not make them available outside of the LAN.
It is acceptable for IPs to be shared knowledge, ie it is ok for the other person to need to know my IP address (and a port on which I am listening) to connect to me.
How does one tell the router to open a socket to the outside world? Presumably this can be done as p2p software such as BitTorrent must do it for communication between clients.
As you have mentioned you have to open a specific port on the router and use that port for communication. As there are many router manufacturers each with a variety of models I suggest you to check the manual for the router you want to use.
for the code, you may check if your code works on LAN and then see if the router let's you white-list some ports. you may find many simple examples online.
this is a code i played sometime ago:
http://www.mediafire.com/download/vef4q4prkr7be2e/python.socket.zip
if you don't want users to mess up with ports and router settings and such, first alternative i can think of is this:
you setup an REST API, in one interface one is able to retrieve the messages providing (chatRoomName, FromTimestamp, ToTimestamp[,optionally chatRoomPassWord]) but this has nothing to do with sockets, you have to use simple HTTP requests(urllib/urllib2). Of course there might exist some workaround for this such as an always-white-listed port(like 80 for browsers, 22 for SSH) but you have to search for such exceptions.
note that ports up to 1024 require special privileges(admin/sudo) to be used.
p.s. in traditional implementation other party(client) have to know your (ip, port) duo to be able to connect to the you(server).
I am running a Graphite server to monitor instruments at remote locations. I have a "perpetual" ssh tunnel to the machines from my server (loving autossh) to map their local ports to my server's local port. This works well, data comes through with no hasstles. However we use a flaky satellite connection to the sites, which goes down rather regularly. I am running a "data crawler" on the instrument that is running python and using socket to send packets to the Graphite server. The problem is, if the link goes down temporarily (or the server gets rebooted, for testing mostly), I cannot re-establish the connection to the server. I trap the error, and then run socket.close(), and then re-open, but I just can't re-establish the connection. If I quit the python program and restart it, the connection comes up just fine. Any ideas how I can "refresh" my socket connection?
It's hard to answer this correctly without a code sample. However, it sounds like you might be trying to reuse a closed socket, which is not possible.
If the socket has been closed (or has experienced an error), you must re-create a new connection using a new socket object. For this to work, the remote server must be able to handle multiple client connections in its accept() loop.
I am little stumped: I have a simple messenger client program (pure python, sockets), and I wanted to add proxy support (http/s, socks), however I am a little confused on how to go about it. I am assuming that the connection on the socket level will be done to the proxy server, at which point the headers should contain a CONNECT + destination IP (of the chat server) and authentication, (if proxy requires so), however the rest is a little beyond me. How is the subsequent connection handled, specifically the reading/writing, etc...
Are there any guides on proxy support implementation for socket based (tcp) programming in Python?
Thank you
Maybe use something like SocksiPy which does all the protocol details for you and would let you connect through a SOCKS proxy as you would without it?
It is pretty simple - after you send the HTTP request: CONNECT example.com:1234 HTTP/1.0\r\nHost: example.com:1234\r\n<additional headers incl. authentication>\r\n\r\n, the server responds with HTTP/1.0 200 Connection established\r\n\r\n and then (after the double line ends) you can communicate just as you would communicate with example.com port 1234 without the proxy (as I understand you already have the client-server communication part done).
Have a look at stunnel.
Stunnel can allow you to secure
non-SSL aware daemons and protocols
(like POP, IMAP, LDAP, etc) by having
Stunnel provide the encryption,
requiring no changes to the daemon's
code