We Keep Coding

Python Requests - Get Server IP

I'm making a small tool that tests CDN performance and would like to check where the response comes from. I thought of getting the host's IP and then using one of the geolocation API's on github to check the country.
I've tried doing so with
import socket
...
raw._fp.fp._sock.getpeername()
...however that only works when i use stream=True for the request and that in turn breaks the tool's functionality.
Is there any other option to get the server ip with requests or in a completely different way?

The socket.gethostbyname() function from Python's socket library should solve your problem. You can check it out in the Python docs here.
Here is an example of how to use it:
import socket
url="cdnjs.cloudflare.com"
print("IP:",socket.gethostbyname(url))
All you need to do is pass the url to socket.gethostbyname() and it will do the rest. Just make sure to remove the http:// before the URL because that will trip it up.

I could not get Akilan's solution to give the IP address of a different host that I was using. socket.gethostbyname() and getpeername() were not working for me. They are not even available. His solution did open the door.
However, navigating the socket object, I did find this:
socket.getaddrinfo('host name', 443)[0][4][0]
I wrapped this in a try/except block.
Maybe there is a prettier way.

Integrating HTML+JS for host system display into Python script running in Docker container (linux subsystem)

I have a basic HTML page with some javascript that I want to use to display the results of a python script at repeated steps during its execution, as well as waiting for some user trigger to continue. The webpage consists of an HTML file, a main javascript file, a secondary javascript file generated by the script containing the custom data, and references to external javascript dependencies (e.g. "https://unpkg.com/d3-graphviz#3.0.5/build/d3-graphviz.js".
To complicate things, the python script is running in a Docker container (Linux subsystem) via Visual Studio Code, so some operations like automatically opening an HTML file on the host system are difficult to accomplish (and there also seems to be a problem, with firefox at least, preventing opening local HTML files that reference other local files - see).
My initial plan was to, at each display step in the script, start a simple python http server, serve the page on localhost by piping it through a forwarded port see, wait for the user to open the localhost in their host browser, then once the page has been loaded close the server and continue with the script. I'm sure there is some fancier way of keeping the server up the entire time using threading and passing the updated data but web dev isn't really my field and this isn't exactly production code either so I felt this approach made more sense.
So assuming that this is a valid approach (and if there is a better one, I am more than happy to hear it), here's my best attempt at it (following this guide):
import http.server # Our http server handler for http requests
import socketserver # Establish the TCP Socket connections
from time import sleep # To sleep before shutting down server
PORT = 8000
count = 0
class requestHandler(http.server.SimpleHTTPRequestHandler):
def do_GET(self):
global count
if self.path == "/":
self.path == ".server/"
count += 1
return http.server.SimpleHTTPRequestHandler.do_GET(self)
# Prevent errors when restarting server and address (port) is still in use
socketserver.TCPServer.allow_reuse_address = True
handler = requestHandler
# Start server
with socketserver.TCPServer(("", PORT), handler) as httpd:
# Checking for request count, 3 expected:
# "GET /.server/ HTTP/1.1"
# "GET /.server/dots.js HTTP/1.1"
# "GET /.server/script.js HTTP/1.1"
# so accept 3 requests and end
while count < 3:
httpd.handle_request()
This is called from the main script with exec(open().read()).
Currently I get the error "NameError: name 'http' is not defined" in line 23. Weirdly, I'm sure a couple of days ago this was working fine, and I've just come back to it so I wouldn't have thought anything had changed. I've also tried a bunch of simple http.server examples with and without custom handlers, with and without the with...as block, with various errors from "Loading failed for the with source "http://localhost:8000/.server/dots.js" (in the browser F12 console) to failures to load the entire ./server directory and others. So I'm pretty much out of ideas at this point and am hoping one of you nice people can help me out.
P.S. Sorry for the long post, had to get it all out.

Okay so I seem to have found a rather inelegant but at least functional solution for the time being. For anyone reading in the future here is the server.py file:
import http.server
import socketserver
PORT = 8000
socketserver.TCPServer.allow_reuse_address = True
httpd = socketserver.TCPServer(("", PORT), http.server.SimpleHTTPRequestHandler)
for i in range(3):
httpd.handle_request()
I'll leave the question up for a little longer in case something better comes along, otherwise I'll mark this as a solution.

can you try to change:
with socketserver.TCPServer(("", PORT), handler) as httpd:
to:
with socketserver.TCPServer(("0.0.0.0", PORT), handler) as httpd:
so your application may need to listen on all interfaces?
EDIT:
as I wrote in the comments, using simple webserver such as flask might be easier for you, only to serve static files.
Here is the first post I found: https://code-maven.com/flask-serve-static-files

Tor, Stem, and Sockets - Changing Identity with TOR

I'm trying to run Tor through python. My goal is to be able to switch exits or otherwise alter my IP from time-to-time at a time of my choosing. I've followed several tutorials and encountered several different errors.
This code prints my IP address
import requests
r = requests.get('http://icanhazip.com/')
r.content
# returns my regular IP
I've downloaded and 'installed' the Tor browser (although it seems to run from an .exe). When it is running, I can use the following code to return my Tor IP, which seems to change every 10 minutes or so, so everything seems to be working so far
import socks
import socket
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9150)
socket.socket = socks.socksocket
r = requests.get('http://icanhazip.com/')
print r.content
# Returns a different IP
Now, I've found instructions here on how to request a new identity programmatically, but this is where things begin to go wrong. I run the following code
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9151) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
and I get the error "SOCKS5Error: 0x01: General SOCKS server failure" at the line that creates the controller. I thought this might be a configuration problem - the instructions say that there should be a config file called 'torrc' that contains the port numbers, apart from other things.
Eventually, in the directory C:\Users\..myname..\Desktop\Tor Browser\Browser\TorBrowser\Data\Tor I found three files, a 'torrc', a 'torrc-defaults', and a 'torrc.orig.1'. My 'torrc-defaults' looks similar to the 'torrc' shown in the documentation, my 'torrc.orig.1' is empty, and my 'torrc' has two lines of comments that look as follows with some more settings afterwards but no port settings
# This file was generated by Tor; if you edit it, comments will not be preserved
# The old torrc file was renamed to torrc.orig.1 or similar, and Tor will ignore it
...
I've tried to make sure that the two ports listed in the 'torrc-defaults' match the ports in the socks statement at the beginning and the controller statment further on. When these are 9150 and 9151 respectively I get the general SOCKS server failure listed above.
When I try and run the Controller with the wrong port (in this case, 9051 which was recommended in other posts but for me leads to Tor browser failing to load when I adjust the 'torrc-defaults') then I instead get the error message "[Errno 10061] No connection could be made because the target machine actively refused it"
Finally, when I run the Controller with Tor browser running in the background but without first running the SOCKS statement, I instead get a lot of warning text and finally an error, as shown in part below:
...
...
250 __OwningControllerProcess
DEBUG:stem:GETCONF __owningcontrollerprocess (runtime: 0.0010)
TRACE:stem:Sent to tor:
SIGNAL NEWNYM
TRACE:stem:Received from tor:
250 OK
TRACE:stem:Received from tor:
650 SIGNAL NEWNYM
TRACE:stem:Sent to tor:
QUIT
TRACE:stem:Received from tor:
250 closing connection
INFO:stem:Error while receiving a control message (SocketClosed): empty socket content
At first I thought this looked quite promising - I put in a quick check line like this both before and after the new user request
print requests.get('http://icanhazip.com/').content
It prints the IP, but unfortunately it's the same both before and afterwards.
I'm pretty lost here - any support would be appreciated!
Thanks.

Get your hashed password :
tor --hash-password my_password
Modify the configuration file :
sudo gedit /etc/tor/torrc
Add these lines
ControlPort 9051
HashedControlPassword 16:E600ADC1B52C80BB6022A0E999A7734571A451EB6AE50FED489B72E3DF
CookieAuthentication 1
Note :- The HashedControlPassword is generated from first command
Refer this link
I haven't tried it on my own because I am using mac, but I hope this helps with configuration

You can use torify to achieve your goal. for example run the python interpreter as this command "sudo torify python" and every connection will be routed through TOR.
Reference on torify: https://trac.torproject.org/projects/tor/wiki/doc/TorifyHOWTO

Handling https requests using a SOCK_STREAM proxy

I'm working on a project that allows a user to redirect his browsing through a proxy. The system works like this - a user runs this proxy on a remote PC and then also runs the proxy on his laptop. The user then changes his browser settings on the laptop to use localhost:8080 to make use of that local proxy, which in turn forwards all browser traffic to the proxy running on the remote PC.
This is where I ran into HTTPS. I was able to get normal HTTP requests working fine and dandy, but as soon as I clicked on google.com, Firefox skipped my proxy and connected to https://google.com directly.
My idea was to watch for browser requests the say CONNECT host:443 and then use the python ssl module to wrap that socket. This would give me a secure connection between the outer proxy and the target server. However, when I run wireshark to see how a browser request looks like before ssl kicks in, it's already there, meaning it looks like the browser connects to port 443 directly, which explains why it omitted my local proxy.
I would like to be able to handle to HTTPS as that would make for a complete browsing experience.
I'd really appreciate any tips that could push in the right direction.

Well, after doing a fair amount of reading on proxies, I found out that my understanding of the problem was insufficient.
For anyone else that might end up in the same spot as me, know that there's a pretty big difference between HTTP, HTTPS, and SOCKS proxies.
HTTP proxies usually take a quick look into the HTTP headers to determine where to forward the whole packet. These are quite easy to code on your own with some basic knowledge of sockets.
HTTPS proxies, on the other hand, have to work differently. They should either be able to do the whole SSL magic for the client or they could try to pass the traffic without changes, however if the latter solution is chosen, the users IP will be known. This is a wee bit more demanding when it comes to coding.
SOCKS proxies are a whole different, albeit really cool, beast. They work on the 5th layer of the OSI model and honestly, I have no clue as to where I would even begin creating one. They achieve both security and anonymity. However, I do know that a person may be able to use SSH to start a SOCKS proxy on their machine, just read this http://www.revsys.com/writings/quicktips/ssh-tunnel.html . That link also gave an idea that it should be possible to use SSH from a Python script to make it much more convenient.
Hope this helps anyone with the same question as I had. Good luck!

Connect to python server using sockets in python 2.7

I created a python server on port 8000 using python -m SimpleHTTPServer.
When I visit this url from my web browser it shows the below content
Now, I want to get the above content using python. So, for that what I did is
>>> import socket
>>> s = socket.socket(
... socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(("localhost", 8000))
>>> s.recv(1024)
But after s.recv(1024) nothing happens it just wait there and prints nothing.
So, my question is how to get above directory content output using python. Also can someone suggest me a tutorial on socket programming with python. I didn't liked the official tutorial that much.
I also observed a strange thing when I try to receive contents using python and nothing happens at that time I cannot access localhost:8000 from my web browser but as soon as I kill my python program I can access it.

Arguably the simplest way to get content over http in python is to use the urllib2 module. For example:
from urllib2 import urlopen
f = urlopen('http://localhost:8000')
for line in f:
print line
This will print out the file hosted by SimpleHTTPServer.

But after s.recv(1024) nothing happens it just wait there and prints nothing.
You simply open a socket and waiting for the data, but it's not how HTTP protocol works. You have to send a request first if you want to receive a response (basically, you have to tell the server which directory you want to list or which file to download). If you really want to, you can send the request using raw sockets to train your skills, but the proper library is highly recommended (see Matthew Adams' response and urllib2 example).
I also observed a strange thing when I try to receive contents using python and nothing happens at that time I cannot access localhost:8000 from my web browser but as soon as I kill my python program I can access it.
This is because SimpleHTTServer is single-threaded and doesn't support multiple connections simultaneously. If you would like to fix it, take a look at the answers here: BasicHTTPServer, SimpleHTTPServer and concurrency.