How do I host an HTTP proxy? - python

I am using Requests to scrape a website. My scraping code runs on one computer, but I need to make the requests come from a different computer (from the perspective of the website being scraped). I understand that I can do this with Requests by passing a proxies= argument when creating my session. I understand that I have two options, either using an HTTP proxy or a SOCKS proxy. I understand how to host an SOCKS proxy, because it just works over SSH, so I just need to make it so that I can SSH into the proxy machine from the machine running the scraping code and use -D, like this
# Generate key
ssh-keygen -o -a 100 -t ed25519 -C ''
# Copy key to proxy machine
ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>#<ip of the computer acting as a proxy>
# Open a connection to that server on some local port (I randomly chose port 14171)
ssh -D 14171 root#<ip of the computer acting as a proxy>
then I can make requests like this
from requests import Session
proxies = {
'http': 'socks5://localhost:14171',
'https': 'socks5://localhost:14171',
}
session = Session()
session.proxies.update(proxies)
session.get('http://example.com')
I understand that with an HTTP proxy it's quite similar, I just do
proxies = {
'http': 'http://user:pass#10.10.1.10:1080',
'https': 'http://user:pass#10.10.1.10:1080',
}
but what do I use on the server to make it act as an HTTP proxy with a password? And are messages sent in the clear or encrypted?

There are many implementations of HTTP proxies to choose from. Squid seems to be the first result on Google. I also tried Tinyproxy. With Squid, you set it up like this:
Install Squid
apt install squid apache2-utils
Create the password file
sudo touch /etc/squid/squid_passwd
sudo chown proxy /etc/squid/squid_passwd
Then edit the configuration file
mv /etc/squid/squid.conf /etc/squid/squid.conf.default # move default file out of the way
vim /etc/squid/squid.conf
and paste the following as the configuration:
http_port 3128
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/squid_passwd
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED
http_access allow authenticated
you can add those lines to the end of the file, but the problem is that their default config is 8000 lines of documentation (about 25 lines of actual default config) and somewhere in there they forbid all connections (probably all connections not from localhost) that you'd have to read and ain't nobody got time for that, so I just cleared and put that config as the default. You should probably take the time to actually learn Squid if you're going to use it though...
Create a password for a user (youruser is a username, you can choose whatever)
htpasswd /etc/squid/squid_passwd youruser
Restart Squid
service squid restart
Open the port in the firewall
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 3128 -j ACCEPT
You can then check that it works with Curl:
curl --proxy <the IP address of your proxy>:3128 --proxy-user youruser:<password> "http://icanhazip.com"
Tinyproxy is pretty similar, it has the advantage that you don't have to download a separate package just to set a password and their default config file is actually short enough to read...
Install Tinyproxy
sudo apt install tinyproxy
Edit the config file
sudo vim /etc/tinyproxy/tinyproxy.conf
These are the options I needed to set:
change the port to some random port Port 17724
comment out the Allow 127.0.0.1 line to allow connections from any IP
add a line to enable a password BasicAuth youruser yourpassword
(optional) disable adding a "Via" header (this is a way to let servers that you're making requests to know that you're using a proxy) with DisableViaHeader yes
(optional) disable everything except for reverse-proxying with ReverseOnly Yes
You may want to read through the entire default config file, maybe there are other options you need for your use-case.
Restart the Tinyproxy systemd service
sudo service tinyproxy restart
Open the port in the firewall
sudo iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 17724 -j ACCEPT
The you can then use your proxy with Requests like this
proxies = {
'http': 'http://<youruser>:<password>#<the IP address of your proxy>:3128',
'https': 'http://<youruser>:<password>#<the IP address of your proxy>:3128'
}
Proxies also allow you to limit connections to only a give IP address, so if the server you're running the code on has a static IP, it would be a good idea to limit connections only from that IP. Note that HTTP proxying is not encrypted, so a man-in-the-middle would be able to see your password and then use your proxy.
Sources:
https://www.vultr.com/docs/how-to-install-squid-proxy-on-centos
https://www.vultr.com/docs/install-squid-proxy-on-ubuntu (a bit outdated)

Related

VS Code Flask temporary hosting with an actual url? [duplicate]

I'm not sure if this is Flask specific, but when I run an app in dev mode (http://localhost:5000), I cannot access it from other machines on the network (with http://[dev-host-ip]:5000). With Rails in dev mode, for example, it works fine. I couldn't find any docs regarding the Flask dev server configuration. Any idea what should be configured to enable this?
While this is possible, you should not use the Flask dev server in production. The Flask dev server is not designed to be particularly secure, stable, or efficient. See the docs on deploying for correct solutions.
The --host option to flask run, or the host parameter to app.run(), controls what address the development server listens to. By default it runs on localhost, change it to flask run --host=0.0.0.0 (or app.run(host="0.0.0.0")) to run on all your machine's IP addresses.
0.0.0.0 is a special value that you can't use in the browser directly, you'll need to navigate to the actual IP address of the machine on the network. You may also need to adjust your firewall to allow external access to the port.
The Flask quickstart docs explain this in the "Externally Visible Server" section:
If you run the server you will notice that the server is only
accessible from your own computer, not from any other in the network.
This is the default because in debugging mode a user of the
application can execute arbitrary Python code on your computer.
If you have the debugger disabled or trust the users on your network,
you can make the server publicly available simply by adding
--host=0.0.0.0 to the command line:
$ flask run --host=0.0.0.0
This tells your operating system to listen on all public IPs.
If you use the flask executable to start your server, use flask run --host=0.0.0.0 to change the default from 127.0.0.1 and open it up to non-local connections.
If you run the server you will notice that the server is only
accessible from your own computer, not from any other in the network.
This is the default because in debugging mode a user of the
application can execute arbitrary Python code on your computer.
If you have the debugger disabled or trust the users on your network,
you can make the server publicly available simply by adding
--host=0.0.0.0 to the command line:
$ flask run --host=0.0.0.0
This tells your operating system to listen on all public IPs.
Reference: https://flask.palletsprojects.com/quickstart/
Try this if the 0.0.0.0 method doesn't work
Boring Stuff
I personally battled a lot to get my app accessible to other devices(laptops and mobile phones) through a local-server. I tried the 0.0.0.0 method, but no luck. Then I tried changing the port, but it just didn't work. So, after trying a bunch of different combinations, I arrived to this one, and it solved my problem of deploying my app on a local server.
Steps
Get the local IPv4 address of your computer.
This can be done by typing ipconfig on Windows and ifconfig on Linux
and Mac.
Please note: The above step is to be performed on the machine you are serving the app on, and on not the machine on which you are accessing it. Also note, that the IPv4 address might change if you disconnect and reconnect to the network.
Now, simply run the flask app with the acquired IPv4 address.
flask run -h 192.168.X.X
E.g. In my case (see the image), I ran it as:
flask run -h 192.168.1.100
On my mobile device
Optional Stuff
If you are performing this procedure on Windows and using Power Shell as the CLI, and you still aren't able to access the website, try a CTRL + C command in the shell that's running the app. Power Shell gets frozen up sometimes and it needs a pinch to revive. Doing this might even terminate the server, but it sometimes does the trick.
That's it. Give a thumbs up if you found this helpful.😉
Some more optional stuff
I have created a short Powershell script that will get you your IP address whenever you need one:
$env:getIp = ipconfig
if ($env:getIp -match '(IPv4[\sa-zA-Z.]+:\s[0-9.]+)') {
if ($matches[1] -match '([^a-z\s][\d]+[.\d]+)'){
$ipv4 = $matches[1]
}
}
echo $ipv4
Save it to a file with .ps1 extension (for PowerShell), and run it on before starting your app. You can save it in your project folder and run it as:
.\getIP.ps1; flask run -h $ipv4
Note: I saved the above shellcode in getIP.ps1.
Cool.👌
Add host='0.0.0.0' to app.run`.
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
If you get OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions on Windows, you either don't have permission to use the port, or something else is using it which you can find with netstat -na|findstr 5000.
Check whether the particular port is open on the server to serve the client or not?
in Ubuntu or Linux distro
sudo ufw enable
sudo ufw allow 5000/tcp //allow the server to handle the request on port 5000
Configure the application to handle remote requests
app.run(host='0.0.0.0' , port=5000)
python3 app.py & #run application in background
If your cool app has it's configuration loaded from an external file, like in the following example, then don't forget to update the corresponding config file with HOST="0.0.0.0"
cool.app.run(
host=cool.app.config.get("HOST", "localhost"),
port=cool.app.config.get("PORT", 9000)
)
If you're having troubles accessing your Flask server, deployed using PyCharm, take the following into account:
PyCharm doesn't run your main .py file directly, so any code in if __name__ == '__main__': won't be executed, and any changes (like app.run(host='0.0.0.0', port=5000)) won't take effect.
Instead, you should configure the Flask server using Run Configurations, in particular, placing --host 0.0.0.0 --port 5000 into Additional options field.
More about configuring Flask server in PyCharm
You can also set the host (to expose it on a network facing IP address) and port via environment variables.
$ export FLASK_APP=app.py
$ export FLASK_ENV=development
$ export FLASK_RUN_PORT=8000
$ export FLASK_RUN_HOST=0.0.0.0
$ flask run
* Serving Flask app "app.py" (lazy loading)
* Environment: development
* Debug mode: on
* Running on https://0.0.0.0:8000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 329-665-000
See How to get all available Command Options to set environment variables?
Go to your project path on CMD(command Prompt) and execute the following command:-
set FLASK_APP=ABC.py
SET FLASK_ENV=development
flask run -h [yourIP] -p 8080
you will get following o/p on CMD:-
Serving Flask app "expirement.py" (lazy loading)
Environment: development
Debug mode: on
Restarting with stat
Debugger is active!
Debugger PIN: 199-519-700
Running on http://[yourIP]:8080/ (Press CTRL+C to quit)
Now you can access your flask app on another machine using http://[yourIP]:8080/ url
For me i followed the above answer and modified it a bit:
Just grab your ipv4 address using ipconfig on command prompt
Go to the file in which flask code is present
In main function write app.run(host= 'your ipv4 address')
Eg:
Create file .flaskenv in the project root directory.
The parameters in this file are typically:
FLASK_APP=app.py
FLASK_ENV=development
FLASK_RUN_HOST=[dev-host-ip]
FLASK_RUN_PORT=5000
If you have a virtual environment, activate it and do a pip install python-dotenv .
This package is going to use the .flaskenv file, and declarations inside it will be automatically imported across terminal sessions.
Then you can do flask run
This answer is not solely related with flask, but should be applicable for all cannot connect service from another host issue.
use netstat -ano | grep <port> to see if the address is 0.0.0.0 or ::. If it is 127.0.0.1 then it is only for the local requests.
use tcpdump to see if any packet is missing. If it shows obvious imbalance, check routing rules by iptables.
Today I run my flask app as usual, but I noticed it cannot connect from other server. Then I run netstat -ano | grep <port>, and the local address is :: or 0.0.0.0 (I tried both, and I know 127.0.0.1 only allows connection from the local host). Then I used telnet host port, the result is like connect to .... This is very odd. Then I thought I would better check it with tcpdump -i any port <port> -w w.pcap. And I noticed it is all like this:
Then by checking iptables --list OUTPUT section, I could see several rules:
these rules forbid output tcp vital packets in handshaking. By deleting them, the problem is gone.
I had the same problem, I use PyCharm as an editor and when I created the project, PyCharm created a Flask Server. What I did was create a server with Python in the following way;
basically what I did was create a new server but flask if not python
I hope it helps you
This finally worked for me.
import os
Then place this at the end of your python app.py or main file.
if __name__ == "__main__":
port = int(os.environ.get("PORT", 5000))
app.run(host='0.0.0.0', port=port)
go to project path
set FLASK_APP=ABC.py
SET FLASK_ENV=development
flask run -h [yourIP] -p 8080
you will following o/p on CMD:-
* Serving Flask app "expirement.py" (lazy loading)
* Environment: development
* Debug mode: on
* Restarting with stat
* Debugger is active!
* Debugger PIN: 199-519-700
* Running on http://[yourIP]:8080/ (Press CTRL+C to quit)
If none of the above solutions are working, try manually adding "http://" to the beginning of the url.
Chrome can distinguish "[ip-address]:5000" from a search query. But sometimes that works for a while, and then stops connecting, seemingly without me changing anything. My hypothesis is that the browser might sometimes automatically prepend https:// (which it shouldn't, but this fixed it in my case).
In case you need to test your app from an external network.
Simply serve it to the whole Internet with ngrok.com
which will deploy it like a dev server but in no time and locally, saved me a lot of time, and no, I'm not related to that company :)
Just make sure to change the port in your flask app:
app.run(host='0.0.0.0', port=80)

Flask Deployment on Local Server [duplicate]

I'm not sure if this is Flask specific, but when I run an app in dev mode (http://localhost:5000), I cannot access it from other machines on the network (with http://[dev-host-ip]:5000). With Rails in dev mode, for example, it works fine. I couldn't find any docs regarding the Flask dev server configuration. Any idea what should be configured to enable this?
While this is possible, you should not use the Flask dev server in production. The Flask dev server is not designed to be particularly secure, stable, or efficient. See the docs on deploying for correct solutions.
The --host option to flask run, or the host parameter to app.run(), controls what address the development server listens to. By default it runs on localhost, change it to flask run --host=0.0.0.0 (or app.run(host="0.0.0.0")) to run on all your machine's IP addresses.
0.0.0.0 is a special value that you can't use in the browser directly, you'll need to navigate to the actual IP address of the machine on the network. You may also need to adjust your firewall to allow external access to the port.
The Flask quickstart docs explain this in the "Externally Visible Server" section:
If you run the server you will notice that the server is only
accessible from your own computer, not from any other in the network.
This is the default because in debugging mode a user of the
application can execute arbitrary Python code on your computer.
If you have the debugger disabled or trust the users on your network,
you can make the server publicly available simply by adding
--host=0.0.0.0 to the command line:
$ flask run --host=0.0.0.0
This tells your operating system to listen on all public IPs.
If you use the flask executable to start your server, use flask run --host=0.0.0.0 to change the default from 127.0.0.1 and open it up to non-local connections.
If you run the server you will notice that the server is only
accessible from your own computer, not from any other in the network.
This is the default because in debugging mode a user of the
application can execute arbitrary Python code on your computer.
If you have the debugger disabled or trust the users on your network,
you can make the server publicly available simply by adding
--host=0.0.0.0 to the command line:
$ flask run --host=0.0.0.0
This tells your operating system to listen on all public IPs.
Reference: https://flask.palletsprojects.com/quickstart/
Try this if the 0.0.0.0 method doesn't work
Boring Stuff
I personally battled a lot to get my app accessible to other devices(laptops and mobile phones) through a local-server. I tried the 0.0.0.0 method, but no luck. Then I tried changing the port, but it just didn't work. So, after trying a bunch of different combinations, I arrived to this one, and it solved my problem of deploying my app on a local server.
Steps
Get the local IPv4 address of your computer.
This can be done by typing ipconfig on Windows and ifconfig on Linux
and Mac.
Please note: The above step is to be performed on the machine you are serving the app on, and on not the machine on which you are accessing it. Also note, that the IPv4 address might change if you disconnect and reconnect to the network.
Now, simply run the flask app with the acquired IPv4 address.
flask run -h 192.168.X.X
E.g. In my case (see the image), I ran it as:
flask run -h 192.168.1.100
On my mobile device
Optional Stuff
If you are performing this procedure on Windows and using Power Shell as the CLI, and you still aren't able to access the website, try a CTRL + C command in the shell that's running the app. Power Shell gets frozen up sometimes and it needs a pinch to revive. Doing this might even terminate the server, but it sometimes does the trick.
That's it. Give a thumbs up if you found this helpful.😉
Some more optional stuff
I have created a short Powershell script that will get you your IP address whenever you need one:
$env:getIp = ipconfig
if ($env:getIp -match '(IPv4[\sa-zA-Z.]+:\s[0-9.]+)') {
if ($matches[1] -match '([^a-z\s][\d]+[.\d]+)'){
$ipv4 = $matches[1]
}
}
echo $ipv4
Save it to a file with .ps1 extension (for PowerShell), and run it on before starting your app. You can save it in your project folder and run it as:
.\getIP.ps1; flask run -h $ipv4
Note: I saved the above shellcode in getIP.ps1.
Cool.👌
Add host='0.0.0.0' to app.run`.
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
If you get OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions on Windows, you either don't have permission to use the port, or something else is using it which you can find with netstat -na|findstr 5000.
Check whether the particular port is open on the server to serve the client or not?
in Ubuntu or Linux distro
sudo ufw enable
sudo ufw allow 5000/tcp //allow the server to handle the request on port 5000
Configure the application to handle remote requests
app.run(host='0.0.0.0' , port=5000)
python3 app.py & #run application in background
If your cool app has it's configuration loaded from an external file, like in the following example, then don't forget to update the corresponding config file with HOST="0.0.0.0"
cool.app.run(
host=cool.app.config.get("HOST", "localhost"),
port=cool.app.config.get("PORT", 9000)
)
If you're having troubles accessing your Flask server, deployed using PyCharm, take the following into account:
PyCharm doesn't run your main .py file directly, so any code in if __name__ == '__main__': won't be executed, and any changes (like app.run(host='0.0.0.0', port=5000)) won't take effect.
Instead, you should configure the Flask server using Run Configurations, in particular, placing --host 0.0.0.0 --port 5000 into Additional options field.
More about configuring Flask server in PyCharm
You can also set the host (to expose it on a network facing IP address) and port via environment variables.
$ export FLASK_APP=app.py
$ export FLASK_ENV=development
$ export FLASK_RUN_PORT=8000
$ export FLASK_RUN_HOST=0.0.0.0
$ flask run
* Serving Flask app "app.py" (lazy loading)
* Environment: development
* Debug mode: on
* Running on https://0.0.0.0:8000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 329-665-000
See How to get all available Command Options to set environment variables?
Go to your project path on CMD(command Prompt) and execute the following command:-
set FLASK_APP=ABC.py
SET FLASK_ENV=development
flask run -h [yourIP] -p 8080
you will get following o/p on CMD:-
Serving Flask app "expirement.py" (lazy loading)
Environment: development
Debug mode: on
Restarting with stat
Debugger is active!
Debugger PIN: 199-519-700
Running on http://[yourIP]:8080/ (Press CTRL+C to quit)
Now you can access your flask app on another machine using http://[yourIP]:8080/ url
For me i followed the above answer and modified it a bit:
Just grab your ipv4 address using ipconfig on command prompt
Go to the file in which flask code is present
In main function write app.run(host= 'your ipv4 address')
Eg:
Create file .flaskenv in the project root directory.
The parameters in this file are typically:
FLASK_APP=app.py
FLASK_ENV=development
FLASK_RUN_HOST=[dev-host-ip]
FLASK_RUN_PORT=5000
If you have a virtual environment, activate it and do a pip install python-dotenv .
This package is going to use the .flaskenv file, and declarations inside it will be automatically imported across terminal sessions.
Then you can do flask run
This answer is not solely related with flask, but should be applicable for all cannot connect service from another host issue.
use netstat -ano | grep <port> to see if the address is 0.0.0.0 or ::. If it is 127.0.0.1 then it is only for the local requests.
use tcpdump to see if any packet is missing. If it shows obvious imbalance, check routing rules by iptables.
Today I run my flask app as usual, but I noticed it cannot connect from other server. Then I run netstat -ano | grep <port>, and the local address is :: or 0.0.0.0 (I tried both, and I know 127.0.0.1 only allows connection from the local host). Then I used telnet host port, the result is like connect to .... This is very odd. Then I thought I would better check it with tcpdump -i any port <port> -w w.pcap. And I noticed it is all like this:
Then by checking iptables --list OUTPUT section, I could see several rules:
these rules forbid output tcp vital packets in handshaking. By deleting them, the problem is gone.
I had the same problem, I use PyCharm as an editor and when I created the project, PyCharm created a Flask Server. What I did was create a server with Python in the following way;
basically what I did was create a new server but flask if not python
I hope it helps you
This finally worked for me.
import os
Then place this at the end of your python app.py or main file.
if __name__ == "__main__":
port = int(os.environ.get("PORT", 5000))
app.run(host='0.0.0.0', port=port)
go to project path
set FLASK_APP=ABC.py
SET FLASK_ENV=development
flask run -h [yourIP] -p 8080
you will following o/p on CMD:-
* Serving Flask app "expirement.py" (lazy loading)
* Environment: development
* Debug mode: on
* Restarting with stat
* Debugger is active!
* Debugger PIN: 199-519-700
* Running on http://[yourIP]:8080/ (Press CTRL+C to quit)
If none of the above solutions are working, try manually adding "http://" to the beginning of the url.
Chrome can distinguish "[ip-address]:5000" from a search query. But sometimes that works for a while, and then stops connecting, seemingly without me changing anything. My hypothesis is that the browser might sometimes automatically prepend https:// (which it shouldn't, but this fixed it in my case).
In case you need to test your app from an external network.
Simply serve it to the whole Internet with ngrok.com
which will deploy it like a dev server but in no time and locally, saved me a lot of time, and no, I'm not related to that company :)
Just make sure to change the port in your flask app:
app.run(host='0.0.0.0', port=80)

Connecting to Zap Proxy Docker Image, "Max retries exceeded with url: Caused by ProxyError('Cannot connect to proxy

I'm trying to use zap proxy via Docker
Pulled it down with:
docker pull owasp/zap2docker-stable
Ran it with the command described in "Accessing the API from outside of the Docker container" section:
docker run -p 8090:8090 -i owasp/zap2docker-stable zap.sh -daemon -port 8090 -host 0.0.0.0
But I don't seem to be able to be able to connect to it. When I run docker inspect <CONTAINER ID> | grep IPAddress I'm getting 172.17.0.2 (EDIT: I can a scan to run and it took me changing ZAP_SERVER_PROXY from 172.17.0.2:8090 to 0.0.0.0:8090 on Mac so editing that into code example below). So the start of my script looks like:
import os
import time
from pprint import pprint
from zapv2 import ZAPv2
BASE_URL = os.getenv('BASE_URL', 'https://example.appspot.com/')
ZAP_SERVER_PROXY = os.getenv('ZAP_SERVER_PROXY', '0.0.0.0:8090')
API_KEY = ''
zap = ZAPv2(
# apikey=API_KEY,
proxies={
'http': "http://%s" % ZAP_SERVER_PROXY,
'https': "https://%s" % ZAP_SERVER_PROXY
}
)
Just trying to run it through terminal using python right now and keep getting connection refused errors. Also I've tried it with the API_KEY parts commented out as well, does anyone know where you find that don't see it in the documentation.
Note: I'm on macos but running docker-machine ip default doesn't do anything, so not sure how to get at bottom of linked page and new to docker. Modeled the test after their own example. Running in a virtualenv -p python3 env not sure if that is effecting it.
Also, you will need to disable ip filtering - see here for details on how to do that, basically just use the following:
docker run -p 8090:8090 -i owasp/zap2docker-stable zap.sh -daemon -port 8090 -host 0.0.0.0 -config api.disablekey=true -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true
Be careful as it will allow connections from any ip to Zaproxy, which is fine while running it in isolated docker container.
As you've commented out the API key parameter in your script,try invoking zap daemon with -config api.disablekey=true.
docker run -p 8090:8090 -i owasp/zap2docker-stable zap.sh -daemon
-port 8090 -host 0.0.0.0 -config api.disablekey=true

Connect to Flask application via public IP [duplicate]

I'm not sure if this is Flask specific, but when I run an app in dev mode (http://localhost:5000), I cannot access it from other machines on the network (with http://[dev-host-ip]:5000). With Rails in dev mode, for example, it works fine. I couldn't find any docs regarding the Flask dev server configuration. Any idea what should be configured to enable this?
While this is possible, you should not use the Flask dev server in production. The Flask dev server is not designed to be particularly secure, stable, or efficient. See the docs on deploying for correct solutions.
The --host option to flask run, or the host parameter to app.run(), controls what address the development server listens to. By default it runs on localhost, change it to flask run --host=0.0.0.0 (or app.run(host="0.0.0.0")) to run on all your machine's IP addresses.
0.0.0.0 is a special value that you can't use in the browser directly, you'll need to navigate to the actual IP address of the machine on the network. You may also need to adjust your firewall to allow external access to the port.
The Flask quickstart docs explain this in the "Externally Visible Server" section:
If you run the server you will notice that the server is only
accessible from your own computer, not from any other in the network.
This is the default because in debugging mode a user of the
application can execute arbitrary Python code on your computer.
If you have the debugger disabled or trust the users on your network,
you can make the server publicly available simply by adding
--host=0.0.0.0 to the command line:
$ flask run --host=0.0.0.0
This tells your operating system to listen on all public IPs.
If you use the flask executable to start your server, use flask run --host=0.0.0.0 to change the default from 127.0.0.1 and open it up to non-local connections.
If you run the server you will notice that the server is only
accessible from your own computer, not from any other in the network.
This is the default because in debugging mode a user of the
application can execute arbitrary Python code on your computer.
If you have the debugger disabled or trust the users on your network,
you can make the server publicly available simply by adding
--host=0.0.0.0 to the command line:
$ flask run --host=0.0.0.0
This tells your operating system to listen on all public IPs.
Reference: https://flask.palletsprojects.com/quickstart/
Try this if the 0.0.0.0 method doesn't work
Boring Stuff
I personally battled a lot to get my app accessible to other devices(laptops and mobile phones) through a local-server. I tried the 0.0.0.0 method, but no luck. Then I tried changing the port, but it just didn't work. So, after trying a bunch of different combinations, I arrived to this one, and it solved my problem of deploying my app on a local server.
Steps
Get the local IPv4 address of your computer.
This can be done by typing ipconfig on Windows and ifconfig on Linux
and Mac.
Please note: The above step is to be performed on the machine you are serving the app on, and on not the machine on which you are accessing it. Also note, that the IPv4 address might change if you disconnect and reconnect to the network.
Now, simply run the flask app with the acquired IPv4 address.
flask run -h 192.168.X.X
E.g. In my case (see the image), I ran it as:
flask run -h 192.168.1.100
On my mobile device
Optional Stuff
If you are performing this procedure on Windows and using Power Shell as the CLI, and you still aren't able to access the website, try a CTRL + C command in the shell that's running the app. Power Shell gets frozen up sometimes and it needs a pinch to revive. Doing this might even terminate the server, but it sometimes does the trick.
That's it. Give a thumbs up if you found this helpful.😉
Some more optional stuff
I have created a short Powershell script that will get you your IP address whenever you need one:
$env:getIp = ipconfig
if ($env:getIp -match '(IPv4[\sa-zA-Z.]+:\s[0-9.]+)') {
if ($matches[1] -match '([^a-z\s][\d]+[.\d]+)'){
$ipv4 = $matches[1]
}
}
echo $ipv4
Save it to a file with .ps1 extension (for PowerShell), and run it on before starting your app. You can save it in your project folder and run it as:
.\getIP.ps1; flask run -h $ipv4
Note: I saved the above shellcode in getIP.ps1.
Cool.👌
Add host='0.0.0.0' to app.run`.
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
If you get OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions on Windows, you either don't have permission to use the port, or something else is using it which you can find with netstat -na|findstr 5000.
Check whether the particular port is open on the server to serve the client or not?
in Ubuntu or Linux distro
sudo ufw enable
sudo ufw allow 5000/tcp //allow the server to handle the request on port 5000
Configure the application to handle remote requests
app.run(host='0.0.0.0' , port=5000)
python3 app.py & #run application in background
If your cool app has it's configuration loaded from an external file, like in the following example, then don't forget to update the corresponding config file with HOST="0.0.0.0"
cool.app.run(
host=cool.app.config.get("HOST", "localhost"),
port=cool.app.config.get("PORT", 9000)
)
If you're having troubles accessing your Flask server, deployed using PyCharm, take the following into account:
PyCharm doesn't run your main .py file directly, so any code in if __name__ == '__main__': won't be executed, and any changes (like app.run(host='0.0.0.0', port=5000)) won't take effect.
Instead, you should configure the Flask server using Run Configurations, in particular, placing --host 0.0.0.0 --port 5000 into Additional options field.
More about configuring Flask server in PyCharm
You can also set the host (to expose it on a network facing IP address) and port via environment variables.
$ export FLASK_APP=app.py
$ export FLASK_ENV=development
$ export FLASK_RUN_PORT=8000
$ export FLASK_RUN_HOST=0.0.0.0
$ flask run
* Serving Flask app "app.py" (lazy loading)
* Environment: development
* Debug mode: on
* Running on https://0.0.0.0:8000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 329-665-000
See How to get all available Command Options to set environment variables?
Go to your project path on CMD(command Prompt) and execute the following command:-
set FLASK_APP=ABC.py
SET FLASK_ENV=development
flask run -h [yourIP] -p 8080
you will get following o/p on CMD:-
Serving Flask app "expirement.py" (lazy loading)
Environment: development
Debug mode: on
Restarting with stat
Debugger is active!
Debugger PIN: 199-519-700
Running on http://[yourIP]:8080/ (Press CTRL+C to quit)
Now you can access your flask app on another machine using http://[yourIP]:8080/ url
For me i followed the above answer and modified it a bit:
Just grab your ipv4 address using ipconfig on command prompt
Go to the file in which flask code is present
In main function write app.run(host= 'your ipv4 address')
Eg:
Create file .flaskenv in the project root directory.
The parameters in this file are typically:
FLASK_APP=app.py
FLASK_ENV=development
FLASK_RUN_HOST=[dev-host-ip]
FLASK_RUN_PORT=5000
If you have a virtual environment, activate it and do a pip install python-dotenv .
This package is going to use the .flaskenv file, and declarations inside it will be automatically imported across terminal sessions.
Then you can do flask run
This answer is not solely related with flask, but should be applicable for all cannot connect service from another host issue.
use netstat -ano | grep <port> to see if the address is 0.0.0.0 or ::. If it is 127.0.0.1 then it is only for the local requests.
use tcpdump to see if any packet is missing. If it shows obvious imbalance, check routing rules by iptables.
Today I run my flask app as usual, but I noticed it cannot connect from other server. Then I run netstat -ano | grep <port>, and the local address is :: or 0.0.0.0 (I tried both, and I know 127.0.0.1 only allows connection from the local host). Then I used telnet host port, the result is like connect to .... This is very odd. Then I thought I would better check it with tcpdump -i any port <port> -w w.pcap. And I noticed it is all like this:
Then by checking iptables --list OUTPUT section, I could see several rules:
these rules forbid output tcp vital packets in handshaking. By deleting them, the problem is gone.
I had the same problem, I use PyCharm as an editor and when I created the project, PyCharm created a Flask Server. What I did was create a server with Python in the following way;
basically what I did was create a new server but flask if not python
I hope it helps you
This finally worked for me.
import os
Then place this at the end of your python app.py or main file.
if __name__ == "__main__":
port = int(os.environ.get("PORT", 5000))
app.run(host='0.0.0.0', port=port)
go to project path
set FLASK_APP=ABC.py
SET FLASK_ENV=development
flask run -h [yourIP] -p 8080
you will following o/p on CMD:-
* Serving Flask app "expirement.py" (lazy loading)
* Environment: development
* Debug mode: on
* Restarting with stat
* Debugger is active!
* Debugger PIN: 199-519-700
* Running on http://[yourIP]:8080/ (Press CTRL+C to quit)
If none of the above solutions are working, try manually adding "http://" to the beginning of the url.
Chrome can distinguish "[ip-address]:5000" from a search query. But sometimes that works for a while, and then stops connecting, seemingly without me changing anything. My hypothesis is that the browser might sometimes automatically prepend https:// (which it shouldn't, but this fixed it in my case).
In case you need to test your app from an external network.
Simply serve it to the whole Internet with ngrok.com
which will deploy it like a dev server but in no time and locally, saved me a lot of time, and no, I'm not related to that company :)
Just make sure to change the port in your flask app:
app.run(host='0.0.0.0', port=80)

Alternative to scrapy proxy

Is there any alternative to using proxy in scrapy. The source site has blocked the server which I'm using for running spiders. I've added ProxyMiddleware in the project and randomized the proxy. But the problem is the proxies are also being blocked by the source site. I've also set the DOWNLOAD_DELAY to 5 but the problem is still alive. Is there any other way to access the site without using proxies other than shifting to new server?
Using tor with privoxy solved my problem of blocking.
Install tor
$ sudo apt-get install tor
Install polipo
$ sudo apt-get install polipo
configure privoxy to use tor socks proxy.
$sudo nano /etc/polipo/config
Add following lines at the end of file.
socksParentProxy = localhost:9050
diskCacheRoot=""
disableLocalInterface=""
Add proxy middleware in middlewares.py.
class ProxyMiddleware(object):
def process_request(self, request, spider):
request.meta['proxy'] = 'http://localhost:8123'
spider.log('Proxy : %s' % request.meta['proxy'])
Activate the proxyMiddleware in Project settings.
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
'project_name.middlewares.ProxyMiddleware': 100
}
You may want the squid.
It will shield failure proxy, use proxy faster, automatic rotation, automatic retry forwarding, and set the rules.
Just set your spider to the same export agent.

Categories