How to check what proxies are used by Python3 Requests module?
I have verified (from responses) that when you set http proxy in system configuration on MacOS but not set the http_proxy environment variable it will also be automatically used by Requests. It seems like it will use the proxies that urllib.request.getproxies() return, but I'm not sure because the document only says that
You can also configure proxies by setting the environment variables HTTP_PROXY and HTTPS_PROXY.
and it seems there aren't any descriptions on system proxy configuration.
Finally I found the solution:
requests.utils.getproxies()
or
requests.utils.get_environ_proxies(url)
Related
I'm in a corporate network and need to use a self-signed certificate for the requests library
in a docker image.
I installed it by putting it in /usr/local/shares/ca-certificates and calling update-ca-certificates like this:
COPY EDAG_Bundle.crt /usr/local/share/ca-certificates/my_cert.crt
RUN update-ca-certificates
ENV REQUESTS_CA_BUNDLE /usr/local/share/ca-certificates/my_cert.crt
Now I am able to access files on a Server in our corporate network without running in a certificate error.
Unfortunately this change caused pip to stop working. As pip is using requests too, it also now uses the self signed certificate instead of the one from certifi.
The requests documentation states the following:
You can pass verify the path to a CA_BUNDLE file with certificates of trusted CAs:
requests.get('https://github.com', verify='/path/to/certfile')
This list of trusted CAs can also be specified through the REQUESTS_CA_BUNDLE environment variable.
As I get this, I can define a List of trusted CAs, not just one.
How can I configure requests to use both CAs? (my self signed one and the one of certifi located in
/site-packages/certifi/cacert.pem).
Setting both in the environment variable by seperating the paths with a colon does not work.
Use /etc/ssl/certs/ca-certificates.crt as your REQUESTS_CA_BUNDLE.
requests.get('https://github.com', verify='/etc/ssl/certs/ca-certificates.crt')
When you put a self-issued CA certificate to /usr/local/shares/ca-certificates, then run update-ca-certificates, it will read those in and append to the global "ca trust file" (ca-certificates.crt). This will hold trust for both publicly trusted and your self-installed CA-s.
Note: Debuan/Ubuntu systems, CentOS/Alpine probably have this in a different location (ref).
Capturing locust traffic in Fiddler with Locust version 0.11 works just fine. Using a python virtual (3.7) and PyCharm.
I created a second python virtual (also 3.7) for the latest version of Locust v1.1.1. Execution of this version of Locust captured no traffic in Fiddler.
In Fiddler I reset The certificate :
Tools -> Options - HTTPS - Actions
Within PyCharm I've tried both Auto-detect proxy and Manual config under:
Settings -> Appearance and Behavior -> System Settings -> HTTP Proxy
Flipping the virtual env back to the locust 0.11 traces traffic in Fiddler just fine. I don't know what v1.1.1 may be doing differently. Fiddler logs do not show any sign an attempt to connect.
Curious if anyone has encountered similar behavior.
I dont think this is a certificate issue at all (if it was, the request would not go thru)
Locust configures requests to ignore proxy settings, see https://docs.locust.io/en/stable/writing-a-locustfile.html#http-proxy-settings
You can probably re-enable it, but you’ll need to check either locust source or requests documentation.
I am currently using python to write some appium test. Because I am behind a corporate firewall my traffic needs to go via a proxy.
I have set my http_proxy and https_proxy variables, but it seems like this is not being picked up by python during execution.
I tried the exact same test using javascript and node and the proxy get picked up and everything works so I am sure the problem is python not following the proxy settings.
How can I make sure python is using correct proxy settings?
I am using python 2.7 on macos mojave
Thanks!
So I figured out that appium currently does not support options to provide a proxy when making a remote connection. As temporary solution I modified the remote_connection module of selenium that appium inherits forcing it to use a proxy url for the connection.
My python knowledge is not that good but I think it shoudnt take much effort for someone to make a module that wraps/override the appium webdriver remote connection to include a proxy option.
I'm looking to configure an upstream proxy for browsermob, preferably programmatically from within a python or shell script.
It doesn't look like the python bindings for browsermob include an upstream-proxy configuration command or method. Is there another method I can use?
The python bindings do actually allow you to configure an upstream proxy. When creating a proxy using create_proxy, you can set the value of httpProxy to the IP address and port of the upstream proxy (see the params parameter on create_proxy for details).
I've tried to create a twill test that changes the proxy server settings of 2 different tests. I need to trigger this change in runtime without relaunching the test script.
I've tried to use the "http_proxy" environment variable by setting os.environ["HTTP_PROXY"], but it's only changes the proxy setting for the first test, and does not works on the second and third tests.
Could you please suggest a way to change twill's proxy settings on runtime ?
Set the proxy environment variable before you run the twill script.
sh/ksh/bash
export HTTP_PROXY=blah:8080
csh
setenv HTTP_PROXY blah:8080
It's worth nothing, this should work by setting os.environ['http_proxy'], but it might not if you set it after you import twill. Twill may be checking this once on startup? The only 100% safe way I would imagine is exporting the variable so that all further child processes will get it as their environment.