Capturing locust traffic in Fiddler with Locust version 0.11 works just fine. Using a python virtual (3.7) and PyCharm.
I created a second python virtual (also 3.7) for the latest version of Locust v1.1.1. Execution of this version of Locust captured no traffic in Fiddler.
In Fiddler I reset The certificate :
Tools -> Options - HTTPS - Actions
Within PyCharm I've tried both Auto-detect proxy and Manual config under:
Settings -> Appearance and Behavior -> System Settings -> HTTP Proxy
Flipping the virtual env back to the locust 0.11 traces traffic in Fiddler just fine. I don't know what v1.1.1 may be doing differently. Fiddler logs do not show any sign an attempt to connect.
Curious if anyone has encountered similar behavior.
I dont think this is a certificate issue at all (if it was, the request would not go thru)
Locust configures requests to ignore proxy settings, see https://docs.locust.io/en/stable/writing-a-locustfile.html#http-proxy-settings
You can probably re-enable it, but you’ll need to check either locust source or requests documentation.
Related
I'm learning how to create an EC2 ubuntu instance on AWS to launch Python based applications, all the server settings were properly specified as directed by the instructor(Tutorial last updated 2020), including configurations for anaconda & jupyter notebook & they run just fine on the terminal. But whenever i try accessing jupyter on the server from a browser (chrome & opera) i get an error description of ERR_SSL_VERSION_OR_CIPHER_MISMATCH (The client and server don't support a common SSL protocol version or cipher suite.). I've tried deleting my windows SSL certificate, reenabling browser access through firewall & other probable fixes through firewall & network security but am beginning to suspect its an issue with EC2 server/instance settings that we were given no longer being supported. How do i change these settings to support all traffic & allow access for all local IPs from my computer (Note that it's just supposed to be a personal demo instance & strict connection security rules are not needed)
I am currently using python to write some appium test. Because I am behind a corporate firewall my traffic needs to go via a proxy.
I have set my http_proxy and https_proxy variables, but it seems like this is not being picked up by python during execution.
I tried the exact same test using javascript and node and the proxy get picked up and everything works so I am sure the problem is python not following the proxy settings.
How can I make sure python is using correct proxy settings?
I am using python 2.7 on macos mojave
Thanks!
So I figured out that appium currently does not support options to provide a proxy when making a remote connection. As temporary solution I modified the remote_connection module of selenium that appium inherits forcing it to use a proxy url for the connection.
My python knowledge is not that good but I think it shoudnt take much effort for someone to make a module that wraps/override the appium webdriver remote connection to include a proxy option.
I am writing a Flask Web-Application and use eventlet as the networking library for that application (eventlet is wrapped by Flask-SocketIO to allow asynchronous operation)
Following this guide I have been successfully creating a SSL key- and cert-file which I pass to the WSGI Server
socket_io.run(app,
host=APP_HOST,
port=APP_PORT,
keyfile='ia.key',
certfile='ia.crt')
This works fine but unfortunately Safari / Chrome says that my SSL-Certificate is not trustworthy when I access the page for the first time.
The Chrome-Failure is the following:
NET::ERR_CERT_COMMON_NAME_INVALID
How to I generate a valid SSL Certificate so that the browsers don't show that error when a user connects to the web application the first time!?
That is because it is something called a "Self Signed Certificate", which is not from any trusted company, so any modern browser auto-detects this as an untrusted site. If you are using a UNIX-based operating system, (Linux, or macOS, Fedora, and more), you can use what I am using. You have to generate new certification from a trusted site.
This is what I use to get a TRUSTED certificate that most browsers can use: https://certbot.eff.org/instructions.
I created a Scrapy project with several spiders to crawl some websites. Now I want to use TOR to:
Hide my ip from the crawled servers;
Associate my requests to different ips, simulating accesses from different users.
I have read some info about this, for example:
using tor with scrapy framework, How to connect to https site with Scrapy via Polipo over TOR?
The answers from these links weren't helpful to me. What are the steps that I should take to make Scrapy work properly with TOR?
EDIT 1:
Considering answer 1, I started by installing TOR. As I am using Windows I downloaded the TOR Expert Bundle (https://www.torproject.org/dist/torbrowser/5.0.1/tor-win32-0.2.6.10.zip) and read the chapter about how to configure TOR as a relay (https://www.torproject.org/docs/tor-doc-windows.html.en). Unfortunately there is little or any information about how to do it on Windows. If I unzip the downloaded archive and run the file Tor\Tor.exe nothing happens. However, I can see in the Task Manager that a new process is instantiated. I don't know what is the best way to proceed from here.
After a lot of research, I found a way to setup my Scrapy project to work with TOR on Windows OS:
Download TOR Expert Bundle for Windows (1) and unzip the files to a folder (ex. \tor-win32-0.2.6.10).
The recent TOR's versions for Windows don't come with a graphical user interface (2). It is probably possible to setup TOR only through config files and cmd commands but for me, the best option was to use Vidalia. Download it (3) and unzip the files to a folder (ex. vidalia-standalone-0.2.21-win32). Run "Start Vidalia.exe" and go to Settings. On the "General" tab, point Vidalia to TOR (\tor-win32-0.2.6.10\Tor\tor.exe).
Check on "Advanced" tab and "Tor Configuration File" section the torrc file. I have the next ports configured:
ControlPort 9151
SocksPort 9050
Click Start Tor on the Vidalia Control Panel UI. After some processing you should se on the status the message "Connected to the Tor network!".
Download Polipo proxy (4) and unzip the files to a folder (ex. polipo-1.1.0-win32). Read about this proxy on the link 5.
Edit the file config.sample and add the next lines to it (in the beginning of the file, for example):
socksParentProxy = "localhost:9050"
socksProxyType = socks5
diskCacheRoot = ""
Start Polipo through cmd. Go to the folder where you unzipped the files and enter the next command "polipo.exe -c config.sample".
Now you have Polipo and TOR up and running. Polipo will redirect any request to TOR through port 9050 with SOCKS protocol. Polipo will receive any HTTP request to redirect trough port 8123.
Now you can follow the rest of the tutorial "Torifying Scrapy Project On Ubuntu" (6). Continue in the step where the tutorial explains how to test the TOR/Polipo communications.
Links:
https://www.torproject.org/download/download.html.en
https://tor.stackexchange.com/questions/6496/tor-expert-bundle-on-windows-no-installation-instructions
https://people.torproject.org/~erinn/vidalia-standalone-bundles/
http://www.pps.univ-paris-diderot.fr/~jch/software/files/polipo/
http://www.pps.univ-paris-diderot.fr/~jch/software/polipo/tor.html
http://blog.privatenode.in/torifying-scrapy-project-on-ubuntu
A detailed step-by-step Explanation is here
http://blog.privatenode.in/torifying-scrapy-project-on-ubuntu/
The Basic steps there are:
Install Tor and Polipo (for linux this might require to add a repository).
Configure Polipo to talk with TOR using SOCK Connection (see above link).
Create a custom Middleware to use tor as a http proxy and to randomly change the scrapy user agent
to suppress depreciation warning from above example, write
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
instead of 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
What is your szenario? Have you thought about renting Proxy Servers?
When trying to open a interactive console on pydev I get an error:
Error initializing console.
Unexpected error connecting to console.
Failed to recive suitable Hello response from pydevconsole. Last msg
received: HTTP server returned unexpected status: Service Unavailable
Command Line used: D:\activestate\python.exe -u
D:\Boulot\pydev\eclipse\plugins\org.python.pydev_3.3.3.201401272249\pysrc\pydevconsole.py
51203 51204 51203 51204
Eclipse:
Version: Kepler Service Release 2
Build id: 20140224-0627
Pydev:
PyDev for Eclipse 3.3.3.201401272249
Java:
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)
Any idea how to make it work?
Try following:
In Eclipse under Window/Preferences/General/Network Connections:
--> set Active Provider: Manual
--> check in the box "Proxy bypass" if localhost and 127.0.0.1 are checked
Click Apply and try to open again.
That worked for me
As documented here, the solution is to update IPython
workon project
pip install -U ipython
I was having the same problem and fixed it with this.
(My solution is inspired by #Sala's answer. And I want to add more reasoning here.)
The root cause for my problem is, pydev console is hosted on localhost/127.0.0.1 and I have configured a proxy in my system. So my eclipse tried to visit it via a proxy but my proxy cannot find it since it is not a valid public address.
With #Sala's solution, one working configuration is like below. It essentially prevents eclipse from using the system (i.e. native) proxy.
And you should notice that in the Proxy bypass area, the Provider for localhost and 127.0.0.1 is Manual. Just in consistent with the Proxy entries area.
But what if you do want to use proxy? You need to bypass your native proxy for pydev console.
You can do it in IE -> Internet Options -> Connection -> LAN Settings -> Advanced:
Then, you don't nee to restart your eclipse, and you will see the Provider for the bypassed localhost and 127.0.0.1 has changed to Native now as shown below:
Now the pydev console should be able to start.