This question already has an answer here:
HTTP Requests using a range of IP address on python
(1 answer)
Closed 8 years ago.
I'm writing a python script that will send http requests concurrently to the urls mentioned in a file using python. The script works fine for a single IP address. The OS I'm using is linux. I've generated virtual IP addresses like eth0:1,eth0:2 etc. I want to send HTTP requests using these virtual IP addresses along with the eth0 IP address concurrently. I use the requests module for http requests and threading module for concurrent requests. Kindly help me. I'm trying to develop a web testing tool.
I think you wanted to avoid "Crawl Delay" and do faster crawl on one server!
In this case, Remote Web Server will recognize request from only one IP!!
I think using parallel + curl + python script is more simple and best way.
or use https://pypi.python.org/pypi/pyparallelcurl/0.0.4
or use a lot of servers.
and refer to https://code.google.com/p/httplib2/issues/detail?id=91
Related
I am currently thinking on a project that automatically executes defensive actions such as adding the IP of a DoS attacker to iptables list to drop their requests permanently.
My question is can I intercept the HTTP requests that are coming for another application, using python? For example, can I count how many times an Apache server running on port 80, recieved a HTTP POST request and extract its sender etc.
I tried looking into requests documentation but couldn't find anything relevant.
This question already has answers here:
Requests, bind to an ip
(3 answers)
Closed 8 months ago.
I have several IP addresses configured on my network interface card. I need to specify through which IP address python requests must connect to the web server. Is it possible?
import requests
r = requests.get("http://example.com/foo/bar")
I need to force requests.get to perform http request through a dasignated IP address.
Is there any other http library that support this feature in python?
Are you running a Django/Django-Rest project? Because, if so, you can just specify a custom host and port into the python manage.py runserver command! Like so:
python manage.py runserver 0.0.0.0:8001
Hope I understood the problem
You can use a SourceAddressAdapter from requests-toolbelt:
import requests
from requests_toolbelt.adapters import source
source = source.SourceAddressAdapter('127.0.0.1')
with requests.Session() as session:
session.mount('http://', source)
r = session.get("http://example.com/foo/bar")
What is the python equivalent for following shell command:
curl --interface 10.91.56.2 http:/10.91.55.3/file0.txt
????
I am using CentOS6.5-Linux and I want to send http request from virtual IP addresses like eth0:0,eth0:1,eth0:2,etc simultaneously with eth0. I am actually trying to make one traffic generator tool using python. I have been successful in sending multiple and concurrent http requests and now my next step is to send such requests from multiple ip addresses.I used following cURL command to send request from eth0:1 " curl--interface 10.91.56.2 http:/10.91.55.3/file0.txt" and I was successful in generating traffic from virtual eth0:1. Can anyone guide me how to do this using python? 10.91.56.2 is my virtual eth0:1 IP interface and 10.91.55.3 is my server address... –
Python Urllib2 provides perfect platform for making any HTTP request. In your case you can use urlopen() function...
More about this libary can be found in the below link:
how-to-use-urllib2-in-python
For me, eth0's ip is 10.91.56.3 and eth0:1's ip is 10.91.56.4 so, to generate traffic using 10.91.56.4(eth0:1)
Followed answer by #AKX here
In above answer in 3rd class write your interface's ip instead of 127.0.0.1 eg in my case i did like this:
class BindableHTTPHandler(urllib2.HTTPHandler):
def http_open(self, req):
return self.do_open(BindableHTTPConnectionFactory('10.91.56.4'), req)
This question already has answers here:
Python requests - print entire http request (raw)?
(9 answers)
Closed 8 years ago.
I am looking for a recipe for writing and reading the raw data generated by a requests transaction from files rather than a socket. By "raw data" I mean the bytes just before they are written to or read from the underlying socket. I've tried:
Using "hooks". This seems to be mostly deprecated as the only remaining hook is "response".
mount()ing a custom Adapter. Some aggressive duck-typing here provides access to the underlying httplib.HTTPConnection objects, but the call stack down there is complicated and quite brittle.
The final solution does not need to be general-purpose as I am only interested in vanilla HTTP functionality. I won't be streaming or using the edgier parts of the protocol.
Thanks!
Spawn a thread (import threading). Run an HTTP server in there. You can generate a unique port on demand by socket.socket().bind(0). In the HTTP server, just write the incoming data to a file (perhaps named by timestamp and incoming port number). Then send your requests there.
I created a python server on port 8000 using python -m SimpleHTTPServer.
When I visit this url from my web browser it shows the below content
Now, I want to get the above content using python. So, for that what I did is
>>> import socket
>>> s = socket.socket(
... socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(("localhost", 8000))
>>> s.recv(1024)
But after s.recv(1024) nothing happens it just wait there and prints nothing.
So, my question is how to get above directory content output using python. Also can someone suggest me a tutorial on socket programming with python. I didn't liked the official tutorial that much.
I also observed a strange thing when I try to receive contents using python and nothing happens at that time I cannot access localhost:8000 from my web browser but as soon as I kill my python program I can access it.
Arguably the simplest way to get content over http in python is to use the urllib2 module. For example:
from urllib2 import urlopen
f = urlopen('http://localhost:8000')
for line in f:
print line
This will print out the file hosted by SimpleHTTPServer.
But after s.recv(1024) nothing happens it just wait there and prints nothing.
You simply open a socket and waiting for the data, but it's not how HTTP protocol works. You have to send a request first if you want to receive a response (basically, you have to tell the server which directory you want to list or which file to download). If you really want to, you can send the request using raw sockets to train your skills, but the proper library is highly recommended (see Matthew Adams' response and urllib2 example).
I also observed a strange thing when I try to receive contents using python and nothing happens at that time I cannot access localhost:8000 from my web browser but as soon as I kill my python program I can access it.
This is because SimpleHTTServer is single-threaded and doesn't support multiple connections simultaneously. If you would like to fix it, take a look at the answers here: BasicHTTPServer, SimpleHTTPServer and concurrency.