For a script I am making, I need to be able to see the parameters that are sent with a request.
This is possible through Fiddler, but I am trying to automate the process.
Here are some screenshots to start with. As you can see in the first picture of Fiddler, I can see the URL of a request and the parameters sent with that request.
I tried to do some packet sniffing with scapy with the code below to see if I can get a similar result, but what I get is in the second picture. Basically, I can get the source and destination of a packet as ip addresses, but the packets themselves are just bytes.
def sniffer():
t = AsyncSniffer(prn = lambda x: x.summary(), count = 10)
t.start()
time.sleep(8)
results = t.results
print(len(results))
print(results)
print(results[0])
From my understanding, after we establish a TCP connection, the request is broken down into several IP packets and then sent over to the destination. I would like to be able to replicate the functionality of Fiddler, where I can see the url of the request and then the values of parameters being sent over.
Would it be feasible to recreate the information of a request through only the information gathered from the packets?
Or is this difference because the sniffing is done on Layer 2, and then maybe Fiddler operates on Layer 3/4 before/after the translation into IP packets is done, so it actually sees the content of the original request itself and the result of the combination of packets? If my understanding is wrong, please correct me.
Basically, my question boils down to: "Is there a python module I can use to replicate the features of Fiddler to identify the destination url of a request and the parameters sent along with that request?"
The sniffed traffic is HTTPS traffic - therefore just by sniffing you won't see any details on the HTTP request/response because it is encrypted via SSL/TLS.
Fiddler is a proxy with HTTPS interception, that is something totally different compared to sniffing traffic on network level. This means that for the client application Fiddler "mimics" the server and for the server Fiddler mimics the client. This allows Fiddler to decrypt the requests/responses and show them to you.
If you want to perform request interception on python level I would recommend to you to use mitmproxy instead of Fiddler. This proxy also can perform HTTPS interception but it is written in Python and therefore much easier to integrate in your Python environment.
Alternatively if you just want to see the request/response details of a Python program it may be easier to do so by setting the log-level in an appropriate way. See for example this question: Log all requests from the python-requests module
Related
im trying to find a way to make my HTTP request in a exact time for my computer example 13:02:00:00.
Any recomendation?
in my opinion, it depends on your network speed. Anyway, I think that's impossible as when you make a GET request, your request will pass through many network devices such as a router, switch, and finally reaches a server where IP address of the site will be sent to you. Even request that is sent will travel with the speed of light you can't get it in exact time, as some switches will make ARP requests to get to know destination address.
If I make a request in Python 3 to grab the text of a webpage (Via Requests module), is it possible that someone using fiddler could change the response?
short answer: Yes.
a bit longer: fiddler is an HTTP proxy, so if someone wants to intercept your traffic with it he needs to either:
a) configure it as a proxy in your code
b) use some program on your computer that sends your traffic to fiddler
c) use some program on your router that sends your traffic to fiddler
notice that only opening fiddler on your computer isn't enough for the script traffic to go through it, you need to configure it as a proxy in your requests at the code (for example: requests.get('http://example.org', proxies=proxies)).
I'm using a service that sends me some data from user over webhooks. If there is any user interaction on this service, it hits my URL with HTTP request, with the data in POST/GET, and then expects text/json response to show back to the user. The response has to be in few seconds, otherwise the HTTP request times out and the service has no way of finding out what should be the response to the user.
The problem here is that now I'm not processing these data on my server with public IP, but I need to do it on my RPi, which keeps moving, which meains it has different IP every few hours, and mostly not public.
I'm sure I will still need to use the server with public IP to redirect these requests to my RPi, and I have few ideas, but I don't know what is reliable or if it even would work.
Let the API talk to my server and save the data. Then have the RPi constantly asking my server if there are any new data. Propably the dumbest idea - not ideal to use over metered connection, propably longer reply, and it will be harder to return the RPi's reply in the HTTP request made from API.
Having (Python) script running on my server, that will a) serve as socket server and RPi will connect to this socket, and b) have running SimpleHTTPRequestHandler to process requests from API and send them to the socket, the reply with RPi's reply. Propably easy way to keep connection between my server and RPi, allowing me to pass data in both directions.
Open SSH tunnel between the RPi and my server. This way, I could process the requests from service directly on my RPi. But how reliable is this solution? (Keeping it alive, opening the tunnel automatically, etc, propably question for superuser forum)
I'm thinking of going with choice 3 if it will be possible, but first I'd like to hear what you guys think. Is this a good and reliable idea? Or are there any better ways I don't know about? Or did anybody already faced this problem?
To sum it up:
Something sends HTTP request to public IP. I need to process this request (and reply to it) in Python script on device without public IP. I have a server with public IP that could be used as a bridge. I much don't care what will run on the server, if it will be able to redirect these requests.
Thanks
I'm trying to intercept all the requests received by a website, to get a certain file. For example, when you use Firefox's network monitor. Can I do that in Python? Sorry for being so vague. I'd like to get all the URLs that the website requests, like you can see in the picture. Example: the favicon, js files, xml files, etc.
Example:
So you probably need a packet sniffer like tcpdump. The best python sniffer I know is scapy. Here is in example of how HTTP may be sniffed with it:
http://www.r00tsec.com/2013/12/simple-sniffer-http-request-and-http.html
Note that you couldn't do that trick with HTTPS. Also packet sniffing usually requires root privileges on a host system.
I am using a server to send some piece of information to another server every second. The problem is that the other server response is few kilobytes and this consumes the bandwidth on the first server ( about 2 GB in an hour ). I would like to send the request and ignore the return ( not even receive it to save bandwidth ) ..
I use a small python script for this task using (urllib). I don't mind using any other tool or even any other language if this is going to make the request only.
A 5K reply is small stuff and is probably below the standard TCP window size of your OS. This means that even if you close your network connection just after sending the request and checking just the very first bytes of the reply (to be sure that request has been really received) probably the server already sent you the whole answer and the packets are already on the wire or on your computer.
If you cannot control (i.e. trim down) what is the server reply for your notification the only alternative I can think to is to add another server on the remote machine waiting for a simple command and doing the real request locally and just sending back to you the result code. This can be done very easily may be even just with bash/perl/python using for example netcat/wget locally.
By the way there is something strange in your math as Glenn Maynard correctly wrote in a comment.
For HTTP, you can send a HEAD request instead of GET or POST:
import urllib2
request = urllib2.Request('https://stackoverflow.com/q/5049244/')
request.get_method = lambda: 'HEAD' # override get_method
response = urllib2.urlopen(request) # make request
print response.code, response.url
Output
200 https://stackoverflow.com/questions/5049244/how-can-i-ignore-server-response-t
o-save-bandwidth
See How do you send a HEAD HTTP request in Python?
Sorry but this does not make much sense and is likely a violation of the HTTP protocol. I consider such an idea as weird and broken-by-design. Either make the remote server shut up or configure your application or whatever is running on the remote server on a different protocol level using a smarter protocol with less bandwidth usage. Everything else is hard being considered as nonsense.