We Keep Coding

Is it possible to recreate a request from the packets programatically?

For a script I am making, I need to be able to see the parameters that are sent with a request.
This is possible through Fiddler, but I am trying to automate the process.
Here are some screenshots to start with. As you can see in the first picture of Fiddler, I can see the URL of a request and the parameters sent with that request.
I tried to do some packet sniffing with scapy with the code below to see if I can get a similar result, but what I get is in the second picture. Basically, I can get the source and destination of a packet as ip addresses, but the packets themselves are just bytes.
def sniffer():
t = AsyncSniffer(prn = lambda x: x.summary(), count = 10)
t.start()
time.sleep(8)
results = t.results
print(len(results))
print(results)
print(results[0])
From my understanding, after we establish a TCP connection, the request is broken down into several IP packets and then sent over to the destination. I would like to be able to replicate the functionality of Fiddler, where I can see the url of the request and then the values of parameters being sent over.
Would it be feasible to recreate the information of a request through only the information gathered from the packets?
Or is this difference because the sniffing is done on Layer 2, and then maybe Fiddler operates on Layer 3/4 before/after the translation into IP packets is done, so it actually sees the content of the original request itself and the result of the combination of packets? If my understanding is wrong, please correct me.
Basically, my question boils down to: "Is there a python module I can use to replicate the features of Fiddler to identify the destination url of a request and the parameters sent along with that request?"

The sniffed traffic is HTTPS traffic - therefore just by sniffing you won't see any details on the HTTP request/response because it is encrypted via SSL/TLS.
Fiddler is a proxy with HTTPS interception, that is something totally different compared to sniffing traffic on network level. This means that for the client application Fiddler "mimics" the server and for the server Fiddler mimics the client. This allows Fiddler to decrypt the requests/responses and show them to you.
If you want to perform request interception on python level I would recommend to you to use mitmproxy instead of Fiddler. This proxy also can perform HTTPS interception but it is written in Python and therefore much easier to integrate in your Python environment.
Alternatively if you just want to see the request/response details of a Python program it may be easier to do so by setting the log-level in an appropriate way. See for example this question: Log all requests from the python-requests module

Changing IP of python requests

How do I change the IP of HTTP requests in python?
My friend built an API for a website, and sometimes it blocks certain IP's and so we need to change the IP of the request... here is an example:
login_req = self.sess.post('https://www.XXX/YYY', params={...}
Now, each request that it sends, is through the computer's IP, and we need it basically to pass through an imaginary VPN.
Thanks for the help. If something isn't clear I will explain.

Short answer: you can't.
Long answer: it seems like you're misunderstanding how IP addresses work. Your IP address is the network address that corresponds to your computer - when you send a request to a server, you attach your IP as a "return address" of sorts, so that the server can send a response back to you.
However, just like a physical address, you don't get to choose what your IP address is – you live on a street, and that's your address, you don't get to change what the street is called or what your house number is. In general, when you send a request from your computer, the message passes through a chain of devices. For example:
Your computer --> Your router --> Your ISP --> The Server
In a lot of cases, each of these assigns a different IP address to whatever's below it. So, when your request passes through your router, your router records your IP address and then forwards the request through your ISP using its own IP address. Hence how several users on the same network can have the same IP address.
There are physical IP addresses, that correspond directly to devices, but there are a limited amount of these. Mostly, each Internet Service Provider has a few blocks of IP addresses that it can attach to things; an ISP can keep a specific IP address pointed to a specific computer all of the time, but they don't have to, and for many of their regular users, they don't.
Your computer has basically no power to determine what its own IP address is, basically. There's nothing python can do about that.
Your Question:
we need [the request] basically to pass through an imaginary VPN.
It'd be easier to actually requisition a real proxy or VPN from somewhere and push your request through it. You'd have to talk with your internet service provider to get them to set something like that up for you specifically, and unless you're representing a reasonably big company they're unlikely to want to put in that effort. Most python libraries that deal with HTTP can easily handle proxy servers, so once you figure it out it shouldn't be a problem.

You can use an IP address from https://www.sslproxies.org/
For example,
import requests
response=requests.get("yourURL", proxies={'https': 'https://219.121.1.93:80', 'http': http://219.121.1.93:80 "})
The IP addresses on that site are pretty crappy and sometimes don't work, so it would be best to find a way to constantly scrape IP addresses from the site so you have a couple to try. Check out this article: https://www.scrapehero.com/how-to-rotate-proxies-and-ip-addresses-using-python-3/
warning: These should not be used for sensitive information as they are not secure. Don't use those IP addresses unless you are ok with anyone in the world knowing what your're doing.

Redirecting HTTP requests to device without static/public IP

I'm using a service that sends me some data from user over webhooks. If there is any user interaction on this service, it hits my URL with HTTP request, with the data in POST/GET, and then expects text/json response to show back to the user. The response has to be in few seconds, otherwise the HTTP request times out and the service has no way of finding out what should be the response to the user.
The problem here is that now I'm not processing these data on my server with public IP, but I need to do it on my RPi, which keeps moving, which meains it has different IP every few hours, and mostly not public.
I'm sure I will still need to use the server with public IP to redirect these requests to my RPi, and I have few ideas, but I don't know what is reliable or if it even would work.
Let the API talk to my server and save the data. Then have the RPi constantly asking my server if there are any new data. Propably the dumbest idea - not ideal to use over metered connection, propably longer reply, and it will be harder to return the RPi's reply in the HTTP request made from API.
Having (Python) script running on my server, that will a) serve as socket server and RPi will connect to this socket, and b) have running SimpleHTTPRequestHandler to process requests from API and send them to the socket, the reply with RPi's reply. Propably easy way to keep connection between my server and RPi, allowing me to pass data in both directions.
Open SSH tunnel between the RPi and my server. This way, I could process the requests from service directly on my RPi. But how reliable is this solution? (Keeping it alive, opening the tunnel automatically, etc, propably question for superuser forum)
I'm thinking of going with choice 3 if it will be possible, but first I'd like to hear what you guys think. Is this a good and reliable idea? Or are there any better ways I don't know about? Or did anybody already faced this problem?
To sum it up:
Something sends HTTP request to public IP. I need to process this request (and reply to it) in Python script on device without public IP. I have a server with public IP that could be used as a bridge. I much don't care what will run on the server, if it will be able to redirect these requests.
Thanks

Send TCP messages at certain rate with Python

I am trying to generate some traffic to a server by sending TCP messages to it.
For this, I am using a Python script which opens a TCP socket and then sends some data over it. After receiving a reply, the TCP connection gets closed.
Question: I would like to be able to predefine a rate with which the script will be sending the requests to the server, eg: 5 messages per second. However, I do not have a clue how to script this via Python :(.
Anyone an idea how to do this (a short example would be super ! ;) ?
Thanks in advance.
Note: I might need to add an extra difficulty: since the server has to reply,
I guess I have to make the script working asynchronously ... That way, I can
send the requests out without having to wait for a reply on the previous request...

What you're looking for is an implementation of the token bucket algorithm. It's analogous to a bucket with a fixed capacity, where each consumer can't perform the action until it gets a token, and the bucket is refilled at a fixed rate.
The algorithm is easy to implement, but the link below has an example:
http://code.activestate.com/recipes/511490-implementation-of-the-token-bucket-algorithm/

How can i ignore server response to save bandwidth?

I am using a server to send some piece of information to another server every second. The problem is that the other server response is few kilobytes and this consumes the bandwidth on the first server ( about 2 GB in an hour ). I would like to send the request and ignore the return ( not even receive it to save bandwidth ) ..
I use a small python script for this task using (urllib). I don't mind using any other tool or even any other language if this is going to make the request only.

A 5K reply is small stuff and is probably below the standard TCP window size of your OS. This means that even if you close your network connection just after sending the request and checking just the very first bytes of the reply (to be sure that request has been really received) probably the server already sent you the whole answer and the packets are already on the wire or on your computer.
If you cannot control (i.e. trim down) what is the server reply for your notification the only alternative I can think to is to add another server on the remote machine waiting for a simple command and doing the real request locally and just sending back to you the result code. This can be done very easily may be even just with bash/perl/python using for example netcat/wget locally.
By the way there is something strange in your math as Glenn Maynard correctly wrote in a comment.

For HTTP, you can send a HEAD request instead of GET or POST:
import urllib2
request = urllib2.Request('https://stackoverflow.com/q/5049244/')
request.get_method = lambda: 'HEAD' # override get_method
response = urllib2.urlopen(request) # make request
print response.code, response.url
Output
200 https://stackoverflow.com/questions/5049244/how-can-i-ignore-server-response-t
o-save-bandwidth
See How do you send a HEAD HTTP request in Python?

Sorry but this does not make much sense and is likely a violation of the HTTP protocol. I consider such an idea as weird and broken-by-design. Either make the remote server shut up or configure your application or whatever is running on the remote server on a different protocol level using a smarter protocol with less bandwidth usage. Everything else is hard being considered as nonsense.