I'm trying to intercept and modify https content using Mitm Proxy.
It works really well using the GUI but I'd like to use a python script.
I tried this script:
from mitmproxy import http
def request(flow: http.HTTPFlow) -> None:
if flow.request.pretty_url == "https://www.facebook.com/?l":
flow.response = http.HTTPResponse.make(
200, # (optional) status code
b"Hello World, this is a test", # (optional) content
{"Content-Type": "text/html"} # (optional) headers
)
def response(flow: http.HTTPFlow):
if flow.response.raw_content and flow.request.pretty_url == "https://www.facebook.com/?lol":
file = open(b"C:\Users\myuser\PycharmProjects\mitmProx\data.txt", "a")
file.write(str(flow.response.) + "\n")
However, the content I intercept is encrypted and not in clear despite the fact that the content is clear on the web GUI !
Does anyone have an idea why my script intercepts encrypted content and the web GUI prints the clear data?
It looks like you want to use response.text or response.content, not response.raw_content. raw_content contains the raw compressed HTTP message body, whereas .content contains the uncompressed version.
Related
I am communicating with an API using HTTP.client in Python 3.6.2.
In order to upload a file it requires a three stage process.
I have managed to talk successfully using POST methods and the server returns data as I expect.
However, the stage that requires the actual file to be uploaded is a PUT method - and I cannot figure out how to syntax the code to include a pointer to the actual file on my storage - the file is an mp4 video file.
Here is a snippet of the code with my noob annotations :)
#define connection as HTTPS and define URL
uploadstep2 = http.client.HTTPSConnection("grabyo-prod.s3-accelerate.amazonaws.com")
#define headers
headers = {
'accept': "application/json",
'content-type': "application/x-www-form-urlencoded"
}
#define the structure of the request and send it.
#Here it is a PUT request to the unique URL as defined above with the correct file and headers.
uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
#get the response from the server
uploadstep2response = uploadstep2.getresponse()
#read the data from the response and put to a usable variable
step2responsedata = uploadstep2response.read()
The response I am getting back at this stage is an
"Error 400 Bad Request - Could not obtain the file information."
I am certain this relates to the body="C:\Test.mp4" section of the code.
Can you please advise how I can correctly reference a file within the PUT method?
Thanks in advance
uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
will put the actual string "C:\Test.mp4" in the body of your request, not the content of the file named "C:\Test.mp4" as you expect.
You need to open the file, read it's content then pass it as body. Or to stream it, but AFAIK http.client does not support that, and since your file seems to be a video, it is potentially huge and will use plenty of RAM for no good reason.
My suggestion would be to use requests, which is a way better lib to do this kind of things:
import requests
with open(r'C:\Test.mp4'), 'rb') as finput:
response = requests.put('https://grabyo-prod.s3-accelerate.amazonaws.com/youruploadpath', data=finput)
print(response.json())
I do not know if it is useful for you, but you can try to send a POST request with requests module :
import requests
url = ""
data = {'title':'metadata','timeDuration':120}
mp3_f = open('/path/your_file.mp3', 'rb')
files = {'messageFile': mp3_f}
req = requests.post(url, files=files, json=data)
print (req.status_code)
print (req.content)
Hope it helps .
This simple Falcon API will take a HTTP POST with enctype=multipart/form-data and a file upload in the file parameter and print the file's content on the console:
# simple_api.py
import cgi
import falcon
class SomeTestApi(object):
def on_post(self, req, resp):
upload = cgi.FieldStorage(fp=req.stream, environ=req.env)
upload = upload['file'].file.read()
print(upload)
app = falcon.API()
app.add_route('/', SomeTestApi())
One might also use the falcon-multipart middleware to achieve the same goal.
To try it out, run it e.g. with gunicorn (pip install gunicorn),
gunicorn simple_api.py
then use cUrl (or any REST client of choice) to upload a text file:
# sample.txt
this is some sample text
curl -F "file=#sample.txt" localhost:8000
I would like to test this API now with Falcon's testing helpers by simulating a file upload. However, I do not understand yet how to do this (if it is possible at all?). The simulate_request method has a file_wrapper parameter which might be useful but from the documentation I do not understand how this is supposed to be filled.
Any suggestions?
This is what I came up with, which tries to simulate what my Chrome does.
Note that this simulates the case when you are uploading only one file, but you can simply modify this function to upload multiple files, each one separated by two new lines.
def create_multipart(data, fieldname, filename, content_type):
"""
Basic emulation of a browser's multipart file upload
"""
boundry = '----WebKitFormBoundary' + random_string(16)
buff = io.BytesIO()
buff.write(b'--')
buff.write(boundry.encode())
buff.write(b'\r\n')
buff.write(('Content-Disposition: form-data; name="%s"; filename="%s"' % \
(fieldname, filename)).encode())
buff.write(b'\r\n')
buff.write(('Content-Type: %s' % content_type).encode())
buff.write(b'\r\n')
buff.write(b'\r\n')
buff.write(data)
buff.write(b'\r\n')
buff.write(boundry.encode())
buff.write(b'--\r\n')
headers = {'Content-Type': 'multipart/form-data; boundary=%s' %boundry}
headers['Content-Length'] = str(buff.tell())
return buff.getvalue(), headers
You can then use this function like the following:
with open('test/resources/foo.pdf', 'rb') as f:
foodata = f.read()
# Create the multipart data
data, headers = create_multipart(foodata, fieldname='uploadFile',
filename='foo.pdf',
content_type='application/pdf')
# Post to endpoint
client.simulate_request(method='POST', path=url,
headers=headers, body=data)
You can craft a suitable request body and Content-Type using the encode_multipart_formdata function in urllib3, documented here. An example usage:
from falcon import testing
import pytest
import myapp
import urllib3
# Depending on your testing strategy and how your application
# manages state, you may be able to broaden the fixture scope
# beyond the default 'function' scope used in this example.
#pytest.fixture()
def client():
# Assume the hypothetical `myapp` package has a function called
# `create()` to initialize and return a `falcon.App` instance.
return testing.TestClient(myapp.create())
# a dictionary mapping the HTML form label to the file uploads
fields = {
'file_1_form_label': ( # label in HTML form object
'file1.txt', # filename
open('path/to/file1.txt').read(), # file contents
'text/plain' # MIME type
),
'file_2_form_label': (
'file2.json',
open('path/to/file2.json').read(),
'application/json'
)
}
# create the body and header
body, content_type_header = urllib3.encode_multipart_formdata(fields)
# NOTE: modify these headers to reflect those generated by your browser
# and/or required by the falcon application you're testing
headers = {
'Content-Type': content_type_header,
}
# craft the mock query using the falcon testing framework
response = client.simulate_request(
method="POST",
path='/app_path',
headers=headers,
body=body)
print(response.status_code)
Note the syntax of the fields object, which is used as input for the encode_multipart_formdata function.
See Tim Head's blog post for another example:
https://betatim.github.io/posts/python-create-multipart-formdata/
Falcon testing example copied from their docs:
https://falcon.readthedocs.io/en/stable/api/testing.html
I'm attempting to pull information from a log file posted online and read through the output. The only information i really need is posted at the end of the file. These files are pretty big and storing the entire socket output to a variable and reading through it is consuming alot of internal memory. is there a was to read the socket from bottom to top?
What I currently have:
socket = urllib.urlopen(urlString)
OUTPUT = socket.read()
socket.close()
OUTPUT = OUTPUT.split("\n")
for line in OUTPUT:
if "xxxx" in line:
print line
I am using Python 2.7. I pretty much want to read about 30 lines from the very end of the output of Socket.
What you want in this use case is the HTTP Range request. Here is tutorial I located:
http://stuff-things.net/2015/05/13/web-scale-http-tail/
I should clarify: the advantage of getting the size with a Head request, then doing a Range request, is that you do not have to transfer all the content. You mentioned you have pretty big file resources, so this is going to be the best solution :)
edit: added this code below...
Here is a demo (simplified) of that blog article, but translated into Python. Please note this will not work with all HTTP servers! More comments inline:
"""
illustration of how to 'tail' a file using http. this will not work on all
webservers! if you need an http server to test with you can try the
rangehttpserver module:
$ pip install requests
$ pip install rangehttpserver
$ python -m RangeHTTPServer
"""
import requests
TAIL_SIZE = 1024
url = 'http://localhost:8000/lorem-ipsum.txt'
response = requests.head(url)
# not all servers return content-length in head, for some reason
assert 'content-length' in response.headers, 'Content length unknown- out of luck!'
# check the the resource length and construct a request header for that range
full_length = int(response.headers['content-length'])
assert full_length > TAIL_SIZE
headers = {
'range': 'bytes={}-{}'.format( full_length - TAIL_SIZE, full_length)
}
# Make a get request, with the range header
response = requests.get(url, headers=headers)
assert 'accept-ranges' in response.headers, 'Accept-ranges response header missing'
assert response.headers['accept-ranges'] == 'bytes'
assert len(response.text) == TAIL_SIZE
# Otherwise you get the entire file
response = requests.get(url)
assert len(response.text) == full_length
Using the poster.encode module, this works when I post a whole file to Solr:
f = open(filePath, 'rb')
datagen, headers = multipart_encode({'file': f})
# use wt=json because it's more convenient to navigate
request = urllib2.Request(SOLR_BASE_URL + 'update/extract?extractOnly=true&extractFormat=text&indent=true&wt=json', datagen, headers) # assumes solrPath ends in '/'
extracted = urllib2.urlopen(request).read()
However, for some files I'd like to send only the first n bytes of the files. I thought this would work:
f = open(filePath, 'rb')
mp = MultipartParam('file', fileobj=f, filesize=f)
datagen, headers = multipart_encode({'file': mp})
# use wt=json because it's more convenient to navigate
request = urllib2.Request(SOLR_BASE_URL + 'update/extract?extractOnly=true&extractFormat=text&indent=true&wt=json', datagen, headers) # assumes solrPath ends in '/'
extracted = urllib2.urlopen(request).read()
...but I get a timed out request (and the odd thing is that I then have to restart apache before requests to my web2py app work again). I get a 'http 400 content missing' error from urlopen() when I leave off the filesize argument. Am I just using MultipartParam incorrectly?
(The point of all this is that I'm using Solr to extract text content and metadata from files. For video and audio files, I'd like to get away with sending just the first 100-300k or so, as presumably the relevant data's all in the file headers.)
The reason you're having trouble is that mime encoding introduces sentinels in the post, if you don't specify the file size - that means that you have to do chunked transfer encoding so that the web server knows when to stop reading the file. But, that's the other problem - if you stop sending a MIME encoded POST to a server mid-stream, it'll just sit there waiting for the block to finish. Chunked transfer encoding and mixed-multipart mime encoding are both dead serious when it comes down to message segment sizes.
If you only want to send 100-300k of data, then only read that much, then every post you make to the server will terminate at the byte you want and the web server is expecting.
I am fairly new to Python and using Python 3.2. I am trying to write a python script that will pick a file from user machine (such as an image file) and submit it to a server using REST based invocation. The Python script should invoke a REST URL and submit the file when the script is called.
This is similar to multipart POST that is done by browser when uploading a file; but here I want to do it through Python script.
If possible do not want to add any external libraries to Python and would like to keep it fairly simple python script using the core Python install.
Can some one guide me? or share some script example that achieve what I want?
Requests library is what you need. You can install with pip install requests.
http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
>>> url = 'http://httpbin.org/post'
>>> files = {'file': open('report.xls', 'rb')}
>>> r = requests.post(url, files=files)
A RESTful way to upload an image would be to use PUT request if you know what the image url is:
#!/usr/bin/env python3
import http.client
h = http.client.HTTPConnection('example.com')
h.request('PUT', '/file/pic.jpg', open('pic.jpg', 'rb'))
print(h.getresponse().read())
upload_docs.py contains an example how to upload a file as multipart/form-data with basic http authentication. It supports both Python 2.x and Python 3.
You could use also requests to post files as a multipart/form-data:
#!/usr/bin/env python3
import requests
response = requests.post('http://httpbin.org/post',
files={'file': open('filename','rb')})
print(response.content)
You can also use unirest . Sample code
import unirest
# consume async post request
def consumePOSTRequestSync():
params = {'test1':'param1','test2':'param2'}
# we need to pass a dummy variable which is open method
# actually unirest does not provide variable to shift between
# application-x-www-form-urlencoded and
# multipart/form-data
params['dummy'] = open('dummy.txt', 'r')
url = 'http://httpbin.org/post'
headers = {"Accept": "application/json"}
# call get service with headers and params
response = unirest.post(url, headers = headers,params = params)
print "code:"+ str(response.code)
print "******************"
print "headers:"+ str(response.headers)
print "******************"
print "body:"+ str(response.body)
print "******************"
print "raw_body:"+ str(response.raw_body)
# post sync request multipart/form-data
consumePOSTRequestSync()
You can check out this post http://stackandqueue.com/?p=57 for more details