I am tasked with developing a mitmproxy script, something I have never done before.
I am trying to modify "Location" headers on the fly during http-flow on a reverse proxy instance in mitmproxy, these headers are often different but my main goal right now is to simply detect them in a static manner.
I am running mitmproxy as such:
mitmdump --set flowdetail:3 -s rproxy.py -p 8080 --mode reverse:http://www.com
^ That is not the real URL of question of course.
Say if I have the header being sent from the server
Location: http://www.com/example-location
But I want to change it to
Location: http://localhost:8080/example-location
I have tried every example I could find on google, including the ones on StackOverFlow.
Some things I have tried without any success.
from mitmproxy import http
import sys
def response(flow: http.HTTPFlow) -> None:
flow.request.host = "http://localhost:8080/example-location"
from mitmproxy import http
import sys
def response(flow):
flow.repsonse.headers['Location'] = ["http://localhost:8080/example-location"]
flow.repsonse.headers['Location'] = "http://localhost:8080/example-location" ## I tried this line separately without the above line
And
from mitmproxy import http
import sys
def response(flow: http.HTTPFlow) -> None:
flow.repsonse.headers['Location'] = ["http://localhost:8080/example-location"]
flow.repsonse.headers['Location'] = "http://localhost:8080/example-location" ## I tried this line separately without the above line
How do I do this correctly?
If it is not possible to manipulate this header (but why shouldn't it be) what are options do I have?
Related
I have created a simple HTTP trigger-based azure function in python which is calling another python script to create a sample file in azure data lake gen 1. My solution structure is given below: -
Requirements.txt contains the following imports: -
azure-functions
azure-mgmt-resource
azure-mgmt-datalake-store
azure-datalake-store
init.py
import logging, os, sys
import azure.functions as func
import json
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
full_path_to_script = os.path.join(os.path.dirname( __file__ ) + '/Test.py')
logging.info(f"Path: - {full_path_to_script}")
os.system(f"python {full_path_to_script}")
return func.HttpResponse(f"Hello {name}!")
else:
return func.HttpResponse(
"Please pass a name on the query string or in the request body",
status_code=400
)
Test.py
import json
from azure.datalake.store import core, lib, multithread
directoryId = ''
applicationKey = ''
applicationId = ''
adlsCredentials = lib.auth(tenant_id = directoryId, client_secret = applicationKey, client_id = applicationId)
adlsClient = core.AzureDLFileSystem(adlsCredentials, store_name = '')
with adlsClient.open('stage1/largeFiles/TestFile.json', 'rb') as input_file:
data = json.load(input_file)
with adlsClient.open('stage1/largeFiles/Result.json', 'wb') as responseFile:
responseFile.write(data)
Test.py is failing with an error that no module found azure.datalake.store
Why other required modules are not working for Test.py since it is inside the same directory?
pip freeze output: -
adal==1.2.2
azure-common==1.1.23
azure-datalake-store==0.0.48
azure-functions==1.0.4
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-resource==6.0.0
azure-nspkg==3.0.2
certifi==2019.9.11
cffi==1.13.2
chardet==3.0.4
cryptography==2.8
idna==2.8
isodate==0.6.0
msrest==0.6.10
msrestazure==0.6.2
oauthlib==3.1.0
pycparser==2.19
PyJWT==1.7.1
python-dateutil==2.8.1
requests==2.22.0
requests-oauthlib==1.3.0
six==1.13.0
urllib3==1.25.6
Problem
os.system(f"python {full_path_to_script}") from your functions project is causing the issue.
Azure Functions Runtime sets up the environment, along with modifying process level variables like os.path so that your function can load any dependencies you may have. When you create a sub-process like that, not all information will flow through. Additionally, you will face issues with logging -- logs from test.py would not show up properly unless explicitly handled.
Importing works locally because you have all your requirements.txt modules installed and available to test.py. This is not the case in Azure. After remotely building as part of publish, your modules are included as part of your code package published. It's not "installed" globally in the Azure environment per se.
Solution
You shouldn't have to run your script like that. In the example above, you could import your test.py from your __init__.py file, and that should behave like it was called python test.py (at least in the case above). Is there a reason you'd want to do python test.py in a sub-process over importing it?
Here's the official guide on how you'd want to structure your app to import shared code -- https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python#folder-structure
Side-Note
I think once you get through the import issue, you may also face problems with adlsClient.open('stage1/largeFiles/TestFile.json', 'rb'). We recommend following the developer guide above to structure your project and using __file__ to get the absolute path (reference).
For example --
import pathlib
with open(pathlib.Path(__file__).parent / 'stage1' / 'largeFiles' /' TestFile.json'):
....
Now, if you really want to make os.system(f"python {full_path_to_script}") work, we have workarounds to the import issue. But, I'd rather not recommend such approach unless you have a really compelling need for it. :)
I'm new to Python. This is my first Ansible module in order to delete the SimpleDB domain from ChaosMonkey deletion.
When tested in my local venv with my Mac OS X, it keeps saying
Module unable to decode valid JSON on stdin. Unable to figure out
what parameters were passed.
Here is the code:
#!/usr/bin/python
# Delete SimpleDB Domain
from ansible.module_utils.basic import *
import boto3
def delete_sdb_domain():
fields = dict(
sdb_domain_name=dict(required=True, type='str')
)
module = AnsibleModule(argument_spec=fields)
client = boto3.client('sdb')
response = client.delete_domain(DomainName='module.params['sdb_domain_name']')
module.exit_json(changed = False, meta = response)
def main():
delete_sdb_domain()
if __name__ == '__main__':
main()
And I'm trying to pass in parameters from this file: /tmp/args.json.
and run the following command to make the local test:
$ python ./delete_sdb_domain.py /tmp/args.json
please note I'm using venv test environment on my Mac.
If you find any syntax error in my module, please also point it out.
This is not how you should test your modules.
AnsibleModule expects to have specific JSON as stdin data.
So the closest thing you can try is:
python ./delete_sdb_domain.py < /tmp/args.json
But I bet you have your json file in wrong format (no ANSIBLE_MODULE_ARGS, etc.).
To debug your modules you can use test-module script from Ansible hacking pack:
./hacking/test-module -m delete_sdb_domain.py -a "sdb_domain_name=zzz"
#!/usr/bin/python3
import cgi
import cgitb
import urllib.request
import os
import sys
def enco_print(string="", encoding = "utf8"):
sys.stdout.buffer.write(string.encode(encoding) + b"\n")
cgitb.enable()
form = cgi.FieldStorage(endcoding="utf8")
name_name= form.getvalue("name")
url_name = form.getvalue("url")
response = urllib.request.urlopen(str(url_name))
html = response.read().decode("utf8")
if not os.path.exists("gecrwalt"):
os.mkdir("gecrwalt")
with open("/gecrwalt/" + str(url_name) + ".html", "w", endcoding="utf8")
as f:
f.write(str(html))
When I try to run this script, I get 500 Status Error on my Website. I can´t see what´s wrong with this code.
I´m very thankful for help.
There are a few typos where you wrote endcoding instead of encoding.
The last segment here
with open("/gecrwalt/" + str(url_name) + ".html", "w", endcoding="utf8")
as f:
f.write(str(html))
has broken indentation, not sure if this was due to a copy-paste error here on Stackoverflow.
Another issue here is that (if I understood your code correlty) url_name will contain a complete URL like http://example.com, which will result in an error because that filename is invalid. You will have to come up with some schema to safely store these files, urlencode the URL or take a hash of the URL. Your save path also starts with /, not sure if intentational, starts at the file system root.
Changing the typo and the last bit has worked for me in a quick test:
with open("gecrwalt/something.html", "w", endcoding="utf8") as f:
f.write(str(html))
Debugging hint: I started a local Python webserver process with this command (from here)
python3 -m http.server --bind localhost --cgi 8000
Accessing http://localhost:8000/cgi-bin/filename.py will show you all errors that occur, not sure about the webserver you are currently using (there should be error logs somewhere I guess).
Can my server module (with http.server.HTTPServer) use something like the RewriteRule for redirect all traffic into a single cgi script? I'd like to be able to do what's shown here in this other question, but for my python server.
Can it be done using something like .htaccess, or is there another way?
Also, can this be done even for a simple localhost development server?
I am serving files for development via, for example, http://localhost:8000/html/index.html, and I would like to hide the /html subfolder from the URL even in development.
How can that be achieved?
You can use a custom script to initialize your server and define your Routes in it, such as suggested in this article:
Python 2:
server.py
import os
import posixpath
import urllib
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
# modify this to add additional routes
ROUTES = ( <- this is the "good stuff", make aliases for your paths
# [url_prefix , directory_path]
['/media', '/var/www/media'],
['', '/var/www/site'] # empty string for the 'default' match
)
class RequestHandler(SimpleHTTPRequestHandler):
def translate_path(self, path):
"""translate path given routes"""
# set default root to cwd
root = os.getcwd()
# look up routes and set root directory accordingly
for pattern, rootdir in ROUTES:
if path.startswith(pattern):
# found match!
path = path[len(pattern):] # consume path up to pattern len
root = rootdir
break
# normalize path and prepend root directory
path = path.split('?',1)[0]
path = path.split('#',1)[0]
path = posixpath.normpath(urllib.unquote(path))
words = path.split('/')
words = filter(None, words)
path = root
for word in words:
drive, word = os.path.splitdrive(word)
head, word = os.path.split(word)
if word in (os.curdir, os.pardir):
continue
path = os.path.join(path, word)
return path
if __name__ == '__main__':
BaseHTTPServer.test(RequestHandler, BaseHTTPServer.HTTPServer)
Then run your script:
python server.py
Python 3:
In Python 3 the BaseHTTPServer and the SimpleHTTPServer modules have been merged into the http.server module. So you will have to modify the above script as follows:
Change the
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
to
import http.server
(if you do this then the calls should be modified to http.server.SimpleHTTPRequest etc.)
or
from http.server import BaseHTTPServer, SimpleHTTPServer, SimpleHTTPRequestHandler
(with this option calls remain the same as the original script)
then run the server script: python server.py
IN YOUR CASE:
You should modify the ROUTES variable to suite your needs. For example, you want to hide the /html folder from your url:
ROUTES = (
['', '/exact/path/to/folder/html'],
['/another_url_path', '/exact/path/to/another/folder'],
...
)
Now if you hit: http://localhost:8000/index.html you will be in your home page.
Note:
This script by default will serve the contained files in the folder that is in when executed to the domain url (ex. I have server.py on the Documents folder, then when I run it, http://localhost:8000 url will server my Documents folder).
You can change this behavior by the Routes (see the 'default' match comment on the code) or by placing the script in your projects root folder and start it from there.
John Moutafis's answer was helpful to get me started, but needed some fine tuning to run on python3, beyond his comments about imports.
The imports should be
from http.server import HTTPServer, SimpleHTTPRequestHandler
and you'll also need to change the urllib import to:
from urllib.parse import unquote
Then the main should be something like:
if __name__ == '__main__':
myServer = HTTPServer(('0.0.0.0', 8000), RequestHandler)
print("Ready to begin serving files.")
try:
myServer.serve_forever()
except KeyboardInterrupt:
pass
myServer.server_close()
print("Exiting.")
Sample server
I have a python script as mentioned below copied to /var/www/cgi-bin folder with permissions set to 775.
#!/usr/bin/env python
print "Content-type: text/plain\n\n";
print "testing...\n";
import cgitb; cgitb.enable()
import cgi
from jsonrpc import handleCGI, ServiceMethod
import json
from datetime import datetime
#ServiceMethod
def echo():
return "Hello"
if __name__ == "__main__":
handleCGI()
Sample Client
Now, Iam accessing this simple echo service using the below client code.
from jsonrpc import ServiceProxy
import json
s = ServiceProxy(`"http://localhost/cgi-bin/t2.py"`)
print s.echo()
1/ Iam getting the below error when i run the above client. Any thoughts?
2/ Is there any issue with httpd.conf settings?
File "/usr/lib/python2.7/site-packages/jsonrpc/proxy.py", line 43, in __call__
resp = loads(respdata)
File "/usr/lib/python2.7/site-packages/jsonrpc/json.py", line 211, in loads
raise JSONDecodeException('Expected []{}," or Number, Null, False or True')
jsonrpc.json.JSONDecodeException: Expected []{}," or Number, Null, False or True
Note: Iam using the example mentioned at the below link using cgi way of handling json.
http://json-rpc.org/wiki/python-json-rpc
Please let me know.
Thanks!
Santhosh
I know this is super late, but I found this question when I had the same problem. In hopes it helps someone else, I will post my solution.
In my case it was as simple (stupid) as making the python file itself executable. i.e. chmod 755 t2.py