I'm using an external service to manage my users (called stormpath) who charges me per api call. Lately the api calls has increased greatly and I want to
log to file or to stdout all the outgoing http calls from the server (debian)
In my localhost I've done this with many hacks involving monkey patching the pips using mitmproxy
obviously , I can't do it in my production server.
How can I log the https calls and then grep only the relevant ones ?
Is there a simple plugin for flask that will do set a callback every time there is an outgoing HTTPS request from the Flask service outside ?
in my localhost the mitmproxy output looks like this
trying to get the same output using python tools in my production server .
After checking through the stormpath-sdk-python library it seems all requests goes through the stormpath.http.HttpExecutor class, specifically the request method of the HttpExecutor class.
And it seems they previously had logging available for requests but chose to remove it due to problems with encoding.
An option would be to fork the stormpath-sdk-python library and add logging or a callback hook that will suit you.
Related
If I develop a REST service hosted in Apache and a Python plugin which services GET, PUT, DELETE, PATCH; and this service is consumed by an Angular client (or other REST interacting browser technology). Then how do I make it scale-able with RabbitMQ (AMQP)?
Potential Solution #1
Multiple Apache's still faces off against the browser's HTTP calls.
Each Apache instance uses an AMQP plugin and then posts message to a queue
Python microservices monitor a queue and pull a message, service it and return response
Response passed back to Apache plugin, in turn Apache generates the HTTP response
Does this mean the Python microservice no longer has any HTTP server code at all. This will change that component a lot. Perhaps best to decide upfront if you want to use this pattern as it seems it would be a task to rip out any HTTP server code.
Other potential solutions? I am genuinely puzzled as to how we're supposed to take a classic REST server component and upgrade it to be scale-able with RabbitMQ/AMQP with minimal disruption.
I would recommend switching wsgi to asgi(nginx can help here), Im not sure why you think rabbitmq is the solution to your problem, as nothing you described seems like that would be solved by using this method.
asgi is not supported by apache as far as I know, but it allows the server to go do work, and while its working it can continue to service new requests that come in. (gross over simplification)
If for whatever reason you really want to use job workers (rabbitmq, etc) then I would suggest returning to the user a "token" (really just the job_id) and then they can call with that token, and it will report back either the current job status or the result
I have built an application utilizing Python that I have integrated successfully with Alexa. Since this application is very specific to my household, I was not too concerned about the deployment process since you can run in development mode forever (or it would seem).
However, in reading about the deployment process, specifically the security requirements to verify that the requests are actually coming from Amazon and not someone else, I learned that those same requirements are a good idea regardless!
So in order to deploy, Amazon requires that you verify requests from Amazon to your Alexa App. Basically, it is broken down into two sections:
Verify that the request is actually coming from Amazon
Verify the timestamp to prevent replay attacks
My entire application is built in Python, but the Alexa front-end is built in Flask-Ask and utilizes Apache2 and WSGI. There are plenty of resources around for learning how to verify Amazon requests utilizing Java, JS, and even some straight Python, but I could not find anything at all on how to accomplish this specifically utilizing Flask-Ask.
So my question is how do I accomplish this utilizing Flask-ask?
Thanks!
According to the source code (and also the documentation, which I cannot access right now because the site is down) there are these relevant config options:
The Ask instance is given the following configuration variables by
calling on Flask's configuration:
ASK_APPLICATION_ID:
Turn on application ID verification by setting this variable to an application ID or a
list of allowed application IDs. By default, application ID verification is disabled and a
warning is logged. This variable should be set in production to ensure
requests are being sent by the applications you specify.
Default: None
ASK_VERIFY_REQUESTS:
Enables or disables Alexa request verification, which ensures requests sent to your skill
are from Amazon's Alexa service. This setting should not be disabled in production.
It is useful for mocking JSON requests in automated tests.
Default: True
So, by default every request is verified already.
In addition, you can add ASK_APPLICATION_ID to make sure the request is from your skill:
app = Flask(__name__)
ask = Ask(app, '/')
app.config["ASK_APPLICATION_ID"] = ["skill-id-1", "skill-id-2"] # List of allowed IDs
Originally, I tried to post an ajax request from my client side to a third party url, but it seems that the browser have security issues with that. I thought about sending an ajax to the server side, from there to send a GET request to the third party, get the response and send it back to the client side. How can I do that with flask?
Install the requests module (much nicer than using urllib2) and then define a route which makes the necessary request - something like:
import requests
from flask import Flask
app = Flask(__name__)
#app.route('/some-url')
def get_data():
return requests.get('http://example.com').content
Depending on your set up though, it'd be better to configure your webserver to reverse proxy to the target site under a certain URL.
Flask alone does not have this capability, but it is a simple matter to write a request handler that makes a request to another server using an HTTP client library and then return that response.
# third-party HTTP client library
import requests
# assume that "app" below is your flask app, and that
# "Response" is imported from flask.
#app.route("/proxy-example")
def proxy_example():
r = requests.get("http://example.com/other-endpoint")
return Response(
r.text,
status=r.status_code,
content_type=r.headers['content-type'],
)
However, this will not achieve exactly the same result as you might expect from a client-side request. Since your server cannot "see" any cookies that the client browser has stored for the target site, your proxied request will be effectively anonymous and so, depending on the target site, may fail or give you a different response than you'd get requesting that resource in the browser.
If you have a relationship with the third-party URL (that is, if you control it or are able to work with the people who do) they can give access for cross-domain requests in the browser using CORS (which is only supported in modern browsers) or JSON-P (an older workaround that predates CORS).
The third-party provider could also give you access to the data you want at an endpoint that is designed to accept requests from other servers and that provides a mechanism for you to authenticate your app. The most popular protocol for this is OAuth.
As the other answers have stated using the requests module for python would be the best way to tackle this from the coding perspective. However as the comments mentioned (and the reason I came to this question) this can give an error that the request was denied. This error is likely cause by SELinux.
To check if this is the issue first make sure SELinux is enabled with this command:
sestatus
If 'Current Mode' is 'enforcing' then SELinux is enabled.
Next get the current bool values that apply directly to apache with this command:
getsebool -a | grep httpd
Look for the setting 'httpd_can_network_connect' this determines if apache is allowed to make TCP requests out to the network. If it is on then all apache TCP requests will be allowed. To turn it on run the following as root:
setsebool -P httpd_can_network_connect 1
If you only need database access (I had this problem before which is why I suspected SELinux here) then it would probably be better to only turn on 'httpd_cna_network_connect'.
The purpose of this policy is that if a hacker was to hijack your apache server they would not be able to get out through the server to the rest of your internal network.
This probably would've been better as a comment but I don't have enough rep..
Sources:
https://tag1consulting.com/blog/stop-disabling-selinux
https://wiki.centos.org/TipsAndTricks/SelinuxBooleans
I have been doing lots of searching and reading to solve this.
The main goal is let a Django-based web management system connecting to a device which runs a http server as well. Django will handle user request and ask device for the real data, then feedback to user.
Now I have a "kinda-work-in-concept" solution:
Browser -> Apache Server: Browser have jQuery and HTML/CSS to collect user request.
Apache Server-> Device HTTP Server:
Apache + mod_python(or somesay Apache + mod_wsgi?) , so I might control the Apache to do stuff like build up a session and cookies to record login.
But, this is the issue actually bugs me.
How to make it work? Using what to build up socket connection between this two servers?
You could use httplib or urllib2 (both supplied in the Python standard library) in your Django view to send HTTP requests to the device running mongoose.
Alternatively you could use the Requests library which provides a less verbose API for generating HTTP requests - see also this blog post.
(Also, I would strongly recommend that you use mod_wsgi rather than mod_python as mod_wsgi is being actively maintained and performs better than mod_python, which was last updated in 2007)
If you have control over what runs on the device side, consider using XML-RPC to talk from client to server.
I am trying to move away from CherryPy for a web service that I am working on and one alternative that I am considering is Tornado. Now, most of my requests look on the backend something like:
get POST data
see if I have it in cache (database access)
if not make multiple HTTP requests to some other web service which can take even a good few seconds depending on the number of requests
I keep hearing that one should not block the tornado main loop; I am wondering if all of the above code is executed in the post() method of a RequestHandler, does this mean that I am blocking the code ? And if so, what's the appropriate approach to use tornado with the above requirements.
Tornado comes shipped with an asynchronous (actually two iirc) http client (AsyncHTTPClient). Use that one if you need to do additional http requests.
The database lookup should also be done using an asynchronous client in order to not block the tornado ioloop/mainloop. I know there are a couple of tornado tailor made database clients (e.g redis, mongodb) out there. The mysql lib is included in the tornado distribution.