I am looking for the most efficient method of hosting a HTTP endpoint to receive Cloudflare log files.
Cloudflare Logpush now supports the ability to send logs to configurable HTTP endpoints.
Note that when using Logpush to HTTP endpoints, Cloudflare customers are expected to perform their own authentication of the pushed logs. For example, customers may specify a secret token in the URL or an HTTP header of the Logpush destination.
Enable HTTP destination
I was looking at nginx to host the end point, but the upload module is no longer supported and it looks like apache does not have such function. I have found various projects that achieve this, however most of them seem to have way more functionality than what I require. I did post the question on the Cloudflare community, but have yet to have a bite.
If you guys can suggest some simple solution to host the http endpoint, I would be very grateful.
Related
I'm working on an API registration and authentication service application using python. Developers will be able to register their application (domain name of the application) and a random API key will be generated for the registered application.
Next, the registered application will send the API key to the API service with each API request. API server will authenticate the domain of the incoming request with the passed API key to confirm that the request is valid. I'm using Forwarded Host to validated the domain name of the API request, however it doesn't work as in some cases (when the opened page is the first page), Forward Host comes blank.
Are there a better approach to authenticate the request or any changes required in the API registration process to reliably authenticate the request? Some pointers will be helpful.
Using Authorization proxy
Samples are "3scale.net", offering free tier, other commercial solutions exist too.
Open source solution I am aware of is ApiAxle, which is much simpler, but still very useful.
The proxy takes care of managing access keys and forwards request back to real application only in case, it is really to be served.
Using Authorization service
Another solution is to have some internal service evaluating set of client provided keys (providerid, appid, accesskey, ...) are authrized or not. For this purpose, you have to:
set up authorization service
modify your code by adding 2-3 lines at the top of each call calling the authentication service.
Sample code for 3scale is here: https://github.com/3scale/3scale_ws_api_for_python
Conclusions
Authentication proxy makes the application simple and not bothering about who is asking. This can be advantage until your application needs to know who is asking.
Authentication service requires changing your code.
I want to have a front-end server where my clients can connect, and depending on the client, be redirected (transparently) to another Flask application that will handle the specific client needs (eg. there can be different applications).
I also want to be able to add / remove / restart those backend clients whenever I want without killing the main server for the other clients.
I'd like the clients to:
not detect that there are other servers in the backend (the URL should be the same host)
not have to reenter their credentials when they are redirected to the other process
What would be the best approach?
The front-end server that you describe is essentially what is known as a reverse proxy.
The reverse proxy receives requests from clients and forwards them to a second line of internal servers that clients cannot reach directly. Typically the decision of which internal server to forward a request to is made based on some aspect of the request URL. For example, you can assign a different sub-domain to each internal application.
After the reverse proxy receives a response from the internal server it forwards it on to the client as if it was its own response. The existence of internal servers is not revealed to the client.
Solving authentication is simple, as long as all your internal servers share the same authentication mechanism and user database. Each request will come with authentication information. This could for example be a session cookie that was set by the login request, direct user credentials or some type of authentication token. In all cases you can validate logins in the same way in all your applications.
Nginx is a popular web server that works well as a reverse proxy.
Sounds like you want a single sign-on setup for a collection of service endpoints with a single entry point.
I would consider deploying all my services as Flask applications with no knowledge of how they are to be architected. All they know is all requests for resources need some kind of credentials associated with them. The manner you pass those credentials can vary. You can use something like the FAS Flask Auth Plugin to handle authentication. Or you can do something simpler like package the credentials provided to your entry service in the HTTP headers of the subsequent requests to other services. Flask.request.headers in your subsequent services will give you access to the right headers to pass to your authentication service.
There are a lot of ways you can go when it comes to details, but I think this general architecture should work for you.
Originally, I tried to post an ajax request from my client side to a third party url, but it seems that the browser have security issues with that. I thought about sending an ajax to the server side, from there to send a GET request to the third party, get the response and send it back to the client side. How can I do that with flask?
Install the requests module (much nicer than using urllib2) and then define a route which makes the necessary request - something like:
import requests
from flask import Flask
app = Flask(__name__)
#app.route('/some-url')
def get_data():
return requests.get('http://example.com').content
Depending on your set up though, it'd be better to configure your webserver to reverse proxy to the target site under a certain URL.
Flask alone does not have this capability, but it is a simple matter to write a request handler that makes a request to another server using an HTTP client library and then return that response.
# third-party HTTP client library
import requests
# assume that "app" below is your flask app, and that
# "Response" is imported from flask.
#app.route("/proxy-example")
def proxy_example():
r = requests.get("http://example.com/other-endpoint")
return Response(
r.text,
status=r.status_code,
content_type=r.headers['content-type'],
)
However, this will not achieve exactly the same result as you might expect from a client-side request. Since your server cannot "see" any cookies that the client browser has stored for the target site, your proxied request will be effectively anonymous and so, depending on the target site, may fail or give you a different response than you'd get requesting that resource in the browser.
If you have a relationship with the third-party URL (that is, if you control it or are able to work with the people who do) they can give access for cross-domain requests in the browser using CORS (which is only supported in modern browsers) or JSON-P (an older workaround that predates CORS).
The third-party provider could also give you access to the data you want at an endpoint that is designed to accept requests from other servers and that provides a mechanism for you to authenticate your app. The most popular protocol for this is OAuth.
As the other answers have stated using the requests module for python would be the best way to tackle this from the coding perspective. However as the comments mentioned (and the reason I came to this question) this can give an error that the request was denied. This error is likely cause by SELinux.
To check if this is the issue first make sure SELinux is enabled with this command:
sestatus
If 'Current Mode' is 'enforcing' then SELinux is enabled.
Next get the current bool values that apply directly to apache with this command:
getsebool -a | grep httpd
Look for the setting 'httpd_can_network_connect' this determines if apache is allowed to make TCP requests out to the network. If it is on then all apache TCP requests will be allowed. To turn it on run the following as root:
setsebool -P httpd_can_network_connect 1
If you only need database access (I had this problem before which is why I suspected SELinux here) then it would probably be better to only turn on 'httpd_cna_network_connect'.
The purpose of this policy is that if a hacker was to hijack your apache server they would not be able to get out through the server to the rest of your internal network.
This probably would've been better as a comment but I don't have enough rep..
Sources:
https://tag1consulting.com/blog/stop-disabling-selinux
https://wiki.centos.org/TipsAndTricks/SelinuxBooleans
I have been trying to do something which I think should be pretty simple. The situation is as follows. The client makes a request for a resource on my web server. My flask application processes the request and determines that this resource is located at a certain location on another web server and the client should make a request of that server instead.
I know I can use the redirect function to tell the client to send a request to the remote location, but my problem is that the remote location is the Amazon Glacier servers. These servers require a request to be made in a certain way, with a special signature (see http://docs.aws.amazon.com/amazonglacier/latest/dev/amazon-glacier-signing-requests.html). My flask application knows how to go about the business of making these requests in the required way. I essentially want to know if it's possible to send a response to my client saying, send this request (generated by my application, with all the required signing) to the Amazon server?
Any ideas?
If the request can be encoded with get params like
http://www.redirecturl.com/?param1=bla¶m2=blub
then it should work no problem. Just construct the request as a string and pass it to redirect().
As far as i know, you can't tell a client to send specific headers to a HTTP redirect URL.
Hitting the Glacier URL serverside would be the easiest. Using javascript on the clientside would only work if Glacier is implementing CORS.
I have an application that has a "private" REST API; I use RESTful URLs when making Ajax calls from my own webpages. However, this is unsecure, and anyone could make those same calls if they knew the URL patterns.
What's the best (or standard) way to secure these calls? Is it worth looking at something like OAuth now if I intend to release an API in the future, or am I mixing two separate strategies together?
I am using Google App Engine for Python and Tipfy.
Definitely take a look at OAuth
It is quickly becoming the "de-facto" standard for securing REST APIs and a lot of big companies are using it, including Google, Twitter and Facebook just to name a few.
For Python on GAE you have two options:
The most straightforward way (IMHO) is using David Larlet's library for OAuth Support in Django available on BitBucket.
But since you're not using Django, maybe you want to take a look at the python-oauth2 library that's available on GitHub, and is considered the most up-to-date and unit-tested implementation of OAuth for Python 2.4+.
Either way I think you'd be much better using OAuth than rolling your own service security solution.
Securing a javascript client is nearly impossible; at the server, you have no fool-proof way to differentiate between a human using a web browser and a well-crafted script.
SSL encrypts data over the wire but decrypts at the edges, so that's no help. It prevents man-in-the-middle attacks, but does nothing to verify the legitimacy of the original client.
OAuth is good for securing requests between two servers, but for a Javascript client, it doesn't really help: anyone reading your javascript code can find your consumer key/secret, and then they can forge signed requests.
Some things you can do to mitigate API scraping:
Generate short-lived session cookies when someone visits your website. Require a valid session cookie to invoke the REST API.
Generate short-lived request tokens and include them in your website HTML; require a valid request token inside each API request.
Require users of your website to log in (Google Accounts / OpenID); check auth cookie before handling API requests.
Rate-limit API requests. If you see too many requests from one client in a short time frame, block them.
OAuth would be overkill in your current scenario (potentially insecure), in that it's designed to authorize a third party service to have access to resources on behave of the user.
Securing AJAX request via an authorized user
AFAICT, you are in control of the client, resources and authentication; so you only need to secure access to the URL's and possibly the communication between client and server via SSL [2].
So, use Tipfy's auth extension to secure your URLs:
from tipfy import RequestHandler, Response
from tipfy.ext.auth import AppEngineAuthMixin, user_required
class MyHandler(RequestHandler, AppEngineAuthMixin):
#user_required
def get(self, **kwargs):
return Response('Only logged in users can see this page.')
Securing AJAX request without an authorized user
If a user is unknown, then one could apply CSRF preventions to help protect the REST service from being called from an "unauthorized" client. Tipfy has this built-in to it's WTForms extension, but it's not AJAX. Instead, the session extension could be used to apply an "authenticity_token" to all calls, that needs to be verified on the server.