I'm writing a Django app that uses a REST api I created. The purpose is to prove api use cases using the web app. In my view, I therefore call the api using the python-requests library like so:
def my_view_method(request):
if request.method == 'POST':
form = MyForm(request.POST)
if form.is_valid():
data = form.cleaned_data
data_to_post = {
'fieldA': data.get('fieldA_in_form'),
'fieldB': data.get('fieldB_in_form'),
}
post_url = "http://%s/%s/" % (request.get_host(), 'entries')
logger.info("request api url: "+ post_url)
r = requests.post(post_url, data=data_to_post)
return HttpResponseRedirect('/')
else:
form = MyForm()
return render(request, 'myview.html', { 'form': form })
I have verified using Unit Tests that POSTing to /entries/ with valid data results in the correct database updates.
url = '/entries/'
#verify initial db state
data = { 'fieldA': value1, 'fieldB': value2 }
response = self.client.post(url, data, format='json')
# verify db was updated
In my Functional Tests, I use LiveServerTestCase and interact with the Form. When the test submits the form, the browser tab shows "Connecting..." in the title and the test case hangs. It wasn't so when I was directly interacting with the database instead of calling the api using requests, so that must be the source of the delay.
Is there something about how LiveServerTestCase works that I'm not understanding here?
Could it be that the LiveServerTestCase server can only handle a single request at a time? So it hangs because it can't deal with a request-from-within-a-request?
The source says it will "handle one request at a time", but then again it says "without blocking", so that's ambiguous...
I think you're going to be best off dropping LiveServerTestCase and just rolling your own test runner. You can use setUp to spin up runserver in a separate process, and tearDown to reset the database (manage.py flush?). If you want to use a test database, you could use a different settings.py, or just move the "real" database out of the way for the duration of the test...
The request is hanging after the test is complete because you're making the request via the requests library and not the official test client described here.
You don't even need to do this though, it makes more sense to test the API directly and not spin up a web server through Django.
This is a decription of LiveServerTestCase from the documentation here
LiveServerTestCase launches a live Django server in the background on
setup, and shuts it down on teardown. This allows the use of automated
test clients such as, for example, the Selenium client, to execute a
series of functional tests inside a browser and simulate a real user’s
actions.
There's no need to even initalise the Django application if you want to test the API itself.
One method for testing I like to follow for problems like this is the component integration methodology, sort of described on Wikipedia here.
In this example we would test the Django application (the front end) separately from the service layer (the API). You may still use the API's when testing the front end, but the separation exists to define how you write the tests.
Since the API is a Flask application, I would use Flasks built in testing tools described here.
The API has nothing to do with Django, yes it is consumed by a Django application, and that application has a hard dependency on that API, but you're wanting to specifically test the API itself.
To aid in UI testing, you could use the API as part of your test fixture/setUp routine to save yourself from using the UI to add any test data needed for execution. However, if you're wanting to test the API itself then doing it through the Django client is not going to work due to the issue mentioned above.
Related
What is the best practice for creating resources while unit testing a GET endpoint in a REST API?
Should I create them directly in the database:
def test_can_get_todos_list(self):
todo1 = Todo(text='todo1', user_id=self.user.id)
todo2 = Todo(text='todo2', user_id=self.user.id)
db.session.add_all([todo1, todo2]) # creating expected response todos directly in the database
db.session.commit()
response = self.client.get('api/todos')
result = [
{'text': 'todo1', 'id': 1},
{'text': 'todo2', 'id': 2},
]
self.assertEqual(response.status_code, 200)
self.assertEqual(result, response.json)
or should I utilize another API endpoint (POST) that I built for creating resources of this type. So instead of doing:
db.session.add_all([todo1, todo2])
db.session.commit()
I could do something like this:
self.client.post(/api/todos, data='todos payload here')
When testing the API as an integration test - i.e. as a completely external test - use the API for every part of the test. Having some parts use the API and some parts use the internal API will make the test more brittle (i.e. changes to either functionality might need updating the test).
Instead the test should use the API of the same abstraction level of the functionality that the tests implement. If you're using the external API, you should use the external API for any feature of the tests. That will also allow you to run the tests externally from your own code, and they will serve as both documentation and examples of how to use the API.
Implementing the tests in this manner will also expose any shortcomings of the API; if you discover functionality that you can't implement through the API but is required for writing the test, then you've found a use case that isn't covered by your current API.
Many frameworks also feature dedicated test clients that sets up the test environment for you, so that the requests doesn't have to actually set up a listening socket and run a http server. This speeds up the tests when testing the external API.
I would like to upgrade my app engine python version from Python 2 to Python 3. But in second generation app engine we cannot use login field in the handler in app.yaml to make certain pages in app engine only accessible to admin.
As per the guidelines Google suggests as follows: The login field is not supported. Use Cloud Identity and Access Management for user management.
I am not able to figure how can I use Identity and Access Management to control login access?
Are you trying to have admin only endpoints that can actually be used called by an admin user? Or are you trying to have admin only endpoints that are only meant to run cron jobs and/or enqueue tasks?
If it is the former (i.e. have pages/handlers that will actually be viewed by admin personnel), then the dcumentation here may be what you're looking for. Unfortunately, as I have noticed with app engine documentation, you may have to read pages upon pages of "theory" and never see sample code you can try to actually use. My guess however is that you will probably end up writing a decorator to check user authorization and authentication, found below.
If you are only trying to limit access to endpoints to secure running cron jobs and queueing tasks, then you are probably looking for this and this solution. Basically, you write a decorator to verify if the endpoint/handler is being called by a cron job or a task queue. Here's working code which should be good to run:
# main.py
from flask import Flask, request, redirect, render_template
app = Flask(__name__)
# Define the decorator to protect your end points
def validate_cron_header(protected_function):
def cron_header_validator_wrapper(*args, **kwargs):
# https://cloud.google.com/appengine/docs/standard/python3/scheduling-jobs-with-cron-yaml#validating_cron_requests
header = request.headers.get('X-Appengine-Cron')
# If you are validating a TASK request from a TASK QUEUE instead of a CRON request, then use 'X-Appengine-TaskName' instead of 'X-Appengine-Cron'
# example:
# header = request.headers.get('X-Appengine-TaskName')
# Other possible headers to check can be found here: https://cloud.google.com/tasks/docs/creating-appengine-handlers#reading_app_engine_task_request_headers
# If the header does not exist, then don't run the protected function
if not header:
# here you can raise an error, redirect to a page, etc.
return redirect("/")
# Run and return the protected function
return protected_function(*args, **kwargs)
# The line below is necessary to allow the use of the wrapper on multiple endpoints
# https://stackoverflow.com/a/42254713
cron_header_validator_wrapper.__name__ = protected_function.__name__
return cron_header_validator_wrapper
#app.route("/example/protected/handler")
#validate_cron_header
def a_protected_handler():
# Run your code here
your_response_or_error_etc = "text"
return your_response_or_error_etc
#app.route("/yet/another/example/protected/handler/<myvar>")
#validate_cron_header
def another_protected_handler(some_var=None):
# Run your code here
return render_template("my_sample_template", some_var=some_var)
Now you have to use Cloud IAM Client Librares in your code in order to provide access. You can find an example of how to use it in Python 3 here.
User authentication in most of Google Cloud Platform is very different from the Python 2 App Engine approach. You could fully implement user login in a variety of ways in your app to make this restriction, but you can't just set a property in a .yaml file to do this.
Or, try putting your admin functions all in a separate App Engine service, then use IAP (Identity-Aware Proxy) to restrict access to that service to desired users. This page should help. And here's a codelab I wrote that has detailed steps to protect an entire App Engine app, not just one service, with IAP.
Things have changed since this question was asked so providing an updated answer
As of March, 2022
You can still use login:required in app.yaml file and it will force a visitor to login with their gmail account.
Python 3 now supports users API (see announcement) which means that if a page is protected by login: required, you can now call is_current_user_admin() on the route handler to confirm it is your administrator
Even if you don't wish to use the users API, you can still get the details of the logged in user (for pages protected by login:required) by checking for any of the following headersX-Appengine-User-Email, X-Appengine-User-Id. You can refer to my response to this other SO question
I currently have a Flask web server that pulls data from a JSON API using the built-in requests object.
For example:
def get_data():
response = requests.get("http://myhost/jsonapi")
...
return response
#main.route("/", methods=["GET"])
def index():
return render_template("index.html", response=response)
The issue here is that naturally the GET method is only run once, the first time get_data is called. In order to refresh the data, I have to stop and restart the Flask wsgi server. I've tried wrapping various parts of the code in a while True / sleep loop but this prevents werkzeug from loading the page.
What is the most Pythonic way to dynamically GET the data I want without having to reload the page or restart the server?
You're discussing what are perhaps two different issues.
Let's assume the problem is you're calling the dynamic data source, get_data(), only once and keeping its (static) value in a global response. This one-time-call is not shown, but let's say it's somewhere in your code. Then, if you are willing to refresh the page (/) to get updates, you could then:
#main.route("/", methods=['GET'])
def index():
return render_template("index.html", response=get_data())
This would fetch fresh data on every page load.
Then toward the end of your question, you ask how to "GET the data I want without having to reload the page or restart the server." That is an entirely different issue. You will have to use AJAX or WebSocket requests in your code. There are quite a few tutorials about how to do this (e.g. this one) that you can find through Googling "Flask AJAX." But this will require an JavaScript AJAX call. I recommend finding examples of how this is done through searching "Flask AJAX jQuery" as jQuery will abstract and simplify what you need to do on the client side. Or, if you wish to use WebSockets for lower-latency connection between your web page, that is also possible; search for examples (e.g. like this one).
To add to Jonathan’s comment, you can use frameworks like stimulus or turbo links to do this dynamically, without having to write JavaScript in some cases as the frameworks do a lot of the heavy lifting. https://stimulus.hotwired.dev/handbook/origin
I'm building a Flask app and started using Flask-Login for authentication. What troubles me is that Flask-Login calls the load_user callback for every request that flask handles. Here is the example from https://flask-login.readthedocs.org/en/latest/#how-it-works:
#login_manager.user_loader
def load_user(userid):
return User.get(userid)
To retrieve the user, I need to pass a session token a remote web service across a VPN, and the remote web service does a db query -- this results in noticeable latency on every web request. My load_user looks something like this:
#login_manager.user_loader
def load_user(userid):
# notice that I don't even use the userid arg
try:
# have to get session_token from session; why not
# just get the entire user object from session???
token = session.get('session_token')
user_profile = RestClient().get_user_profile(token)
return User(user_profile['LDAP_ID'])
except:
return None
Seems like maybe I'm subverting the framework. Could just store/retrieve user from session, so why bother to get it from the web service? This option also seems to subvert Flask-Login, but eliminates latency and makes good use of session.
The best way to handle this is to cache session information using something like memcached or redis (look into Flask-Cache for help).
You should have a key-value cache store that structures the cache like so:
key: sessionID
value: user object
This is what most frameworks do -- Flask-Login is a generic tool -- so you have to implement this yourself.
Incidentally, if you're looking for a way to abstract away that nasty LDAP stuff on the backend, you might want to check out https://stormpath.com -- they sync with LDAP servers, and provide a REST API on top of it. There's also a Flask library for interacting with it: Flask-Stormpath.
I'm building a Django app with an API built on Piston. For the sake of keeping everything as DRY as possible and the API complete, I'd like my internal applications to call the API rather than the models (kind of a proxy-view-controller a la https://github.com/raganwald/homoiconic/blob/master/2010/10/vc_without_m.md but all on one django install for now). So the basic setup is:
Model -> API -> Application -> User Client
I can overload some core Piston classes to create an internal client interface for the application, but I'm wondering if I could just use the Django Test Client to accomplish the same thing. So to create an article, rather than calling the model I would run:
from django.test.client import Client
c = Client()
article = c.post('/api/articles', {
'title' : 'My Title',
'content' : 'My Content'
})
Is there a reason I shouldn't use the test client to do this? (performance, for instance) Is there a better tool that's more tailored for this specific purpose?
After reviewing the code for TestClient, it doesn't appear to have any additional overhead related to testing. Rather, it just functions as a basic client for internal requests. I'll be using the test client as the internal client, and using Piston's DjangoEmitter to get model objects back from the API.
Only testing will tell whether the internal request mechanism is too much of a performance hit.