[django 1.4, python 2.7]
We have a custom session engine that stores session to redis (Pretty much the same as db session engine, just stores in redis).
There are some public endpoints that the session middleware stores a session to redis every time it is invoked (session.modified is set to True), which is not our intention.
I am trying to write unit tests to check which endpoints do generate session.
# This endpoint would always save session to redis
def test_endpoint(request, **kwargs):
request.session['asdfsaf'] = 1
# Or, request.session.modified = True (Both fine)
return httplib.OK, "good"
Attempt 1:
response = self.client.post(reverse('test_endpoint', kwargs=self.kwargs), data={...})
print(response.status_code, response.content) # Ok
print('am i modified?', self.client.session.modified) # False
for i in range(10):
redis = get_redis_instance(db=i)
print("redis key", i, redis.keys()) # All return empty redis
This unit test is unable to detect a session is saved via django session middleware.
probably due to the fact that a new SessionStore is created every time this property is accessed (https://docs.djangoproject.com/en/2.2/topics/testing/tools/#persistent-state)
Attempt 2:
Taking inspiration from SessionMiddlewareTests:
request = RequestFactory().post(reverse('test_endpoint', kwargs=self.kwargs), data={...})
response = HttpResponse('Session test')
middleware = SessionMiddleware()
middleware.process_request(request)
# Handle the response through the middleware
response = middleware.process_response(request, response)
print('am i modified', request.session.modified) # False
for i in range(10):
redis = get_redis_instance(db=i)
print("redis key", i, redis.keys()) # All return empty redis
Again, I am unable to detect this endpoint is in fact storing session to redis.
How can I programmatically check whether an API creating a session? (that somehow creates it via setting session.modified flag True)?
Last thing I want to do is to test manually and checking redis db myself. (Over 200 public end points and unable to prevent similar issue happening in future).
Many thanks in advance!
Related
I built an API using Flask and I'm using a service (as below) to create my database connections.
class DatabaseService:
def __init__(self):
self.connection_string = "foo"
def create_connection(self):
engine = create_engine(self.connection_string)
Session = scoped_session(sessionmaker(bind=engine))
return Session
In my app.py I add and remove these sessions to Flask application context (g) as the docs suggests.
So I can reference to g.session always I need them.
def get_session():
if 'session' not int g:
session = database_service.create_session()
g.session = session
#app.teardown_appcontext
def shutdown_session(exception=None):
if 'session' in g:
g.session.remove()
return None
This way every request has your own session that will close after processing. Am I right?
I don't understand why the connections are still alive on my database after the request is already done.
Always I run the command show processlist I can see multiple connections sleeping from my API.
I see no problem opening and closing sessions per-request
my_session = Session(engine)
my_session.execute(some_query)
my_session.close()
I'm writing a simple web application in google appengine and python. In this application I need to handle two types of sessions:
the "long term session" that stores information about users, current page ecc, with long max_age parameter and the "short term session" with max_age about 20 minutes that keep an access token for the autentication via API.
I have implemented the following BaseHandler:
import webapp2
from webapp2_extras import sessions
class BaseHandler(webapp2.RequestHandler):
def dispatch(self):
# Get a session store for this request.
self.session_store = sessions.get_store(request=self.request)
try:
# Dispatch the request.
webapp2.RequestHandler.dispatch(self)
finally:
# Save all sessions.
self.session_store.save_sessions(self.response)
#webapp2.cached_property
def session(self):
# Returns a session using the default cookie key.
return self.session_store.get_session(backend='memcache')
#webapp2.cached_property
def session_auth(self):
return self.session_store.get_session(backend='memcache', max_age=20*60)<code>
the problem is that all sessions have max_age=20*60 seconds (and not only the sessions accessible by self.session_auth)..
How should I solve this?
thanks
Try setting your config params:
config = {}
config['webapp2_extras.sessions'] = {
'session_max_age': 100000, # try None here also, though that is the default
}
app = webapp2.WSGIApplication([
('/', HomeHandler),
], debug=True, config=config)
For a percentage of production traffic, I want to duplicate the received request to a different version of my application. This needs to happen asynchronously so I don't double service time to the client.
The reason for doing this is so I can compare the responses generated by the prod version and a production candidate version. If their results are appropriately similar, I can be confident that the new version hasn't broken anything. (If I've made a functional change to the application, I'd filter out the necessary part of the response from this comparison.)
So I'm looking for an equivalent to:
class Foo(webapp2.RequestHandler):
def post(self):
handle = make_async_call_to('http://other_service_endpoint.com/', self.request)
# process the user's request in the usual way
test_response = handle.get_response()
# compare the locally-prepared response and the remote one, and log
# the diffs
# return the locally-prepared response to the caller
UPDATE
google.appengine.api.urlfetch was suggested as a potential solution to my problem, but it's synchronous in the dev_appserver, though it behaves the way I wanted in production (the request doesn't go out until get_response() is called, and it blocks). :
start_time = time.time()
rpcs = []
print 'creating rpcs:'
for _ in xrange(3):
rpcs.append(urlfetch.create_rpc())
print time.time() - start_time
print 'making fetch calls:'
for rpc in rpcs:
urlfetch.make_fetch_call(rpc, 'http://httpbin.org/delay/3')
print time.time() - start_time
print 'getting results:'
for rpc in rpcs:
rpc.get_result()
print time.time() - start_time
creating rpcs:
9.51290130615e-05
0.000154972076416
0.000189065933228
making fetch calls:
0.00029993057251
0.000356912612915
0.000473976135254
getting results:
3.15417003632
6.31326603889
9.46627306938
UPDATE2
So, after playing with some other options, I found a way to make completely non-blocking requests:
start_time = time.time()
rpcs = []
logging.info('creating rpcs:')
for i in xrange(10):
rpc = urlfetch.create_rpc(deadline=30.0)
url = 'http://httpbin.org/delay/{}'.format(i)
urlfetch.make_fetch_call(rpc, url)
rpc.callback = create_callback(rpc, url)
rpcs.append(rpc)
logging.info(time.time() - start_time)
logging.info('getting results:')
while rpcs:
rpc = apiproxy_stub_map.UserRPC.wait_any(rpcs)
rpcs.remove(rpc)
logging.info(time.time() - start_time)
...but the important point to note is that none of the async fetch options in urllib work in the dev_appserver. Having discovered this, I went back to try #DanCornilescu's solution and found that it only works properly in production, but not in the dev_appserver.
The URL Fetch service supports asynchronous requests. From Issuing an asynchronous request:
HTTP(S) requests are synchronous by default. To issue an asynchronous
request, your application must:
Create a new RPC object using urlfetch.create_rpc(). This object represents your asynchronous call in subsequent method calls.
Call urlfetch.make_fetch_call() to make the request. This method takes your RPC object and the request target's URL as parameters.
Call the RPC object's get_result() method. This method returns the result object if the request is successful, and raises an exception if
an error occurred during the request.
The following snippets demonstrate how to make a basic asynchronous
request from a Python application. First, import the urlfetch library
from the App Engine SDK:
from google.appengine.api import urlfetch
Next, use urlfetch to make the asynchronous request:
rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, "http://www.google.com/")
# ... do other things ...
try:
result = rpc.get_result()
if result.status_code == 200:
text = result.content
self.response.write(text)
else:
self.response.status_code = result.status_code
logging.error("Error making RPC request")
except urlfetch.DownloadError:
logging.error("Error fetching URL0")
Note: As per Sniggerfardimungus's experiment mentioned in the question's update the async calls might not work as expected on the development server - being serialized instead of concurrent, but they do so when deployed on GAE. Personally I didn't use the async calls yet, so I can't really say.
If the intent is not block at all waiting for the response from the production candidate app you could push a copy of the original request and the production-prepared response on a task queue then answer to the original request - with neglijible delay (that of enqueueing the task).
The handler for the respective task queue would, outside of the original request's critical path, make the request to the staging app using the copy of the original request (async or not, doesn't really matter from the point of view of impacting the production app's response time), get its response and compare it with the production-prepared response, log the deltas, etc. This can be nicely wrapped in a separate module for minimal changes to the production app and deployed/deleted as needed.
When I run py.test --with-gae, I get the following error (I have pytest_gae plugin installed):
def get_current_session():
"""Returns the session associated with the current request."""
> return _tls.current_session
E AttributeError: 'thread._local' object has no attribute 'current_session'
gaesessions/__init__.py:50: AttributeError
I'm using pytest to test my google appengine application. The application runs fine when run in the localhost SDK or when deployed to GAE servers. I just can't figure out how to make pytest work with gaesessions.
My code is below:
test_handlers.py
from webtest import TestApp
import appengine_config
def pytest_funcarg__anon_user(request):
from main import app
app = appengine_config.webapp_add_wsgi_middleware(app)
return TestApp(app)
def test_session(anon_user):
from gaesessions import get_current_session
assert get_current_session()
appengine_config.py
from gaesessions import SessionMiddleware
def webapp_add_wsgi_middleware(app):
from google.appengine.ext.appstats import recording
app = recording.appstats_wsgi_middleware(app)
app = SessionMiddleware(app, cookie_key="replaced-with-this-boring-text")
return app
Relevant code from gaesessions:
# ... more code are not show here ...
_tls = threading.local()
def get_current_session():
"""Returns the session associated with the current request."""
return _tls.current_session
# ... more code are not show here ...
class SessionMiddleware(object):
"""WSGI middleware that adds session support.
``cookie_key`` - A key used to secure cookies so users cannot modify their
content. Keys should be at least 32 bytes (RFC2104). Tip: generate your
key using ``os.urandom(64)`` but do this OFFLINE and copy/paste the output
into a string which you pass in as ``cookie_key``. If you use ``os.urandom()``
to dynamically generate your key at runtime then any existing sessions will
become junk every time your app starts up!
``lifetime`` - ``datetime.timedelta`` that specifies how long a session may last. Defaults to 7 days.
``no_datastore`` - By default all writes also go to the datastore in case
memcache is lost. Set to True to never use the datastore. This improves
write performance but sessions may be occassionally lost.
``cookie_only_threshold`` - A size in bytes. If session data is less than this
threshold, then session data is kept only in a secure cookie. This avoids
memcache/datastore latency which is critical for small sessions. Larger
sessions are kept in memcache+datastore instead. Defaults to 10KB.
"""
def __init__(self, app, cookie_key, lifetime=DEFAULT_LIFETIME, no_datastore=False, cookie_only_threshold=DEFAULT_COOKIE_ONLY_THRESH):
self.app = app
self.lifetime = lifetime
self.no_datastore = no_datastore
self.cookie_only_thresh = cookie_only_threshold
self.cookie_key = cookie_key
if not self.cookie_key:
raise ValueError("cookie_key MUST be specified")
if len(self.cookie_key) < 32:
raise ValueError("RFC2104 recommends you use at least a 32 character key. Try os.urandom(64) to make a key.")
def __call__(self, environ, start_response):
# initialize a session for the current user
_tls.current_session = Session(lifetime=self.lifetime, no_datastore=self.no_datastore, cookie_only_threshold=self.cookie_only_thresh, cookie_key=self.cookie_key)
# create a hook for us to insert a cookie into the response headers
def my_start_response(status, headers, exc_info=None):
_tls.current_session.save() # store the session if it was changed
for ch in _tls.current_session.make_cookie_headers():
headers.append(('Set-Cookie', ch))
return start_response(status, headers, exc_info)
# let the app do its thing
return self.app(environ, my_start_response)
The problem is that your gae sessions is not yet called until the app is also called. The app is only called when you make a request to it. Try inserting a request call before you check for the session value. Check out the revised test_handlers.py code below.
def test_session(anon_user):
anon_user.get("/") # get any url to call the app to create a session.
from gaesessions import get_current_session
assert get_current_session()
I am developing an API using Django-TastyPie.
What API do?
It checks that if two or more requests are there on the server if yes it swap the data of both the requests and return a json response after 7 second delay.
What i need to do is send multiple asynchronous requests to the server to test this API.
I am using Django-Unit Test along with Tasty-Pie to test this functionality.
Problem
Django develpment server is single threaded so it does not support asynchronous requests
Solution tried:
I have tried to solve this by using multiprocessing:
class MatchResourceTest(ResourceTestCase):
def setUp(self):
super(MatchResourceTest, self).setUp()
self.user=""
self.user_list = []
self.thread_list = []
# Create and get user
self.assertHttpCreated(self.api_client.post('/api/v2/user/', format='json', data={'username': '123456','device': 'abc'}))
self.user_list.append( User.objects.get(username='123456') )
# Create and get other_user
self.assertHttpCreated(self.api_client.post('/api/v2/user/', format='json', data={'username': '456789','device': 'xyz'}))
self.user_list.append( User.objects.get(username='456789') )
def get_credentials(self):
return self.create_apikey(username=self.user.username, api_key=self.user.api_key.key)
def get_url(self):
resp = urllib2.urlopen(self.list_url).read()
self.assertHttpOK(resp)
def test_get_list_json(self):
for user in self.user_list:
self.user = user
self.list_url = 'http://127.0.0.1:8000/api/v2/match/?name=hello'
t = multiprocessing.Process(target=self.get_url)
t.start()
self.thread_list.append( t )
for t in self.thread_list:
t.join()
print ContactCardShare.objects.all()
Please suggest me any solution to test this API by sending asychronous requests
or
any APP , Library or any this which allow django development server to handle multiple requests asychronously
As far as I know, django's development server is multi-threaded.
I'm not sure this test is formatted correctly though. The test setUp shouldn't include tests itself, it should be foolproof data insertion by creating entries. The post should have it's own test.
See the tastypie docs for an example test case.