We Keep Coding

Flask: session max size too small

I'm building a data analysis Flask application which takes a ton of user inputs, performs some calculations then projects the results to various web pages. I'm using a pandas Dataframe to store the inputs and perform the calculations. Then I convert the DF to a dictionary and store that in the session object.
I'm having problems due to the fact that the session object can only hold ~4k bytes. Several pages read the data so I need a means to pass this large amount of data (~5k-50k) from one request to another (which the session object does perfectly but for smaller memory sizes).
Can I set the storage limit higher for the session object (I imagine I can't as 4k is the limit for a cookie and the session object is a cookie)? Or is there something else I should do here (store the dict in a database, etc.)?
EDIT:
I'm thinking a viable alternative would be to grab the data from the database (mongodb in my case), store it in a local variable and pass that variable to the template directly. Are there negatives to this? And is there a limit on how much memory I can pass directory to a template? See example below:
#app.route('/results')
def results():
# get data I need from database (~5k-50k bytes)
data = mongo.db[collection_name].find_one({'key': 'query'})
# pass directory to template (instead of storing in session object)
return render_template('results_page.html', data=data)

Yeah this definitely sounds like a case for server-side sessions.
There are code snippets on the official site for the most popular databases.
These shouldn't be hard to migrate to since they use the same SessionMixin interface as the cookie session system.
An even easier approach could be to use Flask-KVSession, which claims that
Integration with Flask is seamless, once the extension is loaded for a Flask application, it transparently replaces Flask’s own Session management. Any application working with sessions should work the same with Flask-KVSession

Flask-Session adds support to Server-side Session.
Here is how you can start Flask-Session (extracted from the official documentation).
from flask import Flask, session
from flask.ext.session import Session
app = Flask(__name__)
# Check Configuration section for more details
SESSION_TYPE = 'redis'
app.config.from_object(__name__)
Session(app)
#app.route('/set/')
def set():
session['key'] = 'value'
return 'ok'
#app.route('/get/')
def get():
return session.get('key', 'not set')
Regarding the size limit on rendering templates, the official Flask documentation has no mention of this.
I don't really believe there is any size limit actually.
If there is any limitation, that might be on Jinja's side.

Regenerating all flask-sessions without logout the users

Basically what I want to do is to regenerate every session with some new set of keys without having users to log in again. How can I do this?
edited for clarity
So let's assume we are using Redis as a backend for sessions and keeping cookies of it on the client-side. Cookie just consists of the session id. This session id corresponds to a session on the Redis. After we have initialized Session by writing Session(APP) in our application, for every request context, we can fetch the session of the current user by
from flask import session
After admin changes some general settings on the application, I am willing to regenerate the session of every current user which can be seen just for the current user by again
from flask import session
This is as far as I know.
For example, let's say there is a value on the user's session determined as
session['arbitrary_key'] = not_important_database_function()
After admin changes some stuff at application, I need to reload a key on the current user's session by
session['arbitrary_key'] = not_important_database_function()
Because after changes admin made, it will yield a different value. After that, I am changing session.modified as true. What I want to learn is how can I change the arbitrary_key on sessions of EVERY USER. Because I am lacking information on how to fetch every session and change them from Flask.
If I delete the sessions from Redis, users are required to reauthenticate. I don't want them to reauthenticate. I just want back-end sessions to be changed because I use some information inside of the user's session which needs to be fetched from Redis so I do not have to call not_important_database_function for every request.
I hope this is enough information for you to at least NOT answer but also NOT downvote so I can continue to seek a solution for my problem.
I am not sharing code snippets because no code snippet is helpful for the case in my opinion.

The question is rather old but it looks like many developers are interested in the answer. There are several approaches which come to mind:
1. Lazy calculations
You need a way to differentiate old and new session values. For example, storing version number in session. Then you can force clients to update their sessions when they are on your resource:
CURRENT_VERSION = '1.2.3'
#app.route('/some_route')
def some_handler():
if session.get('version', '0.0.0') < CURRENT_VERSION:
session['arbitrary_key'] = not_important_database_function()
session['version'] = CURRENT_VERSION
Pros: The method is easy to implement and it is agnostic to the way of storing session data (database, user-agent, etc.)
Cons: Session data is not updated instantly after deploy. You have to give up that some users' session data may not be updated at all.
2. Session storage update
You need to make some kind of a database migration for session storage. It is backend-dependable and will look different for different storages. For Redis it may look like this:
import json
import redis
r = redis.Redis() # Your connection settings here
for key in r.scan_iter():
raw_value = r.get(key)
session = json.loads(raw_value)
session['arbitrary_key'] = not_important_database_function()
r.set(key, json.dumps(session))
Pros: Session data for all users will be updated right after your service deployment.
Cons: The method implementations differ for different storages. It is not applicable if all session data is stored in user-agents.
3. Dropping session data
It is still an option though it is clearly stated in the question that you need to keep the users logged in. It may be a part of the policy of deploying new application versions: session key is regenerated on application deployment and all user sessions are invalidated.
Pros: You don't need to implement any logic to set new user session values.
Cons: There are no new user session values, the data is just wiped out.

Flask-Session extension vs default session

I'm using:
from flask import session
#app.route('/')
def main_page():
if session.get('key'):
print ("session exist" + session.get('key'))
else:
print ("could not find session")
session['key'] = '34544646###########'
return render_template('index.html')
I don't have the Flask-Session extension installed but this still works fine. I'm trying to understand why and when is that extension imp to me. As far as I see, the default session works well for me.

The difference is in where the session data is stored.
Flask's sessions are client-side sessions. Any data that you write to the session is written to a cookie and sent to the client to store. The client will send the cookie back to the server with every request, that is how the data that you write in the session remains available in subsequent requests. The data stored in the cookie is cryptographically signed to prevent any tampering. The SECRET_KEY setting from your configuration is used to generate the signature, so the data in your client-side sessions is secure as long as your secret key is kept private. Note that secure in this context means that the data in the session cannot be modified by a potential attacker. The data is still visible to anybody who knows how to look, so you should never write sensitive information in a client-side session.
Flask-Session and Flask-KVSession are two extensions for Flask that implement server-side sessions. These sessions work exactly in the same way as the Flask native sessions from the point of view of your application, but they store the data in the server. The data is never sent to the client, so there is a bit of increased security. The client still receives a signed cookie, but the only data in the cookie is a session ID that references the file or database index in the server where the data is stored.

from flask import session
Cookies of all session data is stored client-side.
Pros:
Validating and creating sessions is fast (no data storage)
Easy to scale (no need to replicate session data across web servers)
Cons:
Sensitive data cannot be stored in session data, as it's stored on the web browser
Session data is limited by the size of the cookie (usually 4 KB)
Sessions cannot be immediately revoked by the Flask app
from flask_session import Session
Session data is stored server side.
Pros:
Sensitive data is stored on the server, not in the web browser
You can store as much session data as you want without worrying about the cookie size
Sessions can easily be terminated by the Flask app
Cons:
Difficult to set up and scale
Increased complexity since session state must be managed
*this information is from Patrick Kennedy on this excellent tutorial: https://testdriven.io/blog/flask-server-side-sessions/

Session
A session makes it possible to remember information from one request to another. The way Flask does this is by using a signed cookie. Cookie can be modified unless they have SECRET KEY. Save in Client Side unless permanent is set to TRUE(boolean). If Permanent is set True, it's store in the server default 31 days unless it mentioned PERMANENT_SESSION_LIFETIME in flask app.
Flask-Session:
Flask-Session is an extension for Flask that adds support for Server-side Session to your application. It's main goal to store the session in Server side
Server Side method are
- redis: RedisSessionInterface
- memcached: MemcachedSessionInterface
- filesystem: FileSystemSessionInterface
- mongodb: MongoDBSessionInterface
- sqlalchemy: SqlAlchemySessionInterface
Flask-Session is an extension of Session.
Bases on config method it's over write the existing session saving method.
flask.sessions.SessionInterface: SessionInterface is the basic interface you have to implement in order to replace the default session interface which uses flask(werkzeug’s) secure cookie implementation.
The only methods you have to implement are open_session() and save_session(), the others have useful defaults which you don’t need to change.
Based on this, they are updating the session in the selected storage
Session Interface
Reference Links:
Session
flask_session
Session Interface
`

Flask-SQLAlchemy and SQLAlchemy

I'm building a small website and I already have all my models in SQLAlchemy. The website is to publish some information from some calculations which are done offline. Only the results will be published to a slimmed down database i.e. it contains the results, not the raw data, but the website needs to query the results.
I'm going to use Flask, as my models are already driven with Python (and some heavy lifting in C++ via SWIG) and I don't want to use Django.
Now this has been asked before I'm sure and the usual mantra without much justification is to 'use Flask-SQLAlchemy'. The question is why?
If I write some session handling myself, why do I have to go through the additional layer of redefining my database in Flask-SQLAlchemy. Other than having to write some code like here in my Flask app somewhere:
#app.before_request
def before_request():
g.db = connect_db()
#app.teardown_request
def teardown_request(exception):
db = getattr(g, 'db', None)
if db is not None:
db.close()
What else do I need to worry about? SQLAlchemy even does connection pooling for me by default.

Actually, you are building a web application using Flask which do database related kinds of stuff by sqlalchemy. So, when you are dealing with database session with multiple requests which your application handles, then you have to ascertain that, you are creating and closing sessions cautiously.
If you read SQLAlchemy docs, they recommend to keep the lifecycle of the session separate and external from functions and objects that access and/or manipulate database data. This will greatly help with achieving a predictable and consistent transactional scope.
A web application is the easiest case because such an application is already constructed around a single, consistent scope - this is the request, which represents an incoming request from a browser, the processing of that request to formulate a response, and finally the delivery of that response back to the client. Integrating web applications with the Session is then the straightforward task of linking the scope of the Session to that of the request. The Session can be established as the request begins, or using a lazy initialization pattern which establishes one as soon as it is needed. The request then proceeds, with some system in place where application logic can access the current Session in a manner associated with how the actual request object is accessed. As the request ends, the Session is torn down as well, usually through the usage of event hooks provided by the web framework. The transaction used by the Session may also be committed at this point, or alternatively the application may opt for an explicit commit pattern, only committing for those requests where one is warranted, but still always tearing down the Session unconditionally at the end.
In laymen terms, i mean to say that
In SQLAlchemy, above action is mentioned because sessions in web application should be scoped, meaning that each request handler creates and destroys its own session.
This is necessary because web servers can be multi-threaded, so multiple requests might be served at the same time, each working with a different database session.
This means that if you are using SqlAlchemy with Flask, you have to manually handle session like creating scoped session and also remove them on each request cautiously otherwise you may be in deep shit, which adds extra layer of complexity to your web application.
But, there comes Flask-SqlAlchemy (an extension of sqlalchemy library for Flask app) which provides infrastructure to assist in the task of aligning the lifespan of a Session with that of each web request. Actually, you can also found that in SqlAlchmey docs, they also recommend to use this with Flask.
Flask-SQLAlchemy creates a fresh/new scoped session for each request. If you dig further it you will find out here, it also installs a hook on app.teardown_appcontext (for Flask >=0.9), app.teardown_request (for Flask 0.7-0.8), app.after_request (for Flask <0.7) and here is where it calls db.session.remove().

The code you put in the question is not actually valid for Sqlalchemy integration is Flask. I know it is just example, but saying that just in case.
For Sqlalchemy integration all you need to do is to make sure current DbSession is cleaned up at the end of request via something like this:
#app.teardown_appcontext
def shutdown_session(exception=None):
DbSession.remove()
where DbSession is scoped session.
Here is documentation for the case when you dont want to use Flask-Sqlalchemy package.

Flask: setting application and request-specific attributes?

I am writing an application which connects to a database. I want to create that db connection once, and then reuse that connection throughout the life of the application.
I also want to authenticate users. A user's auth will live for only the life of a request.
How can I differentiate between objects stored for the life of a flask app, versus specific to the request? Where would I store them so that all modules (and subsequent blueprints) have access to them?
Here is my sample app:
from flask import Flask, g
app = Flask(__name__)
#app.before_first_request
def setup_database(*args, **kwargs):
print 'before first request', g.__dict__
g.database = 'DATABASE'
print 'after first request', g.__dict__
#app.route('/')
def index():
print 'request start', g.__dict__
g.current_user = 'USER'
print 'request end', g.__dict__
return 'hello'
if __name__ == '__main__':
app.run(debug=True, port=6001)
When I run this (Flask 0.10.1) and navigate to http://localhost:6001/, here is what shows up in the console:
$ python app.py
* Running on http://127.0.0.1:6001/
* Restarting with reloader
before first request {}
after first request {'database': 'DATABASE'}
request start {'database': 'DATABASE'}
request end {'current_user': 'USER', 'database': 'DATABASE'}
127.0.0.1 - - [30/Sep/2013 11:36:40] "GET / HTTP/1.1" 200 -
request start {}
request end {'current_user': 'USER'}
127.0.0.1 - - [30/Sep/2013 11:36:41] "GET / HTTP/1.1" 200 -
That is, the first request is working as expected: flask.g is holding my database, and when the request starts, it also has my user's information.
However, upon my second request, flask.g is wiped clean! My database is nowhere to be found.
Now, I know that flask.g used to apply to the request only. But now that it is bound to the application (as of 0.10), I want to know how to bind variables to the entire application, rather than just a single request.
What am I missing?
edit: I'm specifically interested in MongoDB - and in my case, maintaining connections to multiple Mongo databases. Is my best bet to just create those connections in __init__.py and reuse those objects?

flask.g will only store things for the duration of a request. The documentation mentioned that the values are stored on the application context rather than the request, but that is more of an implementation issue: it doesn't change the fact that objects in flask.g are only available in the same thread, and during the lifetime of a single request.
For example, in the official tutorial section on database connections, the connection is made once at the beginning of the request, then terminated at the end of the request.
Of course, if you really wanted to, you could create the database connection once, store it in __init__.py, and reference it (as a global variable) as needed. However, you shouldn't do this: the connection could close or timeout, and you could not use the connection in multiple threads.
Since you didn't specify HOW you will be using Mongo in Python, I assume you will be using PyMongo, since that handles all of the connection pooling for you.
In this case, you would do something like this...
from flask import Flask
from pymongo import MongoClient
# This line of code does NOT create a connection
client = MongoClient()
app = Flask()
# This can be in __init__.py, or some other file that has imported the "client" attribute
#app.route('/'):
def index():
posts = client.database.posts.find()
You could, if you wish, do something like this...
from flask import Flask, g
from pymongo import MongoClient
# This line of code does NOT create a connection
client = MongoClient()
app = Flask()
#app.before_request
def before_request():
g.db = client.database
#app.route('/'):
def index():
posts = g.db.posts.find()
This really isn't all that different, however it can be helpful for logic that you want to perform on every request (such as setting g.db to a specific database depending on the user that is logged in).
Finally, you can realize that most of the work of setting up PyMongo with Flask is probably done for you in Flask-PyMongo.
Your other question deals with how you keep track of stuff specific to the user that is logged in. Well, in this case, you DO need to store some data that sticks around with the connection. flask.g is cleared at the end of the reuquest, so that's no good.
What you want to use is sessions. This is a place where you can store values that is (with the default implementation) stored in a cookie on the user's browser. Since the cookie will be passed along with every request the user's browser makes to your web site, you will have available the data you put in the session.
Keep in mind, though, that the session is NOT stored on the server. It is turned into a string that is passed back and forth to the user. Therefore, you can't store things like DB connections onto it. You would instead store identifiers (like user IDs).
Making sure that user authentication works is VERY hard to get right. The security concerns that you need to make sure of are amazingly complex. I would strongly recommend using something like Flask-Login to handle this for you. You can still use the session for storing other items as needed, or you can let Flask-Login handle determining the user ID and store the values you need in the database and retrieving them from the database in every request.
So, in summary, there are a few different ways to do what you want to do. Each have their usages.
Globals are good for items that are thread-safe (such as the PyMongo's MongoClient).
flask.g can be used for storing data in the lifetime of a request. With SQLAlchemy-based flask apps, a common thing to do is to ensure that all changes happen at once, at the end of a request using an after_request method. Using flask.g for something like this is very helpful.
The Flask session can be used to store simple data (strings and numbers, not connection objects) that can be used on subsequent requests that come from the same user. This is entirely dependent on using cookies, so at any point the user could delete the cookie and everything in the "session" will be lost. Therefore, you probably want to store much of your data in databases, with the session used to identify the data that relates to the user in the session.

"bound to the application" does not mean what you think it means. It means that g is bound to the currently running request. Quoth the docs:
Flask provides you with a special object that ensures it is only valid for the active request and that will return different values for each request.
It should be noted that Flask's tutorials specifically do not persist database objects, but that this is not normative for any application of substantial size. If you're really interested in diving down the rabbit hole, I suggest a database connection pooling tool. (such as this one, mentioned in the SO answer ref'd above)

I suggest you use session to manage user information. Sessions help you keep information b/w multiple requests and flask provides you a session framework already.
from flask import session
session['usename'] = 'xyz'
Look at the extension Flask-Login. It is well designed to handle user authentications.
For database, I suggest looking at Flask-SQLAlchemy extension. This takes care of initialization, pooling, teardowns etc. for you out of the box. All you need to do is define the database URI in a config and bind it to the application.
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)