We Keep Coding

Caching/reusing a DB connection for later view usage

I am saving a user's database connection. On the first time they enter in their credentials, I do something like the following:
self.conn = MySQLdb.connect (
host = 'aaa',
user = 'bbb',
passwd = 'ccc',
db = 'ddd',
charset='utf8'
)
cursor = self.conn.cursor()
cursor.execute("SET NAMES utf8")
cursor.execute('SET CHARACTER SET utf8;')
cursor.execute('SET character_set_connection=utf8;')
I then have the conn ready to go for all the user's queries. However, I don't want to re-connect every time the view is loaded. How would I store this "open connection" so I can just do something like the following in the view:
def do_queries(request, sql):
user = request.user
conn = request.session['conn']
cursor = request.session['cursor']
cursor.execute(sql)
Update: it seems like the above is not possible and not good practice, so let me re-phrase what I'm trying to do:
I have a sql editor that a user can use after they enter in their credentials (think of something like Navicat or SequelPro). Note this is NOT the default django db connection -- I do not know the credentials beforehand. Now, once the user has 'connected', I would like them to be able to do as many queries as they like without me having to reconnect every time they do this. For example -- to re-iterate again -- something like Navicat or SequelPro. How would this be done using python, django, or mysql? Perhaps I don't really understand what is necessary here (caching the connection? connection pooling? etc.), so any suggestions or help would be greatly appreciated.

You could use an IoC container to store a singleton provider for you. Essentially, instead of constructing a new connection every time, it will only construct it once (the first time ConnectionContainer.connection_provider() is called) and thereafter it will always return the previously constructed connection.
You'll need the dependency-injector package for my example to work:
import dependency_injector.containers as containers
import dependency_injector.providers as providers
class ConnectionProvider():
def __init__(self, host, user, passwd, db, charset):
self.conn = MySQLdb.connect(
host=host,
user=user,
passwd=passwd,
db=db,
charset=charset
)
class ConnectionContainer(containers.DeclarativeContainer):
connection_provider = providers.Singleton(ConnectionProvider,
host='aaa',
user='bbb',
passwd='ccc',
db='ddd',
charset='utf8')
def do_queries(request, sql):
user = request.user
conn = ConnectionContainer.connection_provider().conn
cursor = conn.cursor()
cursor.execute(sql)
I've hardcoded the connection string here, but it is also possible to make it variable depending on a changeable configuration. In that case you could also create a container for the configuration file and have the connection container read its config from there. You then set the config at runtime. As follows:
import dependency_injector.containers as containers
import dependency_injector.providers as providers
class ConnectionProvider():
def __init__(self, connection_config):
self.conn = MySQLdb.connect(**connection_config)
class ConfigContainer(containers.DeclarativeContainer):
connection_config = providers.Configuration("connection_config")
class ConnectionContainer(containers.DeclarativeContainer):
connection_provider = providers.Singleton(ConnectionProvider, ConfigContainer.connection_config)
def do_queries(request, sql):
user = request.user
conn = ConnectionContainer.connection_provider().conn
cursor = conn.cursor()
cursor.execute(sql)
# run code
my_config = {
'host':'aaa',
'user':'bbb',
'passwd':'ccc',
'db':'ddd',
'charset':'utf8'
}
ConfigContainer.connection_config.override(my_config)
request = ...
sql = ...
do_queries(request, sql)

I don't see why do you need a cached connection here and why not just reconnect on every request caching user's credentials somewhere, but anyway I'll try to outline a solution that might fit your requirements.
I'd suggest to look into a more generic task first - cache something between subsequent requests your app needs to handle and can't serialize into django's sessions.
In your particular case this shared value would be a database connection (or multiple connections).
Lets start with a simple task of sharing a simple counter variable between requests, just to understand what's actually happening under the hood.
Amaizingly but neither answer has mentioned anything regarding a web server you might use!
Actually there are multiple ways to handle concurrent connections in web apps:
Having multiple processes, every request comes into one of them at random
Having multiple threads, every request is handled by a random thread
p.1 and p.2 combined
Various async techniques, when there's a single process + event loop handling requests with a caveat that request handlers shouldn't block for a long time
From my own experience p.1-2 are fine for majority of typical webapps.
Apache1.x could only work with p.1, Apache2.x can handle all of 1-3.
Lets start with the following django app and run a single-process gunicorn webserver.
I'm going to use gunicorn because it's fairly easy to configure it unlike apache (personal opinion :-)
views.py
import time
from django.http import HttpResponse
c = 0
def main(self):
global c
c += 1
return HttpResponse('val: {}\n'.format(c))
def heavy(self):
time.sleep(10)
return HttpResponse('heavy done')
urls.py
from django.contrib import admin
from django.urls import path
from . import views
urlpatterns = [
path('admin/', admin.site.urls),
path('', views.main, name='main'),
path('heavy/', views.heavy, name='heavy')
]
Running it in a single process mode:
gunicorn testpool.wsgi -w 1
Here's our process tree - there's only 1 worker that would handle ALL requests
pstree 77292
-+= 77292 oleg /Users/oleg/.virtualenvs/test3.4/bin/python /Users/oleg/.virtualenvs/test3.4/bin/gunicorn testpool.wsgi -w 1
\--- 77295 oleg /Users/oleg/.virtualenvs/test3.4/bin/python /Users/oleg/.virtualenvs/test3.4/bin/gunicorn testpool.wsgi -w 1
Trying to use our app:
curl 'http://127.0.0.1:8000'
val: 1
curl 'http://127.0.0.1:8000'
val: 2
curl 'http://127.0.0.1:8000'
val: 3
As you can see you can easily share the counter between subsequent requests.
The problem here is that you can only serve a single request in parallel. If you request for /heavy/ in one tab, / won't work until /heavy is done
Lets now use 2 worker processes:
gunicorn testpool.wsgi -w 2
This is how the process tree would look like:
pstree 77285
-+= 77285 oleg /Users/oleg/.virtualenvs/test3.4/bin/python /Users/oleg/.virtualenvs/test3.4/bin/gunicorn testpool.wsgi -w 2
|--- 77288 oleg /Users/oleg/.virtualenvs/test3.4/bin/python /Users/oleg/.virtualenvs/test3.4/bin/gunicorn testpool.wsgi -w 2
\--- 77289 oleg /Users/oleg/.virtualenvs/test3.4/bin/python /Users/oleg/.virtualenvs/test3.4/bin/gunicorn testpool.wsgi -w 2
Testing our app:
curl 'http://127.0.0.1:8000'
val: 1
curl 'http://127.0.0.1:8000'
val: 2
curl 'http://127.0.0.1:8000'
val: 1
The first two requests has been handled by the first worker process, and the 3rd one - by the second worker process that has its own memory space so you see 1 instead of 3.
Notice your output may differ because process 1 and 2 are selected at random. But sooner or later you'll hit a different process.
That's not very helpful for us because we need to handle multiple concurrent requests and we need to somehow get our request handled by a specific process that can't be done in general case.
Most pooling technics coming out of the box would only cache connections in the scope of a single process, if your request gets served by a different process - a NEW connection would need to be made.
Lets move to threads
gunicorn testpool.wsgi -w 1 --threads 2
Again - only 1 process
pstree 77310
-+= 77310 oleg /Users/oleg/.virtualenvs/test3.4/bin/python /Users/oleg/.virtualenvs/test3.4/bin/gunicorn testpool.wsgi -w 1 --threads 2
\--- 77313 oleg /Users/oleg/.virtualenvs/test3.4/bin/python /Users/oleg/.virtualenvs/test3.4/bin/gunicorn testpool.wsgi -w 1 --threads 2
Now if you run /heavy in one tab you'll still be able to query / and your counter will be preserved between requests!
Even if the number of threads is growing or shrinking depending on your workload it should still work fine.
Problems: you'll need to synchronize access to the shared variable like this using python threads synchronization technics (read more).
Another problem is that the same user may need to to issue multiple queries in parallel - i.e. open multiple tabs.
To handle it you can open multiple connections on the first request when you have db credentials available.
If a user needs more connections than your app might wait on lock until a connection becomes available.
Back to your question
You can create a class that would have the following methods:
from contextlib import contextmanager
class ConnectionPool(object):
def __init__(self, max_connections=4):
self._pool = dict()
self._max_connections = max_connections
def preconnect(self, session_id, user, password):
# create multiple connections and put them into self._pool
# ...
#contextmanager
def get_connection(sef, session_id):
# if have an available connection:
# mark it as allocated
# and return it
try:
yield connection
finally:
# put it back to the pool
# ....
# else
# wait until there's a connection returned to the pool by another thread
pool = ConnectionPool(4)
def some_view(self):
session_id = ...
with pool.get_connection(session_id) as conn:
conn.query(...)
This is not a complete solution - you'll need to somehow delete outdated connections not used for a long time.
If a user comes back after a long time and his connection have been closed, he'll need to provide his credentials again - hopefully it's ok from your app's perspective.
Also keep in mind python threads have its performance penalties, not sure if this is an issue for you.
I haven't checked it for apache2 (too much configuration burden, I haven't used it for ages and generally use uwsgi), but it should work there too - would be happy to hear back from you
if you manage to run it )
And also don't forget about p.4 (async approach) - unlikely will you be able to use it on apache, but it's worth investigation - keywords: django + gevent, django + asyncio. It has its pros/cons and may greatly affect your app implementation so it's hard to suggest any solution without knowing your app requirements in detail

This is not a good idea to do such a thing synchronously in web app context. Remember that your application may needs to work in multi process/thread fashion, and you could not share connection between processes normally. So if you create a connection for your user on a process, there is no guaranty to receive query request on the same one. May be a better idea is to have a single process background worker which handles connections in multiple threads (a thread per session) to make queries on database and retrieve result on web app. Your application should assign a unique ID to each session and the background worker track each thread using session ID. You may use celery or any other task queues supporting async result. So the design would be something like below:
|<--| |<--------------| |<--|
user (id: x) | | webapp | | queue | | worker (thread x) | | DB
|-->| |-->| |-->| |-->|
Also you could create a queue for each user until they have an active session, as a result you could run a separate background process for each session.

I actually shared my solution to this exact issue. What I did here was create a pool of connections that you can specify the max with, and then queued query requests async through this channel. This way you can leave a certain amount of connections open, but it will queue and pool async and keep the speed you are used to.
This requires gevent and postgres.
Python Postgres psycopg2 ThreadedConnectionPool exhausted

I'm no expert in this field, but I believe that PgBouncer would do the job for you, assuming you're able to use a PostgreSQL back-end (that's one detail you didn't make clear). PgBouncer is a connection pooler, which allows you re-use connections avoiding the overhead of connecting on every request.
According to their documentation:
user, password
If user= is set, all connections to the destination database will be done with the specified user, meaning that there will be only one pool for this database.
Otherwise PgBouncer tries to log into the destination database with client username, meaning that there will be one pool per user.
So, you can have a single pool of connections per user, which sounds just like what you want.
In MySQL land, the mysql.connector.pooling module allows you to do some connection pooling, though I'm not sure if you can do per-user pooling. Given that you can set up the pool name, I'm guessing you could use the user's name to identify the pool.
Regardless of what you use, you will likely have occasions where reconnecting is unavoidable (a user connects, does a few things, goes away for a meeting and lunch, comes back and wants to take more action).

I am just sharing my knowledge over here.
Install the PyMySQL to use the MySql
For Python 2.x
pip install PyMySQL
For Python 3.x
pip3 install PyMySQL
1. If you are open to use Django Framework then it's very easy to run the SQL query without any re-connection.
In setting.py file add the below lines
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'test',
'USER': 'test',
'PASSWORD': 'test',
'HOST': 'localhost',
'OPTIONS': {'charset': 'utf8mb4'},
}
}
In views.py file add these lines to get the data. You can customized your query according to your need
from django.db import connection
def connect(request):
cursor = connection.cursor()
cursor.execute("SELECT * FROM Tablename");
results = cursor.fetchall()
return results
You will get the desire results.
Click here for more information about it
2. For python Tkinter
from Tkinter import *
import MySQLdb
db = MySQLdb.connect("localhost","root","root","test")
# prepare a cursor object using cursor() method
cursor = db.cursor()
cursor.execute("SELECT * FROM Tablename")
if cursor.fetchone() is not None:
print("In If")
else:
print("In Else")
cursor.close()
Refer this for more information
PS: You can check this link for your question to reusing a DB connection for later.
How to enable MySQL client auto re-connect with MySQLdb?

Retrieve current host from within an executed task

I'm using fabric to execute some remote commands on several hosts by setting:
env.hosts = [host1, host2, ...]
There are several tasks I want to perform on some of the hosts, and some I don't.
is there's a way I can retrieve the current hostname the task is executing on?
Any help would be great.
Thanks,
Meny

Why not use different host roles?
from fabric.api import env, roles, run
env.roledefs['webservers'] = ['www1', 'www2', 'www3']
#roles('webservers')
def my_task():
run('ls -l')
Additionally, you can get the current executing host from the env dictionary:
def my_task():
print 'Currently executing on {0}'.format(env.host)

eclaird answer definitely helped me use a better practice when executing tasks on several hosts with different roles.
Though, while playing with it, it seems that within the task, env.host will give the name of the current hostname.
for example:
#parallel(pool_size=len(env.hosts))
def upload_to_s3(to):
awscreds = 'some credentials...'
cmd = '%s aws s3 sync /mnt/backup %s/%s/' % (awscreds, to, env.host)
run(cmd)

Python Fabric decorator

I've some fabric tasks in my fabfile and I need to initialize, the env variable before their execution. I'm trying to use a decorator, it works but fabric always says "no host found Please specify (single)" however if I print the content of my variable "env" all seems good.
Also I call my tasks from another python script.
from fabric.api import *
from instances import find_instances
def init_env(func):
def wrapper(*args, **kwargs):
keysfolder = 'keys/'
env.user = 'admin'
env.key_filename = '%skey_%s_prod.pem'%(keysfolder, args[0])
env.hosts = find_instances(args[1])
return func(args[0], args[1])
return wrapper
#init_env
def restart_apache2(region, groupe):
print(env.hosts)
run('/etc/init.d/apache2 restart')
return True
My script which call the fabfile:
from fabfile import init_env, restart_apache2
restart_apache2('eu-west-1', 'apache2')
Output of print in restart apache2:
[u'10.10.0.1', u'10.10.0.2']
Any idea why my task restart_apache2 doesn't use the env variable?
Thanks
EDIT:
Which is interesting it's if in my script which calls the fabfile, I use settings from fabric.api and set a host ip, it works. This show that my decorator has well initialized the env variable because the key and user are send to fabric. It's only the env.hosts that's not read by fabric...
EDIT2:
I can reach my goal with using settings from fabric.api, like that:
#init_env
def restart_apache2(region, groupe):
for i in env.hosts:
with settings(host_string = '%s#%s' % (env.user, i)):
run('/etc/init.d/apache2 restart')
return True
Bonus question, has there a solution to use directly the env.hosts without settings?

I'm guessing here a little, but I'm assuming you've got into trouble because you're trying to solve two problems at once.
The first issue relates to the issue of multiple hosts. Fabric includes the concepts of roles, which are just groups of machines that you can issue commands to in one go. The information in the find_instances function could be used to populate this data.
from fabric import *
from something import find_instances
env.roledefs = {
'eu-west-1' : find_instances('eu-west-1'),
'eu-west-2' : find_instances('eu-west-2'),
}
#task
def restart_apache2():
run('/etc/init.d/apache2 restart')
The second issue is that you have different keys for different groups of servers. One way to resolve this problem is to use an SSH config file to prevent you from having to mix the details of the keys / users accounts with your fabric code. You can either add an entry per instance into your ~/.ssh/config, or you can use local SSH config (env.use_ssh_config and env.ssh_config_path)
Host instance00
User admin
IdentityFile keys/key_instance00_prod.pem
Host instance01
User admin
IdentityFile keys/key_instance01_prod.pem
# ...
On the command line, you should then be able to issue the commands like:
fab restart_apache2 -R eu-west-1
Or, you can still do single hosts:
fab restart_apache2 -H apache2
In your script, these two are equivalent to the execute function:
from fabric.api import execute
from fabfile import restart_apache2
execute(restart_apache2, roles = ['eu-west-1'])
execute(restart_apache2, hosts = ['apache2'])

How to force the fabric connect to remote host before run() executed?

I am use fabric to write my rsync wrapper, the variable env.host_string will be set by execute() to run task. To get env.host_string, I run('pwd') first, and run rsync.
Is It possible to make sure user set env.hosts before some checkpoint arrived, such as the condition src == env.host_string ?
from fabric.api import run, env, task, abort
from string import Template
#task
def sync_to_same_dir(src, path):
env.host_string
cmd_template = Template("""rsync --dry-run -e ssh -avr $user#$src_host:$path/ $path/""")
if path.endswith("/"):
path = path[:-1]
run("pwd") # it's a work around
if src == env.host_string:
abort("cannot rysnc to the same host")
cmd = cmd_template.substitute(path=path, user=env.user, src_host=src)
run(cmd)
I find the answer from fabric's source code. There is a simple idea: how run check host as needed ?
#needs_host
def put(local_path=None, remote_path=None, use_sudo=False,
mirror_local_mode=False, mode=None):
"""
Upload one or more files to a remote host.
"""
I trace the needs_host it will prompt to ask hosts, when the user don't assign any hosts:
No hosts found. Please specify (single) host string for connection:
We can rewrite the task as:
from fabric.network import needs_host
#task
#needs_host
def the_task_needs_host(): pass

What are you trying to do? A task knows what host it's using w/o having any other fabric calls:
fab -f host-test.py foo
[98.98.98.98] Executing task 'foo'
98.98.98.98
98.98.98.98
Done.
And here is the script eg:
#!/user/bin/env python
from fabric.api import *
env.user = 'mgoose'
#task
#hosts("98.98.98.98")
def foo():
print(env.host)
print(env.host_string)
So you don't have to do anything special to know what host your task is on.

Using an SSH keyfile with Fabric

How do you configure fabric to connect to remote hosts using SSH keyfiles (for example, Amazon EC2 instances)?

Finding a simple fabfile with a working example of SSH keyfile usage isn't easy for some reason. I wrote a blog post about it (with a matching gist).
Basically, the usage goes something like this:
from fabric.api import *
env.hosts = ['host.name.com']
env.user = 'user'
env.key_filename = '/path/to/keyfile.pem'
def local_uname():
local('uname -a')
def remote_uname():
run('uname -a')
The important part is setting the env.key_filename environment variable, so that the Paramiko configuration can look for it when connecting.

Also worth mentioning here that you can use the command line args for this:
fab command -i /path/to/key.pem [-H [user#]host[:port]]

Another cool feature available as of Fabric 1.4 - Fabric now supports SSH configs.
If you already have all the SSH connection parameters in your ~/.ssh/config file, Fabric will natively support it, all you need to do is add:
env.use_ssh_config = True
at the beginning of your fabfile.

For fabric2 in fabfile use the following:
from fabric import task, Connection
#task
def staging(ctx):
ctx.name = 'staging'
ctx.user = 'ubuntu'
ctx.host = '192.1.1.1'
ctx.connect_kwargs.key_filename = os.environ['ENV_VAR_POINTS_TO_PRIVATE_KEY_PATH']
#task
def do_something_remote(ctx):
with Connection(ctx.host, ctx.user, connect_kwargs=ctx.connect_kwargs) as conn:
conn.sudo('supervisorctl status')
and run it with:
fab staging do_something_remote
UPDATE:
For multiple hosts (one host will do also) you can use this:
from fabric2 import task, SerialGroup
#task
def staging(ctx):
conns = SerialGroup(
'user#10.0.0.1',
'user#10.0.0.2',
connect_kwargs=
{
'key_filename': os.environ['PRIVATE_KEY_TO_HOST']
})
ctx.CONNS = conns
ctx.APP_SERVICE_NAME = 'google'
#task
def stop(ctx):
for conn in ctx.CONNS:
conn.sudo('supervisorctl stop ' + ctx.APP_SERVICE_NAME)
and run it with fab or fab2:
fab staging stop

For me, the following didn't work:
env.user=["ubuntu"]
env.key_filename=['keyfile.pem']
env.hosts=["xxx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com"]
or
fab command -i /path/to/key.pem [-H [user#]host[:port]]
However, the following did:
env.key_filename=['keyfile.pem']
env.hosts=["ubuntu#xxx-xx-xxx-xxx-southeast-1.compute.amazonaws.com"]
or
env.key_filename=['keyfileq.pem']
env.host_string="ubuntu#xxx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com"

I had to do this today, my .py file was as simple as possible, like the one posted in the answer of #YuvalAdam but still I kept getting prompted for a password...
Looking at the paramiko (the library used by fabric for ssh) log, I found the line:
Incompatible ssh peer (no acceptable kex algorithm)
I updated paramiko with:
sudo pip install paramiko --upgrade
And now it's working.

None of these answers worked for me on py3.7, fabric2.5.0 and paramiko 2.7.1.
However, using the PKey attribute in the documentation does work: http://docs.fabfile.org/en/2.5/concepts/authentication.html#private-key-objects
from paramiko import RSAKey
ctx.connect_kwargs.pkey = RSAKey.from_private_key_file('path_to_your_aws_key')
with Connection(ctx.host, user, connect_kwargs=ctx.connect_kwargs) as conn:
//etc....

As stated above, Fabric will support .ssh/config file settings after a fashion, but using a pem file for ec2 seems to be problematic. IOW a properly setup .ssh/config file will work from the command line via 'ssh servername' and fail to work with 'fab sometask' when env.host=['servername'].
This was overcome by specifying the env.key_filename='keyfile' in my fabfile.py and duplicating the IdentityFile entry already in my .ssh/config.
This could be either Fabric or paramiko, which in my case was Fabric 1.5.3 and Paramiko 1.9.0.