I'm using 2 mysql database connections. After every page request in django, the connection count ( as displayed by SHOW STATUS LIKE 'Conn%' increases by 2 each time.
Python 3.4.0
django 1.8.2
mysqlclient 1.3.6
( Both Windows and Linux seem to have the issue )
Initially i accessed the second database directly through MySQLdb but now i've switched to using django.db.connections['...'] to access it.
This is my settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'myapp',
'USER': 'root',
'PASSWORD': '',
'HOST': '',
'PORT': '',
},
'analytics': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'myapp_analytics',
'USER': 'root',
'PASSWORD': '',
'HOST': '',
'PORT': '',
},
}
And these are the classes that i wrote to for easy usage:
import MySQLdb as DBConnector
import django
class DictCursorEmulator:
def __init__(self, connection):
self.cursor = connection.cursor()
def _execute(self,query_str,query_args):
self.cursor.execute(query_str,query_args)
def fetchall(self):
"Returns all rows from a cursor as a dict"
desc = self.cursor.description
return [
dict(zip([col[0] for col in desc], row))
for row in self.cursor.fetchall()
]
def fetchone(self):
desc = self.cursor.description
return dict(zip([col[0] for col in desc], row))
#I probably should have used instance methods here but there's too much code to update ;_;
class DatabaseInterface:
#classmethod
def execute(self, queryStr, args):
cursor = DictCursorEmulator(django.db.connections['analytics'])
cursor._execute(queryStr,args)
return cursor
#classmethod
def autocommit(cls, set_to):
django.db.connections['analytics'].autocommit(set_to)
#classmethod
def commit(cls):
django.db.connections['analytics'].commit()
#classmethod
def rollback(cls):
django.db.connections['analytics'].rollback()
I've tried excluding the entire file but it still makes the connections count go up by 2, So i'm starting to think it's not a a problem with my code. Does anyone know how to fix this?
Update: Apparently, Connections shows the total number of connections
Threads connected shows 2, which is expected. That solves my problem. I just had a lot of weird errors popping up on the DB side.
Related
In DB i have three roles: guest, client and admin.
In my django project, there are three connections under these roles
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'test',
'USER': 'guest',
'PASSWORD': 'guest',
'HOST': 'localhost',
'PORT': 5432,
},
'admin': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'test',
'USER': 'admin',
'PASSWORD': 'admin',
'HOST': 'localhost',
'PORT': 5432,
},
'customer': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'test',
'USER': 'customer',
'PASSWORD': 'customer',
'HOST': 'localhost',
'PORT': 5432,
}
}
How and where can I change the connection to the database depending on whether the user is authenticated or not?
I am presuming that you are using psycopg2 to connect to the Postgresql RDBMS. What I would do is specify what Postgresql user you want to use before you execute your query.
For example:
import psycopg2
def func1():
conn = psycopg2.connect(database = "exampledb", user = "user1", password = "user1password", host = "127.0.0.1", port = "5432")
cur = conn.cursor()
cur.execute("SELECT * FROM schema_name.table_name;")
rows = cur.fetchall()
for row in rows:
print(row)
def func2():
conn = psycopg2.connect(database = "exampledb", user = "user2", password = "user2password", host = "127.0.0.1", port = "5432")
cur = conn.cursor()
cur.execute("INSERT INTO schema_name.table_name (col1, col2) VALUES(1, 2);")
rows = cur.fetchall()
for row in rows:
print(row)
I would also be very careful with the admin user, from a security standpoint I would not allow this account to be used for the server-side scripting, this is because if and sql injection is executed, then a lot of harm could be caused. For prevention of sql injections in python I would recommend this: https://realpython.com/prevent-python-sql-injection/
Summary
What is the right syntax to use update_data_source Quicksight boto3 to change credentials ?
Context
I am trying to use update data source method for Quicksight on boto3 to update my Redshift credentials in Quicksight.
My issue is that it is passing a dictionary as key to another dictionary. How can I unpack that to get to the username / password for Redshift ?
Code
My code looks like this :
def main():
qs = boto3.client('quicksight', region_name=region_name)
response = qs.update_data_source(
AwsAccountId='awsaccountid',
DataSourceId='datasourceid',
Name='qs_test',
Credentials={
{
'CredentialPair':{
'Username': 'test_user'
'Password': 'my_pass'
}
}
}
)
print(response)
main()
Also tried the below
response = qs.update_data_source(
AwsAccountId='awsaccountid',
DataSourceId='datasourceid',
Name='qs_test',
Credentials={CredentialPair
{
RedshiftParameters=[
{
'Database': 'dbname',
'ClusterId': 'clusterid'
}
}
],
Credentials={
'CredentialPair': {
'Username': 'test_user',
'Password': 'my_pass'
}
}
)
print(response)
The below syntax works :
def main():
qs = boto3.client('quicksight', region_name=region_name)
response = qs.update_data_source(
AwsAccountId='awsaccountid',
DataSourceId='datasourceid',
Name='qs_test',
DataSourceParameters={
'RedshiftParameters'={
'Database': 'dbname',
'ClusterId': 'clusterid'
}
}
}
Credentials={
'CredentialPair':{
'Username': 'test_user'
'Password': 'my_pass'
}
}
)
print(response)
main()
I want to begin by saying that I am by no mean a python expert so I am sorry if I express myself in an incorrect way.
I am building a script that goes something like this:
from netmiko import ConnectHandler
visw0102 = {
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
}
visw0103 = {
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}
site1_switches = [visw0102, visw0103]
for switch in site1_switches:
... (rest of the script)
I am trying to get the current index name in the FOR loop by using the enumerate() function to get the index name of the site1_switches list but since that list is made of dictionary items, the dictionary keys are returned:
>>> for index, w in enumerate(switch):
... print(w)
...
device_type
ip
username
password
Is there a way the get the actual index name (VISW010X) instead of values that are in the dictionaries?
Thank you
Edit: Nested dictionary was the answer here, thanks Life is complex
So I was able to get further. Here's the code now.
from netmiko import ConnectHandler
site1_switches = {
'visw0102' : {
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
},
'visw0103' : {
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}
}
for key, values in site1_switches.items():
device_type = values.get('device_type', {})
ip_address = values.get('ip', {})
username = values.get('username', {})
password = values.get('password', {})
for key in site1_switches.items():
net_connect = ConnectHandler(**dict(key)) <- The ConnectHandler needs a dictionary
Now the problem is that the dictionary key seems to be converted to a tuple but the ConnectHandler module needs a dictionary to proceed.
Here's what I get:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: dictionary update sequence element #0 has length 8; 2 is required
I would need to find a way to convert the tuple to a dictionary but it seems that dict(key) doesn't work as it puts the tuple in the first dictionary key (or so it seems).
Anyway I can achieve that?
Thanks!
Have you considered using a nested dictionary?
site1_switches = {
'visw0102': {
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
},
'visw0103': {
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}}
for key, value in site1_switches.items():
print (key)
# output
visw0102
visw0103
Here's another way to accomplish this.
for index, (key, value) in enumerate(site1_switches.items()):
print(index, key, value)
# output
0 visw0102 {'device_type': 'hp_comware', 'ip': '192.168.0.241', 'username': 'admin', 'password': 'password'}
1 visw0103 {'device_type': 'hp_comware', 'ip': '192.168.0.242', 'username': 'admin', 'password': 'password'}
A more complete solution
from netmiko import ConnectHandler
# nested dictionary
site1_switches = {
'visw0102': {
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
},
'visw0103': {
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}}
for key, values in site1_switches.items():
device_type = values.get('device_type', {})
ip_address = values.get('ip', {})
username = values.get('username', {})
password = values.get('password', {})
print (f'{key}', {device_type}, {ip_address}, {username}, {password})
# output
visw0102 {'hp_comware'} {'192.168.0.241'} {'admin'} {'password'}
visw0103 {'hp_comware'} {'192.168.0.242'} {'admin'} {'password'}
print (f'Establishing a connection to {key}')
# output
Establishing a connection to visw0102
# pseudo code based on ConnectHandler parameters
switch_connect = ConnectHandler(device_type=device_type, host=ip_address, username=username, password=password)
# checking that the connection has a prompt
switch_connect.find_prompt()
# What you want to do goes here...
# Example
command_output = switch_connect.send_command('display current-configuration')
Unfortunately, there doesn't seem to be a nice, succinct way of accessing the dictionary's name, but Get name of dictionary provides some possible workarounds:
Nesting your switch dictionaries within an overarching dictionary that maps names to dictionaries is one method.
site1_switches = {
"visw0102": visw0102,
"visw0103": visw0103
}
Another would be to add a "name" key to each dictionary, so that you can access the names of each switch in site1_switches by switch['name']
visw0102 = {
'name': 'visw0102',
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
}
visw0103 = {
'name': 'visw0103',
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}
I am writing a multi-tenant application with python-django.
I want to set database connection based on each request.I thought i could write a middleware where we set the database to be used for that particular database.
import re
from django.db import connections
class SetTenantDatabase(object):
def process_request(self, request):
pattern = re.compile("\\b(http://|https://|www.|.com|8000|:|//)\\W\\d+", re.I)
words = request.get_host()
db_name = [pattern.sub("", words)][0].split('.')[0]
connections.databases['new-alias'] = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'store1',
'USER': 'xxx',
'PASSWORD': 'xxx',
'HOST': '127.0.0.1',
}
}
conn = connections['new-alias']
return None
but this is not working.How should i do this.Is the approach wrong or is the solution feasible, and lastly How?
this is the answer, hope it helps someone in future:
import re
import threading
request_cfg = threading.local()
class RouterMiddleware(object):
def process_request( self, request):
pattern = re.compile("\\b(http://|https://|www.|.com|8000|:|//)\\W\\d+", re.I)
words = request.get_host()
db_name = [pattern.sub("", words)][0].split('.')[0]
request_cfg.cfg = db_name
return None
def process_response( self, request, response ):
if hasattr( request_cfg, 'cfg' ):
del request_cfg.cfg
return response
class DatabaseRouter (object):
def _default_db( self ):
if hasattr( request_cfg, 'cfg' ):
return request_cfg.cfg
else:
return 'default'
def db_for_read( self, model, **hints ):
return self._default_db()
def db_for_write( self, model, **hints ):
return self._default_db()
Thanks
Maybe you can use:
https://docs.djangoproject.com/en/dev/topics/db/multi-db/#manually-selecting-a-database-for-a-queryset
Entity.objects.using('context1').all()
Entity.objects.using('context2').all()
To select/use a database depending on the request. You can define multiple DBs in the configurartion:
DATABASES = {
'context1': {
'NAME': 'context1',
'ENGINE': 'db.engine.to.use',
'USER': 'xxx',
'PASSWORD': 'xxx'
},
'context2': {
'NAME': 'context2',
'ENGINE': 'db.engine.to.use',
'USER': 'xxx',
'PASSWORD': 'xxx'
}
}
I'm using Django and having issues exceeding my max number of redis connections. The library I'm using is:
https://github.com/sebleier/django-redis-cache
Here is my settings.py file:
CACHES = {
'default': {
'BACKEND': 'redis_cache.RedisCache',
'LOCATION': "pub-redis-11905.us-east-1-3.1.ec2.garantiadata.com:11905",
'OPTIONS': {
'DB' : 0,
'PASSWORD': "*****",
'PARSER_CLASS': 'redis.connection.HiredisParser'
},
},
}
Then i another file, I do some direct cache access like so:
from django.core.cache import cache
def getResults(self, key):
return cache.get(key)
Looks like this is an outstanding issue with django-redis-cache - perhaps you should consider a different Redis cache backend for Django that does support connection pooling.
Here's django-redis-cache using connection-pool set max_connections.
CACHES = {
'default': {
'OPTIONS': {
'CONNECTION_POOL_CLASS': 'redis.BlockingConnectionPool',
'CONNECTION_POOL_CLASS_KWARGS': {
'max_connections': 50,
'timeout': 20,
...
},
...
},
...
}
}