We Keep Coding

Streamlit Database Error (SQL Sever connection)

There wasn't any problem with connecting a SQL Server to Streamlit. But since I have been changed the SQL Server IP address, I am having a trouble with the connection suddenly.
The errors from Streamlit shown below. (I covered up the table name.)
DatabaseError: Execution failed on sql ' SELECT TOP (100) [SEQ] ,[MCNO] ,[STM] ,[PGNM] ,[MATNR] ,[TBFG] ,[GAMNG] ,[SENDFG] ,[CRTM] ,[PGDAT] FROM Table name is confidential WITH(NOLOCK) ': (208, b"Invalid object name 'Table name is confidential'.DB-Lib error message 20018, severity 16:\nGeneral SQL Server error: Check messages from the SQL Server\n")
Also, I have been used pymssql to connect SQL Server as shown in below.
class DB:
def __init__(self, host, port, db, user, pwd) -> None:
self.conn = None
self.cursor = None
self.init(host, port, db, user, pwd)
def __enter__(self):
return self
def execute(self, query):
self.cursor.execute(query)
def commit(self):
self.conn.commit()
def fetchall(self):
return self.cursor.fetchall()
def read_sql(self, query):
return pd.read_sql(query, self.conn)
def init(self, host, port, db, user, pwd):
self.conn = pymssql.connect(
server=host, database=db, user=user, password=pwd, port=port, charset="utf8"
)
self.cursor = self.conn.cursor()
def close(self):
self.conn.close()
self.cursor = None
def __exit__(self, exc_type, exc_val, exc_tb):
self.close()
Can anyone give me a help with this problem?
Thank you!

How to use one class (for database) with another class when database is not initialized in another class with python?

I have a class named Database, i want to use this class with another class to check information in the DB if it exists and execute certain queries to SQL. It looks something like this:
class Database:
"""Class to connect with the desired DB"""
def __init__(self, host, database, user, password):
self.con = psycopg2.connect(
host=host,
database=database,
user=user,
password=password
)
def select(self, query):
cur = self.con.cursor()
cur.execute(query)
fetched_selects = cur.fetchall()
cur.close()
self.con.close()
return fetched_selects
So i want to do something like this:
def check_hostname(self):
if socket.gethostname() in Database.select(??):
....
It has to be object oriented so that is why i am trying it in another class.
I made another class named Host:
class Host:
def __init__(self):
self.db = Database
self.hostname = socket.gethostname() # The Hostname of the system
self.cores = psutil.cpu_count(logical=True) # Total CPU Core count of Host
self.total_RAM = f'{psutil.virtual_memory()[0] / 1024 ** 3:.2f}' # Total Ram of System
self.uptime = int(time.time() - psutil.boot_time()) # Uptime in Seconds
self.free_ram = f'{psutil.virtual_memory()[4] / 1024 ** 3:.2f}' # Available RAM in MB
self.disk_usage = psutil.disk_usage('/') # Used Disk Space in Percentage
I tried the 'check_hostname' function in here but since the Database class doesn't get initiated in the Host class it doesn't work.

single database connection throughout the python application (following singleton pattern)

My Question is what is the best way to maintain the single database connection in the entire application? Using Singleton Pattern? How?
Conditions that are needed to be taken care of:
In case of multiple requests, I should be using the same connection
In case connection is closed, create a new connection
If the connection has timed-out, on new request my code should create a new connection.
The driver to my Database is not supported by the Django ORM. And due to same driver related issues, I am using pyodbc to connect to the database. Right now I am having below class for creating and managing the DB connections:
class DBConnection(object):
def __init__(self, driver, serve,
database, user, password):
self.driver = driver
self.server = server
self.database = database
self.user = user
self.password = password
def __enter__(self):
self.dbconn = pyodbc.connect("DRIVER={};".format(self.driver) +\
"SERVER={};".format(self.server) +\
"DATABASE={};".format(self.database) +\
"UID={};".format(self.user) +\
"PWD={};".format(self.password) + \
"CHARSET=UTF8",
# "",
ansi=True)
return self.dbconn
def __exit__(self, exc_type, exc_val, exc_tb):
self.dbconn.close()
But the issue with this approach is that it will create new database connection for each query. What will be the better way to do it following singleton pattern? The way I can think of will hold the reference to the connection if the connection is closed. Something like:
def get_database_connection():
conn = DBConnection.connection
if not conn:
conn = DBConnection.connection = DBConnection.create_connection()
return conn
What will be the best way to achieve this? Any suggestion/ideas/examples?
PS: I was checking about using weakref which allows to create weak references to objects. I think it will be good idea to use weakref along with singleton pattern for storing the connection variable. This way I won't have to keep the connection alive when DB is not in use. What you guys say about this?

For now, I am going ahead with the singleton class approach. Anyone seeing the potential flaws in this, feel to mention them :)
DBConnector class for creating a connection
class DBConnector(object):
def __init__(self, driver, server, database, user, password):
self.driver = driver
self.server = server
self.database = database
self.user = user
self.password = password
self.dbconn = None
# creats new connection
def create_connection(self):
return pyodbc.connect("DRIVER={};".format(self.driver) + \
"SERVER={};".format(self.server) + \
"DATABASE={};".format(self.database) + \
"UID={};".format(self.user) + \
"PWD={};".format(self.password) + \
"CHARSET=UTF8",
ansi=True)
# For explicitly opening database connection
def __enter__(self):
self.dbconn = self.create_connection()
return self.dbconn
def __exit__(self, exc_type, exc_val, exc_tb):
self.dbconn.close()
DBConnection class for managing the connections
class DBConnection(object):
connection = None
#classmethod
def get_connection(cls, new=False):
"""Creates return new Singleton database connection"""
if new or not cls.connection:
cls.connection = DBConnector().create_connection()
return cls.connection
#classmethod
def execute_query(cls, query):
"""execute query on singleton db connection"""
connection = cls.get_connection()
try:
cursor = connection.cursor()
except pyodbc.ProgrammingError:
connection = cls.get_connection(new=True) # Create new connection
cursor = connection.cursor()
cursor.execute(query)
result = cursor.fetchall()
cursor.close()
return result

class DBConnector(object):
def __new__(cls):
if not hasattr(cls, 'instance'):
cls.instance = super(DBConnector, cls).__new__(cls)
return cls.instance
def __init__(self):
#your db connection code in constructor
con = DBConnector()
con1 = DBConnector()
con is con1 # output is True
Hope, above code will helpful.

AWS Lambda Python with MySQL

I'm trying to connect to mysql from my AWS Lambda script.I did pip install --allow-external mysql-connector-python mysql-connector-python -t <dir>
to install mysql-connector-python in local directory.
I zipped the file and uploaded it to AWS Lambda where my python files are being executed.
My scripts are executing correctly up to the point where I initialize a mysql connection.
I have this
log('about to set connection for db')
connection = mysql.connector.connect(user=DB_USER, password=DB_PASSWORD, host=DB_HOST, database=DB_DATABASE)
query = "SELECT * FROM customers WHERE customer_email LIKE '%s' LIMIT 1"
log('set connection for DB')
'about to set connection for db' is being logged but 'set connection for DB' is never logged and my program hits a timeout and stops executing.
What might I be doing wrong?
EDIT:
This is my class that I'm calling from lambda_function.py
import mysql.connector
import logging
from mysql.connector import errorcode
class MySql( object ):
USER =None
PASSWORD=None
HOST =None
DATABASE=None
logger = logging.getLogger()
logger.setLevel( logging.INFO )
def __init__(self, user, password, host, database):
global USER, PASSWORD, HOST, DATABASE
USER = user
PASSWORD = password
HOST = host
DATABASE = database
def getId( self, customer_email ):
email_exists = False
connection = mysql.connector.connect(user=USER, password=PASSWORD, host=HOST, database=DATABASE)
query = "SELECT * FROM customers WHERE customer_email LIKE '%s' LIMIT 1"
cursor = connection.cursor()
cursor.execute( query % customer_email )
data = cursor.fetchall()
id = None
for row in data :
id = row[1]
break
cursor.close()
connection.close()
return id
def insertCustomer( self, customer_email, id ):
log('about to set connection for db')
connection = mysql.connector.connect(user=USER, password=PASSWORD, host=HOST, database=DATABASE)
log('set connection for DB')
cursor = connection.cursor()
try:
cursor.execute("INSERT INTO customers VALUES (%s,%s)",( customer_email, id ))
connection.commit()
except:
connection.rollback()
connection.close()
def log( logStr):
logger.info( logStr )
def main():
user = 'xxx'
password = 'xxx'
host = ' xxx'
database = 'xxx'
mysql = MySql( user, password, host, database )
id = mysql.getId('testing')
if id == None:
mysql.insertCustomer('blah','blahblah')
print id
if __name__ == "__main__":
main()
When I execute the MySql.py locally my code works fine. My database gets updated but nothing happens when I run it from AWS.

Is it a MySQL instance on AWS (RDS) or on premise DB? If RDS, check the NACL inbound rules of the vpc associated with your DB instance. Inbound rules can allow/deny from specific IP sources

when you did a zip file create. Did you do a pip to target directory.
I have enclosed the syntax below.This copies the files to your target directory to zip.
That may be the reason you are able to execute it locally but not in a lambda.
This is the syntax
pip install module-name -t /path/to/PythonExampleDir

Python Database connection Close

Using the code below leaves me with an open connection, how do I close?
import pyodbc
conn = pyodbc.connect('DRIVER=MySQL ODBC 5.1 driver;SERVER=localhost;DATABASE=spt;UID=who;PWD=testest')
csr = conn.cursor()
csr.close()
del csr

Connections have a close method as specified in PEP-249 (Python Database API Specification v2.0):
import pyodbc
conn = pyodbc.connect('DRIVER=MySQL ODBC 5.1 driver;SERVER=localhost;DATABASE=spt;UID=who;PWD=testest')
csr = conn.cursor()
csr.close()
conn.close() #<--- Close the connection
Since the pyodbc connection and cursor are both context managers, nowadays it would be more convenient (and preferable) to write this as:
import pyodbc
conn = pyodbc.connect('DRIVER=MySQL ODBC 5.1 driver;SERVER=localhost;DATABASE=spt;UID=who;PWD=testest')
with conn:
crs = conn.cursor()
do_stuff
# conn.commit() will automatically be called when Python leaves the outer `with` statement
# Neither crs.close() nor conn.close() will be called upon leaving the `with` statement!!
See https://github.com/mkleehammer/pyodbc/issues/43 for an explanation for why conn.close() is not called.
Note that unlike the original code, this causes conn.commit() to be called. Use the outer with statement to control when you want commit to be called.
Also note that regardless of whether or not you use the with statements, per the docs,
Connections are automatically closed when they are deleted (typically when they go out of scope) so you should not normally need to call [conn.close()], but you can explicitly close the connection if you wish.
and similarly for cursors (my emphasis):
Cursors are closed automatically when they are deleted (typically when they go out of scope), so calling [csr.close()] is not usually necessary.

You can wrap the whole connection in a context manager, like the following:
from contextlib import contextmanager
import pyodbc
import sys
#contextmanager
def open_db_connection(connection_string, commit=False):
connection = pyodbc.connect(connection_string)
cursor = connection.cursor()
try:
yield cursor
except pyodbc.DatabaseError as err:
error, = err.args
sys.stderr.write(error.message)
cursor.execute("ROLLBACK")
raise err
else:
if commit:
cursor.execute("COMMIT")
else:
cursor.execute("ROLLBACK")
finally:
connection.close()
Then do something like this where ever you need a database connection:
with open_db_connection("...") as cursor:
# Your code here
The connection will close when you leave the with block. This will also rollback the transaction if an exception occurs or if you didn't open the block using with open_db_connection("...", commit=True).

You might try turning off pooling, which is enabled by default. See this discussion for more information.
import pyodbc
pyodbc.pooling = False
conn = pyodbc.connect('DRIVER=MySQL ODBC 5.1 driver;SERVER=localhost;DATABASE=spt;UID=who;PWD=testest')
csr = conn.cursor()
csr.close()
del csr

You can define a DB class as below. Also, as andrewf suggested, use a context manager for cursor access.I'd define it as a member function.
This way it keeps the connection open across multiple transactions from the app code and saves unnecessary reconnections to the server.
import pyodbc
class MS_DB():
""" Collection of helper methods to query the MS SQL Server database.
"""
def __init__(self, username, password, host, port=1433, initial_db='dev_db'):
self.username = username
self._password = password
self.host = host
self.port = str(port)
self.db = initial_db
conn_str = 'DRIVER=DRIVER=ODBC Driver 13 for SQL Server;SERVER='+ \
self.host + ';PORT='+ self.port +';DATABASE='+ \
self.db +';UID='+ self.username +';PWD='+ \
self._password +';'
print('Connected to DB:', conn_str)
self._connection = pyodbc.connect(conn_str)
pyodbc.pooling = False
def __repr__(self):
return f"MS-SQLServer('{self.username}', <password hidden>, '{self.host}', '{self.port}', '{self.db}')"
def __str__(self):
return f"MS-SQLServer Module for STP on {self.host}"
def __del__(self):
self._connection.close()
print("Connection closed.")
#contextmanager
def cursor(self, commit: bool = False):
"""
A context manager style of using a DB cursor for database operations.
This function should be used for any database queries or operations that
need to be done.
:param commit:
A boolean value that says whether to commit any database changes to the database. Defaults to False.
:type commit: bool
"""
cursor = self._connection.cursor()
try:
yield cursor
except pyodbc.DatabaseError as err:
print("DatabaseError {} ".format(err))
cursor.rollback()
raise err
else:
if commit:
cursor.commit()
finally:
cursor.close()
ms_db = MS_DB(username='my_user', password='my_secret', host='hostname')
with ms_db.cursor() as cursor:
cursor.execute("SELECT ##version;")
print(cur.fetchall())

According to pyodbc documentation, connections to the SQL server are not closed by default. Some database drivers do not close connections when close() is called in order to save round-trips to the server.
To close your connection when you call close() you should set pooling to False:
import pyodbc
pyodbc.pooling = False

The most common way to handle connections, if the language does not have a self closing construct like Using in .NET, then you should use a try -> finally to close the objects. Its possible that pyodbc does have some form of automatic closing but here is the code I do just in case:
conn = cursor = None
try:
conn = pyodbc.connect('DRIVER=MySQL ODBC 5.1 driver;SERVER=localhost;DATABASE=spt;UID=who;PWD=testest')
cursor = conn.cursor()
# ... do stuff ...
finally:
try: cursor.close()
except: pass
try: conn.close()
except: pass