We Keep Coding

Connect to SQL server from self made module

I currently have many scripts that connect to the same MSSQL database. I make the connection in each of the scripts, but for ease of use I want to put the connection in a module and call that module from my script. The code in my module connect_to_db.pyc looks like this:
import pyodbc
def sql_connect():
server="some_server.net"
port="1433"
user = "my_username#my_domain"
server="my_server"
database="my_database"
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=my_server,1433',
user=user,
password=password,
database=database)
c=conn.cursor()
Then, in my script I try to call this module and run a query:
from connect_to_db import sql_connect
sql_connect()
c.execute("SELECT * FROM table")
I get the error that the name c is not defined. I tried to define it as a global too, but it don't help. It must have something to do with my lack of understanding modules, but I can't figure out what.

You can return cursor in your sql_connect function
import pyodbc
def sql_connect():
server="some_server.net"
port="1433"
user = "my_username#my_domain"
server="my_server"
database="my_database"
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=my_server,1433',
user=user,
password=password,
database=database)
return conn.cursor()
And then you can use it as
from connect_to_db import sql_connect
c = sql_connect()
c.execute("SELECT * FROM table")

You are indeed missing a bit there:
in your function sql_connect, you assign to a local variable named c.
That variable is not existant outside your function.
If you want a connection variable to exist on module level, maybe try the following attempt:
In your "connect_to_db.py":
import pyodbc
def sql_connect():
server="some_server.net"
port="1433"
user = "my_username#my_domain"
server="my_server"
database="my_database"
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=my_server,1433',
user=user,
password=password,
database=database)
return conn.cursor()
cursor = sql_connect()
This creates a varibale "cursor" on the level of the module.
In another module, simply perform
from connect_to_db import cursor
to import the module's "cursor" member.
This should do the trick.
Hint: Please be advised that this approach may not be very elegant, in terms of software-engineering.
Edit:
Maybe, you may want to dive deeper into object-oriented programming?
class MSSQLConnector(object):
def __init__(self, server, port, database, user, password):
self.server = server
self.port = port
self.conn = pyodbc.connect('DRIVER={SQL Server};SERVER='{0},
{1}.format((self.server, self.port)), user, password, database)
def open_cursor(self):
return self.conn.cursor()
Which would be used in this fashion:
connector = MSSQLConnector("my_server", "1433", "my_database", "username", "secret-password")
cursor = connector.open_cursor()

Python3 psycopg2 commitment issue using variable vs function for connection

Can someone please explain to me why the defined test().commit() does not work as varcon.commit()? Everything else seem to work fine. (using vagrant virtualbox of ubuntu-trusty-32)
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import psycopg2
varcon = psycopg2.connect('dbname=tournament')
def test():
try:
psycopg2.connect("dbname=tournament")
except:
print("Connection to Tournament Database Failed")
else:
return psycopg2.connect('dbname=tournament')
def writer():
#db = psycopg2.connect('dbname=tournament')
c =varcon.cursor()
c.execute('select * from players')
data = c.fetchall()
c.execute("insert into players (name) values ('Joe Smith')")
varcon.commit()
varcon.close
print(data)
def writer2():
#db = psycopg2.connect('dbname=tournament')
c =test().cursor()
c.execute('select * from players')
data = c.fetchall()
c.execute("insert into players (name) values ('Joe Smith')")
test().commit()
test().close
print(data)
writer2() #this seem not commited, but database registers the insert by observing the serial promotion
#writer() # this works as expected

maybe this is because the return statement in a function block (def) is not equal to an assignment like =
The psycopg2 connection function returns a connection object and this is assigned to conn or varcon
conn = psycopg2.connect("dbname=test user=postgres password=secret")
http://initd.org/psycopg/docs/module.html
the test() function also returns the psycopg2 connection object but in writer2 it is not assigned to a variable (memory place) meaning that there is no reference
this also explains why the database connection is established (initialized in the test function) but the commit does not work (broken reference)
(http://www.icu-project.org/docs/papers/cpp_report/the_anatomy_of_the_assignment_operator.html)
maybe try
ami=test()
ami.commit()
to establish a reference

Every time you call psycopg2.connect() you open new connection to database.
So effectively your code executes SQL in one connection, commits another, and then closes third connection. Even in your test() function you are opening two different connections.
I use the following pattern to access PostgreSQL:
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
...
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
...
curs.execute(SQL2)
conn.close()
with statement ensures transaction is opened and properly committed around your SQL. It also automatically rolls transaction back in case your code inside with raises an exception.
Reference: http://initd.org/psycopg/docs/usage.html#with-statement

SQL Server temp table not available in pyodbc code

I'm running a series of complex sql queries in python and it involves temp tables. My auto-commit method doesn't seem to be working to retrieve the data from the temp table. The code snippet I'm using below and this is the output I'm getting:
testQuery="""
Select top 10 *
INTO #Temp1
FROM Table1 t1
JOIN Table2 t2
on t1.key=t2.key
"""
cnxn=pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=DB;UID=UID;PWD=PWD')
cnxn.autocommit=True
cursor=cnxn.cursor()
cursor.execute(testQuery)
cursor.execute("""Select top 10 * from #Temp1""")
<pyodbc.Cursor at 0x8f78930>
cnxn=pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=DB;UID=UID;PWD=PWD')
cnxn.autocommit=True
cursor=cnxn.cursor()
cursor.execute(testQuery)
cursor.execute("""Select top 10 * from #Temp1""")

Even though this question has a "solution", i.e., using global temp table instead of a local temp table, future readers might benefit from understanding why the problem happened in the first place.
A temporary table is automatically dropped when the last connection using said table is closed. The difference between a local temp table (#Temp1) and a global temp table (##Temp1) is that the local temp table is only visible to the connection that created it, while an existing global temp table is available to any connection.
So the following code using a local temp table will fail ...
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT 1 AS foo, 2 AS bar INTO #Temp1
"""
crsr.execute(sql)
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT foo, bar FROM #Temp1
"""
crsr.execute(sql)
row = crsr.fetchone()
print(row)
... while the exact same code using a global temp table will succeed ...
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT 1 AS foo, 2 AS bar INTO ##Temp1
"""
crsr.execute(sql)
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT foo, bar FROM ##Temp1
"""
crsr.execute(sql)
row = crsr.fetchone()
print(row)
... because the second pyodbc.connect call opens a separate second connection to the SQL Server without closing the first one.
The second connection cannot see the local temp table created by the first connection. Note that the local temp table still exists because the first connection was never closed, but the second connection cannot see it.
However, the second connection can see the global temp table because the first connection was never closed and therefore the global temp table continued to exist.
This type of behaviour has implications for ORMs and other mechanisms that may implicitly open and close connections to the server for each SQL statement that it executes.

I asked a colleague about this live and his suggestions worked. So I went and changed the testQuery to create a global temp table instead of a local (##Temp1 instead of #Temp1). And went to sql server to test whether the temp table was actually being created-it was. So I isolated that the problem was the second cursor.execute statement. I modified the code to use pandas read_sql_query instead and it all worked out! Below is the code I used:
testQuery="""
Select top 10 *
INTO ##Temp1
FROM Table1 t1
JOIN Table2 t2
on t1.key=t2.key
"""
cnxn=pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=DB;UID=UID;PWD=PWD')
cnxn.autocommit=True
cursor=cnxn.cursor()
cursor.execute(testQuery)
cnxn.commit()
query1="Select top 10 * from ##Temp1"
data1=pd.read_sql_query(query1, cnxn)
data1[:10]

Best way to go about this is to start your SQL query with:
"SET NOCOUNT ON"
This will output the desired data

The SET NOCOUNT ON is what worked for me.
execute Method () - JDBC Driver for SQL Server | Microsoft Docs
Return Value
true, if the statement returns a result set.
false, if it returns an update count or no result.
If you want a result set, then setting SET NOCOUNT ON s the setting you need in your statements.

Python Flask and sqlite3 commit not working

This is just a small bite of my code but I hope it will be enough so solve my problem. Allow me to explain first.
So I am attempting to store information in a database using python and sqlite3. I can store stuff in the database but the file size of the database never increases and after a restart the database is cleared. I know that .commit works because I have used it in the past but it is not working now. My assumption is that I am out of scope of the database and I don't have the ability to write. Once again, my code runs and provides no errors, but it will not actually write to the database.
My Connection and Init Code:
def connect_db():
conn = sqlite3.connect(app.config['DATABASE'])
conn.row_factory = sqlite3.Row
return conn
def init_db():
conn = connect_db()
cursor = conn.cursor()
sql = 'create table if not exists users (id integer primary key autoincrement, username text not null, password text not null, admin boolean not null)'
cursor.execute(sql)
conn.commit()
So this code connects to the database and gets everything setup.
Here is a small portion of my code to add an item to the database. It works and it will add items to the "database" but the file size of the database does not increase and if I restart it wont see the new items. But as long as the application is open I have access to the items.
#app.route('/user/', methods=['GET', 'POST'])
def users():
conn = connect_db()
cursor = conn.cursor()
isAdmin = False
if (request.form.get('adminuser') != None):
isAdmin = True
cursor.execute('insert into users (username, password, admin) values (?, ?, ?)',
[request.form['username'], request.form['password'], isAdmin])
conn.commit()
return redirect(url_for('index'))
Edit I left This Out
DATABASE = '/tmp/database.db'
Edit A Crazy Simple Mistake.
Change
DATABASE = '/tmp/database.db'
To
DATABASE = './tmp/database.db'

There is an error in this line:
DATABASE = '/tmp/database.db'
Use this instead:
DATABASE = './tmp/database.db'

Closing cursor and connection with Python and MySQLdb

I have a simple web.py-based app that uses MySQLdb. I have a class that handles database operations like so:
class db():
def __init__(self):
db = MySQLdb.connect(host='mysql.server', user='user', passwd='pass', db='app')
self.cur = db.cursor()
def get_data(self):
sql = "SELECT * FROM foobar"
self.cur.execute(sql)
rs = self.cur
r.fetchall()
return rs
I instantiate the class like so DB = db(). Then, in another class, I will refer to it.
class bleh()
def blarg():
DB.get_data()
With something like this, where would I close the cursor and connection? Or am I approaching this completely wrong?

db.close() for connection and cur.close() for cursor.
http://mysql-python.sourceforge.net/MySQLdb.html
EDIT:
But if it give it a bit thought - you won't need to close cursor. Python closes the cursor once the variable is destroyed, so when the instance of your class does not exist anymore -- cursor will be closed.

First of all use different names for class-name and variable as you have used same name ('db') for class-name and connection as well.
Next, you need to define conn (in your question db line no 3) as self.conn.
import MySQLdb
class db():
def __init__(self):
self.conn = MySQLdb.connect(host='mysql.server', user='user', passwd='pass', db='app')
self.cur = self.conn.cursor()
def get_data(self):
sql = "SELECT * FROM test"
self.cur.execute(sql)
rs = self.cur
rs.fetchall()
return rs
class bleh()
def blarg():
data = DB.get_data()
DB.cur.close()
DB.conn.close()
Note: If you have multiple functions in class bleh to get data from database make sure that you close cursor and connection in function, which is to called in last. Or you may have a seperate function, which closes cursor and connection.