I am processing events stored in MS-Access databases using pyodbc.
Each month is a seperate file / database and I would like to process events from multiple months.
Is it possible to create a cursor to a view containing multiple months i.e. database connections?
Edit 1: And without having to write a new database? (Something like UNION VIEW maybe?)
You'll need to make multiple connections and cursors, but you should be able to process the data.
Let's say the files are stored as month_1.mdb, month_2.mdb, etc. in C:\access.
# Set up each connection, must have a way to access each file's name
connect_string = "Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\\access\\month_{}.mdb;"
# Assuming that you'll get the same data from each database
sql = "SELECT column_1, column_2 FROM table"
# Connect to each file
connections = [pyodbc.connect(connect_string.format(n)) for n in range(1, 12 + 1)]
# Create a cursor for each file
cursors = [conn.cursor() for conn in connections]
# Query each file and save the data
data = []
for cur in cursors:
cur.execute(sql)
data.extend(cur.fetchall())
OK, so now you have all the data. You can create an in-memory database with the sqlite3 module and then do queries against it.
import sqlite3
# Create your temporary database
connection = sqlite3.connect(":memory:")
cursor = connection.cursor()
# Set up a place to hold the data fetched previously
_ = cur.execute("CREATE TABLE t(x INTEGER, y INTEGER)")
# Dump all the data into the database
for column_1, column_2 in data:
_ = cursor.execute("INSERT INTO t VALUES (?, ?)", [column_1, column_2])
# Now you can run queries against the new view of your data
sql = "SELECT t.column_1, t.count(*) FROM t GROUP BY t.column_1"
Related
I'm making a project where I need to access to remote databases and get data from it.
I'm connecting to remote postgres database and getting list of all tables in my class-based view like so:
try:
# connect to the PostgreSQL serve
conn = psycopg2.connect(
host='host',
database='db_name',
user='username',
password='password',
port='port',
)
# create a cursor
cursor = conn.cursor()
cursor.execute("select relname from pg_class where relkind='r' and relname !~ '^(pg_|sql_)';")
rows = cursor.fetchall()
# close the communication with the PostgreSQL
cursor.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
Now, in another view I want to make other queries (like retrieving specific rows from specific table). How am I able to do that?
The goal is to take all credentials from user's input in template to connect to db. Then on another template choose which table and which rows to use to get certain data.
This question already has answers here:
Why are some mysql connections selecting old data the mysql database after a delete + insert?
(2 answers)
Closed 3 months ago.
I need to repeatedly query a MySQL DB from Python, as the data is rapidly changing. Each time the data is read, it is transferred into a list.
I had assumed that simply putting the query in a loop would fetch the data from the database on each iteration. It seems not.
import mysql.connector
from mysql.connector import Error
from time import sleep
# Create empty list to store values from database.
listSize = 100
myList = []
for i in range(listSize):
myList.append([[0,0,0]])
# Connect to MySQL Server
mydb = mysql.connector.connect(host='localhost',
database='db',
user='user',
password='pass')
# Main loop
while True:
# SQL query
sql = "SELECT * FROM table"
# Read the database, store as a dictionary
mycursor = mydb.cursor(dictionary=True)
mycursor.execute(sql)
# Store data in rows
myresult = mycursor.fetchall()
# Transfer data into list
for row in myresult:
myList[int(row["rowID"])] = (row["a"], row["b"], row["c"])
print(myList[int(row["rowID"])])
print("---")
sleep (0.1)
I have tried using fetchall, fetchmany, and fetchone.
You need to commit the connection after each query. This commits the current transaction and ensures that the next (implicit) transaction will pick up changes made while the previous transaction was active.
# Main loop
while True:
# SQL query
sql = "SELECT * FROM table"
# Read the database, store as a dictionary
mycursor = mydb.cursor(dictionary=True)
mycursor.execute(sql)
# Store data in rows
myresult = mycursor.fetchall()
# Transfer data into list
for row in myresult:
myList[int(row["rowID"])] = (row["a"], row["b"], row["c"])
print(myList[int(row["rowID"])])
# Commit !
mydb.commit()
print("---")
sleep (0.1)
The concept here is isolation levels. From the docs (emphasis mine):
REPEATABLE READ
This is the default isolation level for InnoDB. Consistent reads within the same transaction read the snapshot established by the first read.
I'd make a few changes. First, declare the cursor before the while loop. I would also do a buffered cursor. And finally, close the cursor and DB after the file is done. Hope this helps.
import mysql.connector
from mysql.connector import Error
from time import sleep
# Create empty list to store values from database.
listSize = 100
myList = []
for i in range(listSize):
myList.append([[0,0,0]])
# Connect to MySQL Server
mydb = mysql.connector.connect(host='localhost',
database='db',
user='user',
password='pass')
mycursor = mydb.cursor(buffered=True, dictionary=True)
# Main loop
while True:
# SQL query
sql = "SELECT * FROM table"
# Read the database, store as a dictionary
mycursor.execute(sql)
# Store data in rows
myresult = mycursor.fetchall()
# Transfer data into list
for row in myresult:
myList[int(row["rowID"])] = (row["a"], row["b"], row["c"])
print(myList[int(row["rowID"])])
print("---")
sleep (0.1)
mycursor.close()
mydb.close()
For SqlAlchemy you need to close the session to get last changes:
try:
results = session.query(TableName).all()
return results
except Exception as e:
print(e)
return e
finally:
session.close() # optional, depends on use case
I am trying to just create a temporary table in my SQL database, where I then want to insert data (from a Pandas DataFrame), and via this temporary table insert the data into a 'permanent' table within the database.
So far I have something like
""" Database specific... """
import sqlalchemy
from sqlalchemy.sql import text
dsn = 'dsn-sql-acc'
database = "MY_DATABASE"
connection_str = """
Driver={SQL Server Native Client 11.0};
Server=%s;
Database=%s;
Trusted_Connection=yes;
""" % (dsn,database)
connection_str_url = urllib.quote_plus(connection_str)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % connection_str_url, encoding='utf8', echo=True)
# Open connection
db_connection = engine.connect()
sql_create_table = text("""
IF OBJECT_ID('[MY_DATABASE].[SCHEMA_1].[TEMP_TABLE]', 'U') IS NOT NULL
DROP TABLE [MY_DATABASE].[SCHEMA_1].[TEMP_TABLE];
CREATE TABLE [MY_DATABASE].[SCHEMA_1].[TEMP_TABLE] (
[Date] Date,
[TYPE_ID] nvarchar(50),
[VALUE] nvarchar(50)
);
""")
db_connection.execute("commit")
db_connection.execute(sql_create_table)
db_connection.close()
The "raw" SQL-snippet within sql_create_table works fine when executed in SQL Server, but when running the above in Python, nothing happens in my database...
What seems to be the issue here?
Later on I would of course want to execute
BULK INSERT [MY_DATABASE].[SCHEMA_1].[TEMP_TABLE]
FROM '//temp_files/temp_file_data.csv'
WITH (FIRSTROW = 2, FIELDTERMINATOR = ',', ROWTERMINATOR='\n');
in Python as well...
Thanks
These statements are out of order:
db_connection.execute("commit")
db_connection.execute(sql_create_table)
Commit after creating your table and your table will persist.
I'm running a series of complex sql queries in python and it involves temp tables. My auto-commit method doesn't seem to be working to retrieve the data from the temp table. The code snippet I'm using below and this is the output I'm getting:
testQuery="""
Select top 10 *
INTO #Temp1
FROM Table1 t1
JOIN Table2 t2
on t1.key=t2.key
"""
cnxn=pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=DB;UID=UID;PWD=PWD')
cnxn.autocommit=True
cursor=cnxn.cursor()
cursor.execute(testQuery)
cursor.execute("""Select top 10 * from #Temp1""")
<pyodbc.Cursor at 0x8f78930>
cnxn=pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=DB;UID=UID;PWD=PWD')
cnxn.autocommit=True
cursor=cnxn.cursor()
cursor.execute(testQuery)
cursor.execute("""Select top 10 * from #Temp1""")
Even though this question has a "solution", i.e., using global temp table instead of a local temp table, future readers might benefit from understanding why the problem happened in the first place.
A temporary table is automatically dropped when the last connection using said table is closed. The difference between a local temp table (#Temp1) and a global temp table (##Temp1) is that the local temp table is only visible to the connection that created it, while an existing global temp table is available to any connection.
So the following code using a local temp table will fail ...
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT 1 AS foo, 2 AS bar INTO #Temp1
"""
crsr.execute(sql)
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT foo, bar FROM #Temp1
"""
crsr.execute(sql)
row = crsr.fetchone()
print(row)
... while the exact same code using a global temp table will succeed ...
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT 1 AS foo, 2 AS bar INTO ##Temp1
"""
crsr.execute(sql)
conn = pyodbc.connect(conn_str, autocommit=True)
crsr = conn.cursor()
sql = """\
SELECT foo, bar FROM ##Temp1
"""
crsr.execute(sql)
row = crsr.fetchone()
print(row)
... because the second pyodbc.connect call opens a separate second connection to the SQL Server without closing the first one.
The second connection cannot see the local temp table created by the first connection. Note that the local temp table still exists because the first connection was never closed, but the second connection cannot see it.
However, the second connection can see the global temp table because the first connection was never closed and therefore the global temp table continued to exist.
This type of behaviour has implications for ORMs and other mechanisms that may implicitly open and close connections to the server for each SQL statement that it executes.
I asked a colleague about this live and his suggestions worked. So I went and changed the testQuery to create a global temp table instead of a local (##Temp1 instead of #Temp1). And went to sql server to test whether the temp table was actually being created-it was. So I isolated that the problem was the second cursor.execute statement. I modified the code to use pandas read_sql_query instead and it all worked out! Below is the code I used:
testQuery="""
Select top 10 *
INTO ##Temp1
FROM Table1 t1
JOIN Table2 t2
on t1.key=t2.key
"""
cnxn=pyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=DB;UID=UID;PWD=PWD')
cnxn.autocommit=True
cursor=cnxn.cursor()
cursor.execute(testQuery)
cnxn.commit()
query1="Select top 10 * from ##Temp1"
data1=pd.read_sql_query(query1, cnxn)
data1[:10]
Best way to go about this is to start your SQL query with:
"SET NOCOUNT ON"
This will output the desired data
The SET NOCOUNT ON is what worked for me.
execute Method () - JDBC Driver for SQL Server | Microsoft Docs
Return Value
true, if the statement returns a result set.
false, if it returns an update count or no result.
If you want a result set, then setting SET NOCOUNT ON s the setting you need in your statements.
I am trying to add all words of a text file into a column such that one row has one word. my code is as :
import MySQLdb
conn = MySQLdb.connect (host = "localhost",user = "root", db = "pcorpora")
c = conn.cursor()
file = open('C:\Users\Admin\Desktop\english.txt', 'r')
words = list(file.read())
i=0
for value in words:
c.execute("""INSERT INTO tenglish (`english words`) VALUES (%s)""" % (words[i]) i=i+1)`
The code run without error but table is still empty.
You should use commit
c.execute("""INSERT INTO tenglish (`english words`) VALUES (%s)""" % (value))
con.commit()
This method sends a COMMIT statement to the MySQL server, committing
the current transaction. Since by default Connector/Python does not
autocommit, it is important to call this method after every
transaction that modifies data for tables that use transactional
storage engines.