I have a school project for a Python script which runs an SQL query and spits out results to a file... I have this so far and wanted some feedback on if this looks right or if I am way off (I'm using adodbapi). Thanks much!
import adodbapi
# Connect to the SQL DB
conn = adodbapi.connect("Provider=SQLDB;SERVER= x.x.x.x ;User Id=user;Password=pass;DATABASE=db database;")
curs = conn.cursor()
# Execute SQL query test_file.sql"
query = 'test_file'
curs.execute("SELECT test_file")
rows = curs.fetchall()
for row in rows:
print test_file | test_file.txt
conn.close()
# Execute SQL query test_file.sql" You are not executing an SQL query from a file. You are executing the SQL query "SELECT test_file".
"SELECT test_file" is not valid SQL syntax for the SELECT query. See this tutorial on the SELECT statement.
rows = curs.fetchall(); for row in rows: ... is not a nice way of iterating through all the results of a query.
If your query returns large number of rows, say a million rows, then all one million rows will have to be transferred from the database to your python program before the loop can start. This could be very slow if the database server is on a remote machine.
Your program will have to allocate memory for the entire data set before work begins. This could be hundreds of megabytes.
The more Pythonic way of doing this is to avoid loading the entire data set into memory unless you have to. Using sqlite3 I would write:
results = curs.execute("SELECT * FROM table_name")
for row in results:
print (row)
This way only one row is loaded at a time.
print test_file | test_file.txt: the print statement does not support the pipe operator for writing to a file. (Python is not a Linux shell!) See Python File I/O.
Additionally, even if this syntax was correct, you have failed to put the file name in 'quote marks'. Without quotes, Python will interpret test_file.txt as the property txt of a variable called test_file. This will get you a NameError because there is no variable called test_file, or possibly an AttributeError.
If you want to test your code without having to connecting to a network database, then use the sqlite3 module. This is a built-in library of Python, implementing a database similar to adodbapi.
import sqlite3
db_conn = sqlite3.connect(":memory:") # connect to temporary database
db_conn.execute("CREATE TABLE colours ( Name TEXT, Red INT, Green INT, Blue INT )")
db_conn.execute("INSERT INTO colours VALUES (?,?,?,?)", ('gray', 128, 128, 128))
db_conn.execute("INSERT INTO colours VALUES (?,?,?,?)", ('blue', 0, 0, 255))
results = db_conn.execute("SELECT * FROM colours")
for row in results:
print (row)
In future please try running your code, or at least testing that individual lines do what you expect. Trying print test_file | test_file.txt in an interpreter would have given you a TypeError: unsupported operand type(s) for |: 'str' and 'str'.
Related
I am using the mysql.connector package in Python in order to query some data from my database.
connection = mysql.connector.connect(
host="host",
user="usr",
password="pw"
)
mycursor = connection.cursor(buffered=True)
command = "SELECT data FROM schema.`table`"
mycursor.execute(command)
fetchy = mycursor.fetchone()
print(fetchy)
connection.close()
The column "data" is of type JSON. This command takes multiple minutes to run for a single row. I have several thousand rows. If I select from a different non-json column, I have no problem returning data instantly, but with this column it is taking absurdly long. It does not take this long when querying from MySQL workbench.
I'm wondering if there is an issue with my machine or if this is normal.
Is there a reason for this? Or anything I can do to fix it?
I'm using python in TestComplete to conduct a db query, but the results seem to be empty strings and do not match the data in the table I queried. The file is a s3db file. Does that matter?
Using:
TestComplete Version 14
imported sqlite3 into python file
I've:
-Tried running the same query in SQLite. It returned the expected result
-Verified the connection is established with the correct db
---python
import sqlite3
def getInfo():
conn = sqlite3.connect(db)
c = conn.cursor()
try:
c.execute('SELECT Column_Name FROM Table_Name')
results = c.fetchall()
except:
Log.Error("Query execution failed")
for x in results:
Log.Message(x) `enter code here`
#Log.Message() works like a print statement in testcomplete.
---
Actual Output:
The program runs without errors, but the results come back as 15 lines of blank rows. 15 is the number of records within the table, so I know it's looking in the right place, but it seems like it's not identifying that there's information stored here.
Expected Output:
15 lines of data contained within the Column I specified in the query.
There is no error with sqlite3 and your DB operations. The issue is with Log.Message and what it expects as an argument. Within TestComplete, Log.Message requires variable arguments of type Variant, which can be any of the supported data types within TestComplete; String, Double/Real, Boolean, Date/Time, Object (i.e. TestComplete-recognised UI objects) and Integer.
Log.Message cannot accept arguments of the type returned by cursor.fetchall's rows.
So you'd need to convert each row into a String, e.g.
for x in results:
msg = ('{0} : {1}'.format(x[0], x[1]))
Log.Message(msg)
1 step: Create a temporary table with pyodbc into sql server for objects
2 step: Select objects from temporary table and load it into pandas dataframe
3 step: print dataframe
for creating a temporary table i work with pyodbc cursor as it trohws errors with pandas.read_sql command. wheras it trohws an error if i try to convert the cursor into a pandas dataframe. even with the special line for handling tuples into dataframes.
my program to connect, create, read and print which works as long as the query stays simple as it is now. (my actual approach has a few hundred lines of sql query statement)
import codecs
import os
import io
import pandas as pd
import pyodbc as po
server = 'sql_server'
database = 'sql_database'
connection = po.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';Trusted_Connection=yes;')
cursor = connection.cursor()
query1 = """
CREATE TABLE #ttobject (object_nr varchar(6), change_date datetime)
INSERT INTO #ttobject (object_nr)
VALUES
('112211'),
('113311'),
('114411');
"""
query2 = """
SELECT *
FROM #ttobject
Drop table if exists #ttobject
"""
cursor.execute(query1)
df = pd.read_sql_query(query2, connection)
print(df)
Because of the lenght of the actually query i save you the trouble but instead post here the error code:
('HY000', '[HY000] [Microsoft][ODBC SQL Server Driver]Connection is busy with results for another hstmt (0) (SQLExecDirectW)')
This error gets thrown at query2 which is a multiple select statement with some joins and pivote functions
When I'm trying to put everything into one cursor i got issues with converting it from cursor to DataFrame (tried several methodes, maybe someone knows one which isn't on SO already or has a special title so i couldn't find it)
same problem if I'm trying to only use pd.read_sql then the creation of the temporary table is not working
I don't know where to go on from here.
Please let me know if i can assist you with further details which i may overwatched in accordance to my lostlyness :S
23.5.19 Further investigating:
According to Gord i tried to add autocommit to true which will work
for simple sql statements but not for my really long and
timeconsuming one.
Secondly i tried to add
"cursor.execute('SET NOCOUNT ON; EXEC schema.proc #muted = 1')
At the moment i guess that the first query takes longer so python already starting with the second and therefore the connection is
blocked. Or that the first query is returing some feedback so python
thinks it is finished before it actually is.
Added a time.sleep(100) after ececution of first query but still getting the hstmt is busy error. Wondering why this is becaus it should have had enough time to process the first
Funfact: The query is running smoothly as long as I'm not trying to output any result from it
this seems like a basic function, but I'm new to Python so maybe I'm not googling this correctly.
In Microsoft SQL Server, when you have a statement like
SELECT top 100 * FROM dbo.Patient_eligibility
you get a result like
Patient_ID | Patient_Name | Patient_Eligibility
67456 | Smith, John | Eligible
...
etc.
Etc.
I am running a connection to SQL through Python as such, and would like the output to look exactly the same as in SQL. Specifically - with column names and all the data rows specified in the SQL query. It doesn't have to appear in the console or the log, I just need a way to access it to see what's in it.
Here is my current code attempts:
import pyodbc
conn = pyodbc.connect(connstr)
cursor = conn.cursor()
sql = "SELECT top 100 * FROM [dbo].[PATIENT_ELIGIBILITY]"
cursor.execute(sql)
data = cursor.fetchall()
#Query1
for row in data :
print (row[1])
#Query2
print (data)
#Query3
data
My understanding is that somehow the results of PATIENT_ELIGIBILITY are stored in the variable data. Query 1, 2, and 3 represent methods of accessing that data that I've googled for - again seems like basic stuff.
The results of #Query1 give me the list of the first column, without a column name in the console. In the variable explorer, 'data' appears as type List. When I open it up, it just says 'Row object of pyodbc module' 100 times, one for each row. Not what I'm looking for. Again, I'm looking for the same kind of view output I would get if I ran it in Microsoft SQL Server.
Running #Query2 gets me a little closer to this end. The results appear like a .csv file - unreadable, but it's all there, in the console.
Running #Query3, just the 'data' variable, gets me the closest result but with no column names. How can I bring in the column names?
More directly, how do i get 'data' to appear as a clean table with column names somewhere? Since this appears a basic SQL function, could you direct me to a SQL-friendly library to use instead?
Also note that neither of the Queries required me to know the column names or widths. My entire method here is attempting to eyeball the results of the Query and quickly check the data - I can't see that the Patient_IDs are loading properly if I don't know which column is patient_ids.
Thanks for your help!
It's more than 1 question, I'll try help you and give advice.
I am running a connection to SQL through Python as such, and would like the output to look exactly the same as in SQL.
You are mixing SQL as language and formatted output of some interactive SQL tool.
SQL itself does not have anything about "look" of data.
Also note that neither of the Queries required me to know the column names or widths. My entire method here is attempting to eyeball the results of the Query and quickly check the data - I can't see that the Patient_IDs are loading properly if I don't know which column is patient_ids.
Correct. cursor.fetchall returns only data.
Field informations can be read from cursor.description.
Read more in PEP-O249
how do i get 'data' to appear as a clean table with column names somewhere?
It depends how do you define "appear".
You want: text output, html page or maybe GUI?
For text output: you can read column names from cursor.description and print them before data.
If you want html/excel/pdf/other - find some library/framework suiting your taste.
If you want an interactive experience similar to SQL tools - I recommend to look on jupyter-notebook + pandas.
Something like:
pandas.read_sql_query(sql)
will give you "clean table" nothing worse than SQLDeveloper/SSMS/DBeaver/other gives.
We don't need any external libraries.
Refer to this for more details.
Print results in MySQL format with Python
However, the latest version of MySQL gives an error to this code. So, I modified it.
Below is the query for the dataset
stri = "select * from table_name"
cursor.execute(stri)
data = cursor.fetchall()
mycon.commit()
Below it will print the dataset in tabular form
def columnnm(name):
v = "SELECT LENGTH("+name+") FROM table_name WHERE LENGTH("+name+") = (SELECT MAX(LENGTH("+name+")) FROM table_name) LIMIT 1;"
cursor.execute(v)
data = cursor.fetchall()
mycon.commit()
return data[0][0]
widths = []
columns = []
tavnit = '|'
separator = '+'
for cd in cursor.description:
widths.append(max(columnnm(cd[0]), len(cd[0])))
columns.append(cd[0])
for w in widths:
tavnit += " %-"+"%ss |" % (w,)
separator += '-'*w + '--+'
print(separator)
print(tavnit % tuple(columns))
print(separator)
for row in data:
print(tavnit % row)
print(separator)
I'm trying to retrieve data from a mysql server using the Python MySQL Connector.
So far, I have the query set up right using example code I found, and other resources. The problem is, I'm not sure how get it to print back all the rows, instead of just a certain row.
The code I'm using is:
def dbConnect(tag):
qrfid = tag
cnx = mysql.connector.connect(user='root', password='root', host='localhost', database='test')
cursor = cnx.cursor(buffered=True)
query = ("SELECT * FROM `user` WHERE `rfid`= %s")
cursor.execute(query, (qrfid,))
if not cursor.rowcount:
print("No results found")
else:
for row in cursor:
print row[1]
cursor.close()
cnx.close()
I'm using a test table that has only 3 columns in it: id, name, and rfid.
The above code only prints out the 2nd column, name, where if I put row[0], I get id, etc.
I'm use to using PHP for queries, but the RFID readers I'm using only have Python, Flash, and C support. Python is the language I know the most out of those 3.
Thanks!
Morning Trevor,
it is a bit late, but your question seems to be unanswered and should not stay like this.
I guess you've mistaken the way the data is returned. You assumed that you access the data of the header using row[0] and the records using row[1], row[2] ...
But actually, the "for row in cursor:" statement creates an iterator for the cursor object, which returns one complete record per loop. The record is represented in form of a tuple, referenced by name row. You can display the while record using "print(record)" or slice out the first column with row[0], second with row[1], etc.
So, everything seems fine. However, be careful with the query where clause. When comparing to a string (%s placeholder), you should always enclose the string in quotes. Some SQL interpreters expect string literals in comparisons to be quoted.