For some odd reason I can't get results from a callproc call in a Python test app. The stored procedure in MqSQL 5.2.47 looks like this:
CREATE PROCEDURE `mytestdb`.`getperson` (IN personid INT)
BEGIN
select person.person_id,
person.person_fname,
person.person_mi,
person.person_lname,
person.persongender_id,
person.personjob_id
from person
where person.person_id = personid;
END
Now, using PyCharm with Python 3.3, I can't seem to retrieve anything when calling this stored procedure. This code gets me the desired results:
import mysql.connector
cnx = mysql.connector.connect(user='root', host='127.0.0.1', database='mytestdb')
cnx._open_connection()
cursor = cnx.cursor()
cursor.execute("select * from person where person.person_id = 1")
people = cursor.fetchall()
for person in people:
print(person)
cnx.close()
But this code with either cursor.fetchall() or cursor.fetchone()...
import mysql.connector
cnx = mysql.connector.connect(user='root', host='127.0.0.1', database='mytestdb')
cnx._open_connection()
cursor = cnx.cursor()
cursor.callproc("getperson", [1])
people = cursor.fetchall()
for person in people:
print(person)
cnx.close()
... returns "mysql.connector.errors.InterfaceError: No result set to fetch from." There's an additional odd behavior using the cursor.execute() method like so...
import mysql.connector
cnx = mysql.connector.connect(user='root', host='127.0.0.1', database='mytestdb')
cnx._open_connection()
cursor = cnx.cursor()
cursor.execute("call getperson(1)")
people = cursor.fetchall()
for person in people:
print(person)
cnx.close()
... because it yields "mysql.connector.errors.InterfaceError: Use cmd_query_iter for statements with multiple queries" followed by "mysql.connector.errors.InterfaceError: Use multi=True when executing multiple statements" despite the fact that I'm only returning one query result rather than multiple result sets. Is the MySQL Python connector treating the execute call on the stored procedure as a double query? How can I just call the stored procedure and get my results back? I really don't want dynamic SQL in my code. Thanks ahead for any advice!
Have you tried picking one of the resultsets?
for result in cursor.stored_results():
people = result.fetchall()
It could be that it's allocating for multiple resultsets even though you only have one SELECT stmt. I know in PHP's MySQLi stored procedures do this to allow for INOUT and OUT variable returns (which again, you have none of, but maybe it's allocating anyways).
The complete code I'm using (which is working) is:
import mysql.connector
cnx = mysql.connector.connect(user='me',password='pw',host='localhost',database='mydb')
cnx._open_connection()
cursor = cnx.cursor()
cursor.callproc("getperson",[1])
for result in cursor.stored_results():
people=result.fetchall()
for person in people:
print person
cnx.close()
Getting the result of a stored procedure after calling cursor.callproc depends on these factors:
whether the result of calling the procedure is assigned to an INOUT or OUT parameter
whether the result consists of a single row or a result set (or result sets)
the python package used to make the call
The DBAPI spec has this to say on cursor.callproc:
Call a stored database procedure with the given name. The sequence of parameters must contain one entry for each argument that the procedure expects. The result of the call is returned as modified copy of the input sequence. Input parameters are left untouched, output and input/output parameters replaced with possibly new values.
The procedure may also provide a result set as output. This must then be made available through the standard .fetch*() methods.
In practice, using the return value of cursor.callproc can only work if the procedure returns a single row, with the number of columns matching the number of INOUT and OUT parameters, so there is some variation in how the results are handled.
Here is how these cases are handled by the principal MySQL Python connector packages - MySQL Connector, mysqlclient (MySQLdb) and PyMySQL.
Single row result, returned via INOUT or OUT parameters
MySQL Connector returns a modified copy of the input sequence as the return value of cursor.callproc; the value is a tuple.
params = [in_param, out_param1, out_param2]
in_, out1, out2 = cursor.callproc("test_proc", params)
mysqlclient and PyMySQL require that the database is queried for the output parameters, and the results then fetched via the cursor; the value is a tuple of tuples. The parameter names to be queried are of the form '#_{procedure_name}_{params.index(param)}'
cursor.callproc("test_proc", params)
cursor.execute("""SELECT #_test_proc_0, #_test_proc_1""")
result = cursor.fetchall()
One or more rows in a single result set, no INOUT or OUT parameters defined
MySQL Connector exposes the result via the cursor's stored_results method (cursor.stored_results is not part of the DBAPI spec)
cursor.callproc("test_proc", params)
results = [r.fetchall() for r in cursor.stored_results()]
mysqlclient and PyMySQL expose the result via the cursor's fetch* methods
cursor.callproc("test_proc", params)
results = cursor.fetchall()
Multiple result sets, no INOUT or OUT parameters defined
MySQL Connector exposes the result via the cursor's stored_results method
cursor.callproc("test_proc", params)
results = [r.fetchall() for r in cursor.stored_results()]
mysqlclient and PyMySQL require that each result set be fetched via the cursor, while calling cursor.nextset to advance to the next result set. Note that an extra, empty result set may be returned, a result of calling the procedure (this would also happen in the previous examples, if the result set were retrieved via cursor.nextset instead of only calling cursor.fetchall once).
cursor.callproc("test_proc", params)
results = [cursor.fetchall()]
while cursor.nextset():
results.append(cursor.fetchall())
Version Info
$ mysql --version
mysql Ver 15.1 Distrib 10.1.41-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
$ pip list | grep -i mysql
mysql-connector-python 8.0.18
mysqlclient 1.4.6
PyMySQL 0.9.3
Why not try it this way
cursor.callproc("getperson", ['1'])
Related
I am trying to query a large data (10 million rows) and try to prevent out of memory, but not familiar with Python and confused with different opinions regarding the execute(), cursor iterator and fetchone()
Am I right to assume that cursor.execute() does not load all data into memory and only when I call fetchone() then it will load 1 row of data
from mysql.connector import MySQLConnection
def query():
conn = MySQLConnection(host=conf['host'],
conf['port'],
conf['user'],
conf['password'],
conf['database'])
cursor = conn.cursor(buffered=True)
cursor.execute('SELECT * FROM TABLE') # 10 million rows
does this cursor iterator does the same with fetchone() ?
for row in cursor:
print(row)
is my code snippet is safe to handle 10 million rows of data? if not, how can I safely iterate the data without out of memory?
My first suggestion is to use from mysql.connector import connect, which by the default will use the C extension (CMySQLConnection), instead of from mysql.connector import MySQLConnection (pure Python).
If you for some reason want to use the pure Python version, you can pass use_pure=True in connect()
The second suggestion is to paginate the results, if you use a buffered cursor, it will fetch the entire result set from the server. I don't know if you want that with 10M rows.
Here's some references:
https://dev.mysql.com/doc/refman/8.0/en/limit-optimization.html
https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorbuffered.html
Taken from MySQL documentation:
The fetchone() method is used by fetchall() and fetchmany(). It is also used when a cursor is used as an iterator.
The following example shows two equivalent ways to process a query result. The first uses fetchone() in a while loop, the second uses the cursor as an iterator:
# Using a while loop
cursor.execute("SELECT * FROM employees")
row = cursor.fetchone()
while row is not None:
print(row)
row = cursor.fetchone()
# Using the cursor as iterator
cursor.execute("SELECT * FROM employees")
for row in cursor:
print(row)
It also stated that:
You must fetch all rows for the current query before executing new statements using the same connection.
If you are worried about performance issues you should use fetchmany(n) in a while loop until you fetch all of the results like so:
'An iterator that uses fetchmany to keep memory usage down'
while True:
results = cursor.fetchmany(arraysize)
if not results:
break
for result in results:
yield result
This behavior adheres to PEP249, which describes how and which methods db connectors should implement. A partial answer is given in this thread.
Basically the implementation of fetchall vs fetchmany vs fetchone would be up to the developers of the library depending on the database capabilities, but it would make sense, in the case of fetchmany and fetchone, that the unfetched/remaining results would be kept server side, until requested by another call or destruction of cursor object.
So in conclusion I think it is safe to assume calling execute method does not, in this case(mysqldb), dump all the data from the query to memory.
I'm using python in TestComplete to conduct a db query, but the results seem to be empty strings and do not match the data in the table I queried. The file is a s3db file. Does that matter?
Using:
TestComplete Version 14
imported sqlite3 into python file
I've:
-Tried running the same query in SQLite. It returned the expected result
-Verified the connection is established with the correct db
---python
import sqlite3
def getInfo():
conn = sqlite3.connect(db)
c = conn.cursor()
try:
c.execute('SELECT Column_Name FROM Table_Name')
results = c.fetchall()
except:
Log.Error("Query execution failed")
for x in results:
Log.Message(x) `enter code here`
#Log.Message() works like a print statement in testcomplete.
---
Actual Output:
The program runs without errors, but the results come back as 15 lines of blank rows. 15 is the number of records within the table, so I know it's looking in the right place, but it seems like it's not identifying that there's information stored here.
Expected Output:
15 lines of data contained within the Column I specified in the query.
There is no error with sqlite3 and your DB operations. The issue is with Log.Message and what it expects as an argument. Within TestComplete, Log.Message requires variable arguments of type Variant, which can be any of the supported data types within TestComplete; String, Double/Real, Boolean, Date/Time, Object (i.e. TestComplete-recognised UI objects) and Integer.
Log.Message cannot accept arguments of the type returned by cursor.fetchall's rows.
So you'd need to convert each row into a String, e.g.
for x in results:
msg = ('{0} : {1}'.format(x[0], x[1]))
Log.Message(msg)
I am querying an MS Access database from Python using the pyodbc module. I am able to do this if I query all records in a table, but when I add a where clause, I am getting an error.
This is my code:
wpc_ids = ['WPCMOOTEST2', 'WPCMOOTEST1']
conn = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=P:\Conservation Programs\Natural Heritage Program\Data Management\ACCESS databases\POND_entry\POND_be.accdb;')
cursor = conn.cursor()
wpc_list = ','.join(str(x) for x in wpc_ids)
cursor.execute('SELECT * FROM pools WHERE wpc_id IN (%s)'%wpc_list)
I am getting the following error:
Error: ('07002', u'[07002] [Microsoft][ODBC Microsoft Access Driver] Too few parameters. Expected 2. (-3010) (SQLExecDirectW)')
I don't get that error without the where clause, so I'm not sure what the second parameter is that I need. Can anyone help with this?
cursor.execute(
'SELECT * FROM pools WHERE wpc_id IN ({})'.format(
','.join('?'*len(wpc_ids))), wpc_ids
)
Explanation:
There is a PEP about databases, PEP249, you can read it here https://www.python.org/dev/peps/pep-0249/
This PEP defines how API of database modules should be. pyodbc is the database module you're using, and it is compatible with PEP249.
One of the things the PEP defines is that each module should have a paramstyle. pyodbc.paramstyle is qmark so that is why you use '?' with pyodbc. More details https://www.python.org/dev/peps/pep-0249/#paramstyle
Now, instead of building a query as a string and sending it to the database, the idea is to use parameter passing, which is a way to send the query and the parameters separately... It uses the paramstyle to put placeholders in the query, then you pass a sequence of parameters as a second parameter to execute. Example:
sql = 'SELECT * FROM foo WHERE id = ? AND text_col = ?'
params = (12, 'testing')
cursor.execute(sql, params)
Note that this is not mixing the params with the string. The code is passing them as two separate arguments to .execute(). That means it will be the database's job to do the interpolation safely.
Since you want to pass multiple values to the query, you must generate a string containing the number of placeholders separated by comma, same number as the elements in the list:
','.join('?'*len(wpc_ids)))
# will generate ?,?,?,?,? according with length of list
I am trying to run some querys that needs to create some temporary tables and then returns a result set, but i am unable to do that with MySQLdb api.
I already dig something about this issue like here but without success.
My query is like this:
create temporary table tmp1
select * from table1;
alter tmp1 add index(somefield);
create temporary table tmp2
select * from table2;
select * from tmp1 inner join tmp2 using(somefield);
This returns immediatly an empty result set. If i go to the mysql client and do a show full processlist i can see my queries executing. They take some minutes to complete.
Why cursor returns immediatly and don't wait to query to run.
If i try to run another query i have a "Commands out of sync; you can't run this command now"
I already tried to put my connection with autocommit to True
db = MySQLdb.connect(host='ip',
user='root',
passwd='pass',
db='mydb',
use_unicode=True
)
db.autocommit(True)
Or put every statement in is own cursor.execute() and between them db.commit() but without success too.
Can you help me to figure what is the problem? I know mysql don't support transactions for some operations like alter table, but why the api don't wait until everything is finished like it does with a select?
By the way i'm trying to do this on a ipython notebook.
I suspect that you're passing your multi-statement SQL string directly to the cursor.execute function. The thing is, each of the statements is a query in its own right so it's unclear what the result set should contain.
Here's an example to show what I mean. The first case is passing a semicolon set of statements to execute which is what I presume you have currently.
def query_single_sql(cursor):
print 'query_single_sql'
sql = []
sql.append("""CREATE TEMPORARY TABLE tmp1 (id int)""")
sql.append("""INSERT INTO tmp1 VALUES (1)""")
sql.append("""SELECT * from tmp1""")
cursor.execute(';'.join(sql))
print list(cursor.fetchall())
Output:
query_single_sql
[]
You can see that nothing is returned, even though there is clearly data in the table and a SELECT is used.
The second case is where each statement is executed as an independent query, and the results printed for each query.
def query_separate_sql(cursor):
print 'query_separate_sql'
sql = []
sql.append("""CREATE TEMPORARY TABLE tmp3 (id int)""")
sql.append("""INSERT INTO tmp3 VALUES (1)""")
sql.append("""SELECT * from tmp3""")
for query in sql:
cursor.execute(query)
print list(cursor.fetchall())
Output:
query_separate_sql
[]
[]
[(1L,)]
As you can see, we consumed the results of the cursor for each query and the final query has the results we expect.
I suspect that even though you've issued multiple queries, the API only has a handle to the first query executed and so immediately returns when the CREATE TABLE is done. I'd suggest serializing your queries as described in the second example above.
How can I access the number of rows affected by:
cursor.execute("SELECT COUNT(*) from result where server_state='2' AND name LIKE '"+digest+"_"+charset+"_%'")
Try using fetchone:
cursor.execute("SELECT COUNT(*) from result where server_state='2' AND name LIKE '"+digest+"_"+charset+"_%'")
result=cursor.fetchone()
result will hold a tuple with one element, the value of COUNT(*).
So to find the number of rows:
number_of_rows=result[0]
Or, if you'd rather do it in one fell swoop:
cursor.execute("SELECT COUNT(*) from result where server_state='2' AND name LIKE '"+digest+"_"+charset+"_%'")
(number_of_rows,)=cursor.fetchone()
PS. It's also good practice to use parametrized arguments whenever possible, because it can automatically quote arguments for you when needed, and protect against sql injection.
The correct syntax for parametrized arguments depends on your python/database adapter (e.g. mysqldb, psycopg2 or sqlite3). It would look something like
cursor.execute("SELECT COUNT(*) from result where server_state= %s AND name LIKE %s",[2,digest+"_"+charset+"_%"])
(number_of_rows,)=cursor.fetchone()
From PEP 249, which is usually implemented by Python database APIs:
Cursor Objects should respond to the following methods and attributes:
[…]
.rowcount
This read-only attribute specifies the number of rows that the last .execute*() produced (for DQL statements like 'select') or affected (for DML statements like 'update' or 'insert').
But be careful—it goes on to say:
The attribute is -1 in case no .execute*() has been performed on the cursor or the rowcount of the last operation is cannot be determined by the interface. [7]
Note:
Future versions of the DB API specification could redefine the latter case to have the object return None instead of -1.
So if you've executed your statement, and it works, and you're certain your code will always be run against the same version of the same DBMS, this is a reasonable solution.
The number of rows effected is returned from execute:
rows_affected=cursor.execute("SELECT ... ")
of course, as AndiDog already mentioned, you can get the row count by accessing the rowcount property of the cursor at any time to get the count for the last execute:
cursor.execute("SELECT ... ")
rows_affected=cursor.rowcount
From the inline documentation of python MySQLdb:
def execute(self, query, args=None):
"""Execute a query.
query -- string, query to execute on server
args -- optional sequence or mapping, parameters to use with query.
Note: If args is a sequence, then %s must be used as the
parameter placeholder in the query. If a mapping is used,
%(key)s must be used as the placeholder.
Returns long integer rows affected, if any
"""
In my opinion, the simplest way to get the amount of selected rows is the following:
The cursor object returns a list with the results when using the fetch commands (fetchall(), fetchone(), fetchmany()). To get the selected rows just print the length of this list. But it just makes sense for fetchall(). ;-)
print len(cursor.fetchall)
# python3
print(len(cur.fetchall()))
To get the number of selected rows I usually use the following:
cursor.execute(sql)
count = len(cursor.fetchall())
when using count(*) the result is {'count(*)': 9}
-- where 9 represents the number of rows in the table, for the instance.
So, in order to fetch the just the number, this worked in my case, using mysql 8.
cursor.fetchone()['count(*)']