How does rowcount work. I am using pyodbc and it's always returning -1.
return_query = conn.query_db_param(query, q_params)
print(return_query.rowcount)
def query_db_param(self, query, params):
self.cursor.execute(query,params)
print(self.cursor.rowcount)
rowcount refers to the number of rows affected by the last operation. So, if you do an insert and insert only one row, then it will return 1. If you update 200 rows, then it will return 200. On the other hand, if you SELECT, the last operation doesn't really affect rows, it is a result set. In that case, 0 would be syntactically incorrect, so the interface returns -1 instead.
It will also return -1 for operations where you do things like set variables or use create/alter commands.
You are connecting to a database that can't give you that number for your query. Many database engines produce rows as you fetch results, scanning their internal table and index data structures for the next matching result as you do so. The engine can't know the final count until you fetched all rows.
When the rowcount is not known, the Python DB-API 2.0 specification for Cursor.rowcount states the number must be set to -1 in that case:
The attribute is -1 in case [...] the rowcount of the last operation is cannot be determined by the interface.
The pyodbc Cursor.rowcount documentation conforms to this requirement:
The number of rows modified by the last SQL statement.
This is -1 if no SQL has been executed or if the number of rows is unknown. Note that it is not uncommon for databases to report -1 immediately after a SQL select statement for performance reasons. (The exact number may not be known before the first records are returned to the application.)
pyodbc is not alone in this, another easy-to-link-to example is the Python standard library sqlite3 module; it's Cursor.rowcount documentation states:
As required by the Python DB API Spec, the rowcount attribute “is -1 in case no executeXX() has been performed on the cursor or the rowcount of the last operation is not determinable by the interface”. This includes SELECT statements because we cannot determine the number of rows a query produced until all rows were fetched.
Note that for subset of database implementations, the rowcount value can be updated after fetching some of the rows. You'll have to check your specific database documentation you are connecting to to see if that implementations can do this, or if the rowcount must remain at -1. You could always experiment, of course.
You could execute a COUNT() select first, or, if the result set is not expected to be too large, use cursor.fetchall() and use len() on the resulting list.
If you are using microsoft sql server, and you want to get the number of rows returned in the prior select statement, you can just execute select ##rowcount.
E.g.:
cursor.execute("select ##rowcount")
rowcount = cursor.fetchall()[0][0]
Related
I have a table, and I want to execute a query that will return the values of two rows:
cursor.execute("""SELECT `egg_id`
FROM `groups`
WHERE `id` = %s;""", (req_id))
req_egg = str(cursor.fetchone())
print req_egg
The column egg_id has two rows it can return for that query, however the above code will only print the first result -- I want it to also show the second, how would I get both values?
Edit: Would there be any way to store each one in a separate variable, with fetchmany?
in this case you can use fetchmany to fetch a specified number of rows:
req_egg = cursor.fetchmany(2)
edit:
but be aware: if you have a table with many rows but only need two, then you should also use a LIMIT in your sql query, otherwise all rows are returned from the database, but only two are used by your program.
Call .fetchone() a second time, and it would return the next result.
Otherwise if you are 100% positively sure that your query would always return exactly two results, even if you've had a bug or inconsistent data in the database, then just do a .fetchall() and capture both results.
Try this:
Cursor.fetchmany(size=2)
Documentation for sqlite3 (which also implements dbapi): http://docs.python.org/library/sqlite3.html#sqlite3.Cursor.fetchmany
I have the following code
cur = db.cursor(cursors.SSDictCursor)
cur.execute("SELECT * FROM large_table")
result_count = cur.rowcount
print result_count
This prints the number 18446744073709551615 which is obviously wrong. If I remove the cursors.SSDictCursor the correct number is shown. Can anyone tell me how I can get the number of records returned while keeping the SSDictCursor?
To get the number of records returned by SSDictCursor or SSCursor, your only options are:
Fetch the entire result and count it using len(), which defeats the purpose of using SSDictCursor or SSCursor in the first place;
Count the rows yourself as you iterate through them, which means you won't know the count until hit the end (not likely to be practical); or,
Run an additional, separate COUNT(*) query.
I highly recommend the third option. It's extremely fast if all you're doing is SELECT COUNT(*) FROM table;. It would be slower for some more complex query, but with proper indexing it should still be quick enough for most purposes.
As an aside, the return value you're seeing is sort of correct; at least, as far as the MySQL C API is concerned.
Per the Python DB API defined in PEP 249, the rowcount attribute is -1 if the rowcount of the last operation cannot be determined by the interface. #glglgl explained why the rowcount can't be determined in their answer:
Internally, SSDictCursor uses mysql_use_result() which allows the server to start transferring the data before the acquiring is complete.
In other words, the server doesn't know how many rows it's ultimately going to fetch. When you execute a query, MySQLdb stores the return value of mysql_affected_rows() in the cursor's rowcount attribute. Because the count is indeterminate, this function returns -1 as an unsigned long long integer (my_ulonglong), a numeric type that's available in the ctypes module of the standard library:
>>> from ctypes import c_ulonglong
>>> n = c_ulonglong(-1)
>>> n.value
18446744073709551615L
A quick-and-dirty alternative to ctypes, when you know you'll always be dealing with a 64-bit unsigned integer, is:
>>> -1 & 0xFFFFFFFFFFFFFFFF
18446744073709551615L
It would be great if MySQLdb checked for this return value and gave you the signed integer you expect to see, but unfortunately it doesn't.
With a SSDictCursor, this value can only be read resp. determined when the cursor is used up.
Internally, SSDictCursor uses mysql_use_result() which allows the server to start transferring the data before the acquiring is complete.
I want to count the total number of rows returned by query. I am able to retrieved rows returned by query but what if i need to work in case when no data exits. that is when query returns no va from database.
the code i used to solve this problem is :
try:
cur.execute(query)
id = cur.fetchone()[0]
if(id is None):
return '-1'
else:
return id
But this doenst help when no values is returned selected from query.(when condition doesnt meet criteria defined in select statement)
cur.fetchall() will give you a sequence of all the rows. You can look at the length of that sequence to see if any rows were returned. This works for small result sets, but may not be ideal for queries that return large amounts of data.
Alternatively, you can look at cur.rowcount. Rowcount will return the number of rows in the query, or -1 if the number cannot be determined. It is up to the implementation to set rowcount; several popular python database modules (most notably sqlite3), do not set rowcount for all queries. For modules that do not set rowcount, the only way to count the number of result rows is to load the full result set into memory.
How can I access the number of rows affected by:
cursor.execute("SELECT COUNT(*) from result where server_state='2' AND name LIKE '"+digest+"_"+charset+"_%'")
Try using fetchone:
cursor.execute("SELECT COUNT(*) from result where server_state='2' AND name LIKE '"+digest+"_"+charset+"_%'")
result=cursor.fetchone()
result will hold a tuple with one element, the value of COUNT(*).
So to find the number of rows:
number_of_rows=result[0]
Or, if you'd rather do it in one fell swoop:
cursor.execute("SELECT COUNT(*) from result where server_state='2' AND name LIKE '"+digest+"_"+charset+"_%'")
(number_of_rows,)=cursor.fetchone()
PS. It's also good practice to use parametrized arguments whenever possible, because it can automatically quote arguments for you when needed, and protect against sql injection.
The correct syntax for parametrized arguments depends on your python/database adapter (e.g. mysqldb, psycopg2 or sqlite3). It would look something like
cursor.execute("SELECT COUNT(*) from result where server_state= %s AND name LIKE %s",[2,digest+"_"+charset+"_%"])
(number_of_rows,)=cursor.fetchone()
From PEP 249, which is usually implemented by Python database APIs:
Cursor Objects should respond to the following methods and attributes:
[…]
.rowcount
This read-only attribute specifies the number of rows that the last .execute*() produced (for DQL statements like 'select') or affected (for DML statements like 'update' or 'insert').
But be careful—it goes on to say:
The attribute is -1 in case no .execute*() has been performed on the cursor or the rowcount of the last operation is cannot be determined by the interface. [7]
Note:
Future versions of the DB API specification could redefine the latter case to have the object return None instead of -1.
So if you've executed your statement, and it works, and you're certain your code will always be run against the same version of the same DBMS, this is a reasonable solution.
The number of rows effected is returned from execute:
rows_affected=cursor.execute("SELECT ... ")
of course, as AndiDog already mentioned, you can get the row count by accessing the rowcount property of the cursor at any time to get the count for the last execute:
cursor.execute("SELECT ... ")
rows_affected=cursor.rowcount
From the inline documentation of python MySQLdb:
def execute(self, query, args=None):
"""Execute a query.
query -- string, query to execute on server
args -- optional sequence or mapping, parameters to use with query.
Note: If args is a sequence, then %s must be used as the
parameter placeholder in the query. If a mapping is used,
%(key)s must be used as the placeholder.
Returns long integer rows affected, if any
"""
In my opinion, the simplest way to get the amount of selected rows is the following:
The cursor object returns a list with the results when using the fetch commands (fetchall(), fetchone(), fetchmany()). To get the selected rows just print the length of this list. But it just makes sense for fetchall(). ;-)
print len(cursor.fetchall)
# python3
print(len(cur.fetchall()))
To get the number of selected rows I usually use the following:
cursor.execute(sql)
count = len(cursor.fetchall())
when using count(*) the result is {'count(*)': 9}
-- where 9 represents the number of rows in the table, for the instance.
So, in order to fetch the just the number, this worked in my case, using mysql 8.
cursor.fetchone()['count(*)']
I have one question concerning Python and the sqlalchemy module. What is the equivalent for cursor.rowcount in the sqlalchemy Python?
ResultProxy objects have a rowcount property as well.
Actually there is no way to know this precisely for postgres.
The closes thing is rowcount. But
rowcount is not the number of affected rows. Its the number of matched rows. See what doc says
This attribute returns the number of rows matched, which is not necessarily the same as the number of rows that were actually modified - an UPDATE statement, for example, may have no net change on a given row if the SET values given are the same as those present in the row already. Such a row would be matched but not modified. On backends that feature both styles, such as MySQL, rowcount is configured by default to return the match count in all cases
So for both of the following scenarios rowcount will report 1. Because of Rows matched: 1
one row changed with update statement.
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
same update statement is executed.
Query OK, 0 row affected (0.00 sec)
Rows matched: 1 Changed: 0 Warnings: 0
What Shiplu says is correct of course, however, to answer the question, in many cases you can easily make the matched columns equal the changed columns by including the condition that the value is different in the WHERE clause, i.e.:
UPDATE table
SET column_to_be_changed = "something different"
WHERE column_to_be_changed != "something different" AND [...your other conditions...]
rowcount will then return the number of affected rows, because it equals the number of matched rows.
Shiplu's analysis is 100% correct.
Pushing the discussion a bit further, here is how to display the updated rowcount and not the matched rowcount using sqlalchemy for MySQL: the engine needs to be created with the flag client_flag=0.
from sqlalchemy import create_engine
engine = create_engine(
'mysql+pymysql://user:password#host:port/db',
connect_args={'client_flag':0}
)
To give a bit more details, the rowcount returned by MySQL depends on the CLIENT_FOUND_ROWS flag provided to the C-API function mysql_real_connect() as stated in MySQL documentation:
For UPDATE statements, the affected-rows value by default is the number of rows actually changed. If you specify the CLIENT_FOUND_ROWS flag to mysql_real_connect() when connecting to mysqld, the affected-rows value is the number of rows “found”; that is, matched by the WHERE clause.
The flag value is 2 (MySQL constants), and is added automatically by sqlalchemy when creating the engine as visible here.
The value of client_flag in connect_args allows to override this value.
Note: this might break sth from the sane_rowcount (only used in the ORM apparently) - in my case, I only use the Core of sqlalchemy. In last link:
# FOUND_ROWS must be set in CLIENT_FLAGS to enable
# supports_sane_rowcount.
You can use .returning to give you the rows which have been updated, and then use result.rowcount to count this
eg
insertstmt = insert(mytable).values(myvalues).returning(mytable.c.mytableid)
with get_engine().begin() as conn:
result = conn.execute(insertstmt)
print(result.rowcount)