This question already has answers here:
sqlalchemy print results instead of objects
(2 answers)
Closed 5 years ago.
I am working on writing a simple test to validate that the number of distinct values in an "id" column matches the number of rows for each table. I am expecting to be able to access particular values of an object, but when I run code and try to print the value of my variable, I can see that my object is a sqlalchemy.engine.result.ResultProxy object at..., as opposed to being something human readable. I have been on the SQLAlchemy website for over an hour, and have googled several permutations of my question, but have not found what I am looking for.
My code, with terminal output, is below:
from sqlalchemy import create_engine
engine = create_engine('postgresql://kyle.pekosh#localhost:5432/testload')
connection = engine.connect()
id_count = connection.execute('SELECT COUNT(DISTINCT(id)) FROM csv.agencies')
id_count
<sqlalchemy.engine.result.ResultProxy object at 0x10357d290>
This is SQLAlchemy's expected behavior. You need to interact with the ResultProxy. As per SQLAlchemy's documentation:
The returned result is an instance of ResultProxy, which references a DBAPI cursor and provides a largely compatible interface with that of the DBAPI cursor. The DBAPI cursor will be closed by the ResultProxy when all of its result rows (if any) are exhausted. A ResultProxy that returns no rows, such as that of an UPDATE statement (without any returned rows), releases cursor resources immediately upon construction.
The ResultProxy API allows you to fetch the data:
results = connection.execute('SELECT COUNT(DISTINCT(id)) FROM csv.agencies')
id_count = results.first()[0]
Related
This question already has answers here:
How to use variables in SQL statement in Python?
(5 answers)
Closed 2 months ago.
def update_inv_quant():
new_quant = int(input("Enter the updated quantity in stock: "))
Hello! I'm wondering how to insert a user variable into an sql statement so that a record is updated to said variable. Also, it'd be really helpful if you could also help me figure out how to print records of the database into the actual python console. Thank you!
I tried doing soemthing like ("INSERT INTO Inv(ItemName) Value {user_iname)") but i'm not surprised it didnt work
It would have been more helpful if you specified an actual database.
First method (Bad)
The usual way (which is highly discouraged as Graybeard said in the comments) is using python's f-string. You can google what it is and how to use it more in-depth.
but basically, say you have two variables user_id = 1 and user_name = 'fish', f-string turns something like f"INSERT INTO mytable(id, name) values({user_id},'{user_name}')" into the string INSERT INTO mytable(id,name) values(1,'fish').
As we mentioned before, this causes something called SQL injection. There are many good youtube videos that demonstrate what that is and why it's dangerous.
Second method
The second method is dependent on what database you are using. For example, in Psycopg2 (Driver for PostgreSQL database), the cursor.execute method uses the following syntax to pass variables cur.execute('SELECT id FROM users WHERE cookie_id = %s',(cookieid,)), notice that the variables are passed in a tuple as a second argument.
All databases use similar methods, with minor differences. For example, I believe SQLite3 uses ? instead of psycopg2's %s. That's why I said that specifying the actual database would have been more helpful.
Fetching records
I am most familiar with PostgreSQL and psycopg2, so you will have to read the docs of your database of choice.
To fetch records, you send the query with cursor.execute() like we said before, and then call cursor.fetchone() which returns a single row, or cursor.fetchall() which returns all rows in an iterable that you can directly print.
Execute didn't update the database?
Statements executing from drivers are transactional, which is a whole topic by itself that I am sure will find people on the internet who can explain it better than I can. To keep things short, for the statement to physically change the database, you call connection.commit() after cursor.execute()
So finally to answer both of your questions, read the documentation of the database's driver and look for the execute method.
This is what I do (which is for sqlite3 and would be similar for other SQL type databases):
Assuming that you have connected to the database and the table exists (otherwise you need to create the table). For the purpose of the example, i have used a table called trades.
new_quant = 1000
# insert one record (row)
command = f"""INSERT INTO trades VALUES (
'some_ticker', {new_quant}, other_values, ...
) """
cur.execute(command)
con.commit()
print('trade inserted !!')
You can then wrap the above into your function accordingly.
This question already has answers here:
sqlalchemy print results instead of objects
(2 answers)
Closed 5 years ago.
I am working on writing a simple test to validate that the number of distinct values in an "id" column matches the number of rows for each table. I am expecting to be able to access particular values of an object, but when I run code and try to print the value of my variable, I can see that my object is a sqlalchemy.engine.result.ResultProxy object at..., as opposed to being something human readable. I have been on the SQLAlchemy website for over an hour, and have googled several permutations of my question, but have not found what I am looking for.
My code, with terminal output, is below:
from sqlalchemy import create_engine
engine = create_engine('postgresql://kyle.pekosh#localhost:5432/testload')
connection = engine.connect()
id_count = connection.execute('SELECT COUNT(DISTINCT(id)) FROM csv.agencies')
id_count
<sqlalchemy.engine.result.ResultProxy object at 0x10357d290>
This is SQLAlchemy's expected behavior. You need to interact with the ResultProxy. As per SQLAlchemy's documentation:
The returned result is an instance of ResultProxy, which references a DBAPI cursor and provides a largely compatible interface with that of the DBAPI cursor. The DBAPI cursor will be closed by the ResultProxy when all of its result rows (if any) are exhausted. A ResultProxy that returns no rows, such as that of an UPDATE statement (without any returned rows), releases cursor resources immediately upon construction.
The ResultProxy API allows you to fetch the data:
results = connection.execute('SELECT COUNT(DISTINCT(id)) FROM csv.agencies')
id_count = results.first()[0]
This question already has answers here:
Python: use mysqldb to import a MySQL table as a dictionary?
(5 answers)
Closed 9 years ago.
After executing a query statement on a MySQL database connection, I perform:
rows = cursor.fetchall()
This gives an array of arrays. I'd like to have an array of dictionaries, where each dictionary takes its keys from the requested column names of my table and associates the values from the table.
How do I do this?
Well, you forgot to mention which mysql library you're using.
If using oursql (which I recommend, it is certainly the best one), use oursql's DictCursor. Example:
conn = oursql.connect(...)
curs = conn.cursor(oursql.DictCursor)
If using MySQLdb (why?) Use MySQLdb's DictCursor. Example:
conn = MySQLdb.connect(..., cursorclass=MySQLdb.cursors.DictCursor)
curs = conn.cursor()
Doing that will give you a cursor that returns dicts for each row. Remember to not have duplicate rownames in your query.
This question already has answers here:
Python MySQLdb string substitution without added quotations
(2 answers)
Closed 9 years ago.
In Python I'm passing in two variables to MySQL, and the second variable is referenced as {0} and works correctly.
How can I make the first parameter do the same thing ? If '{0}' is the second variable. What is the First ? dbname is incorrect, I need to show dbname's value in this string ?
def checkTableExists(dbname,tablename):
sql2 = """SELECT COUNT(*) FROM information_schema.tables
WHERE table_schema = " dbname "
AND table_name = '{0}' """.format(tablename.replace('\'', '\'\''))
A better way to organize your function to create the query might be:
def checkTableExists(dbname, tablename):
query_args = {"table_name":tablename, "database_name":dbname}
exist_query = """
SELECT EXISTS (
SELECT *
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME = '%(table_name)s'
AND TABLE_SCHEMA = '%(database_name)s') AS `Exist`
"""%(query_args)
# Do stuff with the `Exist` column of whatever your db connection returns.
# Should be 1 if True. Might be good to just return bool(...).
This is what EXISTS is for, so you don't need to do a "hack" and check yourself if the COUNT within INFORMATION_SCHEMA.TABLES is greater than 0.
As #jgranger commented on the OP, if you are using MySQLdb, you can follow that link and let it do the argument substitution. But there are many other flavors of interaction with pyodbc, so string-formatted queries will never go away (I wish they would).
Even Better
Write a stored procedure in SQL that checks if tables exist as a function of the database name and schema name, and then do nothing but pass arguments to this stored procedure. Then any component of the system (Python or otherwise) can expect that stored procedure's interface for existence checking.
If the existence-check logic needs to change (say in your application it suddenly matters that a table has more than 0 rows rather than merely having been created with a CREATE TABLE command, or vice versa), then you can change the internals of the stored procedure without needing to go to every downstream location that had embedded string SQL queries and change the logic. It helps add modularity and encapsulation to the system's interaction with the databases.
I need to insert/update bulk rows via SQLAlchemy. And get inserted rows.
I tried to do it with session.execute:
>>> posts = db.session.execute(Post.__table__.insert(), [{'title': 'dfghdfg', 'content': 'sdfgsdf', 'topic': topic}]*2)
>>> posts.fetchall()
ResourceClosedError Traceback (most recent call last)
And with engine:
In [17]: conn = db.engine.connect()
In [18]: result = conn.execute(Post.__table__.insert(), [{'title': 'title', 'content': 'content', 'topic': topic}]*2)
In [19]: print result.fetchall()
ResourceClosedError: This result object does not return rows. It has been closed automatically.
The same response is an object has been closed automatically. How to prevent it?
First answer - on "preventing automatic closing".
SQLAlchemy runs DBAPI execute() or executemany() with insert and do not do any select queries.
So the exception you've got is expected behavior. ResultProxy object returned after insert query executed wraps DB-API cursor that doesn't allow to do .fetchall() on it. Once .fetchall() fails, ResultProxy returns user the exception your saw.
The only information you can get after insert/update/delete operation would be number of affected rows or the value of primary key after auto increment (depending on database and database driver).
If your goal is to receive this kind information, consider checking ResultProxy methods and attributes like:
.inserted_primary_key
.last_inserted_params()
.lastrowid
etc
Second answer - on "how to do bulk insert/update and get resulting rows".
There is no way to load inserted rows while doing single insert query using DBAPI. SQLAlchemy SQL Expression API you are using for doing bulk insert/updates also doesn't provide such functionality.
SQLAlchemy runs DBAPI executemany() call and relies on driver implementation. See this section of documentation for details.
Solution would be to design your table in a way that every record would have natural key to identify records (combination of columns' values that identify record in unique way). So insert/update/select queries would be able to target one record.
After doing it would be possible to do bulk insert/update first and then doing select query by natual key. Thus you won't need to know autoincremented primary key value.
Another option: may be you can use SQLAlchemy Object Relational API for creating objects - then SQLAlchemy may try to optimize insert into doing one query with executemany for you. It worked for me while using Oracle DB.
There won't be any optimization for updates out of the box. Check this SO question for efficient bulk update ideas