Hello everyone I have the following issue,
I am trying to run a simple UPDATE query using sqlalchemy and psycopg2 for Postgres.
the query is
update = f"""
UPDATE {requests_table_name}
SET status = '{status}', {column_db_table} = '{dt.datetime.now()}'
WHERE request_id = '{request_id}'
"""
and then commit the changes using
cursor.execute(update).commit()
But it throws an error that AttributeError: 'NoneType' object has no attribute 'commit'
My connection string is
engine = create_engine(
f'postgresql://{self.params["user"]}:{self.params["password"]}#{self.params["host"]}:{self.params["port"]}/{self.params["database"]}')
conn = engine.connect().connection
cursor = conn.cursor()
The other thing is that cursor is always closed <cursor object at 0x00000253827D6D60; closed: 0>
The connection with the database is ok, I can etch tables and update them using pandas pd_to_sql method, but with commmiting using cursor it does not work. It works perfect with sql server but not with postgres.
In postgres, however, it creates a PID with the status "idle in transaction" and Client: ClientRead, every time I run cursor.execute(update).commit().
I connot get where is the problem, in the code or in the database.
I tried to use different methods to initiate a cursor, like raw_connection(), but without a result.
I checked for Client: ClientRead with idle in transaction but am not sure how to overcome it.
You have to call commit() on the connection object.
According to the documentation, execute() returns None.
Note that even if you use a context manager like this:
with my_connection.cursor() as cur:
cur.execute('INSERT INTO ..')
You may find your database processes still getting stuck in the idle in transaction state. The COMMIT is handled at the connection level, like #laurenz-albe said, so you need to wrap that too:
with my_connection as conn:
with conn.cursor() as cur:
cur.execute('INSERT INTO ..')
It's spelled out clearly in the documentation, but I still managed to overlook it.
Related
I'm writing a Python script to move data from production db to dev db. I'm using vertica-python (something very similar to pyodbc) for db connection and airflow for scheduling.
The script is divided into two files, one for DAG and one for the actual migration job. I use try-except-finally block for all SQL execution functions in the migration job:
try:
# autocommit set to False
# Execute a SQL script
except DatabaseError:
# Logging information
# Rollback
finally:
# autocommit set to False
You can see that setting autocommit and Rollback needs to access the connection, and executing a SQL script needs to access the cursor. The current solution is to simply create two DB connections in DAG and pass them to the migration script. But I also read from a Stackoverflow post that I should pass only the cursor:
Python, sharing mysql connection in multiple functions - pass connection or cursor?
My question is: Is it possible to only pass the cursor from the DAG to the migration script, and still retain the ability to rollback and setting autocommit?
Yes, you can change the autocommit setting via the Cursor:
>>> import pyodbc
>>> cnxn = pyodbc.connect("DSN=mssqlLocal")
>>> cnxn.autocommit
False
>>> crsr = cnxn.cursor()
>>> crsr.connection.autocommit = True
>>> cnxn.autocommit
True
>>>
pyodbc also provides commit() and rollback() methods on the Cursor object, but be aware that they affect all cursors created by the same connection, i.e., crsr.rollback() is exactly the same as calling cnxn.rollback().
This is a sample code I'd like to run:
for i in range(1,2000):
db = create_engine('mysql://root#localhost/test_database')
conn = db.connect()
#some simple data operations
conn.close()
db.dispose()
Is there a way of running this without getting "Too many connections" errors from MySQL?
I already know I can handle the connection otherwise or have a connection pool. I'd just like to understand how to properly close a connection from sqlalchemy.
Here's how to write that code correctly:
db = create_engine('mysql://root#localhost/test_database')
for i in range(1,2000):
conn = db.connect()
#some simple data operations
conn.close()
db.dispose()
That is, the Engine is a factory for connections as well as a pool of connections, not the connection itself. When you say conn.close(), the connection is returned to the connection pool within the Engine, not actually closed.
If you do want the connection to be actually closed, that is, not pooled, disable pooling via NullPool:
from sqlalchemy.pool import NullPool
db = create_engine('mysql://root#localhost/test_database', poolclass=NullPool)
With the above Engine configuration, each call to conn.close() will close the underlying DBAPI connection.
If OTOH you actually want to connect to different databases on each call, that is, your hardcoded "localhost/test_database" is just an example and you actually have lots of different databases, then the approach using dispose() is fine; it will close out every connection that is not checked out from the pool.
In all of the above cases, the important thing is that the Connection object is closed via close(). If you're using any kind of "connectionless" execution, that is engine.execute() or statement.execute(), the ResultProxy object returned from that execute call should be fully read, or otherwise explicitly closed via close(). A Connection or ResultProxy that's still open will prohibit the NullPool or dispose() approaches from closing every last connection.
Tried to figure out a solution to disconnect from database for an unrelated problem (must disconnect before forking).
You need to invalidate the connection from the connection Pool too.
In your example:
for i in range(1,2000):
db = create_engine('mysql://root#localhost/test_database')
conn = db.connect()
# some simple data operations
# session.close() if needed
conn.invalidate()
db.dispose()
I use this one
engine = create_engine('...')
with engine.connect() as conn:
conn.execute(text(f"CREATE SCHEMA IF NOT EXISTS...")
engine.dispose()
In my case these always works and I am able to close!
So using invalidate() before close() makes the trick. Otherwise close() sucks.
conn = engine.raw_connection()
conn.get_warnings = True
curSql = xx_tmpsql
myresults = cur.execute(curSql, multi=True)
print("Warnings: #####")
print(cur.fetchwarnings())
for curresult in myresults:
print(curresult)
if curresult.with_rows:
print(curresult.column_names)
print(curresult.fetchall())
else:
print("no rows returned")
cur.close()
conn.invalidate()
conn.close()
engine.dispose()
I am doing a lot of inserts to a mysql table in a short period of time from python code (using pymysql) that uses a lot of different threads.
Each thread, of which there are many, may or may not end up pushing data to a MySql table.
Here is the code block that causes the issue (this can be called for each thread running):
sql = ("INSERT INTO LOCATIONS (location_id, place_name) VALUES (%s, %s)")
cursor = self.connection.cursor()
cursor.execute(sql, (location_id, place_name))
cursor.close()
and it is specifically this line:
cursor.execute(sql, (location_id, place_name))
That causes this error:
pymysql.err.InterfaceError: (0, '')
Note also that i define self.connection in the init of the class the above block is in. so all threads share a self.connection object but get their own cursor object.
The error seems to happen randomly and only starts appearing (I think) after doing quite a few inserts into the mysql table. It is NOT consistent meaning it does not happen with every single attempt to insert into mysql.
I have googled this specific error and it seems like it could be from the cursor being closed before running the query. but i believe it is obvious i am closing the cursor after the query is executed.
Right now I think this is happening either because of:
Some sort of write limit to the MySql table, although the error of pymysql.err.InterfaceError doesn't seem to say this specifically
The fact that I have a connection defined at a high scope that is having cursors created from in threads could somehow be causing this problem.
Thoughts?
seems like the issue was related to me having a universal connection object. creating one per thread seems to have removed this issue.
I get the same problem. There is a global connection in my project code, and I find that this connection will be timed out if there is no mysql operation for a long time. This error will occur when execute sql tasks, because of the timed-out connection.
My solution is: reconnecting mysql before execute sql tasks.
sql = ("INSERT INTO LOCATIONS (location_id, place_name) VALUES (%s, %s)")
self.connnection.ping() # reconnecting mysql
with self.connection.cursor() as cursor:
cursor.execute(sql, (location_id, place_name))
I've got the same error, than i found that pymysql is threadsafe, so you need to open an connection for every thread as #sometimesiwritecode said.
Source found: https://github.com/PyMySQL/PyMySQL/issues/422
dont close the connection remove the cursor.close() line should continue update your database
After upgrading to a newer PyMySQL version, I was suddenly getting the same error, but without doing a lot of queries. I was also getting an additional error:
[..]
pymysql.err.InterfaceError: (0, '')
During handling of the above exception, another exception occurred:
pymysql.err.Error: Already closed
Since this appears to be only real place where this error is being discussed, I'll post my solution here too.
Per the PyMySQL documentation, I was doing this:
connection = pymysql.connect([...])
with connection:
with connection.cursor() as cursor:
[..]
cursor.execute(sql)
# lots of other code
with connection:
[...]
What I failed to notice is that with connection will automatically close the connection to the database when that context manager finished executing. So subsequent queries would fail, even though I was still able to get a cursor from the connection without error.
The solution was to not use the with connection context manager, but closing the database connection manually:
connection = pymysql.connect([...])
with connection.cursor() as cursor:
[..]
cursor.execute(sql)
# lots of other code
with connection.cursor() as cursor:
[..]
connection.close()
In a database program what does these lines of code mean and do?
conn=sqlite3.connect(filename)
c=conn.cursor()
conn.commit()
You could think of conn = sqlite3.connect(filename) as creating a connection, or a reference, to that database specified in the filename. So anytime you carry out an action with conn, it will be an action performed on the database specified by filename.
c = conn.cursor() is a cursor object, which allows you to carry out SQL queries on the database. It is created using a call on the conn variable created earlier, and so is a cursor object for that specific database. This is most commonly useful for its .execute() method, which is used to execute SQL commands on the database.
conn.commit() 'commits' the changes to the database; that is, when this command is called, any changes that had been made by the cursor will be saved to the database.
I'm using gevent with gevent-mysql (I also used pymysql to the same effect). It does selects just fine but no matter what I do I can't get it to run an insert. I'm out of ideas.
conn = geventmysql.connect(host='localhost', port=3306, user='root', db='content')
cur = conn.cursor()
cur.execute("insert into placement (placement_name, some_id) values ('static', 1)")
cur.close()
conn.close()
If you are using a transactional storage engine (like InnoDB), you should check the value of the autocommit variable: http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_autocommit
If it is 0, you need to commit your transactions, either using a built in commit() method or an execute("COMMIT") call.
If geventmysql works like the rest of the python DB APIs, you need to call commit in order to commit any changes to the database. Unless geventmysql
In the Python DB API, everything is implicitly a transaction. If you close the connection without committing, it gets rolled back. Do this:
conn.commit()
cur.close()
conn.close()