I have the following pattern in my code:
cursor.execute("INSERT INTO main_catalog (1,'mark')")
conn.commit()
item = Catalog.objects.get(pk=1)
item.do_something()
However, I get an error saying that Catalog.DoesNotExist for id=1, even though I can see it in the database. How do I get the ORM to recognize this new value so I can then query it? (Note that I must use raw sql for certain reasons.)
As pointed out by #boxed, the accepted answer no longer works, however the close_old_connections() function he mentions worked for me. I.e.:
from django import db
db.close_old_connections()
The following pattern works to reset the connection and get a 'fresh cursor':
from django import db
try:
item = Catalog.objects.get(pk=1)
except Catalog.DoesNotExist:
db.close_connection()
item = Catalog.objects.get(pk=1)
This question may also be of use: Django multiprocessing and database connections.
Related
I am trying to select data from an ORACLE 12c database using cx_Oracle, but I am getting the exception: "cx_Oracle.OperationalError: ORA-03113: end-of-file on communication channel".
My query behaves fine using Pycharm (jdbc:oracle:thin driver). Using cx_Oracle in python 3.6, however, the query fails unless I reduce the number of IDs in the IN clause from 500 to about 250. The Cursor.fetchall() function is what throws the exception. I do not have privileged access to the database in order to check things like locks or load, but could these be the cause of the issue? According to our DBA, there is nothing wrong on the Oracle db server, and since the query works fine otherwise, I am inclined to believe it. I have messed with the client-side sqlnet.ora as well, which has allowed exceptions to eventually be thrown instead of hanging forever, but I still cannot fetch the data.
def select(self, query, *args):
cur = self.dbh.cursor()
cur.prepare(query)
try:
cur.execute(None, args)
return cur.fetchall()
# my attempt to handle the issue
except (cx_Oracle.OperationalError, cx_Oracle.DatabaseError) as e:
# cx_Oracle.OperationalError: ORA-03113: end-of-file on communication channel
self.logger.error('Oracle Error: {}'.format(traceback.format_exc()))
raise e
The code calls select like this. For brevity, I've omitted the full string IDs
ids = ['1', '2', '3', ...]
query = """\
select * from my_table where id in(:0,:1,:2,:3,:4, ...)
"""
self.select(query, *ids)
The query fails without the placeholders (with the IDs placed directly in the query) as well.
I expect to be able to run any select query using an IN clause with up to 1000 IDs without receiving the ORA-03113 Exception.
Edit:
I installed oracle-instantclient18.5-basic-18.5.0.0.0-3.x86_64.rpm* on ubuntu 18.04.2, have cx_Oracle version 7.1.2, and I am connecting to Oracle 12.1.0.2.0.
The query is on the underlying tables of BMC Software's ARS. I will start working to try to replicate the problem with a local table structure, but it is a mess and will take some time. If I am able to create a local copy of the tables, I'm not sure I'd be able to replicate the issue, as identical queries with different IDs work fine. That makes it seem data driven, however, after I reduced the query to 250 IDs, I swapped the 250 from the first half to the second half, and got the same success result, so it doesn't seem to be just one bad row.
Is there more helpful logging I can enable on the client side to get more information?
Edit2: I should also add that the issue does not just occur with one query. I've seen the same issue with select queries to completely different tables.
Edit3: I just found out that by commenting out some of the columns that I'm selecting also can make the query work. columns like this:
to_char(to_date('1970-01-01','YYYY-MM-DD') + numtodsinterval(EventStart,'SECOND'),'YYYY-MM-DD HH24:MI:SS')
This may indicate that some kind of timeout is being reached which may or may not be configured in my sqlnet.ora:
DISABLE_OOB=on
SQLNET.RECV_TIMEOUT=60
SQLNET.SEND_TIMEOUT=60
TCP.CONNECT_TIMEOUT=300
SQLNET.OUTBOUND_CONNECT_TIMEOUT=300
ENABLE=BROKEN
TRACE_LEVEL_CLIENT=ADMIN
TRACE_FILE_CLIENT=sqlnet
Edit 4: I've tried some more things.
I installed the same version of instant client, except on a windows 7 machine, and ran the same query against the same db instance. The query succeeded.
I also narrowed down that for this particular query, it will accept 499 IDs, but fails with 500. it doesn't matter which ID I comment out from the query.
I also tried tricking the query into thinking there were fewer IDs by using a sub-select instead:
IN(
select regexp_substr(:0,'[^,]+', 1, level) from dual connect by regexp_substr(:0, '[^,]+', 1, level) is not null
)
I got the error "cx_Oracle.DatabaseError: ORA-01460: unimplemented or unreasonable conversion requested", after which I realized made sense because Oracle will only allow a string to be up to 4000 bytes long.
I think I finally found a way to get everything functioning. I finally came across this link:
https://ardentperf.com/2010/09/08/mysterious-oracle-net-errors/
It turns out that this solved my problem. I am still having trouble getting cx_Oracle to honor a connection string of the same format as the tnsnames.ora file, but I changed my code to refer to the tnsnames.ora for now as follows:
connection_info = {
'user': self.config.get(self.db, 'user'),
'pass': self.config.get(self.db, 'password')
}
connection_string = '{user}/{pass}#TEST'\
.format(**connection_info)
connection = cx_Oracle.connect(connection_string)
where my tnsnames.ora contains the following:
TEST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(Host = myhost.com)(Port = 1521))
)
(SDU=1024)
(CONNECT_DATA =
(SID=mysid)
)
)
The key here is the SDU=1024, which inexplicably fixes this issue.
https://docs.oracle.com/cd/B28359_01/network.111/b28317/sqlnet.htm#NETRF184
Documentation from the above link indicates that the default for SDU is 8192 bytes (8 KB) and my understanding is that there is supposed to be auto-negotiation of this value. This does not appear to be the case, and I don't know what past defaults have been.
I'm trying to enter data from python 3.4 into a MySQL database, with both of these entities being within pythonAnywhere. In other words, I'm writing a python 3.4 program in pythonAnywhere and need to connect to a MySQL db also within pythonAnywhere. I've checked other answers here and haven't quite figured it out. I've tried the ( -u -h -p) syntax mentioned by a few posts, but I'm not sure if that is just for gaining access from outside of pythonAnywhere.
Any help would be appreciated.
++++++++++++++++++++
Actually I figured it out (with kudos to Tony Darnell at the Scripting MySQL website, from whom I plagiarized most of this:
import MySQLdb
db=MySQLdb.connect(
host='Your_User_Name.mysql.pythonanywhere-services.com',
user='Your_User_Name',
passwd='Your_pythonAnywhere_password',
db='Your_User_Name$Your_Data_Base')
everything above starting with 'Your' refers to you personal account info with pythonanywhere
everything else gets listed exactly as shown. Watch that $ that follows your user name as part of the database name (db = etc.)
cursor = db.cursor ()
execute the SQL query using execute() method.
cursor.execute ("Enter any MySQL query here. use the quotes. no semi-colon")
fetch a single row from query using fetchone() method.
row = cursor.fetchone ()
print(row)
fetch all the rest of the query using fetchall() method
data = cursor.fetchall()
print(data)
close the cursor object
cursor.close ()
close the connection
db.close ()
I'm trying to populate a couple databases with psycopg2 within a server I am not the root user of (don't know if it's relevant or not). My code looks like
import json
from psycopg2 import connect
cors = connect(user='jungal01', dbname='course')
req = connect(user="jungal01", dbname='requirement')
core = cors.cursor()
reqs = req.cursor()
with open('gened.json') as gens:
geneds = json.load(gens)
for i in range(len(geneds)):
core.execute('''insert into course (number, description, title)
values({0}, {1}, {2});''' .format(geneds[i]["number"], geneds[i]['description'], geneds[i]['title'] ))
reqs.execute('''insert into requirement (fulfills)
values({0});''' .format(geneds[i]['fulfills'] ))
db.commit()
when I execute the code, I get the above pycopg2 error. I know that these particular databases exist, but I just can't figure out why it won't connect to my databases. (side quest, I am also unsure about that commit statement. Should it be in the for loop, or outside of it? It suppose to be database specific?)
First, you have db is not a defined variable, so you code shouldn't run completely anyway.
\list on this server is a bunch of databases full of usernames, of which my username is one
Then the following is how you should connect. To a database, not a table, and the regular pattern is to put the database name, and then the user/pass.
A "schema" is a loose term in relational database. Both tables and databases have schemas, but you seem to be expecting to connect to a table, not a database.
So, try this code with an attempt at fixing your indentation and SQL injection problem -- See this documentation
Note that you first must have created the two tables in the database you are connecting to.
import json
from psycopg2 import connect
username = 'jungal01'
conn = connect(dbname=username, user=username)
cur = conn.cursor()
with open('gened.json') as gens:
geneds = json.load(gens)
for g in geneds:
cur.execute('''insert into course (number, description, title)
values(%(number)s, %(description)s, %(title)s);''', g)
cur.execute('''insert into requirement (fulfills)
values(%(fulfills)s);''', g)
conn.commit()
Allen, you said: "in postgres, tables are databases." That's wrong. Your error message results from this misunderstanding. You want to connect to a database, and insert into a table that exists in that database. You're trying to insert into a database -- a nonsensical operation.
Make sure you are giving the catalog name as database name and not the schema's under catalog.
Catalog is confusing and quite unnecessary. More details below: What's the difference between a catalog and a schema in a relational database?
Here is the answer for getting raw sql from insert and I am wondering whether I can get raw sql if I use session.add() such as:
session.add(ormclass).compile()
If you want SQLAlchemy to log the actual raw SQL queries you can always set echo=True on the create_engine method. I know this seems rudimentary but it's the easiest way to see executed queries.
engine = create_engine("postgresql://localhost/dbname", echo=True)
My temporal solution to this is that when an error happens, there will be an exception and in this exception there are the parameters and statement.
except Exception as inst:
print(inst.statement % inst.params)
But the problem still exists because I cannot get the statement and parameters if there are no exceptions. In addition, there are no quotation marks for strings in the print so the string cannot be executed in mysql directly.
The way that I display the raw sql (which works most [but not all] of the time):
query = [my query that I created]
from sqlalchemy.dialects import mysql
str_query = query.compile(dialect=mysql.dialect(), compile_kwargs={"literal_binds": True})
Then I can just print the sql_query statement and it will show me the query with arguments. Some queries won't display, such as bulk inserts.
I'm trying to run a simple insert query to a database. I have it configured correctly, and it should work, but it doesn't. For some reason I get the following error on this query:
Query:
INSERT INTO searches (query) VALUES ('test')
Error:
(1062, "Duplicate entry 'test' for key 'query'")
The query runs without problems in the MySQL console so it must be a problem with Python? Here's my Python code:
def increase_search_count(search_query):
from django.db import connection, transaction
search_query = search_query.strip()
cursor = connection.cursor()
rows = cursor.execute("INSERT INTO searches (query) VALUES ('test')")
I know there are much better ways to handle databases, but I'm new to Python, and I have a deadline. I'd just like to get this to work, I have another SELECT query in another function and that one runs without any problems!
Any ideas what might be wrong?
The way that query is constructed means you will always be inserting 'test' into the database, and seeing the query is likely the primary key in your table, it will be creating duplicate rows.
The query should be something like "INSERT INTO searches (query) VALUES ('" variable "')" so you don't insert the same value over and over.