I'm trying to figure out why I can't access a particular table in a PostgreSQL database using psycopg2. I am running PostgreSQL 11.5
If I do this, I can connect to the database in question and read all the tables in it:
import psycopg2
try:
connection = psycopg2.connect(user = "postgres", #psycopg2.connect() creates connection to PostgreSQL database instance
password = "battlebot",
host = "127.0.0.1",
port = "5432",
database = "BRE_2019")
cursor = connection.cursor() #creates a cursor object which allows us to execute PostgreSQL commands through python source
#Print PostgreSQL version
cursor.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""")
for table in cursor.fetchall():
print(table)
The results look like this :
('geography_columns',)
('geometry_columns',)
('spatial_ref_sys',)
('raster_columns',)
('raster_overviews',)
('nc_avery_parcels_poly',)
('Zone5e',)
('AllResidential2019',)
#....etc....
The table I am interested in is the last one, 'AllResidential2019'
So I try to connect to it and print the contents by doing the following:
try:
connection = psycopg2.connect(user = "postgres",
#psycopg2.connect() creates connection to PostgreSQL database instance
password = "battlebot",
host = "127.0.0.1",
port = "5432",
database = "BRE_2019")
cursor = connection.cursor() #creates a cursor object which allows us to execute PostgreSQL commands through python source
cursor.execute("SELECT * FROM AllResidential2019;") #Executes a database operation or query. Execute method takes SQL query as a parameter. Returns list of result
record = cursor.fetchall()
print(record)
except (Exception, psycopg2.Error) as error:
print("Error while connecting to PostgreSQL: ", error)
And I get the following error:
Error while connecting to PostgreSQL: relation "allresidential2019" does not exist
LINE 1: SELECT * FROM AllResidential2019;
However, I can successfully connect and get results when attempting to connect to another table in another database I have (this works! and the results are the data in this table):
try:
connection = psycopg2.connect(user = "postgres", #psycopg2.connect() creates connection to PostgreSQL database instance
password = "battlebot",
host = "127.0.0.1",
port = "5432",
database = "ClimbingWeatherApp") . #different database name
cursor = connection.cursor()
cursor.execute("SELECT * FROM climbing_area_info ;")
record = cursor.fetchall()
print(record)
except (Exception, psycopg2.Error) as error:
print("Error while connecting to PostgreSQL: ", error)
I can't figure out why I can retrieve information from one table but not another, using exactly the same code (except names are changes). And I am also not sure how to troubleshoot this. Can anyone offer suggestions?
Your table name is case-sensitive and you have to close it in double quotes:
SELECT * FROM "AllResidential2019";
In Python program it may look like this:
cursor.execute('SELECT * FROM "AllResidential2019"')
or you can use the specialized module SQL string composition:
from psycopg2 import sql
# ...
cursor.execute(sql.SQL("SELECT * FROM {}").format(sql.Identifier('AllResidential2019')))
Note that case-sensitive Postgres identifiers (i.e. names of a table, column, view, function, etc) unnecessarily complicate simple matters. I would advise you not to use them.
Likely, the reason for your issue is Postgres' quoting rules which adheres to the ANSI SQL standard regarding double quoting identifiers. In your table creation, you likely quoted the table:
CREATE TABLE "AllResidential2019" (
...
)
Due to case sensitivity of at least one capital letter, this requires you to always quote the table when referencing the table. Do remember: single and double quotes have different meanings in SQL as opposed to being mostly interchangeable in Python.
SELECT * FROM "AllResidential2019"
DELETE FROM "AllResidential2019" ...
ALTER TABLE "AllResidential2019" ...
It is often recommended, if your table, column, or other identifier does not contain special characters, spaces, or reserved words, to always use lower case or no quotes:
CREATE TABLE "allresidential2019" (
...
)
CREATE TABLE AllResidential2019 (
...
)
Doing so, any combination of capital letters will work
SELECT * FROM ALLRESIDENTIAL2019
SELECT * FROM aLlrEsIdEnTiAl2019
SELECT * FROM "allresidential2019"
See further readings on the subject:
Omitting the double quote to do query on PostgreSQL
PostgreSQL naming conventions
Postgres Docs - 4.1.1. Identifiers and Key Words
Don’t use double quotes in PostgreSQL
What is the difference between single and double quotes in SQL?
I was facing the same error in Ubuntu. But in my case, I accidentally added the tables to the wrong database, which was in turn owned by the root postgres user instead of the new postgres user that I had created for my flask app.
I'm using a SQL file to create and populate the tables. This is the command that I used to be able to create these tables using a .sql file. This allows you to specify the owner of the tables as well as the database in which they should be created:
sudo -u postgres psql -U my_user -d my_database -f file.sql -h localhost
You will then be prompted for my_users's password.
sudo -u postgres is only necessary if you are running this from a terminal as a the root user. It basically runs the psql ... command as the postgres user.
Related
i've been trying to get some data from my db by using below code, but the code is not working. is there any mistake that i made in the code, if so how can i fix it.
NOTE: i took the below code from just a script not a django or flesk web app.
def db():
conn = psycopg2.connect(
"dbname=mydb user=postgres password=****** host=*.*.*.*")
cur = conn.cursor()
cur.execute("""SELECT * FROM MddPublisher""")
query_results = cur.fetchall()
print(query_results)
db()
ERROR: psycopg2.errors.UndefinedTable: relation "mddpublisher" does not exist LINE 1: SELECT * FROM MddPublisher
additionally,i want to show below code to prove that connection is ok. the problem is that i can't receive data from my db whenever i try to execute select command through python.
def print_tables():
conn = psycopg2.connect(
"dbname=mydb user=postgres password=***** host=*.*.*.*.*")
cur = conn.cursor()
cur.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""")
for table in cur.fetchall():
print(table)
print_tables()
OUTPUT:
('MddPublisher',)
This is probably an issue with case sensitivity. Postgresql names are usually normalized to lower case. However, when used inside double quotes, they keep their case. So, to access a table named MddPublisher you must write it like "MddPublisher".
All the gory details are in Section 4.1.1, Identifiers and Key Words in the Postgresql 14 docs.
I have a column called REQUIREDCOLUMNS in a SQL database which contains the columns which I need to select in my Python script below.
Excerpt of Current Code:
db = mongo_client.get_database(asqldb_row.SCHEMA_NAME)
coll = db.get_collection(asqldb_row.TABLE_NAME)
table = list(coll.find())
root = json_normalize(table)
The REQUIREDCOLUMNSin SQL contains values reportId, siteId, price, location
So instead of explicitly typing:
print(root[["reportId","siteId","price","location"]])
Is there a way to do print(root[REQUIREDCOLUMNS])?
Note: (I'm already connected to the SQL database in my python script)
You will have to use cursors if you are using mysql or pymysql , both the syntax are almost similar below i will mention for mysql
import mysql
import mysql.connector
db = mysql.connector.connect(
host = "localhost",
user = "root",
passwd = " ",
database = " "
)
cursor = db.cursor()
sql="select REQUIREDCOLUMNS from table_name"
cursor.execute(sql)
required_cols = cursor.fetchall()#this wll give ["reportId","siteId","price","location"]
cols_as_string=','.join(required_cols)
new_sql='select '+cols_as_string+' from table_name'
cursor.execute(new_sql)
result=cursor.fetchall()
This should probably work, i intentionally split many lines into several lines for understanding.
syntax could be slightly different for pymysql
I am using cx_Oracle module to connect to oracle database. In the script i use two variables schema_name and table_name. The below query works fine
cur1.execute("select owner,table_name from dba_tables where owner ='schema_name'")
But i need to query the num of rows of a table, where i need to qualify the table_name with the schema_name and so the query should be
SELECT count(*) FROM "schema_name"."table_name"
This does not work when using in the code, i have tried to put it in triple quotes, single quotes and other options but it does not format the query as expected and hence errors out with table does not exist.
Any guidance is appreciated.
A prepared statement containing placeholders with variables of the form ...{}.{}".format(sc,tb) might be used
sc='myschema'
tb='mytable'
cur1.execute("SELECT COUNT(*) FROM {}.{}".format(sc,tb))
print(cur1.fetchone()[0])
In this particular case, you could also try setting Connection.current_schema, see the cx_Oracle API doc
For example, if you create table in your own schema:
SQL> show user
USER is "CJ"
SQL> create table ffff (mycol number);
Table created.
SQL> insert into ffff values (1);
1 row created.
SQL> commit;
Commit complete.
Then run Python code that connects as a different user:
import cx_Oracle
import os
import sys, os
if sys.platform.startswith("darwin"):
cx_Oracle.init_oracle_client(lib_dir=os.environ.get("HOME")+"/Downloads/instantclient_19_8")
username = "system"
password = "oracle"
connect_string = "localhost/orclpdb1"
connection = cx_Oracle.connect(username, password, connect_string)
connection.current_schema = 'CJ';
with connection.cursor() as cursor:
sql = """select * from ffff"""
for r in cursor.execute(sql):
print(r)
sql = """select sys_context('USERENV','CURRENT_USER') from dual"""
for r in cursor.execute(sql):
print(r)
the output will be:
(1,)
('SYSTEM',)
The last query shows that it is not the user that is being changed, but just the first query is automatically changed from 'ffff' to 'CJ.ffff'.
I can connect to my database, but psycopg2 fails to find any of my tables. The following will error trying to get my users:
import psycopg2
try:
conn = psycopg2.connect("dbname='pdb' user='postgres' host='localhost' password='password'")
except:
print 'failed to connect'
cur = conn.cursor()
cur.execute(""" SELECT * from Users """)
rows = cur.fetchall()
for row in rows:
print row[0]
#Error:
psycopg2.ProgrammingError: relation "users" does not exist
LINE 1: SELECT * from Users
# This also fails
cur.execute("""SELECT * from pdb.Users """)
If I do:
cur.execute(""" SELECT * from pg_database """)
# Outputs
template1
template0
postgres
pdb
In my admin panel, pdb shows a bunch of tables, one of them being Users, so I'm not sure why psycopg2 can't find it.
Here's a printout from psql for pdb:
List of relations
Schema | Name | Type | Owner
--------+--------------------+-------+----------
public | Companies | table | postgres
public | Users | table | postgres
(2 rows)
Your table names Users and Companies both start with a capitalized letter. PostgreSQL will convert all identifiers to lower case (by default), as you can see from the error message:
psycopg2.ProgrammingError: relation "users" does not exist
Where users is written in all lower case. This is needed if you wish to strictly follow the SQL standard (as PostgreSQL is renowned for). You can solve this in two ways:
Solve it in your database:
Adhere to a common convention and rename your tables to be all lowercase.
Solve it in your code:
Quote your identifiers (your table name in this case) so PostgreSQL will leave them untouched:
cur.execute(""" SELECT * from "Users" """)
This issue also usually happens when you are connected to wrong database. In my case i was connected to wrong database
Below is my code to connect using psycopg2 library
import psycopg2
try:
src_conn = psycopg2.connect("dbname={} user={} host=localhost password={}".format(self.src_db, self.src_db, self.src_url ,self.src_pwd))
if src_conn:
src_curr = src_conn.cursor()
print("Cursor found",src_curr)
src_curr.execute("select * from product_template")
recs = src_curr.fetchall()
print(recs)
else:
print("Cursor not found")
except Exception as ex:
print(ex)
I use this code to retreive an id. It works:
db = MySQLdb.connect("localhost","root","","proyectoacademias" )
cursor = db.cursor()
sql = "SELECT id FROM test WHERE url=\'"
sql = sql + self.start_urls[0]
sql = sql + "\'"
cursor.execute(sql)
data = cursor.fetchone()
for row in data:
self.id_paper_web=str(row)
db.close()
It gives me the id of the current row I have to update...
But then I try to update or to insert, it doesn't work....
def guardarDatos(self):
db = MySQLdb.connect("localhost","root","","proyectoacademias" )
cursor = db.cursor()
sql = "UPDATE test SET abstract=\'"+str(self.abstracto)+"\', fecha_consulta=\'"+str(self.fecha_consulta)+"\', anio_publicacion=\'"+str(self.anio_publicacion)+"\', probabilidad="+str(self.probabilidad)+" WHERE id = "+str(self.id_paper_web)
print "\n\n\n"+sql+"\n\n\n"
cursor.execute(sql)
for i in range (len(self.nombres)):
sql = "INSERT INTO test_autores VALUES (\'"+self.nombres.keys()[i]+"\', "+str(self.id_paper_web)+", \'"+self.instituciones[self.nombres[self.nombres.keys()[i]]]+"\', "+str((i+1))+")"
print "\n\n\n"+sql+"\n\n\n"
cursor.execute(sql)
db.close()
I print every sql query I sent and they seem to be fine... no exceptions thrown, just no updates or inserts in the database...
you must commit ... or set the db to auto commit
db.commit()
lots of py sqlite3 tutorials out there
By default, the sqlite3 module opens transactions implicitly before a
Data Modification Language (DML) statement (i.e.
INSERT/UPDATE/DELETE/REPLACE), and commits transactions implicitly
before a non-DML, non-query statement (i. e. anything other than
SELECT or the aforementioned).
So if you are within a transaction and issue a command like CREATE
TABLE ..., VACUUM, PRAGMA, the sqlite3 module will commit implicitly
before executing that command. There are two reasons for doing that.
The first is that some of these commands don’t work within
transactions. The other reason is that sqlite3 needs to keep track of
the transaction state (if a transaction is active or not).
You can control which kind of BEGIN statements sqlite3 implicitly
executes (or none at all) via the isolation_level parameter to the
connect() call, or via the isolation_level property of connections.
If you want autocommit mode, then set isolation_level to None.
Otherwise leave it at its default, which will result in a plain
“BEGIN” statement, or set it to one of SQLite’s supported isolation
levels: “DEFERRED”, “IMMEDIATE” or “EXCLUSIVE”.
http://docs.python.org/library/sqlite3.html Section 11.13.6