We Keep Coding

psycopg2 - 'Relation does not exist' in Postgresql database

I'm trying to figure out why I can't access a particular table in a PostgreSQL database using psycopg2. I am running PostgreSQL 11.5
If I do this, I can connect to the database in question and read all the tables in it:
import psycopg2
try:
connection = psycopg2.connect(user = "postgres", #psycopg2.connect() creates connection to PostgreSQL database instance
password = "battlebot",
host = "127.0.0.1",
port = "5432",
database = "BRE_2019")
cursor = connection.cursor() #creates a cursor object which allows us to execute PostgreSQL commands through python source
#Print PostgreSQL version
cursor.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""")
for table in cursor.fetchall():
print(table)
The results look like this :
('geography_columns',)
('geometry_columns',)
('spatial_ref_sys',)
('raster_columns',)
('raster_overviews',)
('nc_avery_parcels_poly',)
('Zone5e',)
('AllResidential2019',)
#....etc....
The table I am interested in is the last one, 'AllResidential2019'
So I try to connect to it and print the contents by doing the following:
try:
connection = psycopg2.connect(user = "postgres",
#psycopg2.connect() creates connection to PostgreSQL database instance
password = "battlebot",
host = "127.0.0.1",
port = "5432",
database = "BRE_2019")
cursor = connection.cursor() #creates a cursor object which allows us to execute PostgreSQL commands through python source
cursor.execute("SELECT * FROM AllResidential2019;") #Executes a database operation or query. Execute method takes SQL query as a parameter. Returns list of result
record = cursor.fetchall()
print(record)
except (Exception, psycopg2.Error) as error:
print("Error while connecting to PostgreSQL: ", error)
And I get the following error:
Error while connecting to PostgreSQL: relation "allresidential2019" does not exist
LINE 1: SELECT * FROM AllResidential2019;
However, I can successfully connect and get results when attempting to connect to another table in another database I have (this works! and the results are the data in this table):
try:
connection = psycopg2.connect(user = "postgres", #psycopg2.connect() creates connection to PostgreSQL database instance
password = "battlebot",
host = "127.0.0.1",
port = "5432",
database = "ClimbingWeatherApp") . #different database name
cursor = connection.cursor()
cursor.execute("SELECT * FROM climbing_area_info ;")
record = cursor.fetchall()
print(record)
except (Exception, psycopg2.Error) as error:
print("Error while connecting to PostgreSQL: ", error)
I can't figure out why I can retrieve information from one table but not another, using exactly the same code (except names are changes). And I am also not sure how to troubleshoot this. Can anyone offer suggestions?

Your table name is case-sensitive and you have to close it in double quotes:
SELECT * FROM "AllResidential2019";
In Python program it may look like this:
cursor.execute('SELECT * FROM "AllResidential2019"')
or you can use the specialized module SQL string composition:
from psycopg2 import sql
# ...
cursor.execute(sql.SQL("SELECT * FROM {}").format(sql.Identifier('AllResidential2019')))
Note that case-sensitive Postgres identifiers (i.e. names of a table, column, view, function, etc) unnecessarily complicate simple matters. I would advise you not to use them.

Likely, the reason for your issue is Postgres' quoting rules which adheres to the ANSI SQL standard regarding double quoting identifiers. In your table creation, you likely quoted the table:
CREATE TABLE "AllResidential2019" (
...
)
Due to case sensitivity of at least one capital letter, this requires you to always quote the table when referencing the table. Do remember: single and double quotes have different meanings in SQL as opposed to being mostly interchangeable in Python.
SELECT * FROM "AllResidential2019"
DELETE FROM "AllResidential2019" ...
ALTER TABLE "AllResidential2019" ...
It is often recommended, if your table, column, or other identifier does not contain special characters, spaces, or reserved words, to always use lower case or no quotes:
CREATE TABLE "allresidential2019" (
...
)
CREATE TABLE AllResidential2019 (
...
)
Doing so, any combination of capital letters will work
SELECT * FROM ALLRESIDENTIAL2019
SELECT * FROM aLlrEsIdEnTiAl2019
SELECT * FROM "allresidential2019"
See further readings on the subject:
Omitting the double quote to do query on PostgreSQL
PostgreSQL naming conventions
Postgres Docs - 4.1.1. Identifiers and Key Words
Don’t use double quotes in PostgreSQL
What is the difference between single and double quotes in SQL?

I was facing the same error in Ubuntu. But in my case, I accidentally added the tables to the wrong database, which was in turn owned by the root postgres user instead of the new postgres user that I had created for my flask app.
I'm using a SQL file to create and populate the tables. This is the command that I used to be able to create these tables using a .sql file. This allows you to specify the owner of the tables as well as the database in which they should be created:
sudo -u postgres psql -U my_user -d my_database -f file.sql -h localhost
You will then be prompted for my_users's password.
sudo -u postgres is only necessary if you are running this from a terminal as a the root user. It basically runs the psql ... command as the postgres user.

python-mysql cursor.execute failing with access denied error

I have two machines: local_machine, server_machine. I have mysql server on server_machine and sftp server on local_machine. I am trying to send sritest.csv file (UTF-8) from local_machine to server_machine using python. These are the contents of sritest.csv:
1,2,3
I have the sql query saved in sritest.sql and these are the contents of the file:
LOAD DATA INFILE '{}'
INTO TABLE TESTBED_STAGING.test
COLUMNS TERMINATED BY ','
;
This is the python script I have now:
import MySQLdb
import os
import string
# Open database connection
db = MySQLdb.connect (host="1.2.3.4",port=3306,user="app_1",\
passwd="passwd",db="TESTBED_STAGING")
cursor=db.cursor()
#Query under testing
sql = open('sritest.sql','r').read()
print sql
l = os.listdir(".")
for file_name in l:
if file_name.endswith('sritest.csv'):
print 'the csv file we are reading is: '+file_name
#try:
cursor = db.cursor()
print 'filename is '+sql.format(file_name)
cursor.execute(sql.format(file_name))
db.commit()
'''
except Exception:
# Rollback in case there is any error
db.rollback()
print 'ERROR - So, rollback :( :( '
'''
# disconnect from server
db.close()
In the above script, I commented try,except so I can see the error where it breaks. Currently the code is breaking at cursor.execute(sql.format(file_name)) line with this error:
OperationalError: (1045, "Access denied for user 'app_1'#'%' (using password: YES)")
I have been playing around but not able to fix it. Any suggestions/ideas?

For starters, creating cursor at every loop is not a good idea. You've already created a cursor earlier, so you can remove the cursor declaration in the for loop.
Second, I think your error is due to lack of access on MySQL server at 1.2.3.4 remotely using user app_1. Try this on the server's MySQL console,
GRANT ALL PRIVILEGES ON TESTBED_STAGING.* TO 'app_1'#'%';
Lastly, try and avoid using print "line" notation and start switching to the print("line") notation for compatibility with Python 3.x

I figured out the answer and decided to leave this question open for those who might face the similar problem:
In the MySQL server (server_machine), make sure you do this after you start mysql:
mysql>grant all privileges on *.* to 'app_1'#'%' identified by 'passwd';
change LOAD DATA INFILE '{}' in sritest.sql to LOAD DATA LOCAL INFILE '{}'
In the python code, edit the MySQLdb.connect statement as:
db = MySQLdb.connect (host="1.2.3.4",port=3306,user="app_1",\
passwd="passwd",db="TESTBED_STAGING", local_infile=1)
All errors are eliminated and data is transferred.

PyMysql encoding issue

I have MySql database (MyISAM engine, utf-8 charset and utf8_general_ci collation). Also I use PyMysql driver to connect my Python3 code to that database.
But when I am trying to fetch some rows with following code
import pymysql
conn = pymysql.connect(host='localhost',
port=3306,
user='root',
passwd='',
db='db_name')
cur = conn.cursor()
cur.execute('SELECT * FROM my_table')
print(cur.description)
print()
for row in cur:
print(row)
cur.close()
conn.close()
I have bad output for Russian column
(1, '???????????????????????????????')
I have tried to provide different parameters such as use_unicode, charset but none of their combinations do not bring desired result. With English columns all is ok. So how can I get well-encoded string?
Output of create table command
'CREATE TABLE my_table (id int(11) NOT NULL AUTO_INCREMENT,
ruColumn varchar(155) DEFAULT NULL, PRIMARY KEY (id))
ENGINE=MyISAM DEFAULT CHARSET=utf8'

There are multiple ways that could have happened.
Your bytes in your client were not utf8-encoded.
You did not specify utf8 when connecting to the database. This is done via set_charset('utf8'), SET NAMES utf8, or some other client-specific method. (What client are you using?)
The data is lost. Delete the data from the tables. (The table definition is ok.) So, clean up the code and start over.
For more details on Python, see my blog.

I solved this problem by switching connection driver from pymysql to mysql-connector-python
You can install it via pip
pip install mysql-connector-python --allow-external mysql-connector-python
So change the code to
import mysql
conn = mysql.connector.connect(host='localhost',
port=3306,
user='root',
passwd='',
db='db_name')
and output will be correct without additional specifying of charset or other params.
Seems like it's internal encoding problem of PyMySQL driver.

Empty table in MySQL even though Python can insert data into table

I'm new to mySQL and Python.
I have code to insert data from Python into mySQL,
conn = MySQLdb.connect(host="localhost", user="root", passwd="kokoblack", db="mydb")
for i in range(0,len(allnames)):
try:
query = "INSERT INTO resumes (applicant, jobtitle, lastworkdate, lastupdate, url) values ("
query = query + "'"+allnames[i]+"'," +"'"+alltitles[i]+"',"+ "'"+alldates[i]+"'," + "'"+allupdates[i]+"'," + "'"+alllinks[i]+"')"
x = conn.cursor()
x.execute(query)
row = x.fetchall()
except:
print "error"
It seems to be working fine, because "error" never appears. Instead, many rows of "1L" appear in my Python shell. However, when I go to MySQL, the "resumes" table in "mydb" remains completely empty.
I have no idea what could be wrong, could it be that I am not connected to MySQL's server properly when I'm viewing the table in MySQL? Help please.
(I only use import MySQLdb, is that enough?)

use commit to commit the changes that you have done
MySQLdb has autocommit off by default, which may be confusing at first
You could do commit like this
conn.commit()
or
conn.autocommit(True) Right after the connection is created with the DB

Python Mysqldb Returns Question Marks

I am using the following code in python and it seems to be returning the "Name" as question marks since the name is in russian. Any help would be much appreciated.
import MySQLdb
db = MySQLdb.connect(host="localhost", user="root",passwd="*****")
cur = db.cursor()
cur.execute("USE WebCorpusStatus;")
cur.execute("SELECT Name, Source, Date(dateScraped) FROM russian WHERE status = 1;")
for row in cur:
print row

MySQLdb.connect takes a use_unicode parameter, which may solve the problem. If not, you may also need to set charset to whatever your table uses:
MySQLdb.connect(host="localhost", user="root",passwd="*****", use_unicode=True, charset='xxxxx')
http://mysql-python.sourceforge.net/MySQLdb.html