We Keep Coding

Python SQLAlchemy Importing Table Names in Lowercase (Snowflake)

Using both pandas.read_sql as well as pandas.read_sql_table, I keep getting the entire table back with all the column names in lowercase.
Is there anyway around this?
I wanted to do some transformations on the data then replace the table in the DB, but it's a pain if doing so changes all the column names to lowercase.
#both of these produce the same lowercase columns
sql = 'SELECT * from "DB"."SCHEMA"."'+"tablename"+'"; '
df = pd.read_sql(
sql,
con=engine
)
df = pd.read_sql_table(
"tablename",
con=engine
)

Snowflake stores all case-insensitive object names in uppercase text.
In contrast, SQLAlchemy considers all lowercase object names to be
case-insensitive. Snowflake SQLAlchemy converts the object name case
during schema-level communication, i.e. during table and index
reflection. If you use uppercase object names, SQLAlchemy assumes they
are case-sensitive and encloses the names with quotes. This behavior
will cause mismatches agaisnt data dictionary data received from
Snowflake, so unless identifier names have been truly created as case
sensitive using quotes, e.g., "TestDb", all lowercase names should be
used on the SQLAlchemy side.
https://github.com/snowflakedb/snowflake-sqlalchemy
lowercase is default behavior in Sqlalchemy. You should not using uppercase in Sqlalchemy or pandas if it is not necessary to. You can either use "..." or quote_names in Sqlalchemy to specify case-sensitive.
If you insists on getting uppercase columns for all, this post of using events listener could be helpful https://stackoverflow.com/a/34322171/12032355

Are you sure its an issue? https://docs.snowflake.com/en/sql-reference/identifiers-syntax.html says:
Unquoted object identifiers are stored and resolved as uppercase
characters (e.g. id is stored and resolved as ID).

ProgrammingError: (psycopg2.errors.UndefinedColumn), while working with sqlalchemy

I have trouble querying a table, created with sqlalchemy on postgres db (local).
While I am able to execute, and receive query result with:
SELECT * FROM olympic_games
I am getting an error message when I'm trying to access single column, or perform any other operation on table:
SELECT games FROM olympic_games
The error message is (couple sentences translated from Polish):
ProgrammingError: (psycopg2.errors.UndefinedColumn) BŁĄD: column "games" does not exist
LINE 1: SELECT COUNT(Sport)
^
HINT: maybe you meant "olympic_games.Games".
SQL: SELECT games FROM olympic_games LIMIT 5;]
(Background on this error at: http://sqlalche.me/e/f405)
It pretty much sums to that program doesn't see, or can access specific column, and display that it doesn't exist.
I tried accessing with table.column format, it didn't work as well. I am also able to see column names, via information_schema.columns
Data (.csv) was loaded with pd.read_csv, and then DataFrame.to_sql. Code below, thanks for help!
engine = create_engine('postgresql://:#:/olympic_games')
with open('olympic_athletes_2016_14.csv', 'r') as file:
df = pd.read_csv(file, index_col='ID')
df.to_sql(name = 'olympic_games', con = engine, if_exists = 'replace', index_label = 'ID')
Both execute commands returned with same error:
with engine.connect() as con:
rs = con.execute("SELECT games FROM olympic_games LIMIT 5;")
df_fetch = pd.DataFrame(rs.fetchall())
df_fetch2 = engine.execute("""SELECT games FROM olympic_games LIMIT 5;""").fetchall()

Essentially, this is the double quoting issue of column identifiers as mentioned in the PostgreSQL manual:
Quoting an identifier also makes it case-sensitive, whereas unquoted names are always folded to lower case. For example, the identifiers FOO, foo, and "foo" are considered the same by PostgreSQL, but "Foo" and "FOO" are different from these three and each other.
When any of your Pandas data frame columns have mixed cases, the DataFrame.to_sql preserves the case sensitivity by creating columns with double quotes at CREATE TABLE stage. Specifically, the below Python Pandas code when using replace
df.to_sql(name='olympic_games', con=engine, if_exists='replace', index_label='ID')
Translates as below in Postgres if Sport was a titled case column in data frame:
DROP TABLE IF EXISTS public."olympic_games";
CREATE TABLE public."olympic_games"
(
...
"Sport" varchar(255)
"Games" varchar(255)
...
);
Once an identifier is quoted with mixed cases, it must always be referred to in that manner. Therefore sport is not the same as "Sport". Remember in SQL, double quotes actually is different than single quotes which can be interchangeable in Python.
To fix, consider rendering all your Pandas columns to lower case since "games" is the same as games, Games or GAMES (but not "Games" or "GAMES").
df.columns = df.columns.str.lower()
df.to_sql(name='olympic_games', con=engine, if_exists='replace', index_label='ID')
Alternatively, leave as is and quote appropriately:
SELECT "Games" FROM olympic_games

Try SELECT "games" FROM olympic_games. In some cases PostgreSQL create the quotes around a columns names. For example if the column name contained mixed register. I have to remind you: PostgreSQL is case sensitive

Read table in pandas with lowercase column using sqlalchemy

I would like to read a table in my database as pandas dataframe. I am working with sqlalchemy and it seems to me that it only executes queries in uppercase.
The table XYZ in my schema has a column name "pred_pred" in lowercase. When I do the following:
import pandas as pd
import cx_Oracle as ora
from sqlalchemy import create_engine
from sqlalchemy.engine import url
connect_url = url.URL(...)
engine = create_engine(connect_url)
connection = engine.connect()
input = pd.read_sql_query('SELECT pred_pred FROM XYZ', connection)
I get the following error:
DatabaseError: ORA-00904: "PRED_PRED": invalid identifier
Is there a workaround?
EDIT: as a workaround at the moment I am simply importing all columns using * and then working on them in pandas because the table has only few columns. I would still like to know if it's possible to solve this problem in a more direct way.

As also described in comment, you should just add double quotes to wrap your columns as oracle converts it to upper case if it is not wrapped with double quotes.
I think you need something like following:
input = pd.read_sql_query('SELECT "pred_pred" FROM XYZ', connection)
Because you must have created the xyz table with column wrapped in double quotes, it is stored as case sensitive name i.e lowercase.
See this db<>fiddle demo for more clarification.
Cheers!!

It depends how the table was created in the first place. By default, it doesn't matter if the DDL for that table was written in uppercase or lowercase, Oracle will change it all to uppercase and store it in the database.
That means, that bellow DDL statements are equal for Oracle:
create table table1 (column1 VARCHAR2(20));
CREATE TABLE TABLE1 (COLUMN1 VARCHAR2(20));
Table with such creation DDL can be simply queried both by:
SELECT COLUMN1 FROM TABLE1;
SELECT column1 FROM table1;
However, different issue is, when the table name or column is specified with double quotes.
create table table1 ("column1" VARCHAR2(20));
Then everytime you're querying that column, it has to be always queried again by those quotes and with the exact casing as it was created:
SELECT "column1" FROM TABLE1;
Regarding to Python code, in REPL it seems that you can easily combine double quotes with single quotes:
>>>input = 'SELECT "pred_pred" FROM XYZ'
>>>input
>>>'SELECT "pred_pred" FROM XYZ'
So the correct code you can just change your code with:
input = pd.read_sql_query('SELECT "pred_pred" FROM XYZ', connection)
To be sure that we're approaching the correct issue in here, you might want to connect to your database via for example SQL developer and query:
SELECT COLUMN_NAME FROM ALL_TAB_COLUMNS WHERE UPPER(TABLE_NAME) = 'XYZ'
If the column_name is not all upperacase, then it was created with double quotes and must be therefore queried the same way.
Further reading regarding to naming rules https://docs.oracle.com/database/121/SQLRF/sql_elements008.htm#SQLRF00223
You already mentioned the workaround with * in the comments - but it seems to be bad idea to query with * and do the projecting part on the Python side, since it would be raising the needed IO operations on the database side.

Too many server roundtrips w/ psycopg2

I am making a script, that should create a schema for each customer. I’m fetching all metadata from a database that defines how each customer’s schema should look like, and then create it. Everything is well defined, the types, names of tables, etc. A customer has many tables (fx, address, customers, contact, item, etc), and each table has the same metadata.
My procedure now:
get everything I need from the metadataDatabase.
In a for loop, create a table, and then Alter Table and add each metadata (This is done for each table).
Right now my script runs in about a minute for each customer, which I think is too slow. It has something to do with me having a loop, and in that loop, I’m altering each table.
I think that instead of me altering (which might be not so clever approach), I should do something like the following:
Note that this is just a stupid but valid example:
for table in tables:
con.execute("CREATE TABLE IF NOT EXISTS tester.%s (%s, %s);", (table, "last_seen date", "valid_from timestamp"))
But it gives me this error (it seems like it reads the table name as a string in a string..):
psycopg2.errors.SyntaxError: syntax error at or near "'billing'"
LINE 1: CREATE TABLE IF NOT EXISTS tester.'billing' ('last_seen da...

Consider creating tables with a serial type (i.e., autonumber) ID field and then use alter table for all other fields by using a combination of sql.Identifier for identifiers (schema names, table names, column names, function names, etc.) and regular format for data types which are not literals in SQL statement.
from psycopg2 import sql
# CREATE TABLE
query = """CREATE TABLE IF NOT EXISTS {shm}.{tbl} (ID serial)"""
cur.execute(sql.SQL(query).format(shm = sql.Identifier("tester"),
tbl = sql.Identifier("table")))
# ALTER TABLE
items = [("last_seen", "date"), ("valid_from", "timestamp")]
query = """ALTER TABLE {shm}.{tbl} ADD COLUMN {col} {typ}"""
for item in items:
# KEEP IDENTIFIER PLACEHOLDERS
final_query = query.format(shm="{shm}", tbl="{tbl}", col="{col}", typ=i[1])
cur.execute(sql.SQL(final_query).format(shm = sql.Identifier("tester"),
tbl = sql.Identifier("table"),
col = sql.Identifier(item[0]))
Alternatively, use str.join with list comprehension for one CREATE TABLE:
query = """CREATE TABLE IF NOT EXISTS {shm}.{tbl} (
"id" serial,
{vals}
)"""
items = [("last_seen", "date"), ("valid_from", "timestamp")]
val = ",\n ".join(["{{}} {typ}".format(typ=i[1]) for i in items])
# KEEP IDENTIFIER PLACEHOLDERS
pre_query = query.format(shm="{shm}", tbl="{tbl}", vals=val)
final_query = sql.SQL(pre_query).format(*[sql.Identifier(i[0]) for i in items],
shm = sql.Identifier("tester"),
tbl = sql.Identifier("table"))
cur.execute(final_query)
SQL (sent to database)
CREATE TABLE IF NOT EXISTS "tester"."table" (
"id" serial,
"last_seen" date,
"valid_from" timestamp
)

However, this becomes heavy as there are too many server roundtrips.
How many tables with how many columns are you creating that this is slow? Could you ssh to a machine closer to your server and run the python there?
I don't get that error. Rather, I get an SQL syntax error. A values list is for conveying data. But ALTER TABLE is not about data, it is about metadata. You can't use a values list there. You need the names of the columns and types in double quotes (or no quotes) rather than single quotes. And you can't have a comma between name and type. And you can't have parentheses around each pair. And each pair needs to be introduced with "ADD", you can't have it just once. You are using the wrong tool for the job. execute_batch is almost the right tool, except it will use single quotes rather than double quotes around the identifiers. Perhaps you could add a flag to it tell it to use quote_ident.
Not only is execute_values the wrong tool for the job, but I think python in general might be as well. Why not just load from a .sql file?

How do you escape strings for SQLite table/column names in Python?

The standard approach for using variable values in SQLite queries is the "question mark style", like this:
import sqlite3
with sqlite3.connect(":memory:") as connection:
connection.execute("CREATE TABLE foo(bar)")
connection.execute("INSERT INTO foo(bar) VALUES (?)", ("cow",))
print(list(connection.execute("SELECT * from foo")))
# prints [(u'cow',)]
However, this only works for substituting values into queries. It fails when used for table or column names:
import sqlite3
with sqlite3.connect(":memory:") as connection:
connection.execute("CREATE TABLE foo(?)", ("bar",))
# raises sqlite3.OperationalError: near "?": syntax error
Neither the sqlite3 module nor PEP 249 mention a function for escaping names or values. Presumably this is to discourage users from assembling their queries with strings, but it leaves me at a loss.
What function or technique is most appropriate for using variable names for columns or tables in SQLite? I'd would strongly prefer to do able to do this without any other dependencies, since I'll be using it in my own wrapper.
I looked for but couldn't find a clear and complete description of the relevant part of SQLite's syntax, to use to write my own function. I want to be sure this will work for any identifier permitted by SQLite, so a trial-and-error solution is too uncertain for me.
SQLite uses " to quote identifiers but I'm not sure that just escaping them is sufficient. PHP's sqlite_escape_string function's documentation suggests that certain binary data may need to be escaped as well, but that may be a quirk of the PHP library.

To convert any string into a SQLite identifier:
Ensure the string can be encoded as UTF-8.
Ensure the string does not include any NUL characters.
Replace all " with "".
Wrap the entire thing in double quotes.
Implementation
import codecs
def quote_identifier(s, errors="strict"):
encodable = s.encode("utf-8", errors).decode("utf-8")
nul_index = encodable.find("\x00")
if nul_index >= 0:
error = UnicodeEncodeError("NUL-terminated utf-8", encodable,
nul_index, nul_index + 1, "NUL not allowed")
error_handler = codecs.lookup_error(errors)
replacement, _ = error_handler(error)
encodable = encodable.replace("\x00", replacement)
return "\"" + encodable.replace("\"", "\"\"") + "\""
Given a string single argument, it will escape and quote it correctly or raise an exception. The second argument can be used to specify any error handler registered in the codecs module. The built-in ones are:
'strict': raise an exception in case of an encoding error
'replace': replace malformed data with a suitable replacement marker, such as '?' or '\ufffd'
'ignore': ignore malformed data and continue without further notice
'xmlcharrefreplace': replace with the appropriate XML character reference (for encoding only)
'backslashreplace': replace with backslashed escape sequences (for encoding only)
This doesn't check for reserved identifiers, so if you try to create a new SQLITE_MASTER table it won't stop you.
Example Usage
import sqlite3
def test_identifier(identifier):
"Tests an identifier to ensure it's handled properly."
with sqlite3.connect(":memory:") as c:
c.execute("CREATE TABLE " + quote_identifier(identifier) + " (foo)")
assert identifier == c.execute("SELECT name FROM SQLITE_MASTER").fetchone()[0]
test_identifier("'Héllo?'\\\n\r\t\"Hello!\" -☃") # works
test_identifier("北方话") # works
test_identifier(chr(0x20000)) # works
print(quote_identifier("Fo\x00o!", "replace")) # prints "Fo?o!"
print(quote_identifier("Fo\x00o!", "ignore")) # prints "Foo!"
print(quote_identifier("Fo\x00o!")) # raises UnicodeEncodeError
print(quote_identifier(chr(0xD800))) # raises UnicodeEncodeError
Observations and References
SQLite identifiers are TEXT, not binary.
SQLITE_MASTER schema in the FAQ
Python 2 SQLite API yelled at me when I gave it bytes it couldn't decode as text.
Python 3 SQLite API requires queries be strs, not bytes.
SQLite identifiers are quoted using double-quotes.
SQL as Understood by SQLite
Double-quotes in SQLite identifiers are escaped as two double quotes.
SQLite identifiers preserve case, but they are case-insensitive towards ASCII letters. It is possible to enable unicode-aware case-insensitivity.
SQLite FAQ Question #18
SQLite does not support the NUL character in strings or identifiers.
SQLite Ticket 57c971fc74
sqlite3 can handle any other unicode string as long as it can be properly encoded to UTF-8. Invalid strings could cause crashes between Python 3.0 and Python 3.1.2 or thereabouts. Python 2 accepted these invalid strings, but this is considered a bug.
Python Issue #12569
Modules/_sqlite/cursor.c
I tested it a bunch.

The psycopg2 documentation explicitly recommends using normal python % or {} formatting to substitute in table and column names (or other bits of dynamic syntax), and then using the parameter mechanism to substitute values into the query.
I disagree with everyone who is saying "don't ever use dynamic table/column names, you're doing something wrong if you need to". I write programs to automate stuff with databases every day, and I do it all the time. We have lots of databases with lots of tables, but they are all built on repeated patterns, so generic code to handle them is extremely useful. Hand-writing the queries every time would be far more error prone and dangerous.
It comes down to what "safe" means. The conventional wisdom is that using normal python string manipulation to put values into your queries is not "safe". This is because there are all sorts of things that can go wrong if you do that, and such data very often comes from the user and is not in your control. You need a 100% reliable way of escaping these values properly so that a user cannot inject SQL in a data value and have the database execute it. So the library writers do this job; you never should.
If, however, you're writing generic helper code to operate on things in databases, then these considerations don't apply as much. You are implicitly giving anyone who can call such code access to everything in the database; that's the point of the helper code. So now the safety concern is making sure that user-generated data can never be used in such code. This is a general security issue in coding, and is just the same problem as blindly execing a user-input string. It's a distinct issue from inserting values into your queries, because there you want to be able to safely handle user-input data.
So my recommendation is: do whatever you want to dynamically assemble your queries. Use normal python string templating to sub in table and column names, glue on where clauses and joins, all the good (and horrible to debug) stuff. But make sure you're aware that whatever values such code touches has to come from you, not your users[1]. Then you use SQLite's parameter substitution functionality to safely insert user-input values into your queries as values.
[1] If (as is the case for a lot of the code I write) your users are the people who have full access to databases anyway and the code is to simplify their work, then this consideration doesn't really apply; you probably are assembling queries on user-specified tables. But you should still use SQLite's parameter substitution to save yourself from the inevitable genuine value that eventually contains quotes or percent signs.

If you're quite certain that you need to specify column names dynamically, you should use a library that can do so safely (and complains about things that are wrong). SQLAlchemy is very good at that.
>>> import sqlalchemy
>>> from sqlalchemy import *
>>> metadata = MetaData()
>>> dynamic_column = "cow"
>>> foo_table = Table('foo', metadata,
... Column(dynamic_column, Integer))
>>>
foo_table now represents the table with the dynamic schema, but you can only use it in the context of an actual database connection (so that sqlalchemy knows the dialect, and what to do with the generated sql).
>>> metadata.bind = create_engine('sqlite:///:memory:', echo=True)
You can then issue the CREATE TABLE .... with echo=True, sqlalchemy will log the generated sql, but in general, sqlalchemy goes out of its way to keep the generated sql out of your hands (lest you consider using it for evil purposes).
>>> foo_table.create()
2011-06-28 21:54:54,040 INFO sqlalchemy.engine.base.Engine.0x...2f4c
CREATE TABLE foo (
cow INTEGER
)
2011-06-28 21:54:54,040 INFO sqlalchemy.engine.base.Engine.0x...2f4c ()
2011-06-28 21:54:54,041 INFO sqlalchemy.engine.base.Engine.0x...2f4c COMMIT
>>>
and yes, sqlalchemy will take care of any column names that need special handling, like when the column name is a sql reserved word
>>> dynamic_column = "order"
>>> metadata = MetaData()
>>> foo_table = Table('foo', metadata,
... Column(dynamic_column, Integer))
>>> metadata.bind = create_engine('sqlite:///:memory:', echo=True)
>>> foo_table.create()
2011-06-28 22:00:56,267 INFO sqlalchemy.engine.base.Engine.0x...aa8c
CREATE TABLE foo (
"order" INTEGER
)
2011-06-28 22:00:56,267 INFO sqlalchemy.engine.base.Engine.0x...aa8c ()
2011-06-28 22:00:56,268 INFO sqlalchemy.engine.base.Engine.0x...aa8c COMMIT
>>>
and can save you from possible badness:
>>> dynamic_column = "); drop table users; -- the evil bobby tables!"
>>> metadata = MetaData()
>>> foo_table = Table('foo', metadata,
... Column(dynamic_column, Integer))
>>> metadata.bind = create_engine('sqlite:///:memory:', echo=True)
>>> foo_table.create()
2011-06-28 22:04:22,051 INFO sqlalchemy.engine.base.Engine.0x...05ec
CREATE TABLE foo (
"); drop table users; -- the evil bobby tables!" INTEGER
)
2011-06-28 22:04:22,051 INFO sqlalchemy.engine.base.Engine.0x...05ec ()
2011-06-28 22:04:22,051 INFO sqlalchemy.engine.base.Engine.0x...05ec COMMIT
>>>
(apparently, some strange things are perfectly legal identifiers in sqlite)

The first thing to understand is that table/column names cannot be escaped in the same sense than you can escape strings stored as database values.
The reason is that you either have to:
accept/reject the potential table/column name, i.e. it is not guaranteed that a string is an acceptable column/table name, contrarily to a string to be stored in some database; or,
sanitize the string which will have the same effect as creating a digest: the function used is surjective, not bijective (once again, the inverse is true for a string that is to be stored in some database); so not only can't you be certain of going from the sanitized name back to the original name, but you are at risk of unintentionally trying to create two columns or tables with the same name.
Having understood that, the second thing to understand is that how you will end up "escaping" table/column names depends on your specific context, and so there is more than one way to do this, but whatever the way, you'll need to dig up to figure out exactly what is or is not an acceptable column/table name in sqlite.
To get you started, here is one condition:
Table names that begin with "sqlite_" are reserved for internal use. It is an error to attempt to create a table with a name that starts with "sqlite_".
Even better, using certain column names can have unintended side effects:
Every row of every SQLite table has a 64-bit signed integer key that
uniquely identifies the row within its table. This integer is usually
called the "rowid". The rowid value can be accessed using one of the
special case-independent names "rowid", "oid", or "rowid" in place
of a column name. If a table contains a user defined column named
"rowid", "oid" or "rowid", then that name always refers the
explicitly declared column and cannot be used to retrieve the integer
rowid value.
Both quoted texts are from http://www.sqlite.org/lang_createtable.html

From the sqlite faq, question 24 (the formulation of the question of course does not give a clue that the answer may be useful to your question):
SQL uses double-quotes around identifiers (column or table names) that contains special characters or which are keywords. So double-quotes are a way of escaping identifier names.
If the name itself contains double quotes, escape that double quote with another one.

Placeholders are only for values. Column and table names are structural, and are akin to variable names; you can't use placeholders to fill them in.
You have three options:
Appropriately escape/quote the column name everywhere you use it. This is fragile and dangerous.
Use an ORM like SQLAlchemy, which will take care of escaping/quoting for you.
Ideally, just don't have dynamic column names. Tables and columns are for structure; anything dynamic is data and should be in the table rather than part of it.

I made some research because I was unsatisfied with the current unsafe answers, and I would recommend using the internal printf function of sqlite to do that. It is made to escape any identifier (table name, column table...) and make it safe for concatenation.
In python, it should be something like that (I'm not a python user, so there may be mistakes, but the logic itself works):
table = "bar"
escaped_table = connection.execute("SELECT printf('%w', ?)", (table,)).fetchone()[0]
connection.execute("CREATE TABLE \""+escaped_table+"\" (bar TEXT)")
According to the documentation of %w:
This substitution works like %q except that it doubles all double-quote characters (") instead of single-quotes, making the result suitable for using with a double-quoted identifier name in an SQL statement.
The %w substitution is an SQLite enhancements, not found in most other printf() implementations.
Which means you can alternatively do the same with single quotes using %q:
table = "bar"
escaped_table = connection.execute("SELECT printf('%q', ?)", (table,)).fetchone()[0]
connection.execute("CREATE TABLE '"+escaped_table+"' (bar TEXT)")

If you find that you need a variable entity name (either relvar or field) then you probably are doing something wrong. an alternative pattern would be to use a property map, something like:
CREATE TABLE foo_properties(
id INTEGER NOT NULL,
name VARCHAR NOT NULL,
value VARCHAR,
PRIMARY KEY(id, name)
);
Then, you just specify the name dynamically when doing an insert instead of a column.