How do I convert an sqlalchemy hstore value to a string?
from sqlalchemy.dialects.postgresql import array, hstore
hs = hstore(array(['key1', 'key2', 'key3']), array(['value1', 'value2', 'value3']))
# this triggers sqlalchemy.exc.UnsupportedCompilationError
str(hs)
I expect something like "key1"=>"value1", "key2"=>"value2", "key3"=>"value3"
I would like to use an sqlalchemy api rather than write a custom string formatting function that approximates what I want. I'm working with a legacy code base that uses sqlalchemy: I need to preserve any internal quirks and escaping logic that formatting does.
However, the existing code base uses sqlalchemy via an ORM table insert, while I want to directly convert an sqlalchemy hstore value to a string?
UPDATE: I am trying to do something like this:
I have an existing table with schema
create table my_table
(
id bigint default nextval('my_table_id_seq'::regclass),
ts timestamp default now(),
text_col_a text,
text_col_b text
);
I want to get the following Python sqlalchemy code working:
str_value = some_function()
# Existing code is building an sqlalchemy hstore and inserting
# into a column of type `text`, not an `hstore` column.
# I want it to work with hstore text formatting
hstore_value = legacy_build_my_hstore()
# as is this triggers error:
# ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'hstore'
return db_connection.execute(
"""
insert into my_table(text_col_a, text_col_b) values (%s, %s)
returning id, ts
""",
(str_value, hstore_value).first()
Let Postgresql do the cast for you instead of trying to manually convert the hstore construct to a string, and SQLAlchemy handle the conversion to suitable text representation:
return db_connection.execute(
my_table.insert().
values(text_col_a=str_value,
text_col_b=cast(hstore_value, Text)).
returning(my_table.c.id, my_table.c.ts)).first()
As soon as you can, alter your schema to use hstore type instead of text, if that is what the column contains.
Related
I have a large dictionary of items - ~400 columns. I have a script to detect and create the type of each item so I have {'age' = INT, "Name" = String,etc..) but I'm not sure how to use that to create a table in SQLAlchemy or directly creating the query?
I am using postgres but I am familiar with mysql & sqlite so anything that works for those I would be able to apply to my usecase.
What about this:
from sqlalchemy import create_engine, Table, Column, MetaData
metadata = MetaData()
fields = (Column(colname, coltype) for colname, coltype in your_dict.items())
t = Table(name, metadata, *fields)
engine = create_engine(database)
metadata.create_all(engine)
You need to have objects from sqlalchemy.sql.sqltypes rather than strings as values in your_dict:
from sqlalchemy.sql.sqltypes import String, Integer
See https://docs.sqlalchemy.org/en/13/core/type_basics.html for the whole list.
I want to use SQLAlchemy to create a view in my PostgreSQL database. I'm using the CreateView compiler from sqlalchemy-views. I'm using the answer to this question as a reference:
How to create an SQL View with SQLAlchemy?
My code for creating the view looks like this:
def create_view(self, myparameter):
mytable = Table('mytable', metadata, autoload=True)
myview = Table('myview', metadata)
engine.execute(CreateView(myview, mytable.select().where(mytable.c.mycolumn==myparameter)))
However, when I attempt to run this query, the following exception is thrown:
KeyError: 'mycolumn_1'
Looking at the compiled query, it seems that a placeholder for my parameter value is not being replaced:
'\nCREATE VIEW myview AS SELECT mytable.mycolumn \nFROM mytable \nWHERE mytable.mycolumn = %(mycolumn_1)s\n\n'
Since the placeholder is not being replaced, the query obviously fails. However, I do not understand why the replacement does not happen, since my code does not differ much from the example.
My first suspicion was that maybe the type of the parameter and the column were incompatible. Currently, the parameter comes in as a unicode string, which should be mapped to a text column in my database. I have also tried mapping the parameter as a long to a bigint column with the same (failed) result.
Does anyone have another suggestion?
From the SQLAlchemy documentation, I can see that when one wants to pass the actual value that will be ultimately used at expression time, the bindparam() is used. A nice example is also provided:
from sqlalchemy import bindparam
stmt = select([users_table]).\
where(users_table.c.name == bindparam('username'))
I have a DataFrame which consists of a column of strings. If I do df.to_sql() to save it as a table into an Oracle database, the column is of CLOB type and I need to convert it. I wonder if I can specify the type (say varchar2) when I create the table?
You can specify SQLAlchemy Type explicitly:
import cx_Oracle
from sqlalchemy import types, create_engine
engine = create_engine('oracle://user:password#host_or_scan_address:1521/ORACLE_SERVICE_NAME')
df.to_sql('table_name', engine, if_exists='replace',
dtype={'str_column': types.VARCHAR(df.str_column.str.len().max())})
df.str_column.str.len().max() - will calculate the maximum string length
NOTE: types.VARCHAR will be mapped to VARCHAR2 for Oracle (see working example here)
You have to options, the first is to create the table manually and then use the if_exists parameter to tell pandas to append to the table rather than to drop and recreate
Option two is to use the dtype pass a dictionary of column names so that the table can be created appropriately. These are SQL Alchemy types so you should
from sqlalchemy.dialects.oracle import VARCHAR2
and pass that in the dictionary as
{'mycolumn': VARCHAR2(256) }
or suitable length.
Ref: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html
I am working in python with a MySQL database. I have a table that uses the MySQL geometry extension, so I need to call the GeomFromText MySQL function during an update statement, something like this:
UPDATE myTable SET Location=GeomFromText('Point(39.0 55.0)') where id=1;
UPDATE myTable SET Location=GeomFromText('Point(39.0 55.0)') where id=2;
Originally, I was using the low-level MySQLdb library. I am switching to using the SQLAlchemy core library (I cannot use the SQLAlchemy ORM for speed and other reasons).
If I were using the lower-level MySQLdb library directly, I would do something like this:
import MySQLdb as mysql
commandTemplate = "UPDATE myTable SET Location=GeomFromText(%s) where id=%s"
connection = mysql.connect(host="myhost",user="user",passwd="password",db="my_schema")
cursor = connection.cursor(mysql.cursors.DictCursor)
data = [
("Point(39.0 55.0)",1),
("Point(39.0 55.0)",2),
]
cursor.executemany(commandTemplate,data)
How do I get the equivalent functionality with SQLAlchemy core?
Without the GeomFromText, I think it would look something like this (thanks to this answer):
from sqlalchemy.sql.expression import bindparam
updateCommand = myTable.update().where(id=bindparam("idToChange"))
data = [
{'idToChange':1,'Location':"Point(39.0 55.0)"},
{'idToChange':2,'Location':"Point(39.0 55.0)"},
]
connection.execute(updateCommand,data)
I can't just textually replace "Point(39.0 55.0)" with "GeomFromText('Point(39.0 55.0)')", or I get:
Cannot get geometry object from data you send to the GEOMETRY field
The easiest way I have found so far involves the use of text (i.e. constructing TextClause objects), which lets you enter SQL syntax (almost) literally.
My example would work something like this:
from sqlalchemy.sql.expression import bindparam
from sqlalchemy import text
updateCommand = myTable.update().where(id=bindparam("idToChange"))
valuesDict = {'idToChange':':idToChange',
'Location':text("GeomFromText(:_location)")
}
updateCommand = updateCommand.values(**valuesDict)
data = [
{'idToChange':1,'_location':"Point(39.0 55.0)"},
{'idToChange':2,'_location':"Point(39.0 55.0)"},
]
#see the MySQL command as it will be executed (except for data)
print(connection.compile(bind=connection))
#actually execute the statement
connection.execute(updateCommand,data)
The key points:
calling updateCommand.values replaces the VALUES part of the SQL clause. Only the columns that you give as kwargs to this call will actually be put into the final UPDATE statement
the values of the keyword arguments to updateCommand.values can either be a literal set of data (if you are only updating one row), or it can be a string giving the names of keys in the data dictionary that will eventually be passed with the command to the connection.execute method. The format to use is ColumnName=":dictionaryKeyName".
the values of the keyword arguments can also be the result of a text clause, which can itself contain field names in the same ":dictionaryKeyName" format.
I'm using sqlalchemy-migrate to alter the type of one of the columns in a table in a Postgre SQL database. The upgrade script I'm using is:
# -*- cofing: utf-8 -*-
from sqlalchemy import MetaData, Table, Column, String, Integer
from migrate import changeset
metadata = MetaData()
def upgrade(migrate_engine):
# ALTER TABLE courses ALTER COLUMN number SET DATA TYPE character varying;
metadata.bind = migrate_engine
courses = Table('courses', metadata, Column("number", Integer), extend_existing=True)
courses.c.number.alter(type=String)
def downgrade(migrate_engine):
# ALTER TABLE courses ALTER COLUMN number SET DATA TYPE integer;
metadata.bind = migrate_engine
courses = Table('courses', metadata, Column("number", String), extend_existing=True)
courses.c.number.alter(type=Integer, cast='numeric')
The upgrade part seems to work but the downgrade always fails with the following error:
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "number" cannot be cast to type integer
'\nALTER TABLE courses ALTER COLUMN number TYPE INTEGER' {}
Now, if I were using plain SQL I could use ALTER TABLE courses ALTER COLUMN number TYPE INTEGER USING number::numeric to alter the column type back from character varying to integer, but I don't know how to achieve that using sqlalchemy-migrate.
Is there a way to force sqlalchemy to include USING number::numeric in the ALTER clause? or is there another way to avoid the error I posted above?
I appreciate your help.
Looks like sqlalchemy.migrate doesn't have support for rendering valid query in case of altering from String to Integer column types for postgresql.
In your case I would implement it as a direct query execution and move on.
def downgrade(migrate_engine):
# ALTER TABLE courses ALTER COLUMN number SET DATA TYPE integer;
migrate_engine.execute('ALTER TABLE courses ALTER COLUMN number TYPE INTEGER USING number::numeric')
BTW migrating from String to Integer may fail for different reasons - when column value would contain some value that cannot be converted to number. So I would add some extra validation into application logic to keep downgrade migration possible later on.