I want to use SQLAlchemy to create a view in my PostgreSQL database. I'm using the CreateView compiler from sqlalchemy-views. I'm using the answer to this question as a reference:
How to create an SQL View with SQLAlchemy?
My code for creating the view looks like this:
def create_view(self, myparameter):
mytable = Table('mytable', metadata, autoload=True)
myview = Table('myview', metadata)
engine.execute(CreateView(myview, mytable.select().where(mytable.c.mycolumn==myparameter)))
However, when I attempt to run this query, the following exception is thrown:
KeyError: 'mycolumn_1'
Looking at the compiled query, it seems that a placeholder for my parameter value is not being replaced:
'\nCREATE VIEW myview AS SELECT mytable.mycolumn \nFROM mytable \nWHERE mytable.mycolumn = %(mycolumn_1)s\n\n'
Since the placeholder is not being replaced, the query obviously fails. However, I do not understand why the replacement does not happen, since my code does not differ much from the example.
My first suspicion was that maybe the type of the parameter and the column were incompatible. Currently, the parameter comes in as a unicode string, which should be mapped to a text column in my database. I have also tried mapping the parameter as a long to a bigint column with the same (failed) result.
Does anyone have another suggestion?
From the SQLAlchemy documentation, I can see that when one wants to pass the actual value that will be ultimately used at expression time, the bindparam() is used. A nice example is also provided:
from sqlalchemy import bindparam
stmt = select([users_table]).\
where(users_table.c.name == bindparam('username'))
Related
When I try to reflect all tables in my Sybase DB
metadata = MetaData()
metadata.reflect(bind=engine)
SQLAlchemy runs the following query:
SELECT o.name AS name
FROM sysobjects o JOIN sysusers u ON o.uid = u.uid
WHERE u.name = #schema_name
AND o.type = 'U'
I then try to print the contents of metadata.tables, and this yields nothing.
I've tried creating an individual Table object and using the autoload=True option, but this yields a TableDoesNotExist error.
accounts = Table('Accounts', metadata, autoload=True, autoload_with=engine)
I looked into this query and it seems the #schema_name is becoming my username, and none of the tables which come from "sysobjects" appear to have a "name" attribute set to my username. They are all set to "dbo", which means the Database Owner, and thus the query returns nothing, and nothing is ever reflected. Is there any way to force SQLAlchemy to use something different as schema_name?
I've found two questions regarding table reflection using the Sybase dialect. Both were asked 6 years ago and seem to indicate that table reflection with Sybase was unsupported. However, it seems that SQLAlchemy tries to run a genuine sybase reflection query as above, so I don't think this is the case now.
I've solved this by setting the schema parameter on the MetaData object. I had to set it to dbo. You can also specify this in the reflect function.
I have a stored procedure in a postgresql database.
I'm trying to use the function within a python flask app with sqlalchemy. That query looks like this:
from sqlalchemy import func
appts = db.session.execute(func.getopenappointments(current_user.id))
for appt in appts:
# work with each appt
The result from this query is an object of type sqlalchemy.engine.result.ResultProxy. Each iteration of that object looks like this:
('(2,"2017-09-15 10:00:00",6,cleaning,available,5)',)
The problem is I am used to referring to the columns with something like:
for appt in appts:
print(appt.id)
But this fails due to id not existing. What I have realized is the output is pretty much a string that I have to parse with python split() just to get the values I need. How I can keep this a stored procedure but be able to refer to the output by columns, or at least as a tuple and not a regular string?
Take a look at this question. There is a construct called from_statement that can be used to interpret the results of a SQL statement as an SQLAlchemy ORM model.
So I'm assuming that you have an Appointment class that is an ORM mapper, either because you used declarative_base or because you used the mapper function directly.
Then you can do something like
appts = db.session.query(Appointment).from_statement(func.getopenappointments(current_user.id))
That will run your SQL stored procedure and interpret the result if it is a set of Appointment objects.
I am working in python with a MySQL database. I have a table that uses the MySQL geometry extension, so I need to call the GeomFromText MySQL function during an update statement, something like this:
UPDATE myTable SET Location=GeomFromText('Point(39.0 55.0)') where id=1;
UPDATE myTable SET Location=GeomFromText('Point(39.0 55.0)') where id=2;
Originally, I was using the low-level MySQLdb library. I am switching to using the SQLAlchemy core library (I cannot use the SQLAlchemy ORM for speed and other reasons).
If I were using the lower-level MySQLdb library directly, I would do something like this:
import MySQLdb as mysql
commandTemplate = "UPDATE myTable SET Location=GeomFromText(%s) where id=%s"
connection = mysql.connect(host="myhost",user="user",passwd="password",db="my_schema")
cursor = connection.cursor(mysql.cursors.DictCursor)
data = [
("Point(39.0 55.0)",1),
("Point(39.0 55.0)",2),
]
cursor.executemany(commandTemplate,data)
How do I get the equivalent functionality with SQLAlchemy core?
Without the GeomFromText, I think it would look something like this (thanks to this answer):
from sqlalchemy.sql.expression import bindparam
updateCommand = myTable.update().where(id=bindparam("idToChange"))
data = [
{'idToChange':1,'Location':"Point(39.0 55.0)"},
{'idToChange':2,'Location':"Point(39.0 55.0)"},
]
connection.execute(updateCommand,data)
I can't just textually replace "Point(39.0 55.0)" with "GeomFromText('Point(39.0 55.0)')", or I get:
Cannot get geometry object from data you send to the GEOMETRY field
The easiest way I have found so far involves the use of text (i.e. constructing TextClause objects), which lets you enter SQL syntax (almost) literally.
My example would work something like this:
from sqlalchemy.sql.expression import bindparam
from sqlalchemy import text
updateCommand = myTable.update().where(id=bindparam("idToChange"))
valuesDict = {'idToChange':':idToChange',
'Location':text("GeomFromText(:_location)")
}
updateCommand = updateCommand.values(**valuesDict)
data = [
{'idToChange':1,'_location':"Point(39.0 55.0)"},
{'idToChange':2,'_location':"Point(39.0 55.0)"},
]
#see the MySQL command as it will be executed (except for data)
print(connection.compile(bind=connection))
#actually execute the statement
connection.execute(updateCommand,data)
The key points:
calling updateCommand.values replaces the VALUES part of the SQL clause. Only the columns that you give as kwargs to this call will actually be put into the final UPDATE statement
the values of the keyword arguments to updateCommand.values can either be a literal set of data (if you are only updating one row), or it can be a string giving the names of keys in the data dictionary that will eventually be passed with the command to the connection.execute method. The format to use is ColumnName=":dictionaryKeyName".
the values of the keyword arguments can also be the result of a text clause, which can itself contain field names in the same ":dictionaryKeyName" format.
I am trying to do a simple filter operation on a query in sqlalchemy, like this:
q = session.query(Genotypes).filter(Genotypes.rsid.in_(inall))
where
inall is a list of strings
Genotypes is mapped to a table:
class Genotypes(object):
pass
Genotypes.mapper = mapper(Genotypes, kg_table, properties={'rsid': getattr(kg_table.c, 'rs#')})
This seems pretty straightforward to me, but I get the following error when I execute the above query by doing q.first():
"sqlalchemy.exc.OperationalError: (OperationalError) too many SQL
variables u'SELECT" followed by a list of the 1M items in the inall
list. But they aren't supposed to be SQL variables, just a list whose
membership is the filtering criteria.
Am I doing the filtering incorrectly?
(the db is sqlite)
If the table where you are getting your rsids from is available in the same database I'd use a subquery to pass them into your Genotypes query rather than passing the one million entries around in your Python code.
sq = session.query(RSID_Source).subquery()
q = session.query(Genotypes).filter(Genotypes.rsid.in_(sq))
The issue is that in order to pass that list to SQLite (or any database, really), SQLAlchemy has to pass over each entry for your in clause as a variable. The SQL translates roughly to:
-- Not valid SQLite SQL
DECLARE #Param1 TEXT;
SET #Param1 = ?;
DECLARE #Param2 TEXT;
SET #Param2 = ?;
-- snip 999,998 more
SELECT field1, field2, -- etc.
FROM Genotypes G
WHERE G.rsid IN (#Param1, #Param2, /* snip */)
The below workaround worked for me:
q = session.query(Genotypes).filter(Genotypes.rsid.in_(inall))
query_as_string = str(q.statement.compile(compile_kwargs={"literal_binds": True}))
session.execute(query_as_string).first()
This basically forces the query to compile as a string before execution, which bypasses the whole variables issue. Some details on this are available in SQLAlchemy's docs here.
BTW, if you're not using SQLite you can make use of the ANY operator to pass the list object as a single parameter (see my answer to this question here).
I would like to be able to full text search across several text fields of one of my SQLAlchemy mapped objects. I would also like my mapped object to support foreign keys and transactions.
I plan to use MySQL to run the full text search. However, I understand that MySQL can only run full text search on a MyISAM table, which does not support transactions and foreign keys.
In order to accomplish my objective I plan to create two tables. My code will look something like this:
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String(50))
description = Column(Text)
users_myisam = Table('users_myisam', Base.metadata,
Column('id', Integer),
Column('name', String(50)),
Column('description', Text),
mysql_engine='MyISAM')
conn = Base.metadata.bind.connect()
conn.execute("CREATE FULLTEXT INDEX idx_users_ftxt \
on users_myisam (name, description)")
Then, to search I will run this:
q = 'monkey'
ft_search = users_myisam.select("MATCH (name,description) AGAINST ('%s')" % q)
result = ft_search.execute()
for row in result: print row
This seems to work, but I have a few questions:
Is my approach of creating two tables to solve my problem reasonable? Is there a standard/better/cleaner way to do this?
Is there a SQLAlchemy way to create the fulltext index, or am I best to just directly execute "CREATE FULLTEXT INDEX ..." as I did above?
Looks like I have a SQL injection problem in my search/match against query. How can I do the select the "SQLAlchemy way" to fix this?
Is there a clean way to join the users_myisam select/match against right back to my user table and return actual User instances, since this is what I really want?
In order to keep my users_myisam table in sync with my mapped object user table, does it make sense for me to use a MapperExtension on my User class, and set the before_insert, before_update, and before_delete methods to update the users_myisam table appropriately, or is there some better way to accomplish this?
Thanks,
Michael
Is my approach of creating two tables to solve my problem reasonable?
Is there a standard/better/cleaner way to do this?
I've not seen this use case attempted before, as developers who value transactions and constraints tend to use Postgresql in the first place. I understand that may not be possible in your specific scenario.
Is there a SQLAlchemy way to create the fulltext index, or am I best
to just directly execute "CREATE FULLTEXT INDEX ..." as I did above?
conn.execute() is fine though if you want something slightly more integrated you can use the DDL() construct, read through http://docs.sqlalchemy.org/en/rel_0_8/core/schema.html?highlight=ddl#customizing-ddl for details
Looks like I have a SQL injection problem in my search/match against query. How can I do the
select the "SQLAlchemy way" to fix this?
note: this recipe is only for MATCH against multiple columns simultaneously - if you have just one column, use the match() operator more simply.
most basically you could use the text() construct:
from sqlalchemy import text, bindparam
users_myisam.select(
text("MATCH (name,description) AGAINST (:value)",
bindparams=[bindparam('value', q)])
)
more comprehensively you could define a custom construct:
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql.expression import ClauseElement
from sqlalchemy import literal
class Match(ClauseElement):
def __init__(self, columns, value):
self.columns = columns
self.value = literal(value)
#compiles(Match)
def _match(element, compiler, **kw):
return "MATCH (%s) AGAINST (%s)" % (
", ".join(compiler.process(c, **kw) for c in element.columns),
compiler.process(element.value)
)
my_table.select(Match([my_table.c.a, my_table.c.b], "some value"))
docs:
http://docs.sqlalchemy.org/en/rel_0_8/core/compiler.html
Is there a clean way to join the users_myisam select/match against right back
to my user table and return actual User instances, since this is what I really want?
you should probably create a UserMyISAM class, map it just like User, then use relationship() to link the two classes together, then simple operations like this are possible:
query(User).join(User.search_table).\
filter(Match([UserSearch.x, UserSearch.y], "some value"))
In order to keep my users_myisam table in sync with my mapped object
user table, does it make sense for me to use a MapperExtension on my
User class, and set the before_insert, before_update, and
before_delete methods to update the users_myisam table appropriately,
or is there some better way to accomplish this?
MapperExtensions are deprecated, so you'd at least use the event API, and in most cases we want to try applying object mutations outside of the flush process. In this case, I'd be using the constructor for User, or alternatively the init event, as well as a basic #validates decorator which will receive values for the target attributes on User and copy those values into User.search_table.
Overall, if you've been learning SQLAlchemy from another source (like the Oreilly book), its really out of date by many years, and I'd be focusing on the current online documentation.