I'm using Python's Peewee ORM to work with a MySQL database. Peewee supplies an object called "fn" that allows you to make certain types of calls to the database. One of those calls I want to make is the following:
Blocks.select(Blocks, fn.Count(Blocks.height))
Where Blocks is a table in my database, which has a column named height. This syntax is taken straight from Peewee's documentation, namely
User.select(
User, fn.Count(Tweet.id))
located here http://peewee.readthedocs.org/en/latest/peewee/querying.html. Note that I also have the following lines at the top of my python file
import peewee
from peewee import *
from peewee import fn
Yet when I run this code, it doesn't work, and it spits out this
<class '__main__.Blocks'> SELECT t1.`height`, t1.`hash`, t1.`time`, t1.`confirmations`, t1.`size`, t1.`version`, t1.`merkleRoot`, t1.`numTX`, t1.`nonce`, t1.`bits`, t1.`difficulty`, t1.`chainwork`, t1.`previousBlockHash`, t1.`nextBlockHash`, Count(t1.`height`) FROM `blocks` AS t1 []
So this is really just printing out the column names that are returned by the select query.
What peewee code do I have to write to return the count of the number of rows in a table? I regret using peewee because it makes what should be simple queries impossibly hard to find the right syntax for.
Peewee lazily evaluates queries, so you need to coerce it to a list or iterate through it in order to retrieve results, e.g.
query = User.select(User, fn.Count(Tweet.id).alias('num_tweets'))
for user in query:
print user.username, user.num_tweets
users = list(query)
Related
I am new to this and trying to learn python. I wrote a select statement in python where I used a parameter
Select """cln.customer_uid = """[(num_cuid_number)])
TypeError: string indices must be integers
Agree with the others, this doesn't look really like Python by itself.
I will see even without seeing the rest of that code I'll guess the [(num_cuid_number)] value(s) being returned is a string, so you'll want to convert it to integer for the select statement to process.
num_cuid_number is most likely a string in your code; the string indices are the ones in the square brackets. So please first check your data variable to see what you received there. Also, I think that num_cuid_number is a string, while it should be in an integer value.
Let me give you an example for the python code to execute: (Just for the reference: I have used SQLAlchemy with flask)
#app.route('/get_data/')
def get_data():
base_sql="""
SELECT cln.customer_uid='%s' from cln
""" % (num_cuid_number)
data = db.session.execute(base_sql).fetchall()
Pretty sure you are trying to create a select statement with a "where" clause here. There are many ways to do this, for example using raw sql, the query should look similar to this:
query = "SELECT * FROM cln WHERE customer_uid = %s"
parameters = (num_cuid_number,)
separating the parameters from the query is secure. You can then take these 2 variables and execute them with your db engine like
results = db.execute(query, parameters)
This will work, however, especially in Python, it is more common to use a package like SQLAlchemy to make queries more "flexible" (in other words, without manually constructing an actual string as a query string). You can do the same thing using SQLAlchemy core functionality
query = cln.select()
query = query.where(cln.customer_uid == num_cuid_number)
results = db.execute(query)
Note: I simplified "db" in both examples, you'd actually use a cursor, session, engine or similar to execute your queries, but that wasn't your question.
So the question is "is it possible to make the following query using Django ORM without raw statements?"
SELECT
my_table.column_a + my_table.column_b
FROM
my_table
The example for which it would be suitable from my point of view:
We have a model:
class MyOperations(models.Model):
operation_start_time = models.DateTimeField()
At some point we create a record and set the field value to Now (or we update some existing record. it doesn't matter):
MyOperations.objects.create(operation_start_time=functions.Now())
Now we want to know how much time has already passed. I would expect that Django ORM produces the following SQL statement to request data from the database (let's assume that we use MySQL backend):
SELECT
TIMESTAMPDIFF(MICROSECOND, `myapp_myoperations`.`operation_start_time`, CURRENT_TIMESTAMP) AS `time_spent`
FROM
`myapp_myoperations`
WHERE ...
So is it a way to achieve this without raw statements?
For now I settled on the following solution:
MyOperations.objects.values('operation_start_time').annotate(
diff=ExpressionWrapper(functions.Now() - F('operation_start_time'),
output_field=DurationField()
)).filter(...)
It produces the following SQL statement:
SELECT
`myapp_myoperations`.`operation_start_time`,
TIMESTAMPDIFF(MICROSECOND, `myapp_myoperations`.`operation_start_time`, CURRENT_TIMESTAMP) AS `time_spent`
FROM
`myapp_myoperations`
WHERE ...
Or in the Python response object representation:
{'operation_start_time': datetime(...), 'diff': timedelta(...)}
Is it a way to get the response dict with only diff since this is the only field I am interested in?
Django ORM produced the query which requests operation_start_time just as we had written. But in case I remove the call to values at all it produces query which requests all table columns
Solution which produces the expected SQL
We should just put the call to values to the place in which diff is already known to the query
MyOperations.objects.annotate(
diff=ExpressionWrapper(functions.Now() - F('operation_start_time'),
output_field=DurationField()
)).values('diff').filter(...)
You can use values() on your calculated field, so a query like
MyOperations.objects.values('operation_start_time').annotate(
diff=ExpressionWrapper(functions.Now() - F('operation_start_time'),
output_field=DurationField()
)).values('diff')
should give you a resulting queryset containing only your calculated 'diff'.
I want to use SQLAlchemy to create a view in my PostgreSQL database. I'm using the CreateView compiler from sqlalchemy-views. I'm using the answer to this question as a reference:
How to create an SQL View with SQLAlchemy?
My code for creating the view looks like this:
def create_view(self, myparameter):
mytable = Table('mytable', metadata, autoload=True)
myview = Table('myview', metadata)
engine.execute(CreateView(myview, mytable.select().where(mytable.c.mycolumn==myparameter)))
However, when I attempt to run this query, the following exception is thrown:
KeyError: 'mycolumn_1'
Looking at the compiled query, it seems that a placeholder for my parameter value is not being replaced:
'\nCREATE VIEW myview AS SELECT mytable.mycolumn \nFROM mytable \nWHERE mytable.mycolumn = %(mycolumn_1)s\n\n'
Since the placeholder is not being replaced, the query obviously fails. However, I do not understand why the replacement does not happen, since my code does not differ much from the example.
My first suspicion was that maybe the type of the parameter and the column were incompatible. Currently, the parameter comes in as a unicode string, which should be mapped to a text column in my database. I have also tried mapping the parameter as a long to a bigint column with the same (failed) result.
Does anyone have another suggestion?
From the SQLAlchemy documentation, I can see that when one wants to pass the actual value that will be ultimately used at expression time, the bindparam() is used. A nice example is also provided:
from sqlalchemy import bindparam
stmt = select([users_table]).\
where(users_table.c.name == bindparam('username'))
I have a stored procedure in a postgresql database.
I'm trying to use the function within a python flask app with sqlalchemy. That query looks like this:
from sqlalchemy import func
appts = db.session.execute(func.getopenappointments(current_user.id))
for appt in appts:
# work with each appt
The result from this query is an object of type sqlalchemy.engine.result.ResultProxy. Each iteration of that object looks like this:
('(2,"2017-09-15 10:00:00",6,cleaning,available,5)',)
The problem is I am used to referring to the columns with something like:
for appt in appts:
print(appt.id)
But this fails due to id not existing. What I have realized is the output is pretty much a string that I have to parse with python split() just to get the values I need. How I can keep this a stored procedure but be able to refer to the output by columns, or at least as a tuple and not a regular string?
Take a look at this question. There is a construct called from_statement that can be used to interpret the results of a SQL statement as an SQLAlchemy ORM model.
So I'm assuming that you have an Appointment class that is an ORM mapper, either because you used declarative_base or because you used the mapper function directly.
Then you can do something like
appts = db.session.query(Appointment).from_statement(func.getopenappointments(current_user.id))
That will run your SQL stored procedure and interpret the result if it is a set of Appointment objects.
I would like to be able to full text search across several text fields of one of my SQLAlchemy mapped objects. I would also like my mapped object to support foreign keys and transactions.
I plan to use MySQL to run the full text search. However, I understand that MySQL can only run full text search on a MyISAM table, which does not support transactions and foreign keys.
In order to accomplish my objective I plan to create two tables. My code will look something like this:
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String(50))
description = Column(Text)
users_myisam = Table('users_myisam', Base.metadata,
Column('id', Integer),
Column('name', String(50)),
Column('description', Text),
mysql_engine='MyISAM')
conn = Base.metadata.bind.connect()
conn.execute("CREATE FULLTEXT INDEX idx_users_ftxt \
on users_myisam (name, description)")
Then, to search I will run this:
q = 'monkey'
ft_search = users_myisam.select("MATCH (name,description) AGAINST ('%s')" % q)
result = ft_search.execute()
for row in result: print row
This seems to work, but I have a few questions:
Is my approach of creating two tables to solve my problem reasonable? Is there a standard/better/cleaner way to do this?
Is there a SQLAlchemy way to create the fulltext index, or am I best to just directly execute "CREATE FULLTEXT INDEX ..." as I did above?
Looks like I have a SQL injection problem in my search/match against query. How can I do the select the "SQLAlchemy way" to fix this?
Is there a clean way to join the users_myisam select/match against right back to my user table and return actual User instances, since this is what I really want?
In order to keep my users_myisam table in sync with my mapped object user table, does it make sense for me to use a MapperExtension on my User class, and set the before_insert, before_update, and before_delete methods to update the users_myisam table appropriately, or is there some better way to accomplish this?
Thanks,
Michael
Is my approach of creating two tables to solve my problem reasonable?
Is there a standard/better/cleaner way to do this?
I've not seen this use case attempted before, as developers who value transactions and constraints tend to use Postgresql in the first place. I understand that may not be possible in your specific scenario.
Is there a SQLAlchemy way to create the fulltext index, or am I best
to just directly execute "CREATE FULLTEXT INDEX ..." as I did above?
conn.execute() is fine though if you want something slightly more integrated you can use the DDL() construct, read through http://docs.sqlalchemy.org/en/rel_0_8/core/schema.html?highlight=ddl#customizing-ddl for details
Looks like I have a SQL injection problem in my search/match against query. How can I do the
select the "SQLAlchemy way" to fix this?
note: this recipe is only for MATCH against multiple columns simultaneously - if you have just one column, use the match() operator more simply.
most basically you could use the text() construct:
from sqlalchemy import text, bindparam
users_myisam.select(
text("MATCH (name,description) AGAINST (:value)",
bindparams=[bindparam('value', q)])
)
more comprehensively you could define a custom construct:
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql.expression import ClauseElement
from sqlalchemy import literal
class Match(ClauseElement):
def __init__(self, columns, value):
self.columns = columns
self.value = literal(value)
#compiles(Match)
def _match(element, compiler, **kw):
return "MATCH (%s) AGAINST (%s)" % (
", ".join(compiler.process(c, **kw) for c in element.columns),
compiler.process(element.value)
)
my_table.select(Match([my_table.c.a, my_table.c.b], "some value"))
docs:
http://docs.sqlalchemy.org/en/rel_0_8/core/compiler.html
Is there a clean way to join the users_myisam select/match against right back
to my user table and return actual User instances, since this is what I really want?
you should probably create a UserMyISAM class, map it just like User, then use relationship() to link the two classes together, then simple operations like this are possible:
query(User).join(User.search_table).\
filter(Match([UserSearch.x, UserSearch.y], "some value"))
In order to keep my users_myisam table in sync with my mapped object
user table, does it make sense for me to use a MapperExtension on my
User class, and set the before_insert, before_update, and
before_delete methods to update the users_myisam table appropriately,
or is there some better way to accomplish this?
MapperExtensions are deprecated, so you'd at least use the event API, and in most cases we want to try applying object mutations outside of the flush process. In this case, I'd be using the constructor for User, or alternatively the init event, as well as a basic #validates decorator which will receive values for the target attributes on User and copy those values into User.search_table.
Overall, if you've been learning SQLAlchemy from another source (like the Oreilly book), its really out of date by many years, and I'd be focusing on the current online documentation.