I am a novice in Python. Based on this SO post, I created a SQL query using PYODBC to search a MSSQL table of historic option prices and select the option symbol with a strike value closest to the desired value I specified. However, I am now trying to teach myself OOP by re-factoring this program, and to that end I am trying to implement the ORM in SQLAlchemy.
I cannot figure out how to implement a calculated Order_By statement. I don't think a calculated column would work because desired_strike is an argument that that is specified by the user(me) at each method call.
Here is the (simplified) original code:
import pyodbc
def get_option_symbol(stock, entry_date, exp_date, desired_strike):
entry_date = entry_date.strftime('%Y-%m-%d %H:%M:%S')
exp_date = exp_date.strftime('%Y-%m-%d %H:%M:%S')
cursor.execute("""select top(1) optionsymbol
from dbo.options_pricestore
where underlying=?
and quotedate=?
and expiration=?
and exchange='*'
and option_type=?
order by abs(strike - ?)""",
stock,
entry_date,
exp_date,
desired_strike,
)
row = cursor.fetchone()
return row
Maybe not the most Pythonic, but it worked. I am now encapsulating my formerly procedural code into classes, and to use SQLAlchemy's ORM, except that in this one case I cannot figure out how to represent abs(strike - desired_strike) in the Order_By clause. I have not used lambda functions much in the past, but here is what I came up with:
import sqlalchemy
class Option(Base):
__tablename__= 'options_pricestore'
<column definitions go here>
def get_option_symbol(stock, entry_date, exp_date, desired_strike):
entry_date = entry_date.strftime('%Y-%m-%d %H:%M:%S')
exp_date = exp_date.strftime('%Y-%m-%d %H:%M:%S')
qry = session.query(Option.optionsymbol).filter(and_
(Option.underlying == stock,
Option.quotedate == entry_date,
Option.expiration == exp_date,
Option.option_type== "put",
Option.exchange == "*")
).order_by(lambda:abs(Option.strike - desired_strike))
return qry
I get "ArgumentError: SQL expression object or string expected" - Any help would be greatly appreciated.
order_by wants a string - give it to it:
qry = session.query(Option.optionsymbol).filter(and_
(Option.underlying == stock,
Option.quotedate == entry_date,
Option.expiration == exp_date,
Option.option_type== "put",
Option.exchange == "*")
).order_by('abs(strike - %d)' % desired_strike)
Related
TLDR:
I tried for the past hours to convert these 2 SQL queries into SQL alchemy db query statements (as I have read that this is the preferred way to do in Python). Thus, I am not writing queries with the text SQLAlchemy API.
The main goal is to create an intermediate table to get the max date from the weekly table and then, use the 2nd query to group by code,name,week the sum of the grades.
drop temporary table if exists latest;
create temporary table latest_d
(index ix_1 (code), index ix_2 (date)) as
select fw.Code, max(date) as date
from weekly fw
join (
select code, max(week) as max_week
from weekly
group by code
) max_f_wk on fw.code = max_fcast_wk.code and fw.Week = max_f_wk.max_week
group by code
And the next one:
drop temporary table if exists latest_grouped
create temporary table latest_grouped
(index ix_1 (code), index ix_2 (Name)) as
select fw.code, Name, Week,
sum(if(temp = 2, grade, 0)) as grade_2w,
sum(if(temp = 4, grade, 0)) as grade_4w,
sum(if(temp = 8, grade, 0)) as grade_8w
from weekly fw
join latest_d lf on fw.code = lf.code and fw.Date = lf.Date
where model_name in ('John Doe', 'Doe John')
group by fw.code, Name, Week;
My best try is - all the DBs and tables are declared in another python file like:
class Weekly(Base):
__tablename__ = "weekly"
code = Column(CHAR(5))
Name = Column(VARCHAR(250))
Date = Column(DATE)
And my best try for query nr 1
logger.info(f"Try launching trunc table {table_name}")
try:
truncate_table(logger, db, table_name)
except Exception:
logger.exception(f"Failed to truncate {table_name}!")
logger.info(f"Table trunc successful!")
try:
latest_dates = (
db.query(
Weekly.code,
func.max((Weekly.date) as date,)
.distinct()
.join(Weekly, Weekly.code == max_f_wk.code)
.join(
db.query( <second table query> ))
.where(fw.name in [<list>])
except Exception:
logger.exception("Failed")
return None
Not very professional but I am really stuck - I am not used to ORM maps
As the title says, I'm having problems retrieving data from a SQLite DB when using WHERE statement.
Here is the piece of code that tries to get a row where an ID is given:
def check_attendance(self, cred):
query = """SELECT * FROM clients WHERE dni=?"""
self.conn.cursor().execute(query, (cred,))
record = self.conn.cursor().fetchone()
The var cred is already inside a tuple as specified by SQLite API for Python. Sadly, the query returns None when executed here.
If I do the same but using sqlite.exe, then I do get the right row back. In fact, this is the only query I cannot execute properly from my python script, everything else return rows normally.
Here it is executing from the Python script
And here is in sqlite.exe
Here is the piece that stores values in the DB:
def new_client(self, *args):
success = False
# Check if all inputs are filled
if self.dialog.content_cls.ids.user_name.text and self.dialog.content_cls.ids.user_surname.text and len(self.dialog.content_cls.ids.user_dni.text) == 8 and self.dialog.content_cls.ids.user_date.text:
# Convert str date to a datetime obj in order to use it with timedelta
paid_date = datetime.strptime(self.dialog.content_cls.ids.user_date.text, "%d-%m-%Y")
# paid_date is now YYYY-MM-DD HH-MM-SS format
# Add 30 days to paid_date
exp_date = paid_date + timedelta(days=30)
# Convert YYYY-MM-DD HH-MM-DD to string YYYY-MM-DD as we don't need clock
paid_date = datetime.strptime(str(paid_date), "%Y-%m-%d %H:%M:%S").strftime("%Y-%m-%d")
exp_date = datetime.strptime(str(exp_date), "%Y-%m-%d %H:%M:%S").strftime("%Y-%m-%d")
# Create query blueprint and try executing
query = """INSERT INTO clients (name, surname, dni, membership_date, expiration_date) VALUES (?,?,?,?,?)"""
try:
self.conn.execute(query, (self.dialog.content_cls.ids.user_name.text,
self.dialog.content_cls.ids.user_surname.text,
self.dialog.content_cls.ids.user_dni.text,
paid_date,
exp_date
)
)
success = True
except sqlite3.IntegrityError:
pass
if success:
self.conn.commit()
The try/except was used for other reasons. Adding to the database from the Python script works fine as shown in the second screenshot.
And the table clients is as follows:
c.execute(''' CREATE TABLE clients (id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
surname TEXT NOT NULL,
dni INTEGER NOT NULL UNIQUE,
membership_date date NOT NULL,
expiration_date date NOT NULL); ''')
Using Python v3.7.7 32bit.
Thanks!
In your code, the cursor is initialized two times (cursor()).
You should either get the results from the same cursor you used to execute the SELECT statement:
def check_attendance(self, cred):
query = """SELECT * FROM clients WHERE dni=?"""
cur = self.conn.cursor()
cur.execute(query, (cred,))
record = cur.fetchone()
...or you can avoid the implicit cursor creation by using execute method directly on Connection object:
def check_attendance(self, cred):
query = """SELECT * FROM clients WHERE dni=?"""
record = self.conn.execute(query, (cred,)).fetchone()
You can read more about this approach in the documentation (https://docs.python.org/3/library/sqlite3.html#using-sqlite3-efficiently):
Using the nonstandard execute(), executemany() and executescript() methods of the Connection object, your code can be written more concisely because you don’t have to create the (often superfluous) Cursor objects explicitly. Instead, the Cursor objects are created implicitly and these shortcut methods return the cursor objects.
The query below works as raw sql
SELECT WORK_ORDER.*,(SELECT COMPLETE FROM SAMPLE WHERE COMPLETE = 'TRUE' AND
ARF_ID = WORK_ORDER.ARF_ID AND ROWNUM <= 1) AS SAMPLE_COMPLETE, (DUE_DATE -
SYSDATE) AS DUE_IN FROM WORK_ORDER WHERE COMPLETE = 'FALSE' ORDER BY
DUE_DATE ASC
The following Django Queryset does not work
subquery = Sample.objects.filter(complete = 'TRUE', arf_id = models.OuterRef('arf_id'))[:1]
workOrderList = WorkOrder.objects.annotate(sample_complete= models.Subquery(subquery.values('complete'))).annotate(due_in= models.F('due_date') - datetime.now()).filter(complete = 'FALSE').order_by('due_date')
which produces this query when running workOrderList.query
SELECT "WORK_ORDER"."ARF_ID", "WORK_ORDER"."COMPANY_NAME",
"WORK_ORDER"."COMPANY_ADDRESS", "WORK_ORDER"."CONTACT_TELEPHONE",
"WORK_ORDER"."ORDER_DATE", "WORK_ORDER"."DUE_DATE",
"WORK_ORDER"."ARF_NUMBER", "WORK_ORDER"."COMPLETE",
"WORK_ORDER"."COMPLETE_DATE", "WORK_ORDER"."REPORTED",
"WORK_ORDER"."REPORTED_DATE", "WORK_ORDER"."COMPANY_CODE", (SELECT * FROM
(SELECT "_SUB".* FROM (SELECT U0."COMPLETE" AS Col1 FROM "SAMPLE" U0 WHERE
(U0."COMPLETE" = TRUE AND U0."ARF_ID" = ("WORK_ORDER"."ARF_ID"))) "_SUB"
WHERE ROWNUM <= 1)) AS "SAMPLE_COMPLETE", ("WORK_ORDER"."DUE_DATE" - 2019-
01-28 13:00:51.043013) AS "DUE_IN" FROM "WORK_ORDER" WHERE
"WORK_ORDER"."COMPLETE" = FALSE ORDER BY "WORK_ORDER"."DUE_DATE" ASC
This returns error
cx_Oracle.DatabaseError: ORA-00904: "WORK_ORDER"."ARF_ID": invalid identifier
I am using Django 1.11.13 and this is a legacy database, I am comfortable using raw sql to query data but would like to learn/utilize the Django ORM the correct way so any fix or explanation why this won't work is helpful to me.
Using the Exists() subclass in this case works is a solution to accomplish the the overall goal:
workOrderList = WorkOrder.objects.annotate(sample_complete= models.Exists(subquery.values('complete')))
.annotate(due_in= models.F('due_date') - datetime.now()).filter(complete = 'FALSE').order_by('due_date')
I am still looking for a solution that uses the Subquery() method
Please help me understand behavior of peewee 2.4.5 when talking to MySQL 5.5. I'm running a simple query to count children associated with a parent; in this case documents at a path. As plain SQL it boils down to this:
select p.name, count(d.file) as child_count
from path as p, doc as d
where p.id = d.path_id
group by p.name
The Peewee code uses the fn.COUNT feature, see below for a self-contained example. The result comes back just fine and with the results I expect, with one exception: the query result object attribute "child_count" is of type unicode instead of integer. In this little example there's 1 row and I get back a string (essentially) '1' instead of the number 1.
I'm confused because in other queries I have done with fn.COUNT the result is of type integer. Is this a feature? Am I making a silly python mistake here? Thanks in advance.
'''
Example of accessing MySQL from Python using Peewee.
Developed with peewee 2.4.5, pymysql 0.6.3, MySql 5.5
'''
from __future__ import print_function
from peewee import MySQLDatabase, Model, CharField, ForeignKeyField, fn
db = MySQLDatabase(database="test", host="localhost", user="mumble", password="foo")
class MySQLModel(Model):
'''
Base class to associate the database object
'''
class Meta:
database = db
class Path(MySQLModel):
# peewee adds primary key field 'id'
name = CharField()
class Doc(MySQLModel):
# peewee adds primary key field 'id'
path = ForeignKeyField(Path)
file = CharField()
def main():
db.connect()
db.create_tables([Path, Doc], True)
newpath = Path(name='ab/23')
newpath.save()
newdoc1 = Doc(path=newpath.id, file='file1.txt')
newdoc1.save()
newdoc2 = Doc(path=newpath.id, file='file2.txt')
newdoc2.save()
for row in Path.select():
print("Path: id=%d, name=%s" % (row.id, row.name))
for row in Doc.select():
print("Doc: id=%d, file=%s" % (row.id, row.file))
# query in plain old SQL:
# select p.name, count(d.file) from path as p, doc as d where p.id = d.path_id group by p.name
path_doc_result = (Path
.select(Path.name, fn.COUNT(Doc.file).alias('child_count'))
.join(Doc, on=(Path.id == Doc.path))
.group_by(Path.name))
path_doc_count = len(list(path_doc_result))
print("Path-doc parent-child result count is %d" % path_doc_count)
if path_doc_count == 0:
print("Programmer error, no results!")
else:
# get the first one
d_row = path_doc_result[0]
#### Why is the child_count attribute not integer? ###
print("Type of child_count attribute is %s" % type(d_row.child_count))
print("Path-Doc result: name=%s child_count=%d" % (d_row.name, int(d_row.child_count)))
newdoc1.delete_instance()
newdoc2.delete_instance()
newpath.delete_instance()
# order matters for foreign keys!
db.drop_table(Doc)
db.drop_table(Path)
db.close()
if __name__ == "__main__":
main()
Peewee functions look at the type of the first argument and attempt to coerce the return value to that type. This makes sense in most cases but I can see why it's causing an issue here.
To work around, just call fn.COUNT(Doc.file).coerce(False).alias('child_count')
path_doc_result = (Path
.select(Path.name, fn.COUNT(Doc.file).coerce(False).alias('child_count'))
.join(Doc, on=(Path.id == Doc.path))
.group_by(Path.name))
I am using SQLAlchemy without the ORM, i.e. using hand-crafted SQL statements to directly interact with the backend database. I am using PG as my backend database (psycopg2 as DB driver) in this instance - I don't know if that affects the answer.
I have statements like this,for brevity, assume that conn is a valid connection to the database:
conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)")
Assume also that the user table consists of the columns (id [SERIAL PRIMARY KEY], name, country_id)
How may I obtain the id of the new user, ideally, without hitting the database again?
You might be able to use the RETURNING clause of the INSERT statement like this:
result = conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)
RETURNING *")
If you only want the resulting id:
result = conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)
RETURNING id")
[new_id] = result.fetchone()
User lastrowid
result = conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)")
result.lastrowid
Current SQLAlchemy documentation suggests
result.inserted_primary_key should work!
Python + SQLAlchemy
after commit, you get the primary_key column id (autoincremeted) updated in your object.
db.session.add(new_usr)
db.session.commit() #will insert the new_usr data into database AND retrieve id
idd = new_usr.usrID # usrID is the autoincremented primary_key column.
return jsonify(idd),201 #usrID = 12, correct id from table User in Database.
this question has been asked many times on stackoverflow and no answer I have seen is comprehensive. Googling 'sqlalchemy insert get id of new row' brings up a lot of them.
There are three levels to SQLAlchemy.
Top: the ORM.
Middle: Database abstraction (DBA) with Table classes etc.
Bottom: SQL using the text function.
To an OO programmer the ORM level looks natural, but to a database programmer it looks ugly and the ORM gets in the way. The DBA layer is an OK compromise. The SQL layer looks natural to database programmers and would look alien to an OO-only programmer.
Each level has it own syntax, similar but different enough to be frustrating. On top of this there is almost too much documentation online, very hard to find the answer.
I will describe how to get the inserted id AT THE SQL LAYER for the RDBMS I use.
Table: User(user_id integer primary autoincrement key, user_name string)
conn: Is a Connection obtained within SQLAlchemy to the DBMS you are using.
SQLite
======
insstmt = text(
'''INSERT INTO user (user_name)
VALUES (:usernm) ''' )
# Execute within a transaction (optional)
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = result.lastrowid
txn.commit()
MS SQL Server
=============
insstmt = text(
'''INSERT INTO user (user_name)
OUTPUT inserted.record_id
VALUES (:usernm) ''' )
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = result.fetchone()[0]
txn.commit()
MariaDB/MySQL
=============
insstmt = text(
'''INSERT INTO user (user_name)
VALUES (:usernm) ''' )
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = conn.execute(text('SELECT LAST_INSERT_ID()')).fetchone()[0]
txn.commit()
Postgres
========
insstmt = text(
'''INSERT INTO user (user_name)
VALUES (:usernm)
RETURNING user_id ''' )
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = result.fetchone()[0]
txn.commit()
result.inserted_primary_key
Worked for me. The only thing to note is that this returns a list that contains that last_insert_id.
Make sure you use fetchrow/fetch to receive the returning object
insert_stmt = user.insert().values(name="homer", country_id="123").returning(user.c.id)
row_id = await conn.fetchrow(insert_stmt)
For Postgress inserts from python code is simple to use "RETURNING" keyword with the "col_id" (name of the column which you want to get the last inserted row id) in insert statement at end
syntax -
from sqlalchemy import create_engine
conn_string = "postgresql://USERNAME:PSWD#HOSTNAME/DATABASE_NAME"
db = create_engine(conn_string)
conn = db.connect()
INSERT INTO emp_table (col_id, Name ,Age)
VALUES(3,'xyz',30) RETURNING col_id;
or
(if col_id column is auto increment)
insert_sql = (INSERT INTO emp_table (Name ,Age)
VALUES('xyz',30) RETURNING col_id;)
result = conn.execute(insert_sql)
[last_row_id] = result.fetchone()
print(last_row_id)
#output = 3
ex -