Using plain Oracle SQL, I would use:
select *
from mytable
where
to_date(date_d, 'dd-mon-yyyy') == to_date('01-Jan-2013', 'dd-mon-yyyy')
Now using SQLAlchemy, I'm at a loss.
from datetime import datetime
dt = myTable.c.date_d == datetime(2013, 1, 1)
q = session.query(myTable).filter(dt)
q.first()
Will give me incorrect results, because date_d is varchar, so it won't translate to date unless I run an Oracle function to convert the datatype within the query. How can I do that?
Turns out I didn't need to convert the datatype. SQLAlchemy seems to do that transparently. However, If I feel like doing it explicitly:
from sqlalchemy import func
dt = func.to_date(myTable.c.date_d, 'dd-mon-yyyy') == datetime(2013, 1, 1)
q = session.query(dt)
q.first()
My incorrect results were due to an unrelated mistake.
Related
My database is SQL Server 2008.
The type of time character I want to query in the database (such as finishdate) is datetime2
I just want data between "10-11" and "10-17".
When using Sqlalchemy, I use
cast(FinishDate, DATE).between(cast(time1, DATE),cast(time2, DATE))
to query dates, but it does not return any data (I confirm that there must be some data statements meet the query time range)
==============================================
from sqlalchemy import DATE
bb = "2021-10-11 12:21:23"
cc = "2021-10-17 16:12:34"
record = session.query(sa.Name cast(sa.FinishDate, DATE)).filter(
cast(sa.SamplingTime, DATE).between(cast(bb, DATE), cast(cc, DATE)),
sa.SamplingType != 0
).all()
or
record = session.query(sa.Name cast(sa.FinishDate, DATE)).filter(
cast(sa.SamplingTime, DATE)>= cast(bb, DATE),
sa.SamplingType != 0
).all()
Both return []
Something is wrong with my code and I don't know what the trouble is.
It is working for me, I only changed the DATE that you are using to Date
from sqlalchemy import Date
record = session.query(
sa.Name cast(sa.FinishDate, Date)
).filter(
cast(sa.SamplingTime, Date).between(
cast(bb, Date), cast(cc, Date)
),
sa.SamplingType != 0
).all()
As a matter of fact first parameter of cast can be a string also, so in this case its fine to pass date as string in cast.
:param expression: A SQL expression, such as a
:class:`_expression.ColumnElement`
expression or a Python string which will be coerced into a bound
literal value.
The Sqlite documentation states:
SQLite has no DATETIME datatype. Instead, dates and times can be stored in any of these ways:
As a TEXT string in the ISO-8601 format. Example: '2018-04-02 12:13:46'.
As an INTEGER number of seconds since 1970 (also known as "unix time").
...
so I decided to use an INTEGER unix timestamp:
import sqlite3, time
conn = sqlite3.connect(':memory:')
conn.execute("CREATE TABLE data(datetime INTEGER, t TEXT);")
conn.execute("INSERT INTO data VALUES (CURRENT_TIMESTAMP, 'hello')")
Why does the following query return no result?
ts = int(time.time()) + 31*24*3600 # unix timestamp 1 month in the future
print(list(conn.execute("SELECT * FROM data WHERE datetime <= ?", (ts, ))))
More generally, how to do a SELECT query with a comparison with a unix timestamp with Sqlite?
PS:
I have already read SQLite DateTime comparison and similar questions, which offer other comparison methods, but here I'd like to precisely discuss why this unix timestamp comparison does not work.
For performance reasons, I'd like to:
do a query that compares integers (which is super fast if many rows): WHERE datetime <= unix_timestamp,
avoid to convert unix_timestamp into string, and then compare datetime to this string (I guess it'll be far slower)
You use CURRENT_TIMESTAMP when inserting new rows.
This means that in your column the values are not stored as unix timestamps becuase CURRENT_TIMESTAMP returns the current date in the format of YYYY-MM-DD hh:mm:ss.
You can transform the unix timestamp to datetime in the format of YYYY-MM-DD hh:mm:ss with the function datetime() and the unixepoch modifier:
conn.execute("SELECT * FROM data WHERE datetime <= datetime(?, 'unixepoch')", (ts, ))
If your unix timestamp contains milliseconds you must strip them off:
conn.execute("SELECT * FROM data WHERE datetime <= datetime(? / 1000, 'unixepoch')", (ts, ))
Or, you can transform the string datetime in the column datetime to a unix timestamp with the function strftime():
conn.execute("SELECT * FROM data WHERE strftime('%s', datetime) + 0 <= ?", (ts, ))
If you want to store integer values in the column, use strftime() like this:
INSERT INTO data VALUES (strftime('%s', CURRENT_TIMESTAMP) + 0, 'hello')
I am using Python to connect to SQL Server database and execute several 'select' type of queries that contain date range written in a particular way. All these queries have the same date range, so instead of hard-coding it, I'd prefer to have it as a string and change it in one place only when needed.
So far, I found out that I can use datetime module and the following logic to convert dates to strings:
from datetime import datetime
start_date = datetime(2020,1,1).strftime("%Y-%m-%d")
end_date = datetime(2020,1,31).strftime("%Y-%m-%d")
Example of the query:
select * from where xxx='yyy' and time between start_date and end_date
How can I make it work?
EDIT
my code:
import pyodbc
import sqlalchemy
from sqlalchemy import create_engine
from datetime import datetime
start_date = datetime(2020,1,1).strftime("%Y-%m-%d")
end_date = datetime(2020,1,31).strftime("%Y-%m-%d")
engine = create_engine("mssql+pyodbc://user:pwd#server/monitor2?driver=SQL+Server+Native+Client+11.0")
sql_query = """ SELECT TOP 1000
[mtime]
,[avgvalue]
FROM [monitor2].[dbo].[t_statistics_agg]
where place = 'Europe' and mtime between 'start_date' and 'end_date'
order by [mtime] asc;"""
df = pd.read_sql(sql_query, engine)
print(df)
Thank you all for your input, I have found the answer to make the query work. The variables should look like:
start_date = date(2020, 1, 1)
end_date = date(2020, 1, 31)
and SQL query like:
sql_query = f""" SELECT TOP 1000
[mtime]
,[avgvalue]
FROM [monitor2].[dbo].[t_statistics_agg]
where place = 'Europe' and mtime between '{start_date}' and '{end_date}'
order by [mtime] asc;"""
I have a Python(3) script that is supposed to run each morning. In it, I call some SQL. However I'm getting an error message:
Error while connecting to PostgreSQL operator does not exist: date = integer
The SQL is based on the concatenation of a string:
ecom_dashboard_query = """
with
days_data as (
select
s.date,
s.user_type,
s.channel_grouping,
s.device_category,
sum(s.sessions) as sessions,
count(distinct s.dimension2) as daily_users,
sum(s.transactions) as transactions,
sum(s.transaction_revenue) as revenue
from ga_flagship_ecom.sessions s
where date = """ + run.start_date + """
group by 1,2,3,4
)
insert into flagship_reporting.ecom_dashboard
select *
from days_data;
"""
Here is the full error:
09:31:25 Error while connecting to PostgreSQL operator does not exist: date = integer
09:31:25 LINE 14: where date = 2020-01-19
09:31:25 ^
09:31:25 HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
I tried wrapping run.start_date within str like so: str(run.start_date) but I received the same error message.
I suspect it may be to do with the way I concatenate the SQL query string, but I am not sure.
The query runs fine in SQL directly with a hard coded date and no concatenation:
where date = '2020-01-19'
How can I get the query string to work correctly?
It's more better to pass query params to cursor.execute method. From docs
Warning Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
So instead of string concatenation pass run.start_date as second argument of cursor.execute.
In your query instead of concatenation use %s:
where date = %s
group by 1,2,3,4
In your python code add second argument to execute method:
cur.execute(ecom_dashboard_query , (run.start_date,))
Your sentece is wrong:
where date = """ + run.start_date + """
try to compare a date and a string and this is not posible, you need to convert "run.start_date" to datetime and compare simply:
date_format = datetime.strptime(your_date_string, '%y-%m-%d')
and with this date converted to datetime do:
where date = date_format
Final code:
date_format = datetime.strptime(your_date_string, '%y-%m-%d')
ecom_dashboard_query = """
with
days_data as (
select
s.date,
s.user_type,
s.channel_grouping,
s.device_category,
sum(s.sessions) as sessions,
count(distinct s.dimension2) as daily_users,
sum(s.transactions) as transactions,
sum(s.transaction_revenue) as revenue
from ga_flagship_ecom.sessions s
where date = {}
group by 1,2,3,4
)
insert into flagship_reporting.ecom_dashboard
select *
from days_data;
""".format(date_format)
I am running a query that I plan on using multiple times. However when running this query the 'my-job1a' has to be different everytime so I was planning on making this go by the date time. Does anybody know how to implement the date time function for this?
from google.cloud import bigquery
client = bigquery.Client('dataworks-356fa')
query = query
dataset = client.dataset('FirebaseArchive')
table = dataset.table(name='test1')
tbl = dataset.table(name='test12')
job = client.run_async_query('my-job1a', query)
job.destination = tbl
job.write_disposition= 'WRITE_TRUNCATE'
job.begin()
i believe "my-job1a" is a constant string. and you want to change the string for new query.
import datetime
# "my-job1a" replace this with "my-job1a" + datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
job = client.run_async_query("my-job1a-" + datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), query)
this will change for each second . if you want in millisecond then change the strftime function parameter. if you don't want such a big string , then change strftime parameter as per your choice.