I am having some issues with finding documents with a today's date function through python.
I use the following function:
datetime.datetime.now().date().strftime('%Y-%m-%d')
which gives the following value: 2017-05-21
However, the values in my documents (mongo) have double strings attached around it like the following: "2017-05-21"
So literally filling in the above string ("2017-05-21") exactly in my pymongo query works like a charm. However, I need the dynamicness of the datetime function, but unfortunately, that doesn't match with the double quoted date strings which I need for the query.
Does anyone know any workaround? I have already tried a replace function etc. It either creates double quotes within the single quotes or doesn't do anything.
It sounds like your MongoDB documents were inserted incorrectly, using text for date fields instead of BSON datetimes.
PyMongo automatically converts between Python datetimes and BSON datetimes, so you can insert a document that contains a BSON datetime like this:
dt = datetime.datetime.utcnow()
collection.insert_one({'myDate': dt})
Prove that date comparisons work now like so:
# There is a document with myDate in the past, now.
print(collection.find_one({'myDate': {'$lt': datetime.datetime.utcnow()}}))
# No document with myDate in the future.
print(collection.find_one({'myDate': {'$gt': datetime.datetime.utcnow()}}))
Related
class Test(Model):
time = DateTimeField()
# ...
row = Test.select()[0]
test.time
This returns a string that looks like this: 2017-01-23 01:01:39+01:00. How can I get it as a datetime object instead? Do I have to parse it manually?
Also I would be interested if there is any documentation on how to use the DateTimeField. The official documentation doesn't have anything on it.
Are you using SQLite? If so, SQLite doesn't have a dedicated datetime type, so datetimes are stored as strings in the DB. What peewee will do is recognize certain datetime formats coming out of the DB and convert them to datetime objects. What you need to do is ensure that either:
When you create/save your object, that you assign a datetime object to the field.
When reading back pre-existing data, that the data is in a recognized format.
The formats peewee supports out-of-the-box for datetime field are:
YYYY-mm-dd HH:MM:SS.ffffff
YYYY-mm-dd HH:MM:SS
YYYY-mm-dd
It looks like your has zone info. I'd suggest converting to UTC and dropping the zone info. That should fix it.
Have you tried adding a default like this?
time = DateTimeField(default=datetime.datetime.now())
Or when adding an entry add it as a datetime.datetime object directly:
test = Test(....., time=datetime.datetime.strptime("2018-3-15", '%Y-%m-%d'))
In the second case you don't need to specify anything in the class definition...
I am moving some of my code onto sqlalchemy from using raw MySQL queries.
The current issue I am having is that the datetime was saved in a string format by a C# tool. Unfortunately, the representation does not match up with Python's (as well as that it has an extra set of single quotes), thus making filtering somewhat cumbersome.
Here is an example of the format that the date was saved in:
'2016-07-01T17:27:01'
Which I was able to convert to a usable datetime using the following MySQL command:
STR_TO_DATE(T.PredicationGeneratedTime, \"'%%Y-%%m-%%dT%%H:%%i:%%s'\")
However, I cannot find any documentation that describes how to invoke built-in functions such as STR_TO_DATE when filtering with sqlalchemy
The following Python code:
session.query(Train.Model).filter(cast(Train.Model.PredicationGeneratedTime, date) < start)
is giving me:
TypeError: Required argument 'year' (pos 1) not found
There does not seem to be a way to specify the format for the conversion.
Note: I realize the solution is to fix the way the datetime is stored, but in the mean time I'd like to run queries against the existing data.
You can try to use func.str_to_date(COLUMN, FORMAT_STRING) instead of cast
In the cast() you should be using sqlalchemy.DateTime, not (what I assume is) a datetime.date - that is the cause of the exception.
However, fixing that will not really help because of the embedded single quotes.
You are fortunate that the dates stored in your table are in ISO format. That means that lexicographic comparisons will work on the date strings themselves, without casting. As long as you use a string for start with the surrounding single quotes, it will work.
from datetime import datetime
start = "'{}'".format(datetime.now().isoformat())
session.query(Train.Model).filter(Train.Model.PredicationGeneratedTime < start)
I have a Python 2.7 API that queries a SQL db and delivers a JSON list of dictionaries that is then used in a bootstrap/Django site.
Dates in the DB are strings in the format '2017-04-20 00:00:00', but sometimes the time of the source data instead has a decimal, which causes trouble with strptime, so I'm removing the seconds by keeping only the first 10 characters of the string.
import datetime
dict_list = response['my_list_of_dicts']
for dt_to_cmpr in dict_list:
dt_to_cmpr['date_key'] = dt_to_cmpr['date_key'][:10]
Before I can compare date ranges, the dates need to be date time not strings. (Note: For production, I plan to account for exceptions such as null values.)
dt_to_cmpr['date_key'] = datetime.datetime.strptime(dt_to_cmpr['date_key'],
'%Y-%m-%d')
I want to know things about dictionaries where date_key is roughly no more than 90 days from today. (i.e. the total number in the time frame, or the sum of every dictionary's price_key.)
under_days = datetime.timedelta(days=-1)
over_days = datetime.timedelta(days=91)
now = datetime.datetime.now()
ttl_within_90days = sum(1 for d in response['my_list_of_dicts'] if (under_days <
(d.get('date_key')-now) < over_days))
One problem is now that I've converted my dates, the are not JSON serializable. So, now I have to put them back into a string again
for dt_to_cmpr in dict_list:
dt_to_cmpr['date_key'] = dt_to_cmpr['date_key'].strftime("%Y-%m-%d")
I cleaned up the above for simplicity, but that should all work. When it gets to Django, the view is going to covert them all back to date time again for use in a template.
Can I have Python just treat my date strings as time for the 90 day comparison, but leave them alone. Or, maybe have JSON use the Python date times? That much iteration every page load is slow, and can't be the best way.
The main problem is the way you're storing the datetimes. You should probably be storing them as actual datetimes in your database, not strings. You can't do date queries on string fields. Instead, you have to use the inefficient method of querying all the records and then filtering all of them in python after the fact. Database data types were created for a reason, use them.
There's no reason to convert datetimes to strings except at the very last moment when you need to format it for json or html, and the only bit of code that should need to do that is the Django app. That means:
Your Django app should almost entirely be using datetimes. It only coverts to strings when it needs to render out html or json.
Your API should only use python datetimes.
Your database should only use datetimes as well.
If you don't control the database, the best case is going to be 2 conversions
string -> datetime when pulling data out of the database.
datetime -> string when serializing to html or json.
If you can fix the database, then you only need to do the 2nd conversion.
I have having a problem with inserting date values into an SQL query. I am using sqlite3 and python. The query is:
c.execute("""SELECT tweeterHash.* FROM tweeterHash, tweetDates WHERE
Date(tweetDates.start) > Date(?) AND
Date(tweetDates.end) > Date(?)""",
(start,end,))
The query doesn't return any values, and there is no error message. If I use this query:
c.execute("""SELECT tweeterHash.* FROM tweeterHash, tweetDates WHERE
Date(tweetDates.start) > Date(2014-01-01) AND
Date(tweetDates.end) > Date(2015-01-01)""")
Then I get the values that I want, which is as expected?
The values start and end come from a text file:
f = open('dates.txt','r')
start = f.readline().strip('\n')
end = f.readline().strip('\n')
but I have also just tried declaring it as well:
start = '2014-01-01'
end = '2015-01-01'
I guess I don't understand why passing the string in from the start and end variables doesn't work? What is the best way to pass a date variable into a SQL query? Any help is greatly appreciated.
These aren't the same dates—and it's the non-parameterized ones you've got wrong.
Date(2014-01-01) calculates the arithmetic expression 2014 - 01 - 01, then constructs a Date from the resulting number 2012, which will get you something in 4707 BC.
Date('2014-01-01'), or Date(?) where the parameter is the string '2014-01-01', constructs the date you want, in 2014 AD.
You can see this more easily by just selecting dates directly:
>>> cur.execute('SELECT Date(2014-01-01), Date(?)', ['2014-01-01'])
>>> print(cur.fetchone())
('-4707-05-28', '2014-01-01')
Meanwhile:
What is the best way to pass a date variable into a SQL query?
Ideally, use actual date objects instead of strings. The sqlite3 library knows how to handle datetime.datetime and datetime.date. And don't call Date on the values, just compare them. (Yes, sqlite3 might then compare them as strings instead of dates, but the whole point of using ISO8601-like formats is that this always gives the same result… unless of course you have a bunch of dates from 4707 BC lying around.) So:
start = datetime.date(2014, 1, 1)
end = datetime.date(2015, 1, 1)
c.execute("""SELECT tweeterHash.* FROM tweeterHash, tweetDates WHERE
tweetDates.start > ? AND
tweetDates.end > ?""",
(start,end,))
And would this also mean that when I create the table, I would want: " start datetime, end datetime, "?
That would work, but I wouldn't do that. Python will convert date objects to ISO8601-format strings, but not convert back on SELECT, and SQLite will let you transparently compare those strings to the values returned by the Date function.
You could get the same effect with TEXT, but I believe you'd find it less confusing, DATETIME will set the column affinity to NUMERIC, which can confuse both humans and other tools when you're actually storing strings.
Or you could use the type DATE—which is just as meaningless to SQLite as DATETIME, but it can tell Python to transparently convert return values into datetime.date objects. See Default adapters and converters in the sqlite3 docs.
Also, if you haven't read Datatypes in SQLite Version 3 and SQLite and Python types, you really should; there are a lot of things that are both surprising (even—or maybe especially—if you've used other databases), and potentially very useful.
Meanwhile, if you think you're getting the "right" results from passing Date(2014-01-01) around, that means you've actually got a bunch of garbage values in your database. And there's no way to fix them, because the mistake isn't reversible. (After all, 2014-01-01 and 2015-01-02 are both 2012…) Hopefully you either don't need the old data, or can regenerate it. Otherwise, you'll need some kind of workaround that lets you deal with existing data as usefully as possible under the circumstances.
I have a SQL Server table that contains some datetime data. I'm using a Python script with pymssql to connect to the database and execute queries.
The problem I'm having is that when my queries return datetime data, the dates are shifted and are no longer accurate. For example, one data entry in my table is for the date '2012-03-27', but when I retrieve it and store it in my script as a datetime object in Python, the date is stored as '2012-01-03'.
Any ideas on how to fix this? Should I be using a library like pytz?
Thanks!
I am having the same issue with pymssql version 1.02. Although its not the cleanest aproach, I am using the workaround suggested by the OP and converting the string value to a python datetime object with the correct values using the dateutil.parser module. Here is the code I am using, provided that iso_datetime_string contains the string with the datetime in ISO format, as returned by the workaround:
import dateutil.parser as ps
correct_datetime_obj = ps.parse(iso_datetime_string)
I'm still not sure what is causing this problem, but I found a solution: convert the datetime data to type varchar when the query is performed:
select convert(varchar(20),event_time,120) from event_detail;
Previously I was performing this query:
select event_time from event_detail;
where event_time is of type datetime