I'm using sqlalchemy to query a database and have a Table.c.Field object from inspect, where Field is of type integer (e.g., 1596657600). How do I cast that field to a datetime? In the query I want to cast Field as datetime, then later extract and groupby dayofweek (or some other aspect of datetime). But first I need to cast Field as a date or datetime.
I have tried a few ways, but all fail. In the code below I first import sqlalchemy as sa.
I tried
sa.func.to_timestamp(Table.c.Field / 1.0).cast(sa.Date)
as suggested here. But it produces the error: no such function: to_timestamp.
I tried
Table.c.Field.cast(sa.DateTime)
as suggested here. But as in that link it produces the error: Couldn't parse datetime string '1596657600' - value is not a string. If it helps, the database I am querying is also Sqlite, but I would like my solution to work for any integer field from any database.
I also tried
sa.cast(Table.c.Field, SQA.Interval)
but it produced the same error: Couldn't parse datetime string '1596657600' - value is not a string.
If the solution is to create some special function (e.g., Fx) to do the conversion, to be used as
(Table.c.Field).Fx()
what would that function look like? Or what kind of function would do the trick? I cannot write to the database, and need to do all calculations (casting, grouping, filtering, etc.) via a single sqlalchemy query.
I found a solution in the docs, but I wasn't sure at first how to apply it in my case.
First, create a new type that accepts integers, then simply cast Field to that type. I first import sqlalchemy as SQA and import tzinfo and datetime and timedelta from datetime.
class MyEpochType(SQA.types.TypeDecorator):
impl = SQA.types.Integer
epoch = datetime(1970, 1, 1, 0, 0, 0, tzinfo=pytz.timezone('UTC'))
def process_bind_param(self, value, dialect):
pass # not needed in my use case
def process_result_value(self, value, dialect):
# either of the two returns below, or similar ones
#return self.epoch + timedelta(seconds=value )
return datetime.fromtimestamp(value).isoformat()
Table.c.Field.cast(MyEpochType)
The above solution will not allow for a group_by on the transformed values (e.g., a datetime or string). For this, a better solution is to use the native functions of the underlying database. In my case it is Sqlite, and would be called similar to that below:
SQA.func.strftime('%H%M',
SQA.func.datetime(Table.c.Field, 'unixepoch')).label('someLabel')
Here as an example I am calling two different Sqlite functions and giving the result a label. Each database has its own set of particular functions that can be called.
Related
class Test(Model):
time = DateTimeField()
# ...
row = Test.select()[0]
test.time
This returns a string that looks like this: 2017-01-23 01:01:39+01:00. How can I get it as a datetime object instead? Do I have to parse it manually?
Also I would be interested if there is any documentation on how to use the DateTimeField. The official documentation doesn't have anything on it.
Are you using SQLite? If so, SQLite doesn't have a dedicated datetime type, so datetimes are stored as strings in the DB. What peewee will do is recognize certain datetime formats coming out of the DB and convert them to datetime objects. What you need to do is ensure that either:
When you create/save your object, that you assign a datetime object to the field.
When reading back pre-existing data, that the data is in a recognized format.
The formats peewee supports out-of-the-box for datetime field are:
YYYY-mm-dd HH:MM:SS.ffffff
YYYY-mm-dd HH:MM:SS
YYYY-mm-dd
It looks like your has zone info. I'd suggest converting to UTC and dropping the zone info. That should fix it.
Have you tried adding a default like this?
time = DateTimeField(default=datetime.datetime.now())
Or when adding an entry add it as a datetime.datetime object directly:
test = Test(....., time=datetime.datetime.strptime("2018-3-15", '%Y-%m-%d'))
In the second case you don't need to specify anything in the class definition...
I am moving some of my code onto sqlalchemy from using raw MySQL queries.
The current issue I am having is that the datetime was saved in a string format by a C# tool. Unfortunately, the representation does not match up with Python's (as well as that it has an extra set of single quotes), thus making filtering somewhat cumbersome.
Here is an example of the format that the date was saved in:
'2016-07-01T17:27:01'
Which I was able to convert to a usable datetime using the following MySQL command:
STR_TO_DATE(T.PredicationGeneratedTime, \"'%%Y-%%m-%%dT%%H:%%i:%%s'\")
However, I cannot find any documentation that describes how to invoke built-in functions such as STR_TO_DATE when filtering with sqlalchemy
The following Python code:
session.query(Train.Model).filter(cast(Train.Model.PredicationGeneratedTime, date) < start)
is giving me:
TypeError: Required argument 'year' (pos 1) not found
There does not seem to be a way to specify the format for the conversion.
Note: I realize the solution is to fix the way the datetime is stored, but in the mean time I'd like to run queries against the existing data.
You can try to use func.str_to_date(COLUMN, FORMAT_STRING) instead of cast
In the cast() you should be using sqlalchemy.DateTime, not (what I assume is) a datetime.date - that is the cause of the exception.
However, fixing that will not really help because of the embedded single quotes.
You are fortunate that the dates stored in your table are in ISO format. That means that lexicographic comparisons will work on the date strings themselves, without casting. As long as you use a string for start with the surrounding single quotes, it will work.
from datetime import datetime
start = "'{}'".format(datetime.now().isoformat())
session.query(Train.Model).filter(Train.Model.PredicationGeneratedTime < start)
I have having a problem with inserting date values into an SQL query. I am using sqlite3 and python. The query is:
c.execute("""SELECT tweeterHash.* FROM tweeterHash, tweetDates WHERE
Date(tweetDates.start) > Date(?) AND
Date(tweetDates.end) > Date(?)""",
(start,end,))
The query doesn't return any values, and there is no error message. If I use this query:
c.execute("""SELECT tweeterHash.* FROM tweeterHash, tweetDates WHERE
Date(tweetDates.start) > Date(2014-01-01) AND
Date(tweetDates.end) > Date(2015-01-01)""")
Then I get the values that I want, which is as expected?
The values start and end come from a text file:
f = open('dates.txt','r')
start = f.readline().strip('\n')
end = f.readline().strip('\n')
but I have also just tried declaring it as well:
start = '2014-01-01'
end = '2015-01-01'
I guess I don't understand why passing the string in from the start and end variables doesn't work? What is the best way to pass a date variable into a SQL query? Any help is greatly appreciated.
These aren't the same dates—and it's the non-parameterized ones you've got wrong.
Date(2014-01-01) calculates the arithmetic expression 2014 - 01 - 01, then constructs a Date from the resulting number 2012, which will get you something in 4707 BC.
Date('2014-01-01'), or Date(?) where the parameter is the string '2014-01-01', constructs the date you want, in 2014 AD.
You can see this more easily by just selecting dates directly:
>>> cur.execute('SELECT Date(2014-01-01), Date(?)', ['2014-01-01'])
>>> print(cur.fetchone())
('-4707-05-28', '2014-01-01')
Meanwhile:
What is the best way to pass a date variable into a SQL query?
Ideally, use actual date objects instead of strings. The sqlite3 library knows how to handle datetime.datetime and datetime.date. And don't call Date on the values, just compare them. (Yes, sqlite3 might then compare them as strings instead of dates, but the whole point of using ISO8601-like formats is that this always gives the same result… unless of course you have a bunch of dates from 4707 BC lying around.) So:
start = datetime.date(2014, 1, 1)
end = datetime.date(2015, 1, 1)
c.execute("""SELECT tweeterHash.* FROM tweeterHash, tweetDates WHERE
tweetDates.start > ? AND
tweetDates.end > ?""",
(start,end,))
And would this also mean that when I create the table, I would want: " start datetime, end datetime, "?
That would work, but I wouldn't do that. Python will convert date objects to ISO8601-format strings, but not convert back on SELECT, and SQLite will let you transparently compare those strings to the values returned by the Date function.
You could get the same effect with TEXT, but I believe you'd find it less confusing, DATETIME will set the column affinity to NUMERIC, which can confuse both humans and other tools when you're actually storing strings.
Or you could use the type DATE—which is just as meaningless to SQLite as DATETIME, but it can tell Python to transparently convert return values into datetime.date objects. See Default adapters and converters in the sqlite3 docs.
Also, if you haven't read Datatypes in SQLite Version 3 and SQLite and Python types, you really should; there are a lot of things that are both surprising (even—or maybe especially—if you've used other databases), and potentially very useful.
Meanwhile, if you think you're getting the "right" results from passing Date(2014-01-01) around, that means you've actually got a bunch of garbage values in your database. And there's no way to fix them, because the mistake isn't reversible. (After all, 2014-01-01 and 2015-01-02 are both 2012…) Hopefully you either don't need the old data, or can regenerate it. Otherwise, you'll need some kind of workaround that lets you deal with existing data as usefully as possible under the circumstances.
I have chapter times in the form of HH:MM:SS. I am parsing them from a document, and I will have times as a string in the format of '00:12:14'. How would I store this in a mysql column, and then retrieve it in the required format to be able to:
1) order by time;
2) convert to a string in the above format.
I suggest you look at the MySQL time type. It will allow you to sort and format as you wish.
http://dev.mysql.com/doc/refman/5.0/en/time.html
Use the TIME type.
It allows "time values to be represented in several formats, such as quoted strings or as numbers, depending on the exact type of the value and other factors." In addition, you can perform various functions to manipulate the time.
If I have such a simple task, I choose a simple solution: I would choose the python datetime.time module (see: datetime.time) and store a TIME object using strftime.
Loading it back in is a little painful as you would have to split your string at : and then pass the values to the time constructor. Example:
def load(timestr):
hours,minutes,seconds = timestr.split(":")
return datetime.time(hours,minutes,seconds)
Hope this helps.
In my MySQL database I have dates going back to the mid 1700s which I need to convert somehow to ints in a format similar to Unix time. The value of the int isn't important, so long as I can take a date from either my database or from user input and generate the same int. I need to use MySQL to generate the int on the database side, and python to transform the date from the user.
Normally, the UNIX_TIMESTAMP function, would accomplish this in MySQL, but for dates before 1970, it always returns zero.
The TO_DAYS MySQL function, also could work, but I can't take a date from user input and use Python to create the same values as this function creates in MySQL.
So basically, I need a function like UNIX_TIMESTAMP that works in MySQL and Python for dates between 1700-01-01 and 2100-01-01.
Put another way, this MySQL pseudo-code:
select 1700_UNIX_TIME(date) from table;
Must equal this Python code:
1700_UNIX_TIME(date)
I don't have MySQL here installed, but when I look here: http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_to-days - I see an example TO_DAYS('2008-10-07') returning 733687.
The following Python function returns datetime(2008,10,7).toordinal() = 733322, which is 365 less than the MySQL's output.
So take this:
from datetime import datetime
query = '2008-10-07'
nbOfDays = datetime.strptime(query, '%Y-%m-%d').toordinal() + 365
and it should work for dates between 1700 and 2100.
According to the link that you gave,
Given a date date, returns a day number (the number of days since year 0).
mysql> SELECT TO_DAYS(950501);
-> 728779
mysql> SELECT TO_DAYS('2007-10-07');
-> 733321
Corresponding numbers in Python:
>>> import datetime
>>> datetime.date(1995,5,1).toordinal()
728414
>>> datetime.date(2007,10,7).toordinal()
732956
So the relationship is : mySQL_int == Python_int + 365 and you can convert in the other direction by using the fromordinal class method:
>>> datetime.date.fromordinal(728779 - 365)
datetime.date(1995, 5, 1)