We Keep Coding

How to cast integer (epoch) column to datetime in sqlalchemy?

I'm using sqlalchemy to query a database and have a Table.c.Field object from inspect, where Field is of type integer (e.g., 1596657600). How do I cast that field to a datetime? In the query I want to cast Field as datetime, then later extract and groupby dayofweek (or some other aspect of datetime). But first I need to cast Field as a date or datetime.
I have tried a few ways, but all fail. In the code below I first import sqlalchemy as sa.
I tried
sa.func.to_timestamp(Table.c.Field / 1.0).cast(sa.Date)
as suggested here. But it produces the error: no such function: to_timestamp.
I tried
Table.c.Field.cast(sa.DateTime)
as suggested here. But as in that link it produces the error: Couldn't parse datetime string '1596657600' - value is not a string. If it helps, the database I am querying is also Sqlite, but I would like my solution to work for any integer field from any database.
I also tried
sa.cast(Table.c.Field, SQA.Interval)
but it produced the same error: Couldn't parse datetime string '1596657600' - value is not a string.
If the solution is to create some special function (e.g., Fx) to do the conversion, to be used as
(Table.c.Field).Fx()
what would that function look like? Or what kind of function would do the trick? I cannot write to the database, and need to do all calculations (casting, grouping, filtering, etc.) via a single sqlalchemy query.

I found a solution in the docs, but I wasn't sure at first how to apply it in my case.
First, create a new type that accepts integers, then simply cast Field to that type. I first import sqlalchemy as SQA and import tzinfo and datetime and timedelta from datetime.
class MyEpochType(SQA.types.TypeDecorator):
impl = SQA.types.Integer
epoch = datetime(1970, 1, 1, 0, 0, 0, tzinfo=pytz.timezone('UTC'))
def process_bind_param(self, value, dialect):
pass # not needed in my use case
def process_result_value(self, value, dialect):
# either of the two returns below, or similar ones
#return self.epoch + timedelta(seconds=value )
return datetime.fromtimestamp(value).isoformat()
Table.c.Field.cast(MyEpochType)
The above solution will not allow for a group_by on the transformed values (e.g., a datetime or string). For this, a better solution is to use the native functions of the underlying database. In my case it is Sqlite, and would be called similar to that below:
SQA.func.strftime('%H%M',
SQA.func.datetime(Table.c.Field, 'unixepoch')).label('someLabel')
Here as an example I am calling two different Sqlite functions and giving the result a label. Each database has its own set of particular functions that can be called.

Query issue in pymongo

I am having some issues with finding documents with a today's date function through python.
I use the following function:
datetime.datetime.now().date().strftime('%Y-%m-%d')
which gives the following value: 2017-05-21
However, the values in my documents (mongo) have double strings attached around it like the following: "2017-05-21"
So literally filling in the above string ("2017-05-21") exactly in my pymongo query works like a charm. However, I need the dynamicness of the datetime function, but unfortunately, that doesn't match with the double quoted date strings which I need for the query.
Does anyone know any workaround? I have already tried a replace function etc. It either creates double quotes within the single quotes or doesn't do anything.

It sounds like your MongoDB documents were inserted incorrectly, using text for date fields instead of BSON datetimes.
PyMongo automatically converts between Python datetimes and BSON datetimes, so you can insert a document that contains a BSON datetime like this:
dt = datetime.datetime.utcnow()
collection.insert_one({'myDate': dt})
Prove that date comparisons work now like so:
# There is a document with myDate in the past, now.
print(collection.find_one({'myDate': {'$lt': datetime.datetime.utcnow()}}))
# No document with myDate in the future.
print(collection.find_one({'myDate': {'$gt': datetime.datetime.utcnow()}}))

Python SQLite, passing date values in sql query

I have having a problem with inserting date values into an SQL query. I am using sqlite3 and python. The query is:
c.execute("""SELECT tweeterHash.* FROM tweeterHash, tweetDates WHERE
Date(tweetDates.start) > Date(?) AND
Date(tweetDates.end) > Date(?)""",
(start,end,))
The query doesn't return any values, and there is no error message. If I use this query:
c.execute("""SELECT tweeterHash.* FROM tweeterHash, tweetDates WHERE
Date(tweetDates.start) > Date(2014-01-01) AND
Date(tweetDates.end) > Date(2015-01-01)""")
Then I get the values that I want, which is as expected?
The values start and end come from a text file:
f = open('dates.txt','r')
start = f.readline().strip('\n')
end = f.readline().strip('\n')
but I have also just tried declaring it as well:
start = '2014-01-01'
end = '2015-01-01'
I guess I don't understand why passing the string in from the start and end variables doesn't work? What is the best way to pass a date variable into a SQL query? Any help is greatly appreciated.

These aren't the same dates—and it's the non-parameterized ones you've got wrong.
Date(2014-01-01) calculates the arithmetic expression 2014 - 01 - 01, then constructs a Date from the resulting number 2012, which will get you something in 4707 BC.
Date('2014-01-01'), or Date(?) where the parameter is the string '2014-01-01', constructs the date you want, in 2014 AD.
You can see this more easily by just selecting dates directly:
>>> cur.execute('SELECT Date(2014-01-01), Date(?)', ['2014-01-01'])
>>> print(cur.fetchone())
('-4707-05-28', '2014-01-01')
Meanwhile:
What is the best way to pass a date variable into a SQL query?
Ideally, use actual date objects instead of strings. The sqlite3 library knows how to handle datetime.datetime and datetime.date. And don't call Date on the values, just compare them. (Yes, sqlite3 might then compare them as strings instead of dates, but the whole point of using ISO8601-like formats is that this always gives the same result… unless of course you have a bunch of dates from 4707 BC lying around.) So:
start = datetime.date(2014, 1, 1)
end = datetime.date(2015, 1, 1)
c.execute("""SELECT tweeterHash.* FROM tweeterHash, tweetDates WHERE
tweetDates.start > ? AND
tweetDates.end > ?""",
(start,end,))
And would this also mean that when I create the table, I would want: " start datetime, end datetime, "?
That would work, but I wouldn't do that. Python will convert date objects to ISO8601-format strings, but not convert back on SELECT, and SQLite will let you transparently compare those strings to the values returned by the Date function.
You could get the same effect with TEXT, but I believe you'd find it less confusing, DATETIME will set the column affinity to NUMERIC, which can confuse both humans and other tools when you're actually storing strings.
Or you could use the type DATE—which is just as meaningless to SQLite as DATETIME, but it can tell Python to transparently convert return values into datetime.date objects. See Default adapters and converters in the sqlite3 docs.
Also, if you haven't read Datatypes in SQLite Version 3 and SQLite and Python types, you really should; there are a lot of things that are both surprising (even—or maybe especially—if you've used other databases), and potentially very useful.
Meanwhile, if you think you're getting the "right" results from passing Date(2014-01-01) around, that means you've actually got a bunch of garbage values in your database. And there's no way to fix them, because the mistake isn't reversible. (After all, 2014-01-01 and 2015-01-02 are both 2012…) Hopefully you either don't need the old data, or can regenerate it. Otherwise, you'll need some kind of workaround that lets you deal with existing data as usefully as possible under the circumstances.

MySQL data type for movie times

I have chapter times in the form of HH:MM:SS. I am parsing them from a document, and I will have times as a string in the format of '00:12:14'. How would I store this in a mysql column, and then retrieve it in the required format to be able to:
1) order by time;
2) convert to a string in the above format.

I suggest you look at the MySQL time type. It will allow you to sort and format as you wish.
http://dev.mysql.com/doc/refman/5.0/en/time.html

Use the TIME type.
It allows "time values to be represented in several formats, such as quoted strings or as numbers, depending on the exact type of the value and other factors." In addition, you can perform various functions to manipulate the time.

If I have such a simple task, I choose a simple solution: I would choose the python datetime.time module (see: datetime.time) and store a TIME object using strftime.
Loading it back in is a little painful as you would have to split your string at : and then pass the values to the time constructor. Example:
def load(timestr):
hours,minutes,seconds = timestr.split(":")
return datetime.time(hours,minutes,seconds)
Hope this helps.

python sql interval

With PostgreSQL, one of my tables has an 'interval' column, values of which I would like to extract as something I can manipulate (datetime.timedelta?); however I am using PyGreSQL which seems to be returning intervals as strings, which is less than helpful.
Where should I be looking to either parse the interval or make PyGreSQL return it as a <something useful>?

Use Psycopg 2. It correctly converts between Postgres's interval data type and Python's timedelta.