Found this issues working with timestamp conversions where the .timestamp() seem to compensate for time offset even though I created the datetime object with utcnow() method. So what is the reason for the utcnow() method if it actually fails when converting to timestamp later on?
def test_timestamp_issues():
now = datetime.now()
utc_now = datetime.utcnow()
utc_now_with_tz = datetime.now(timezone.utc)
print(f"Now: {now} -- As Timestamp: {now.timestamp()}")
print(f"Utc Now: {utc_now} -- As TimeStamp: {utc_now.timestamp()} "
f"with builtin: {datetime.timestamp(utc_now)}, with setting timezone hard: "
f"{utc_now.replace(tzinfo=timezone.utc).timestamp()}")
print(f"Utc Now: {utc_now_with_tz} -- As TimeStamp: {utc_now_with_tz.timestamp()} "
f"with builtin: {datetime.timestamp(utc_now_with_tz)}, with setting timezone hard: "
f"{utc_now_with_tz.replace(tzinfo=timezone.utc).timestamp()}")
Output:
Now: 2022-04-25 09:12:16.881608 -- As Timestamp: 1650870736.881608
Utc Now: 2022-04-25 07:12:16.881613 -- As TimeStamp: 1650863536.881613 with builtin: 1650863536.881613, with setting timezone hard: 1650870736.881613
Utc Now: 2022-04-25 07:12:16.881753+00:00 -- As TimeStamp: 1650870736.881753 with builtin: 1650870736.881753, with setting timezone hard: 1650870736.881753
The expected timestamp here should be 1650870736.881608 but for some reason converting the utcnow() gives 2 hours backward again.
So for some reason I get another 2 hours withdrawal when taking timestamp of a utcnow() created datetime object. I am currently running a macbook m1 pro, with Timezone Norway (+1, +2 atm with daylight), and python 3.9.12
So can this be the arm architecture messing up things or is it just not recommended to use the utcnow() and then convert to timestamp?
The answer to your question is managing timezones (TZ). When you create a datetime instance with now() or utcnow(), it does not store info about TZ. And, according to python docs for utcnow() (bolding by author):
Return the current UTC date and time, with tzinfo None.
This is like now(), but returns the current UTC date and time, as a naive datetime object. An aware current UTC datetime can be obtained by calling datetime.now(timezone.utc). See also now().
Warning: Because naive datetime objects are treated by many datetime methods as local times, it is preferred to use aware datetimes to represent times in UTC. As such, the recommended way to create an object representing the current time in UTC is by calling datetime.now(timezone.utc).
This means that, although you received time for UTC TZ with utcnow(), since no TZ info is stored (naive datetime object), python thinks of it as a local time. And, when converting it to timestamp, will add/subtract time difference once again. Proof for this weird behaviour may be found in python docs for timestamp() method:
Naive datetime instances are assumed to represent local time
The reason actually is the one, you stated in the title of the question, not the ARM architecture
I'm trying to perform a, to me, simple task of generating a current date/time combo at a speficic time zone. All I see is suggestions to use pytz even though datetime includes the tzinfo class to deal with timezones. However, if I try to use tzinfo, it does not work:
>>> from datetime import datetime, tzinfo
>>> d = datetime.now(tzinfo.tzname("EDT"))
TypeError: descriptor 'tzname' requires a 'datetime.tzinfo' object but received a 'str'
The docs say you can use a time zone name like "EDT" or "GMT". What's wrong with this?
The function tzinfo.tzname does the opposite of what you think it does.
It takes a datetime object and returns a string indicating the time zone.
Is there a way I can obtain a datetime aware object out of a string in Python using only the standard library modules ?
I know that I can use dateutil.parser.parse, but unfortunately that's not a good enough reason to add it as a dependency to my project. I already have the mx.DateTime module as a dependency, buuttttt:
>>> dateutil.parser.parse('2011-10-24T06:51:47-07:00')
datetime.datetime(2011, 10, 24, 6, 51, 47, tzinfo=tzoffset(None, -25200))
>>> mx.DateTime.ISO.ParseDateTimeUTC('2011-10-24T06:51:47-07:00')
<mx.DateTime.DateTime object for '2011-10-24 13:51:47.00' at 29c7e48>
the ParseDateTimeUTC fails to detect the offset, even though in its documentation says that:
Returns a DateTime instance in UTC reflecting the given ISO
date. A time part is optional and must be delimited from the
date by a space or 'T'. Timezones are honored.
mx.DateTime.ISO.ParseDateTimeUTC is doing the right thing - it is applying the specified timezone to adjust the time to UTC. The resulting UTC time doesn't have a timezone because it isn't a local time anymore.
The standard Python library doesn't contain any concrete timezone classes according to the documentation:
tzinfo is an abstract base clase, meaning that this class should not
be instantiated directly. You need to derive a concrete subclass, and
(at least) supply implementations of the standard tzinfo methods
needed by the datetime methods you use. The datetime module does not
supply any concrete subclasses of tzinfo.
I've always been surprised that they didn't at least include a class for UTC, or a generic implementation like dateutil's tzoffset.
I'm trying to make one of my libraries compatible with Python 2.6. It uses the method datetime.timedelta.total_seconds which was added in 2.7, so to make it work with 2.6, I wanted to monkey patch it into timedelta like this:
import datetime
if not hasattr(datetime.timedelta, 'total_seconds'):
class timedelta(datetime.timedelta):
def total_seconds(self):
return self.days * 86400.0 + self.seconds + self.microseconds * 1e-6
datetime.timedelta = timedelta
This does actually work in some cases, e.g. if I create a timedelta variable, it does have this method. But if a new timedelta object is produced by subtracting two datetime.datetime values, the method is missing from the resulting object.
What's going wrong and how do I fix this?
The datetime module is written entirely in C.
This includes the function that subtracts one datetime.datetime object from another. That function -- called new_delta() -- directly instantiates PyDateTime_DeltaType, which is the original datetime.timedelta class (written in C). It won't notice that you've rebound datetime.timedelta to your own class in Python-land.
I don't see any easy workarounds.
When I write with business logic, my code often depends on the current time. For example the algorithm which looks at each unfinished order and checks if an invoice should be sent (which depends on the no of days since the job was ended). In these cases creating an invoice is not triggered by an explicit user action but by a background job.
Now this creates a problem for me when it comes to testing:
I can test invoice creation itself easily
However it is hard to create an order in a test and check that the background job identifies the correct orders at the correct time.
So far I found two solutions:
In the test setup, calculate the job dates relative to the current date. Downside: The code becomes quite complicated as there are no explicit dates written anymore. Sometimes the business logic is pretty complex for edge cases so it becomes hard to debug due to all these relative dates.
I have my own date/time accessor functions which I use throughout my code. In the test I just set a current date and all modules get this date. So I can simulate an order creation in February and check that the invoice is created in April easily. Downside: 3rd party modules do not use this mechanism so it's really hard to integrate+test these.
The second approach was way more successful to me after all. Therefore I'm looking for a way to set the time Python's datetime+time modules return. Setting the date is usually enough, I don't need to set the current hour or second (even though this would be nice).
Is there such a utility? Is there an (internal) Python API that I can use?
Monkey-patching time.time is probably sufficient, actually, as it provides the basis for almost all the other time-based routines in Python. This appears to handle your use case pretty well, without resorting to more complex tricks, and it doesn't matter when you do it (aside from the few stdlib packages like Queue.py and threading.py that do from time import time in which case you must patch before they get imported):
>>> import datetime
>>> datetime.datetime.now()
datetime.datetime(2010, 4, 17, 14, 5, 35, 642000)
>>> import time
>>> def mytime(): return 120000000.0
...
>>> time.time = mytime
>>> datetime.datetime.now()
datetime.datetime(1973, 10, 20, 17, 20)
That said, in years of mocking objects for various types of automated testing, I've needed this approach only very rarely, as most of the time it's my own application code that needs the mocking, and not the stdlib routines. After all, you know they work already. If you are encountering situations where your own code has to handle values returned by library routines, you may want to mock the library routines themselves, at least when checking how your own app will handle the timestamps.
The best approach by far is to build your own date/time service routine(s) which you use exclusively in your application code, and build into that the ability for tests to supply fake results as required. For example, I do a more complex equivalent of this sometimes:
# in file apptime.py (for example)
import time as _time
class MyTimeService(object):
def __init__(self, get_time=None):
self.get_time = get_time or _time.time
def __call__(self):
return self.get_time()
time = MyTimeService()
Now in my app code I just do import apptime as time; time.time() to get the current time value, whereas in test code I can first do apptime.time = MyTimeService(mock_time_func) in my setUp() code to supply fake time results.
Update: Years later there's an alternative, as noted in Dave Forgac's answer.
The freezegun package was made specifically for this purpose. It allows you to change the date for code under test. It can be used directly or via a decorator or context manager. One example:
from freezegun import freeze_time
import datetime
#freeze_time("2012-01-14")
def test():
assert datetime.datetime.now() == datetime.datetime(2012, 1, 14)
For more examples see the project: https://github.com/spulec/freezegun
You can patch the system, by creating a custom datetime module (even a fake one - see example below) acting as a proxy and then insert it in sys.modules dictionary. From there on, each import to the datetime module will return your proxy.
There is still the caveat of datetime class, especially when someone does from datetime import datetime; for that, you can simply add another proxy only for that class.
Here is an example of what I am saying - of course it is just something I've thrown in 5 minutes, and may have several issues (for instance, the type of datetime class is not correct); but hopefully it may already be of use.
import sys
import datetime as datetime_orig
class DummyDateTimeModule(sys.__class__):
""" Dummy class, for faking datetime module """
def __init__(self):
sys.modules["datetime"] = self
def __getattr__(self, attr):
if attr=="datetime":
return DummyDateTimeClass()
else:
return getattr(datetime_orig, attr)
class DummyDateTimeClass(object):
def __getattr__(self, attr):
return getattr(datetime_orig.datetime, attr)
dt_fake = DummyDateTimeModule()
Finally - is it worth?
Frankly speaking, I like our second solution much more than this one :-).
Yes, python is a very dynamic language, where you can do quite a lot of interesting things, but patching code in this way has always a certain degree of risk, even if we are talking here of test code.
But mostly, I think the accessory function would make test patching more explicit, and also your code would be more explicit in terms of what it is going to be tested, thus increasing readability.
Therefore, if the change is not too expensive, I would go for your second approach.
I would use the helpers from the 'testfixtures' package to mock out the date, datetime or time calls you're making:
http://packages.python.org/testfixtures/datetime.html
Well one way to do it is to dynamic patch the time /datetime module
something like
import time
import datetime
class MyDatetime:
def now(self):
return time.time()
datetime.datetime = MyDatetime
print datetime.datetime().now()
there might be few ways of doing this, like creating the orders (with the current timestamp) and then changing that value in the DB directly by some external process (assuming data is in the DB).
I'll suggest something else. Have you though about running your application in a virtual machine, setting the time to say Feb, creating orders, and then just changing the VMs time? This approach is the closest as you can get to the real-life situation.