I've read a bunch of posts on how flaky parsing time can be. I believe I have come up with a reliable way of converting an ISO8601-formatted timestamp here:
https://gist.github.com/3702066
The most important part being the astimezone(LOCALZONE) call when the date is parsed. This allowed time.mktime() to do the right thing and appears to handle daylight savings properly.
Are there obvious gotchas I've missed?
Your code does seem to work even for times that fall just before or just after daylight savings time transitions, but I am afraid it might still fail on those rare occasions when a location's timezone offset actually changes. I don't have an example to test with though.
So even if if does work (or almost always work), I think it's crazy to convert a UTC time string to a UTC timestamp in a manner which involves or passed through local time in any way. The local time zone should be irrelevant. It's an unwanted dependency. I'm not saying that you're crazy. You're just trying to work with the APIs you are given, and the C library's time APIs are badly designed.
Luckily, Python provides an alternative to mktime() that is what the C library should have provided: calendar.timegm(). With this function, I can rewrite your function like this:
parsed = parse_date(timestamp)
timetuple = parsed.timetuple()
return calendar.timegm(timetuple)
Because local time is not involved, this also removes the dependency on pytz and the nagging doubt that an obscure artifact of somebody's local timezone will cause an unwanted effect.
Related
There are several Python packages that implement the datetime.tzinfo interface, including pytz and dateutil. If someone hands me a timezone object and wants me to apply it to a datetime, the procedure is different depending on what kind of timezone object it is:
def apply_tz_to_datetime(dt: datetime.datetime, tz: datetime.tzinfo, ambiguous, nonexistent):
if isinstance(tz, dateutil.tz._common._tzinfo):
# do dt.replace(tz, fold=...)
elif isinstance(tz, pytz.tzinfo.BaseTzInfo):
# do tz.localize(dt, is_dst=...)
# other cases here
(The dateutil.tz case is a lot more complicated than I've shown, because there are a lot of cases to consider for non-existent or ambiguous datetimes, but the gist is always to either call dt.replace(tz, fold=...) or raise an exception.)
Checking dateutil.tz._common._tzinfo seems like a no-no, though, is there a better way?
I apologize for having to be the guy to say, "You shouldn't be doing that in the first place", but you indeed should not be trying to detect whether a time zone is a dateutil zone, you should instead just use it as a tzinfo object.
From your comments, it seems like the only reason you want to detect this is because pytz has a separate localization / normalization stage, but since pytz is the only library with this unusual interface, it should suffice to detect pytz zones.
As I mentioned in my comment on the dateutil issue, my recommendations are to either:
Not support pytz at all, if that is possible. It is effectively legacy software at this point, and if you have a new library you at least don't have any users who are already expecting to use it with pytz.
If that is not feasible, something like pytz-deprecation-shim might be a useful abstraction here. For a new library, I wouldn't recommend introducing time zones like those provided that also expose a pytz-like interface, but the helper functions (which don't require a dependency on pytz!) can be profitably used or re-implemented to either detect pytz zones or seamlessly upgrade them to their modern equivalents. You could also use this in combination with #1 by detecting if a zone is a pytz zone and throwing an error.
In any case, there is no particular reason to enumerate all the different time zone providers, since all except pytz use the standard interface.
It appears from the ratio of comments to answers (currently 9/0 = ∞), there is no available answer to the surface-level question (how to determine whether something is a dateutil.tz-style timezone object). I'll open a feature request ticket with the maintainers of the library.
I am developing an application to manage shifts of operation of Internet of Things devices which can be anywhere on the planet using different time zones, I am using python, mongodb and pymongo.
As we all know, when we store a date in mongodb it automatically converts the date of our time zone to UTC by changing the time but keeping the same absolute value below.
The problem I have is that when I query the database to get the date using pymongo I get exactly the same date and time that is stored in mongodb in UTC but in my time zone without having made the conversion change;
for example, Let's suppose I create a datetime in python using the "datetime" library:
MY_TIMEZONE = pytz.timezone('America/Bogota')
datetime.now().astimezone(MY_TIMEZONE)
and I get this date in my time zone:
'2021-02-25 00:00:00-05:00'
look at -05:00 at the end
when I save that date in my mongodb collection in atlas, it transforms the date to UTC which looks like this:
'2021-02-25 05:00:00.000+00:00'
look at +00:00 at the end
So far so good, the problem comes when I query the database to obtain that same date using pymongo method "find_one"
what i get is this:
'2021-02-25 05:00:00-05:00'
look at -05:00 at the end
the same time and date as UTC but in my time zone, which in theory should not happen, I should get the date in UTC or in the local time zone but with the change. That bug can be critical in the system I'm developing.
Does anyone know what could be happening, if it is some kind of bug in the current versions?
I am using python 3.8.5, pymongo 3.11.3 and mongodb 4.4.4
I have looked everywhere about this problem, but it seems that I am the only one that has happened, adding or subtracting the time difference in local is not an option, I guess I will have to choose to use timestamps to avoid this problem or another headache.
If your application is working with multiple time zones, you should configure your system (generally preferable) or your application to use UTC as the time zone.
If you do this, you have two time zones you generally need to keep track of: UTC, which is what every timestamp is stored in, and a local time zone, which is used for output and presentation.
If you don't do this (i.e. your system and/or application time zone is anything other than UTC), you need to keep track of 3 time zones and more than 3 possible combinations of how things can go wrong:
The time zone that your input is in (is it system time zone or the non-system local time?)
When you store a timestamp, is it provided in UTC, system local time or non-system local time?
etc.
If your application is working with a single time zone, AND all of your software is built for this correctly, you can generally assume all times are in local time and ignore time zones.
This last bit isn't always explicitly documented. For example in Ruby, the base driver timezone behavior is described here and it doesn't say that all times are returned in local time zone. Mongoid documentation has a more extensive treatise of time zones, because they are explicitly configurable.
For pymongo I suggest reviewing the entirety of documentation carefully looking for equivalent statements (and make sure you review the bson (de)serialization docs as well), and failing that you might need to read the source code to ascertain behavior.
In any event, working with multiple time zones when your system/application time zone is not UTC is really more difficult than it should be and I don't recommend it (if not for yourself, then whoever will have to work with your code after you).
In Python, I want a Python program to be able to determine the current date and time in NYC . Is that practical? While datetime.datetime.now() can tell me the local time, and datetime.utcnow() can tell me the UTC (GMT). However just looking at the difference will not help me as DST changes.
I try things like "dt=datetime.now() " and "dt.timetuple()"
I get tm_isdst=-1 even if I change the computer date.
I change my computer clock from a January date to a July date. I still get tm_isdst=-1
Why not use pytz? I want the users to not have to go thru the step of downloading an extra library.
I suspect some sort of problems in your use of the datetime, time, etc. modules, but without knowing more, not much help can be provided.
The following suggestion has some definite drawbacks, and I really recommend more pursuit to solving the problems with datetime, etc. However, if you're sure to have a web connection and need to get something done fast, you could query USNO time with something like:
import urllib
f = urllib.urlopen("http://tycho.usno.navy.mil/cgi-bin/timer.pl")
time_page = f.readlines()
for line in time_page:
if line.find("Eastern Time") != -1:
ny_time = line[4:24]
break
print ny_time
The output looks like:
Jan. 19, 05:18:04 PM
This makes use of the fact that NYC is in the Eastern Time zone. Also, it assumes the USNO server is available to your user. Furthermore, it has assumptions about the format of the content returned. I don't know if/how frequently that format changes. Also, if this is going to be used a lot, please find another server, as you don't want to sink the USNO server! (Pun not originally intended, but recognized and kept. :-).
If you are not in the same timezone as NYC, it's in practice impossible without knowing the timezone and when DST changes. You can't hardcode it for NYC, of course, but it is way easier to just install pytz or dateutil, and then you aren't limited to NYC.
I'm working on making a small ban system, and the snippet below will tell the client how much time of their ban is remaining.
The problem:
When you call Bans.timeleft_str(), rather then showing something less then a day, it will show the timestamp + 18 hours.
Snippet: http://pastebin.com/Zumn0tLv
This problem occurs if I change self.length = WEEK, etc. Rather then 7d 00h 00m, it will be 7d 18h 00m.
I originally tested this on my ubuntu vbox, and then tried it on my windows python shell, and still got the same result.
You may need to change self.timestamp to a time in the past.
Thanks in advance.
time.time, as the docs I just pointed to say, works in UTC (once known as "Greenwich" time, now "universal time coordinate"). mktime, again as said in its docs, takes as argument
9-tuple [...] which expresses the time in local time, not UTC.
strptime may work either way (but you're not supplying a timezone, so it's going to use local time).
So, overall, you're getting deep into timezone confusion;-).
I recommend (as always) that you standardize on UTC (the local timezone of your server can well not be the same as that of its users, after all), e.g. with a %Z directive in the format you pass to strptime and a corresponding timezone of 'UTC' (which is guaranteed to be recognized on all platforms) in the corresponding part of the string you're parsing.
I currently have setup a Python script that uses feedparser to read a feed and parse it. However, I have recently come across a problem with the date parsing. The feed I am reading contains <modified>2010-05-05T24:17:54Z</modified> - which comes up in Python as a datetime object - 2010-05-06 00:17:54. Notice the discrepancy: the feed entry was modified on the 5th of may, while python reads it as the 6th.
So the question is why this is happening. Is the ATOM feed (that is, the one who created the feed) wrong by putting the time as 24:17:54, or is my python script wrong in the way it treats it.
And can I solve this?
There are some interesting special cases in the rfc here (https://www.rfc-editor.org/rfc/rfc3339), however, typically its for the 00:00:60 vs 00:00:59 to allow for leap seconds. It may be though that that is legal. My guess is that its doing the "right thing". In all honesty, date/time things get really messy due to things like DST and local timezones. If its 24:17:54, that might be the right thing after all.
I think today at 24:17 is intelligently parsed as tomorrow at 00:17.... I'm thinking you are well handling the producer's bug.