I am currently working on Django project with a postgres DB.
The data stored in db with time-stamp using Naive time (user's local time).
However, in the setting.py, we have
USE_TZ = True
which means all the time-stamps retrieved by Django ORM are converted to UTC.
Generally, it is ok. However, for the function I am building right now does need the real time (uesr's local time). Certainly, I would fetch the data and convert the time to whatever time I want, with two problems:
1. I can convert time-stamps to EST or anything, but I still don't know the original time;
2. I want to do the converting during the ORM query rather than after, since it will be more efficient.
Does anyone have a clue about this?
Thanks in advance!
1) If the original datetimes really are naive then I would assume they got stored as whatever timezone you have set in your TIME_ZONE setting (defaults to 'America/Chicago' but might be something else in your case). So converting back to that timezone will probably give you the original time. From the django docs:
"When USE_TZ is False, this is the time zone in which Django will
store all datetimes. When USE_TZ is True, this is the default time
zone that Django will use to display datetimes in templates and to
interpret datetimes entered in forms."
(https://docs.djangoproject.com/en/1.6/ref/settings/#std:setting-TIME_ZONE)
Normally the timezone information is actually set on the db connection (https://docs.djangoproject.com/en/dev/ref/databases/#optimizing-postgresql-s-configuration), so it's expected that you don't get your datetime in UTC when connecting to postgres through psql since the timezone used will default to your system's timezone, I'm not sure why this doesn't happen with raw queries in django though.
2) I can't say I've done many hours of research, however I believe that setting USE_TZ kind of dooms you to have to convert post-query. You can possibly override the connection timezone, but I don't know of an easy way to do this at runtime since it will default to UTC due to USE_TZ.
On previous projects I've worked on and ran into similar issues we have either passed the responsibility over to the front-end as someone suggested in your comments(frontend returns UTC datetimes and converts data back from UTC) or we also store the user timezone and do a post-query conversion. This didn't prove to be inefficient in our use case.
Related
I have a Django app which needs to support different time zones. I'm saving events to my database which have timestamps and based on those timestamps I want to send notifications. Because the user can move into different time zone, I want to store timestamps as UTC time and convert them to user current time zone later. If I'm correct, PostgreSQL converts datetimes to UTC automatically? At the moment I get an error RuntimeWarning: DateTimeField event.timestamp received a naive datetime (2022-02-15 15:00:00) while time zone support is active.. I used to use Django's make_aware when saving the timestamp and didn't get this warning then. The questions I'd like to get an answer are:
Should I save aware datetimes to the database if I want to support multiple time zones?
What should I use in variables TIME_ZONE and USE_TZ in settings.py? UTC and True?
I store the time zone info to the database, what is the correct method to activate it when I want to get the local time for the user? django.utils.timezone.activate + get_current_timezone or is there some function like astimezone(string_from_db)?
Should I save aware datetimes to the database if I want to support multiple time zones?
The thing that helped me understand timezones is the fact, that time when presented as unix timestamp is a constant and does not depend on where the user is on the globe. Timestamp is saved into Postgres as such. Timezone is just an optional extra information, nothing more. One can take unix timestamp number and transform it into Timezone A, B or whatever he wishes. This depends totally on the request.
Storing timezone information to the database only says that you have to provide timezone when persisting model attribute, otherwise you will get that warning. Warning means that database doesn't know how to treat it and will consider it to be in default timezone, which is usually wrong.
I find it very convenient to store timezone information into the database, but not because it "supports multiple time zones", but because it is less confusing and cleaner. Your users are already in multiple time zones and they are all expecting to see time in their own timezone. If you use timezones, then you don't need to convert to and from UTC and ponder if this number is already converted or not.
I faced issues such as: when a user creates an item, after a single second he sees that an item was created 2 hours ago because you did not include timezone in creation time.
What should I use in variables?
USE_TZ = True
TIME_ZONE = "UTC"
In Django 5.0 the USE_TZ will become True by default.
what is the correct method to get the local time for the user
If you return time values with timezone, the JavaScript easily transforms them to user's timezone by default. If you are using Django templates and all times should appear in different timezone than the user is in, then it is a bit more involved: need to save user's preferred timezone in his profile and activate it with timezone.activate as you correctly pointed out.
I am using flask sqlalchemy and postgreSQL and I have issues with the displayed datetimes, while investigating this issue I found an other weird thing:
Creating a DB entry in incognito mode (chrome browser tab) gives a different/wrong time.
EDIT: It has nothing to do with incognito mode, both cases happen in normal mode aswell. I have not figured out yet why.
This is the code:
I changed the default time zone of my database:
ALTER DATABASE postgres SET timezone TO 'Europe/Berlin';
the model:
class User(UserMixin, Base):
__tablename__ = 'users'
date_added = Column(DateTime(timezone=True), nullable=False)
The method I use to add datetime to DB:
date_added=datetime.today()
How it looks in the DB (my localtime at this time was 13:53:46):
Creating entry not in incognito
timestamp with time zone
2019-02-01 13:53:46.73817+01
Creating entry in incognito
timestamp with time zone
2019-02-01 12:53:46.73817+01
This really makes me worry. It is simply wrong. Even if I would convert the datetime objects to localtime. Both entries have been done at the same time, but show different results, how is this possible?
Also when viewing these dates in HTML, postgreSQL does not apply the offset, so the first date looks correct, but the second is wrong.
Initially I just wanted to find a way to store ALL datetime objects in Europe/Berlin and return them in Europe/Berlin time, so I dont have to convert UTC to Europe/Berlin, but now I think something went horribly wrong.
I also have double checked my code everywhere, I am not using other methods to manipulate datetime objects.
EDIT
I am saving a datetime everytime a user logs in. Currently I tried this on not incognito. My localtime is 14:13:33 BUT it saved into the DB:
2019-02-01 13:13:33.804339+01. How is this even possible? I know it cant be random, but right now it looks like its saving times randomly sometimes UTC with offset and sometimes Europe/Berlin with offset.
EDIT
I double checked all tables in question with SHOW timezone; and they all correctly return Europe/Berlin
datetime.today() returns a timestamp without timezone info of the current local time (the returned value is timezone naive). The root of your problem is that somewhere between SQL Alchemy's postgres adapter and postgres itself it has to guess at a timezone. As you might imagine, computer systems tend toward assuming UTC if no timezone is explicitly provided, but the precise logic for a toolsuite set can be complex and hard to debug (and my depend on your local time zone settings on your computer, system-level settings in the db, session level settings, and the preference of the toolmaker). You can sidestep this entire can of worms by either:
Storing all of your timestamps without a time zone in UTC, then converting to the desired timestamp as appropriate
Always using timestamps with time zone (i.e. replace datetime.today() with datetime.now() and pass in the desired time zone so that you are always dealing with a timezone aware value ), so there is no need for the computer to assume a timezone.
Note that in postgres, the timestamp with time zone type is still always stored as UTC with no extra information, the database simply uses the session-level configuration to decide which time zone to display it as when outputting it.
This is my first time developing a timezone aware web app. I'm using Django 1.11.7. I have looked through many posts regarding timezones and honestly, I'm starting to feel like I just don't understand the concept of timezones... 0_o
I'm making a booking app with start_time and end_time for each reservation.
I expect the user to be able to enter dates and times as he experiences them in his timezone. On the other hand, I want these dates and times stored in a postgresql database in UTC format.
If I was choosing to reserve, for example, between 12am and 2am (today), the result in the database should show between 5pm and 7pm (today).
Right?
Django Internationalization settings:
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_TZ = True
USE_L10N = True
USE_I18N = True
During database creation:
start_time TIMESTAMP WITH TIME ZONE NOT NULL,
end_time TIMESTAMP WITH TIME ZONE NOT NULL
Using Django's built in admin page feature, I enter the following information:
But when I look up the results in the database, here's what I get:
2017-12-05 19:00:00-05 | 2017-12-05 20:59:59-05
The stored result changed in the opposite direction of what I was expecting. I was expecting:
2017-12-06 05:00:00+05 | 2017-12-06 07:00:00+05
If my expectations are wrong and everything is working fine, I would be grateful if you could explain to me how I should interpret the timezone when I read it.
Otherwise, if my expectations where right, I would appreciate any advice.
Thank you.
The basic issue here is that you've misunderstood what the admin is doing for you ("I expect the user to be able to enter dates and times as he experiences them in his timezone"). In fact, the admin interprets all datetimes in the server time zone (UTC in your case).
Note: You are 5 hours behind server time
actually means:
Warning: The values you enter here will be interpreted in the server timezone, which is 5 hours ahead of you.
(I agree with you that this wording is ambiguous. You might want to file a ticket about improving it.)
For background on this issue, see this ticket. The root of the problem is that there's no way to get the user's timezone in the browser. What you can do is compare the current time in the browser to the current time on the server and post the warning above if they are different.
If you want to interpret datetimes in the user's timezone, there's no alternative but to ask the user to indicate their timezone explicitly, record it somewhere, and then activate() it to make it the current timezone. This is the technique described in the documentation. I believe that there are third-party packages that provide timezone selection widgets based on the list of timezones in pytz.
You're also misreading the time display. I think what you were expecting is 2017-12-06 05:00:00+00. Instead the result is 2017-12-06 00:00:00+00 in UTC or, equivalently, 2017-12-05 19:00:00-05 in Ottawa time, which is what you reported.
https://www.google.co.jp/search?dcr=0&q=ottawa+timezone&spell=1&sa=X&ved=0ahUKEwjX0LnA5_bXAhVFjJQKHXCdBDYQvwUIJigA&biw=1394&bih=803
Ottawa timezone is -5.
You can manually test in a template that's the timezones are properly registered by showing the time you've registered in different timezones with this filter:
https://docs.djangoproject.com/en/1.11/topics/i18n/timezones/#std:templatefilter-timezone
I had a very similar issue with Hotel bookings. This answer put me on the right track.
https://stackoverflow.com/a/18307581/4660189
The key here is in what you said:
I expect the user to be able to enter dates and times as he experiences them in his timezone.
"his timezone" would need to be handled via a middleware. So that each user can be assigned a timezone, and thus the system would handle all times/dates for this user accordingly.
This should get you going. Happy coding and welcome to the mind bending reality of timezones.
My Oracle DB has a table with a column of type TIMESTAMP WITH TIMEZONE. The Django model converts it to datetime but loses the original TZ info and assumes UTC.
DB: 01-JAN-17 12.10.01.809000 AM -05:00
Django:2017-01-01 00:10:01.809000+00:00
USE_TZ = True
The problem is that the table contains timestamps from different timezones. So if I change the standard timezone with activate() it won't help.
Is there a way for Django to explicitly read the timezone from the database without assuming it?
PS.: I'm aware that it would be better/easiest if my database would only contain UTC values but this is what I got.
I faced the similar problem earlier, what i did was, I saved the time and timezone separately and used pytz and tz_local, which provides a timezone database and local timezone time respectively. While saving the time use one of these library to get the timezone and save time and timezone separately.
I am using: datetime.now() to get the current time in an Event app that lets you create an event that has an end date, then all of the events are displayed in a calendar and if an event is passed due it is displayed in red.
My issue is that I have some users in different timezones than me saying that the events are ending at the wrong time. They should end at midnight on the day they are due.
I have the timezone setup in my django settings.py. When I use: datetime.now() is that going off of the users local timezone or is it going off of what timezone I have setup in django?
What I want is to find midnight for the users current timezone, so if my method above is wrong, how do I go about doing that?
Thanks
You will need your users to specify their timezone in their user profile. This can then be used to calculate local times correctly.
Check out Relativity of time – shortcomings in Python datetime, and workaround for some good information (and concrete examples).
UPDATE: From Django 1.4 it comes up with timezone support. Check it out.
I worked on a library to work with user timezones transparently. You should just use a Field and some timezone utils and you should get everything converted to the user timezone everytime you do a request or a query.
The library is named django-timezones, and is a modification of the one that Brosner first made.
Give it a try to see if it works for you.
Try storing the UTC Datetime instead then make the necessary adjustments based on the user's timezone:
import datetime
def getUtcNow():
return datetime.datetime(*time.gmtime()[:6])