How do I parse a time like this '/Date(-62135596800000)/' with Python?
I searched the internet, but was unable to find a solution.
Link:http://211.137.19.74:8089/Ajax/GetStationDetailList?cityName=%E6%B2%88%E9%98%B3
TimePoint:"/Date(-62135596800000)/"
and the time should be 2016:12:31: 01:00
/Date(N)/ is usually the format for dates serialized by the built-in WCF JSON serializer in .NET. The value can then be reconstructed by calling new Date(N) in JavaScript.
Running that in JavaScript will show why this date is very likely just wrong:
new Date(-62135596800000)
// Date 0001-01-01T00:00:00.000Z
As you can see, you get the 0001-01-01 00:00:00.00. This also happens to be the default value in .NET for empty DateTime objects:
default(DateTime).ToString("u") // 0001-01-01 00:00:00Z
new DateTime(0).ToString("u") // 0001-01-01 00:00:00Z
So, sorry to say it like that, but that server is just broken.
That all being said, once you get a valid date, you can parse the number in Python using datetime.fromtimestamp or datetime.utcfromtimestamp:
>>> from datetime import datetime
>>> datetime.fromtimestamp(1483120713887 // 1000)
datetime.datetime(2016, 12, 30, 18, 58, 33)
Using python 3.x
Looks like epoch time (in nano seconds starting January 1, 1970 00:00:00.000 GMT). You could use something like this:
from datetime import datetime
dt = datetime.fromtimestamp(62135596800000 // 1000000000)
print(dt)
As for the value you're getting, you can't have a negative epoch time stamp (-62135596800000 would be invalid). If the - indicates negative, it might be another time stamp format.
Source: ASP.NET FORUMS
var dateString = "/Date(1526409000000)/".substr(6);
var currentTime = new Date(parseInt(dateString));
var month = currentTime.getMonth() + 1;
var day = currentTime.getDate();
var year = currentTime.getFullYear();
var date = day + "/" + month + "/" + year;
alert(date);
I have a timestamp such 1474398821633L that I think is in utc. I want to compare it to datetime.datetime.now() to verify if it is expired.
I am using python 2.7
from datetime import datetime
timestamp = 1474398821633L
now = datetime.now()
if datetime.utcfromtimestamp(timestamp) < now:
print "timestamp expired"
However I got this error when trying to create a datetime object from the timestamp: ValueError: timestamp out of range for platform localtime()/gmtime() function
What can I do?
It looks like your timestamp is in milliseconds. Python uses timestamps in seconds:
>>> datetime.datetime.utcfromtimestamp(1474398821.633)
datetime.datetime(2016, 9, 20, 19, 13, 41, 633000)
In other words, you might need to divide your timestamp by 1000. in order to get it in the proper range.
Also, you'll probably want to compare datetime.utcnow() instead of datetime.now() to make sure that you're handling timezones correctly :-).
As #mgilson pointed out your input is likely "milliseconds", not "seconds since epoch".
Use time.time() instead of datetime.now():
import time
if time.time() > (timestamp_in_millis * 1e-3):
print("expired")
If you need datetime then use datetime.utcnow() instead of datetime.now(). Do not compare .now() that returns local time as a naive datetime object with utcfromtimestamp() that returns UTC time also as a naive datetime object (it is like comparing celsius and fahrenheit directly: you should convert to the same unit first).
from datetime import datetime
now = datetime.utcnow()
then = datetime.utcfromtimestamp(timestamp_in_millis * 1e-3)
if now > then:
print("expired")
See more details in Find if 24 hrs have passed between datetimes - Python.
I'm running a Django application on top of a MySQL (actually MariaDB) database.
My Django Model looks like this:
from django.db import models
from django.db.models import Avg, Max, Min, Count
class myModel(models.Model):
my_string = models.CharField(max_length=32,)
my_date = models.DateTimeField()
#staticmethod
def get_stats():
logger.info(myModel.objects.values('my_string').annotate(
count=Count("my_string"),
min=Min('my_date'),
max=Max('my_date'),
avg=Avg('my_date'),
)
)
When I run get_stats(), I get the following log line:
[2015-06-21 09:45:40] INFO [all_logs:96] [{'my_string': u'A', 'count': 2, 'avg': 20080507582679.5, 'min': datetime.datetime(2007, 8, 2, 11, 33, 53, tzinfo=<UTC>), 'max': datetime.datetime(2009, 2, 13, 5, 20, 6, tzinfo=<UTC>)}]
The problem I have with this is that the average of the my_date field returned by the database is: 20080507582679.5. Look carefully at that number. It is an invalid date format.
Why doesn't the database return a valid value for the average of these two dates? How do I get the actual average of this field if the way described fails? Is Django DateTimeField not setup to do handle averaging?
Q1: Why doesn't the database return a valid value for the average of these two dates?
A: The value returned is expected, it's well defined MySQL behavior.
MySQL automatically converts a date or time value to a number if the value is used in a numeric context and vice versa.
MySQL Reference Manual: https://dev.mysql.com/doc/refman/5.5/en/date-and-time-types.html
In MySQL, the AVG aggregate function operates on numeric values.
In MySQL, a DATE or DATETIME expression can be evaluated in a numeric context.
As a simple demonstration, performing an numeric addition operation on a DATETIME implicitly converts the datetime value into a number. This query:
SELECT NOW(), NOW()+0
returns a result like:
NOW() NOW()+0
------------------- -----------------------
2015-06-23 17:57:48 20150623175748.000000
Note that the value returned for the expression NOW()+0 is not a DATETIME, it's a number.
When you specify a SUM() or AVG() function on a DATETIME expression, that's equivalent to converting the DATETIME into a number, and then summing
or averaging the number.
That is, the return from this expression AVG(mydatetimecol) is equivalent to the return from this expression: AVG(mydatetimecol+0)
What is being "averaged" is a numeric value. And you have observed, the value returned is not a valid datetime; and even in cases where it happens to look like a valid datetime, it's likely not a value you would consider a true "average".
Q2: How do I get the actual average of this field if the way described fails?
A2: One way to do that is to convert the datetime into a numeric value that can be "accurately" averaged, and then convert that back into a datetime.
For example, you could convert the datetime into a numeric value representing a number of seconds from some fixed point in time, e.g.
TIMESTAMPDIFF(SECOND,'2015-01-01',t.my_date)
You could then "average" those values, to get an average number of seconds from a fixed point in time. (NOTE: beware of adding up an extremely large number of rows, with extremely large values, and exceeding the limit (maximum numeric value), numeric overflow issues.)
AVG(TIMESTAMPDIFF(SECOND,'2015-01-01',t.my_date))
To convert that back to a datetime, add that value as a number of seconds back to a the fixed point in time:
'2015-01-01' + INTERVAL AVG(TIMESTAMPDIFF(SECOND,'2015-01-01',t.my_date)) SECOND
(Note that the DATEIME values are evaluated in the timezone of the MySQL session; so there are edge cases where the setting of the time_zone variable in the MySQL session will have some influence on the value returned.)
MySQL also provides a UNIX_TIMESTAMP() function which returns a unix-style integer value, number of seconds from the beginning of the era (midnight Jan. 1, 1970 UTC). You can use that to accomplish the same operation more concisely:
FROM_UNIXTIME(AVG(UNIX_TIMESTAMP(t.my_date)))
Note that this final expression is really doing the same thing... converting the datetime value into a number of seconds since '1970-01-01 00:00:00' UTC, taking a numeric average of that, and then adding that average number of seconds back to '1970-01-01' UTC, and finally converting that back to a DATETIME value, represented in the current session time_zone.
Q3: Is Django DateTimeField not setup to do handle averaging?
A: Apparently, the authors of Django are satisfied with the value returned from the database for a SQL expression AVG(datetime).
Plan A: Use a TIMESTAMP field instead of a DATETIME field
Plan B: Convert DATETIME to TIMESTAMP during the computation:
FROM_UNIXTIME(ROUND(AVG(UNIX_TIMESTAMP(`my_date`))))
(Sorry, I don't know the Django syntax needed.)
When you use values(), Django will not convert the value it got from the database-python connector. It's up to the connector to determine how the value is returned.
In this case, it seems that the MySQL connector returns a string-representation with the separators removed. You can try to use datetime.strptime() with a matching format to parse it into a datetime object.
I want to get a date and time in the form of 1/1/2000. The program I am making reads from a compressed document that stores a table of information. The program gets to a date time cell and pulls out
946652400 as a int
or 11110000110001000110110000111000 as binary
This should be interpreted as Fri Dec 31 15:00:00 UTC 1999 which I want it to display as 1/1/2000. Is there a method of some form in python that I can use? I can also display this information as 12/31/1999 as well. If not can someone explain how to create this from the example above?
Thank you!
Using datetime:
>>> import datetime
>>> datetime.datetime.fromtimestamp(946652400)
datetime.datetime(2000, 1, 1, 0, 0)
Timezones are a mess, especially if you have to worry about the future.
But if you're just dealing with the past, you can always apply a static offset to adjust the value to the desired timezone.
The value you have given is already in 1/1/2000, you can adjust it back to UTC if you know the appropriate offset.
>>> dt = datetime.datetime.fromtimestamp(946652400)
>>> OFFSET = datetime.timedelta(hours=-9)
>>> adjusted_dt = dt + OFFSET
datetime.datetime(1999, 12, 31, 15, 0)
import time
time_struct = time.gmtime(946652400)
print time.strftime('%d/%m/%Y', time_struct)
The time module is part of the stdlib: documented here
And here's a table showing the %m format directives you can use in strftime.
Is there a way to get the UTC timestamp by specifying the date? What I would expect:
datetime(2008, 1, 1, 0, 0, 0, 0)
should result in
1199145600
Creating a naive datetime object means that there is no time zone information. If I look at the documentation for datetime.utcfromtimestamp, creating a UTC timestamp means leaving out the time zone information. So I would guess, that creating a naive datetime object (like I did) would result in a UTC timestamp. However:
then = datetime(2008, 1, 1, 0, 0, 0, 0)
datetime.utcfromtimestamp(float(then.strftime('%s')))
results in
2007-12-31 23:00:00
Is there still any hidden time zone information in the datetime object? What am I doing wrong?
Naïve datetime versus aware datetime
Default datetime objects are said to be "naïve": they keep time information without the time zone information. Think about naïve datetime as a relative number (ie: +4) without a clear origin (in fact your origin will be common throughout your system boundary).
In contrast, think about aware datetime as absolute numbers (ie: 8) with a common origin for the whole world.
Without timezone information you cannot convert the "naive" datetime towards any non-naive time representation (where does +4 targets if we don't know from where to start ?). This is why you can't have a datetime.datetime.toutctimestamp() method. (cf: http://bugs.python.org/issue1457227)
To check if your datetime dt is naïve, check dt.tzinfo, if None, then it's naïve:
datetime.now() ## DANGER: returns naïve datetime pointing on local time
datetime(1970, 1, 1) ## returns naïve datetime pointing on user given time
I have naïve datetimes, what can I do ?
You must make an assumption depending on your particular context:
The question you must ask yourself is: was your datetime on UTC ? or was it local time ?
If you were using UTC (you are out of trouble):
import calendar
def dt2ts(dt):
"""Converts a datetime object to UTC timestamp
naive datetime will be considered UTC.
"""
return calendar.timegm(dt.utctimetuple())
If you were NOT using UTC, welcome to hell.
You have to make your datetime non-naïve prior to using the former
function, by giving them back their intended timezone.
You'll need the name of the timezone and the information about
if DST was in effect when producing the target naïve datetime (the
last info about DST is required for cornercases):
import pytz ## pip install pytz
mytz = pytz.timezone('Europe/Amsterdam') ## Set your timezone
dt = mytz.normalize(mytz.localize(dt, is_dst=True)) ## Set is_dst accordingly
Consequences of not providing is_dst:
Not using is_dst will generate incorrect time (and UTC timestamp)
if target datetime was produced while a backward DST was put in place
(for instance changing DST time by removing one hour).
Providing incorrect is_dst will of course generate incorrect
time (and UTC timestamp) only on DST overlap or holes. And, when
providing
also incorrect time, occuring in "holes" (time that never existed due
to forward shifting DST), is_dst will give an interpretation of
how to consider this bogus time, and this is the only case where
.normalize(..) will actually do something here, as it'll then
translate it as an actual valid time (changing the datetime AND the
DST object if required). Note that .normalize() is not required
for having a correct UTC timestamp at the end, but is probably
recommended if you dislike the idea of having bogus times in your
variables, especially if you re-use this variable elsewhere.
and AVOID USING THE FOLLOWING: (cf: Datetime Timezone conversion using pytz)
dt = dt.replace(tzinfo=timezone('Europe/Amsterdam')) ## BAD !!
Why? because .replace() replaces blindly the tzinfo without
taking into account the target time and will choose a bad DST object.
Whereas .localize() uses the target time and your is_dst hint
to select the right DST object.
OLD incorrect answer (thanks #J.F.Sebastien for bringing this up):
Hopefully, it is quite easy to guess the timezone (your local origin) when you create your naive datetime object as it is related to the system configuration that you would hopefully NOT change between the naive datetime object creation and the moment when you want to get the UTC timestamp. This trick can be used to give an imperfect question.
By using time.mktime we can create an utc_mktime:
def utc_mktime(utc_tuple):
"""Returns number of seconds elapsed since epoch
Note that no timezone are taken into consideration.
utc tuple must be: (year, month, day, hour, minute, second)
"""
if len(utc_tuple) == 6:
utc_tuple += (0, 0, 0)
return time.mktime(utc_tuple) - time.mktime((1970, 1, 1, 0, 0, 0, 0, 0, 0))
def datetime_to_timestamp(dt):
"""Converts a datetime object to UTC timestamp"""
return int(utc_mktime(dt.timetuple()))
You must make sure that your datetime object is created on the same timezone than the one that has created your datetime.
This last solution is incorrect because it makes the assumption that the UTC offset from now is the same than the UTC offset from EPOCH. Which is not the case for a lot of timezones (in specific moment of the year for the Daylight Saving Time (DST) offsets).
Another possibility is:
d = datetime.datetime.utcnow()
epoch = datetime.datetime(1970,1,1)
t = (d - epoch).total_seconds()
This works as both "d" and "epoch" are naive datetimes, making the "-" operator valid, and returning an interval. total_seconds() turns the interval into seconds. Note that total_seconds() returns a float, even d.microsecond == 0
Also note the calendar.timegm() function as described by this blog entry:
import calendar
calendar.timegm(utc_timetuple)
The output should agree with the solution of vaab.
A simple solution without using external modules:
from datetime import datetime, timezone
dt = datetime(2008, 1, 1, 0, 0, 0, 0)
int(dt.replace(tzinfo=timezone.utc).timestamp())
If input datetime object is in UTC:
>>> dt = datetime(2008, 1, 1, 0, 0, 0, 0)
>>> timestamp = (dt - datetime(1970, 1, 1)).total_seconds()
1199145600.0
Note: it returns float i.e., microseconds are represented as fractions of a second.
If input date object is in UTC:
>>> from datetime import date
>>> utc_date = date(2008, 1, 1)
>>> timestamp = (utc_date.toordinal() - date(1970, 1, 1).toordinal()) * 24*60*60
1199145600
See more details at Converting datetime.date to UTC timestamp in Python.
I feel like the main answer is still not so clear, and it's worth taking the time to understand time and timezones.
The most important thing to understand when dealing with time is that time is relative!
2017-08-30 13:23:00: (a naive datetime), represents a local time somewhere in the world, but note that 2017-08-30 13:23:00 in London is NOT THE SAME TIME as 2017-08-30 13:23:00 in San Francisco.
Because the same time string can be interpreted as different points-in-time depending on where you are in the world, there is a need for an absolute notion of time.
A UTC timestamp is a number in seconds (or milliseconds) from Epoch (defined as 1 January 1970 00:00:00 at GMT timezone +00:00 offset).
Epoch is anchored on the GMT timezone and therefore is an absolute point in time. A UTC timestamp being an offset from an absolute time therefore defines an absolute point in time.
This makes it possible to order events in time.
Without timezone information, time is relative, and cannot be converted to an absolute notion of time without providing some indication of what timezone the naive datetime should be anchored to.
What are the types of time used in computer system?
naive datetime: usually for display, in local time (i.e. in the browser) where the OS can provide timezone information to the program.
UTC timestamps: A UTC timestamp is an absolute point in time, as mentioned above, but it is anchored in a given timezone, so a UTC timestamp can be converted to a datetime in any timezone, however it does not contain timezone information. What does that mean? That means that 1504119325 corresponds to 2017-08-30T18:55:24Z, or 2017-08-30T17:55:24-0100 or also 2017-08-30T10:55:24-0800. It doesn't tell you where the datetime recorded is from. It's usually used on the server side to record events (logs, etc...) or used to convert a timezone aware datetime to an absolute point in time and compute time differences.
ISO-8601 datetime string: The ISO-8601 is a standardized format to record datetime with timezone. (It's in fact several formats, read on here: https://en.wikipedia.org/wiki/ISO_8601) It is used to communicate timezone aware datetime information in a serializable manner between systems.
When to use which? or rather when do you need to care about timezones?
If you need in any way to care about time-of-day, you need timezone information. A calendar or alarm needs time-of-day to set a meeting at the correct time of the day for any user in the world. If this data is saved on a server, the server needs to know what timezone the datetime corresponds to.
To compute time differences between events coming from different places in the world, UTC timestamp is enough, but you lose the ability to analyze at what time of day events occured (ie. for web analytics, you may want to know when users come to your site in their local time: do you see more users in the morning or the evening? You can't figure that out without time of day information.
Timezone offset in a date string:
Another point that is important, is that timezone offset in a date string is not fixed. That means that because 2017-08-30T10:55:24-0800 says the offset -0800 or 8 hours back, doesn't mean that it will always be!
In the summer it may well be in daylight saving time, and it would be -0700
What that means is that timezone offset (+0100) is not the same as timezone name (Europe/France) or even timezone designation (CET)
America/Los_Angeles timezone is a place in the world, but it turns into PST (Pacific Standard Time) timezone offset notation in the winter, and PDT (Pacific Daylight Time) in the summer.
So, on top of getting the timezone offset from the datestring, you should also get the timezone name to be accurate.
Most packages will be able to convert numeric offsets from daylight saving time to standard time on their own, but that is not necessarily trivial with just offset. For example WAT timezone designation in West Africa, is UTC+0100 just like CET timezone in France, but France observes daylight saving time, while West Africa does not (because they're close to the equator)
So, in short, it's complicated. VERY complicated, and that's why you should not do this yourself, but trust a package that does it for you, and KEEP IT UP TO DATE!
There is indeed a problem with using utcfromtimestamp and specifying time zones. A nice example/explanation is available on the following question:
How to specify time zone (UTC) when converting to Unix time? (Python)
The accepted answer seems not work for me. My solution:
import time
utc_0 = int(time.mktime(datetime(1970, 01, 01).timetuple()))
def datetime2ts(dt):
"""Converts a datetime object to UTC timestamp"""
return int(time.mktime(dt.utctimetuple())) - utc_0
Simplest way:
>>> from datetime import datetime
>>> dt = datetime(2008, 1, 1, 0, 0, 0, 0)
>>> dt.strftime("%s")
'1199163600'
Edit: #Daniel is correct, this would convert it to the machine's timezone. Here is a revised answer:
>>> from datetime import datetime, timezone
>>> epoch = datetime(1970, 1, 1, 0, 0, 0, 0, timezone.utc)
>>> dt = datetime(2008, 1, 1, 0, 0, 0, 0, timezone.utc)
>>> int((dt-epoch).total_seconds())
'1199145600'
In fact, its not even necessary to specify timezone.utc, because the time difference is the same so long as both datetime have the same timezone (or no timezone).
>>> from datetime import datetime
>>> epoch = datetime(1970, 1, 1, 0, 0, 0, 0)
>>> dt = datetime(2008, 1, 1, 0, 0, 0, 0)
>>> int((dt-epoch).total_seconds())
1199145600
I think the correct way to phrase your question is
Is there a way to get the timestamp by specifying the date in UTC?, because timestamp is just a number which is absolute, not relative. The relative (or timezone aware) piece is the date.
I find pandas very convenient for timestamps, so:
import pandas as pd
dt1 = datetime(2008, 1, 1, 0, 0, 0, 0)
ts1 = pd.Timestamp(dt1, tz='utc').timestamp()
# make sure you get back dt1
datetime.utcfromtimestamp(ts1)