I am trying to extract data from a netcdf file using wrf-python. The data is for every hour. The date is being extracted as a number, and not a calendar-date-time. First I extract the data, convert it to a flat np array, then try to save the file. The format is saved as '%s'
np.savetxt((stn + 'WRF_T2_T10_WS_WD.csv'), np.transpose(arr2D), %s, delimiter=',', header=headers, comments='')
it looks like this:
but it needs to look like this:
Thanks
By convention, dates are frequently stored as an offset in seconds from Jan 1, 1970
For the case of converting seconds, this answer Python Numpy Loadtxt - Convert unix timestamp suggests converting them by changing their datatype (should be as efficient as possible as it dodges by-row loops, copying data, etc.)
x = np.asarray(x, dtype='datetime64[s]')
However, the E+18 postfix implies that if you really have a date, your timestamps are in nanoseconds, so datetime64[ns] may work for you
import time
import numpy as np
>>> a = np.array([time.time() * 10**9]) # epoch seconds to ns
>>> a # example array
array([1.60473147e+18])
>>> a = np.asarray(a, dtype='datetime64[ns]')
>>> a
array(['2020-11-07T06:44:29.714103040'], dtype='datetime64[ns]')
Related
This link shows my csv file and graph.
I want to represent the AVG number (which are seconds actually) as hour(s):minute(s) on y axis.
I think, it cannot be solved because I spent 3 days wit this problem.
But to be more precise, aside of lot of conversations with dateime, timedelta, timestamp nothing worked.
Either the data could no be shown on y axis because it did not represent number like variable to plot or I've got not proper representation of the data.
I was trying to create something like converting seconds to calculate with divmod
than put them on the top of the bars with annonate.
Later I have used Timple.
I do not understand how should I create an acceptable datatype for this.
I've made some related and use pandasDataFrame.plot
>>> import pandas as pd
>>> df = pd.DataFrame()
>>> df["activity"] = ['run', 'swim', 'drive']
>>> df["avg"] = [86400,43200,21600]
>>> df
activity avg
0 run 86400
1 swim 43200
2 drive 21600
>>> df.plot.bar(x="activity")
<AxesSubplot: xlabel='activity'>
>>> plt.show()
To represent time transcurred for a certain number of seconds you can use fromtimestamp and formatting strftime but it might not be compatible with matplotlib, then using Timple is something related, but graph could not be properly plotted or maybe it is needed to perform something like explore data or apply a certain statistical procedure.
>>> import datetime
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import timple
>>> tmpl = timple.Timple()
>>> tmpl.enable()
>>> timedeltas = np.array([datetime.timedelta(seconds=(s)) for s in df["avg"]])
>>> timedeltas
array([datetime.timedelta(days=1), datetime.timedelta(seconds=43200),
datetime.timedelta(seconds=21600)], dtype=object)
>>> plt.plot(timedeltas, df["activity"])
[<matplotlib.lines.Line2D object at 0x0000026FAC3F5B40>]
>>> plt.show()
I have an large 2D array contains seconds from year 2000, I want to convert to an array of datetime. I could not find a good way to do it. I used a loop. But it did not work and it produced an error as:
TypeError: float() argument must be a string or a number, not 'datetime.datetime'
I give the example code as below. Would you please give me any suggestions?
Thank you.
import numpy as np
import datetime as dt
secs_from_2000 = np.array([[6.833232e+08, 6.833233e+08, 6.833235e+08], [6.833239e+08, 6.833242e+08, 6.833244e+08]])
dt_from_1970 = np.empty_like(secs_from_2000)
for i in range(secs_from_2000.shape[0]):
for j in range(secs_from_2000.shape[1]):
dt_from_1970[i,j] = dt.datetime.utcfromtimestamp((dt.datetime(2000,1,1)- dt.datetime(1970,1,1)).total_seconds() + secs_from_2000[i,j])
There are three parts of this problem:
Convert "seconds from 2000" to standard Unix timestamps (seconds after 1970)
Convert Unix timestamp to datetime
Do this for every element of the array
For 1, if we call the "seconds from 2000" figure t', and the standard Unix time is t, you can see that t - t' = x where x is a constant adjustment factor, such that t = t' + x (t' is what you have, t is what you want). Moreover, x is equal to the number of seconds between 1970 and 2000. Thus you can calculate it with:
>>> from datetime import datetime
>>> datetime(year=2000, month=1, day=1).timestamp()
946710000.0
Now you just have to add this to your t':
def unix_time(secs_from_2000: float) -> float:
return secs_from_2000 + 946710000
For 3, I believe this is covered in Apply function to all elements in NumPy matrix so I won't duplicate it here.
I just started moving from Matlab to Python 2.7 and I have some trouble reading my .mat-files. Time information is stored in Matlab's datenum format. For those who are not familiar with it:
A serial date number represents a calendar date as the number of days that has passed since a fixed base date. In MATLAB, serial date number 1 is January 1, 0000.
MATLAB also uses serial time to represent fractions of days beginning at midnight; for example, 6 p.m. equals 0.75 serial days. So the string '31-Oct-2003, 6:00 PM' in MATLAB is date number 731885.75.
(taken from the Matlab documentation)
I would like to convert this to Pythons time format and I found this tutorial. In short, the author states that
If you parse this using python's datetime.fromordinal(731965.04835648148) then the result might look reasonable [...]
(before any further conversions), which doesn't work for me, since datetime.fromordinal expects an integer:
>>> datetime.fromordinal(731965.04835648148)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: integer argument expected, got float
While I could just round them down for daily data, I actually need to import minutely time series. Does anyone have a solution for this problem? I would like to avoid reformatting my .mat files since there's a lot of them and my colleagues need to work with them as well.
If it helps, someone else asked for the other way round. Sadly, I'm too new to Python to really understand what is happening there.
/edit (2012-11-01): This has been fixed in the tutorial posted above.
You link to the solution, it has a small issue. It is this:
python_datetime = datetime.fromordinal(int(matlab_datenum)) + timedelta(days=matlab_datenum%1) - timedelta(days = 366)
a longer explanation can be found here
Using pandas, you can convert a whole array of datenum values with fractional parts:
import numpy as np
import pandas as pd
datenums = np.array([737125, 737124.8, 737124.6, 737124.4, 737124.2, 737124])
timestamps = pd.to_datetime(datenums-719529, unit='D')
The value 719529 is the datenum value of the Unix epoch start (1970-01-01), which is the default origin for pd.to_datetime().
I used the following Matlab code to set this up:
datenum('1970-01-01') % gives 719529
datenums = datenum('06-Mar-2018') - linspace(0,1,6) % test data
datestr(datenums) % human readable format
Just in case it's useful to others, here is a full example of loading time series data from a Matlab mat file, converting a vector of Matlab datenums to a list of datetime objects using carlosdc's answer (defined as a function), and then plotting as time series with Pandas:
from scipy.io import loadmat
import pandas as pd
import datetime as dt
import urllib
# In Matlab, I created this sample 20-day time series:
# t = datenum(2013,8,15,17,11,31) + [0:0.1:20];
# x = sin(t)
# y = cos(t)
# plot(t,x)
# datetick
# save sine.mat
urllib.urlretrieve('http://geoport.whoi.edu/data/sine.mat','sine.mat');
# If you don't use squeeze_me = True, then Pandas doesn't like
# the arrays in the dictionary, because they look like an arrays
# of 1-element arrays. squeeze_me=True fixes that.
mat_dict = loadmat('sine.mat',squeeze_me=True)
# make a new dictionary with just dependent variables we want
# (we handle the time variable separately, below)
my_dict = { k: mat_dict[k] for k in ['x','y']}
def matlab2datetime(matlab_datenum):
day = dt.datetime.fromordinal(int(matlab_datenum))
dayfrac = dt.timedelta(days=matlab_datenum%1) - dt.timedelta(days = 366)
return day + dayfrac
# convert Matlab variable "t" into list of python datetime objects
my_dict['date_time'] = [matlab2datetime(tval) for tval in mat_dict['t']]
# print df
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 201 entries, 2013-08-15 17:11:30.999997 to 2013-09-04 17:11:30.999997
Data columns (total 2 columns):
x 201 non-null values
y 201 non-null values
dtypes: float64(2)
# plot with Pandas
df = pd.DataFrame(my_dict)
df = df.set_index('date_time')
df.plot()
Here's a way to convert these using numpy.datetime64, rather than datetime.
origin = np.datetime64('0000-01-01', 'D') - np.timedelta64(1, 'D')
date = serdate * np.timedelta64(1, 'D') + origin
This works for serdate either a single integer or an integer array.
Just building on and adding to previous comments. The key is in the day counting as carried out by the method toordinal and constructor fromordinal in the class datetime and related subclasses. For example, from the Python Library Reference for 2.7, one reads that fromordinal
Return the date corresponding to the proleptic Gregorian ordinal, where January 1 of year 1 has ordinal 1. ValueError is raised unless 1 <= ordinal <= date.max.toordinal().
However, year 0 AD is still one (leap) year to count in, so there are still 366 days that need to be taken into account. (Leap year it was, like 2016 that is exactly 504 four-year cycles ago.)
These are two functions that I have been using for similar purposes:
import datetime
def datetime_pytom(d,t):
'''
Input
d Date as an instance of type datetime.date
t Time as an instance of type datetime.time
Output
The fractional day count since 0-Jan-0000 (proleptic ISO calendar)
This is the 'datenum' datatype in matlab
Notes on day counting
matlab: day one is 1 Jan 0000
python: day one is 1 Jan 0001
hence an increase of 366 days, for year 0 AD was a leap year
'''
dd = d.toordinal() + 366
tt = datetime.timedelta(hours=t.hour,minutes=t.minute,
seconds=t.second)
tt = datetime.timedelta.total_seconds(tt) / 86400
return dd + tt
def datetime_mtopy(datenum):
'''
Input
The fractional day count according to datenum datatype in matlab
Output
The date and time as a instance of type datetime in python
Notes on day counting
matlab: day one is 1 Jan 0000
python: day one is 1 Jan 0001
hence a reduction of 366 days, for year 0 AD was a leap year
'''
ii = datetime.datetime.fromordinal(int(datenum) - 366)
ff = datetime.timedelta(days=datenum%1)
return ii + ff
Hope this helps and happy to be corrected.
I am working with a netcdf format code and I need to convert the time from seconds from the starting time (2016-01-01 00:00:00.0) to time in UTC. I'm fairly new to all of this so I am really struggling!
I have tried using the num2date from netCDF4.
from netCDF4 import date2num , num2date, Dataset
time=f.variables['time'][:]
dates=netCDF4.num2date(time[:],time.units)
print(dates.strftime('%Y%m%d%H') for date in dates)
AttributeError: 'MaskedArray' object has no attribute 'units'
Since you extract time from the variables in time=f.variables['time'][:], it will lose it's associated unit (time is just a masked array, as the error says).
What you have to feed to num2date() is variables['time'].units, e.g.
from netCDF4 import date2num, num2date, Dataset
file = ... # your nc file
with Dataset(file) as root:
time = root.variables['time'][:]
dates = num2date(time, root.variables['time'].units)
## directly get UTC hours here:
# unit_utchours = root.variables['time'].units.replace('seconds', 'hours')
## would e.g. be 'hours since 2019-08-15 00:00:00'
# utc_hours = date2num(dates, unit_utchours)
# check:
print(dates[0].strftime('%Y%m%d%H'))
# e.g. prints 2019081516
...to get the dates as a number, you could e.g. do
num_dates = [int(d.strftime('%Y%m%d%H')) for d in dates]
# replace int with float if you need floating point numbers etc.
...to get the dates in UTC hours, see the commented section in the first code block. Since the dates array contains objects of type datetime.datetime, you could also do
utc_hours = [d.hour+(d.minute/60)+(d.second/3600) for d in dates]
I'm confused by the interoperation between numpy and pandas date objects (or maybe just by numpy's datetime64 in general).
I was trying to count business days using numpy's built-in functionality like so:
np.busday_count("2016-03-01", "2016-03-31", holidays=[np.datetime64("28/03/2016")])
However, numpy apparently can't deal with the inverted date format:
ValueError: Error parsing datetime string "28/03/2016" at position 2
To get around this, I thought I'd just use pandas to_datetime, which can. However:
np.busday_count("2016-03-01", "2016-03-31", holidays=[np.datetime64(pd.to_datetime("28/03/2016"))])
ValueError: Cannot safely convert provided holidays input into an array of dates
Searching around for a bit, it seemed that this was caused by the fact that the chaining of to_datetime and np.datetime64 results in a datetime64[us] object, which apparently the busday_count function cannot accept (is this intended behaviour or a bug?). Thus, my next attempt was:
np.busday_count("2016-03-01", "2016-03-31", holidays=[np.datetime64(pd.Timestamp("28"), "D")])
But:
TypeError: Cannot cast datetime.datetime object from metadata [us] to [D] according to the rule 'same_kind'
And that's me out - why are there so many incompatibilities between all these datetime formats? And how can I get around them?
I've been having a similar issue, using np.is_busday()
The type of datetime64 is vital to get right. Checking the numpy datetime docs, you can specify the numpy datetime type to be D.
This works:
my_holidays=np.array([datetime.datetime.strptime(x,'%m/%d/%y') for x in holidays.Date.values], dtype='datetime64[D]')
day_flags['business_day'] = np.is_busday(days,holidays=my_holidays)
Whereas this throws the same error you got:
my_holidays=np.array([datetime.datetime.strptime(x,'%m/%d/%y') for x in holidays.Date.values], dtype='datetime64')
The only difference is specifying the type of datetime64.
dtype='datetime64[D]'
vs
dtype='datetime64'
Docs are here:
https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.datetime.html
I had the same issue while using np.busday_count, later I figured out the problem was with the hours, minutes, seconds, and milliseconds getting added while converting it to datetime object or numpy datetime object.
I just converted to datetime object with only date and not the hours, minutes, seconds, and milliseconds.
The following was my code:
holidays_list.json file:
{
"holidays_2019": [
"04-Mar-2019",
"21-Mar-2019",
"17-Apr-2019",
"19-Apr-2019",
"29-Apr-2019",
"01-May-2019",
"05-Jun-2019",
"12-Aug-2019",
"15-Aug-2019",
"02-Sep-2019",
"10-Sep-2019",
"02-Oct-2019",
"08-Oct-2019",
"28-Oct-2019",
"12-Nov-2019",
"25-Dec-2019"
],
"format": "%d-%b-%Y"
}
code file:
import json
import datetime
import numpy as np
with open('holidays_list.json', 'r') as infile:
data = json.loads(infile.read())
# the following is where I convert the datetime object to date
holidays = list(map(lambda x: datetime.datetime.strptime(
x, data['format']).date(), data['holidays_2019']))
start_date = datetime.datetime.today().date()
end_date = start_date + datetime.timedelta(days=30)
holidays = [start_date + datetime.timedelta(days=1)]
print(np.busday_count(start_date, end_date, holidays=holidays))