Related
I have downloaded temperature data from a model. Because I chose the whole temporal range the data was split into different netcdf files. However, when I am trying to access these data for some I get an error which I do not understand. First of all I have imported the following packages (and installed correctly)
import netCDF4 as nc
import h5py
import xarray as xr
import numpy as np
import matplotlib.pyplot as plt
#first read the first file of the same model
c1 = xr.open_dataset(path+'tas_day_CMCC-ESM2_historical_r1i1p1f1_gn_18500101-18741231_v20210114.nc')
#then read the second file
c2 = xr.open_dataset(path+'tas_day_CMCC-ESM2_historical_r1i1p1f1_gn_18750101-18991231_v20210114.nc')
#then take only the longitude and latitude values which I am interested
cc1 = c1.sel(lon = 35.86,lat = 14.375,method='nearest')
cc2 = c2.sel(lon = 35.86,lat = 14.375,method='nearest')
However when I am trying to access the temperature variable for the second file I get an error. For example let s print the first xarray
#for the first Xarray
time , (time) , object, ,1850-01-01 12:00:00 ... 1874-12-...
lat, () ,float64, ,14.61
lon, () ,float64. ,36.25
height, (). ,float64. ,2.0
Data variables:
time_bnds. , (time, bnds). , object. , ...
lat_bnds. , (bnds). , float64. , ...
lon_bnds. , (bnds). , float64. , ...
tas. ,(time). , float32. , 297.9 298.0 297.9 ... 297.8 298.2
but when I try to print the second one
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
File ~/miniforge3/lib/python3.9/site-packages/IPython/core/formatters.py:343, in BaseFormatter.__call__(self, obj)
341 method = get_real_method(obj, self.print_method)
342 if method is not None:
--> 343 return method()
344 return None
345 else:
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/dataset.py:2058, in Dataset._repr_html_(self)
2056 if OPTIONS["display_style"] == "text":
2057 return f"<pre>{escape(repr(self))}</pre>"
-> 2058 return formatting_html.dataset_repr(self)
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/formatting_html.py:297, in dataset_repr(ds)
290 obj_type = f"xarray.{type(ds).__name__}"
292 header_components = [f"<div class='xr-obj-type'>{escape(obj_type)}</div>"]
294 sections = [
295 dim_section(ds),
296 coord_section(ds.coords),
--> 297 datavar_section(ds.data_vars),
298 attr_section(ds.attrs),
299 ]
301 return _obj_repr(ds, header_components, sections)
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/formatting_html.py:179, in _mapping_section(mapping, name, details_func, max_items_collapse, expand_option_name, enabled)
172 expanded = _get_boolean_with_default(
173 expand_option_name, n_items < max_items_collapse
174 )
175 collapsed = not expanded
177 return collapsible_section(
178 name,
--> 179 details=details_func(mapping),
180 n_items=n_items,
181 enabled=enabled,
182 collapsed=collapsed,
183 )
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/formatting_html.py:138, in summarize_vars(variables)
137 def summarize_vars(variables):
--> 138 vars_li = "".join(
139 f"<li class='xr-var-item'>{summarize_variable(k, v)}</li>"
140 for k, v in variables.items()
141 )
143 return f"<ul class='xr-var-list'>{vars_li}</ul>"
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/formatting_html.py:139, in <genexpr>(.0)
137 def summarize_vars(variables):
138 vars_li = "".join(
--> 139 f"<li class='xr-var-item'>{summarize_variable(k, v)}</li>"
140 for k, v in variables.items()
141 )
143 return f"<ul class='xr-var-list'>{vars_li}</ul>"
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/formatting_html.py:115, in summarize_variable(name, var, is_index, dtype, preview)
113 preview = preview or escape(inline_variable_array_repr(variable, 35))
114 attrs_ul = summarize_attrs(var.attrs)
--> 115 data_repr = short_data_repr_html(variable)
117 attrs_icon = _icon("icon-file-text2")
118 data_icon = _icon("icon-database")
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/formatting_html.py:30, in short_data_repr_html(array)
28 if hasattr(internal_data, "_repr_html_"):
29 return internal_data._repr_html_()
---> 30 text = escape(short_data_repr(array))
31 return f"<pre>{text}</pre>"
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/formatting.py:572, in short_data_repr(array)
570 return limit_lines(repr(array.data), limit=40)
571 elif array._in_memory or array.size < 1e5:
--> 572 return short_numpy_repr(array)
573 else:
574 # internal xarray array type
575 return f"[{array.size} values with dtype={array.dtype}]"
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/formatting.py:548, in short_numpy_repr(array)
547 def short_numpy_repr(array):
--> 548 array = np.asarray(array)
550 # default to lower precision so a full (abbreviated) line can fit on
551 # one line with the default display_width
552 options = {"precision": 6, "linewidth": OPTIONS["display_width"], "threshold": 200}
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/common.py:143, in AbstractArray.__array__(self, dtype)
142 def __array__(self: Any, dtype: DTypeLike = None) -> np.ndarray:
--> 143 return np.asarray(self.values, dtype=dtype)
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/variable.py:512, in Variable.values(self)
509 #property
510 def values(self):
511 """The variable's data as a numpy.ndarray"""
--> 512 return _as_array_or_item(self._data)
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/variable.py:252, in _as_array_or_item(data)
238 def _as_array_or_item(data):
239 """Return the given values as a numpy array, or as an individual item if
240 it's a 0d datetime64 or timedelta64 array.
241
(...)
250 TODO: remove this (replace with np.asarray) once these issues are fixed
251 """
--> 252 data = np.asarray(data)
253 if data.ndim == 0:
254 if data.dtype.kind == "M":
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/indexing.py:552, in MemoryCachedArray.__array__(self, dtype)
551 def __array__(self, dtype=None):
--> 552 self._ensure_cached()
553 return np.asarray(self.array, dtype=dtype)
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/indexing.py:549, in MemoryCachedArray._ensure_cached(self)
547 def _ensure_cached(self):
548 if not isinstance(self.array, NumpyIndexingAdapter):
--> 549 self.array = NumpyIndexingAdapter(np.asarray(self.array))
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/indexing.py:522, in CopyOnWriteArray.__array__(self, dtype)
521 def __array__(self, dtype=None):
--> 522 return np.asarray(self.array, dtype=dtype)
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/indexing.py:423, in LazilyIndexedArray.__array__(self, dtype)
421 def __array__(self, dtype=None):
422 array = as_indexable(self.array)
--> 423 return np.asarray(array[self.key], dtype=None)
File ~/miniforge3/lib/python3.9/site-packages/xarray/coding/variables.py:70, in _ElementwiseFunctionArray.__array__(self, dtype)
69 def __array__(self, dtype=None):
---> 70 return self.func(self.array)
File ~/miniforge3/lib/python3.9/site-packages/xarray/coding/variables.py:137, in _apply_mask(data, encoded_fill_values, decoded_fill_value, dtype)
133 def _apply_mask(
134 data: np.ndarray, encoded_fill_values: list, decoded_fill_value: Any, dtype: Any
135 ) -> np.ndarray:
136 """Mask all matching values in a NumPy arrays."""
--> 137 data = np.asarray(data, dtype=dtype)
138 condition = False
139 for fv in encoded_fill_values:
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/indexing.py:423, in LazilyIndexedArray.__array__(self, dtype)
421 def __array__(self, dtype=None):
422 array = as_indexable(self.array)
--> 423 return np.asarray(array[self.key], dtype=None)
File ~/miniforge3/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:93, in NetCDF4ArrayWrapper.__getitem__(self, key)
92 def __getitem__(self, key):
---> 93 return indexing.explicit_indexing_adapter(
94 key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
95 )
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/indexing.py:712, in explicit_indexing_adapter(key, shape, indexing_support, raw_indexing_method)
690 """Support explicit indexing by delegating to a raw indexing method.
691
692 Outer and/or vectorized indexers are supported by indexing a second time
(...)
709 Indexing result, in the form of a duck numpy-array.
710 """
711 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support)
--> 712 result = raw_indexing_method(raw_key.tuple)
713 if numpy_indices.tuple:
714 # index the loaded np.ndarray
715 result = NumpyIndexingAdapter(np.asarray(result))[numpy_indices]
File ~/miniforge3/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:106, in NetCDF4ArrayWrapper._getitem(self, key)
104 with self.datastore.lock:
105 original_array = self.get_array(needs_lock=False)
--> 106 array = getitem(original_array, key)
107 except IndexError:
108 # Catch IndexError in netCDF4 and return a more informative
109 # error message. This is most often called when an unsorted
110 # indexer is used before the data is loaded from disk.
111 msg = (
112 "The indexing operation you are attempting to perform "
113 "is not valid on netCDF4.Variable object. Try loading "
114 "your data into memory first by calling .load()."
115 )
File src/netCDF4/_netCDF4.pyx:4406, in netCDF4._netCDF4.Variable.__getitem__()
File src/netCDF4/_netCDF4.pyx:5350, in netCDF4._netCDF4.Variable._get()
File src/netCDF4/_netCDF4.pyx:1927, in netCDF4._netCDF4._ensure_nc_success()
RuntimeError: NetCDF: HDF error
<xarray.Dataset>
Dimensions: (time: 9125, bnds: 2)
Coordinates:
* time (time) object 1875-01-01 12:00:00 ... 1899-12-31 12:00:00
lat float64 14.61
lon float64 36.25
height float64 ...
Dimensions without coordinates: bnds
Data variables:
time_bnds (time, bnds) object ...
lat_bnds (bnds) float64 ...
lon_bnds (bnds) float64 ...
tas (time) float32 ...
Attributes: (12/48)
Conventions: CF-1.7 CMIP-6.2
activity_id: CMIP
branch_method: standard
branch_time_in_child: 0.0
branch_time_in_parent: 0.0
comment: none
... ...
title: CMCC-ESM2 output prepared for CMIP6
variable_id: tas
variant_label: r1i1p1f1
license: CMIP6 model data produced by CMCC is licensed und...
cmor_version: 3.6.0
tracking_id: hdl:21.14100/88ee8b49-0196-4c37-b1cd-a3cc6e2e598e
It should be noted that when I try to print the second Xarray without selecting a smaller domain (basically the .sel) it can be printed. Why does it not work when I use the sel command? I have confirmed that the coordinates I want are indeed inside c2 (my second Xarray). Also all my xarrays, with or without the sel command, are of type xarray.core.dataset.Dataset.
In order to identify the problem, I changed the .sel command. First I wanted to see the coordinates that are available in my Xarray dataset. For Lon the nearest was 35 and for lat the closes was 14.60733. First I tried to .sel only the Lon. The following is the command
d1 = cc2.sel(lon = 35.)
d1
and the result I get is
time, (time), object, 1875-01-01 12:00:00 ... 1899-12-...
lat, (lat), float64, -90.0 -89.06 -88.12 ... 89.06 90.0
lon, (), float64, 35.0
height, (), float64, 2.0
Data variables:
time_bnds, (time, bnds), object, 1875-01-01 00:00:00... 1900-01-...
lat_bnds
(lat, bnds)
float64
-90.0 -89.53 -89.53 ... 89.53 90.0
lon_bnds, (bnds), float64, ...
tas, (time, lat), float32, ...
In this dataset I saw that the coordinate for the lat = 14.60732984
can be found in the index 111 which I found by trial and error by printing using the following command
cc2['lat'][111].
which gives the following outcome.
array(14.60732984)
lat, (), float64, 14.61
lon, (), float64, 35.0
height, (), float64, 2.0
Now I tried the same procedure for lat both for values 14.60732984 and 14.61 but I get the following error
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/coordinates.py:421, in remap_label_indexers(obj, indexers, method, tolerance, **indexers_kwargs)
414 indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "remap_label_indexers")
416 v_indexers = {
417 k: v.variable.data if isinstance(v, DataArray) else v
418 for k, v in indexers.items()
419 }
--> 421 pos_indexers, new_indexes = indexing.remap_label_indexers(
422 obj, v_indexers, method=method, tolerance=tolerance
423 )
424 # attach indexer's coordinate to pos_indexers
425 for k, v in indexers.items():
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/indexing.py:121, in remap_label_indexers(data_obj, indexers, method, tolerance)
119 for dim, index in indexes.items():
120 labels = grouped_indexers[dim]
--> 121 idxr, new_idx = index.query(labels, method=method, tolerance=tolerance)
122 pos_indexers[dim] = idxr
123 if new_idx is not None:
File ~/miniforge3/lib/python3.9/site-packages/xarray/core/indexes.py:241, in PandasIndex.query(self, labels, method, tolerance)
237 raise KeyError(
238 f"not all values found in index {coord_name!r}"
239 )
240 else:
--> 241 indexer = self.index.get_loc(label_value)
242 elif label.dtype.kind == "b":
243 indexer = label
File ~/miniforge3/lib/python3.9/site-packages/pandas/core/indexes/base.py:3623, in Index.get_loc(self, key, method, tolerance)
3621 return self._engine.get_loc(casted_key)
3622 except KeyError as err:
-> 3623 raise KeyError(key) from err
3624 except TypeError:
3625 # If we have a listlike key, _check_indexing_error will raise
3626 # InvalidIndexError. Otherwise we fall through and re-raise
3627 # the TypeError.
3628 self._check_indexing_error(key)
KeyError: 14.60732984
I'm taking an online python course (EpiSkills, which uses the Jupyter notebook) that was written in Python 2.7, and I'm on Python 3.6.4 so I have run into a few compatibility issues along the way. Most of the time I've been able to stumble through, but can't figure out this one, so was hoping someone might be able to help.
I start with the following packages:
import pandas as pd
import epipy
import seaborn as sns
%pylab inline
import statsmodels.api as sm
from scipy import stats
import numpy as np
And use the following code to create a pandas series and model:
multivar_model = sm.formula.glm('age ~ onset_to_hospital + onset_to_death +
data=my_data).fit()
new_data = pd.Series([6, 8, 'male'], index=['onset_to_hospital', 'onset_to_death', 'sex'])
When I try to use this to the following code, I throw the error that I've attached:
multivar_model.predict(new_data)
NameError part1
NameError part2
The intended output is meant to be this:
array([ 60.6497459])
I know that a lot of NameErrors are because something has been specified in the local, not global, environment but I'm unsure how to correct it in this instance. Any help is much appreciated.
Thanks!
C
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\compat.py in call_and_wrap_exc(msg, origin, f, *args, **kwargs)
116 try:
--> 117 return f(*args, **kwargs)
118 except Exception as e:
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\eval.py in eval(self, expr, source_name, inner_namespace)
165 return eval(code, {}, VarLookupDict([inner_namespace]
--> 166 + self._namespaces))
167
<string> in <module>()
NameError: name 'onset_to_death' is not defined
The above exception was the direct cause of the following exception:
PatsyError Traceback (most recent call last)
<ipython-input-79-e0364e267da7> in <module>()
----> 1 multivar_model.predict(new_data)
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\statsmodels\base\model.py in predict(self, exog, transform, *args, **kwargs)
774 exog_index = exog.index
775 exog = dmatrix(self.model.data.design_info.builder,
--> 776 exog, return_type="dataframe")
777 if len(exog) < len(exog_index):
778 # missing values, rows have been dropped
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\highlevel.py in dmatrix(formula_like, data, eval_env, NA_action, return_type)
289 eval_env = EvalEnvironment.capture(eval_env, reference=1)
290 (lhs, rhs) = _do_highlevel_design(formula_like, data, eval_env,
--> 291 NA_action, return_type)
292 if lhs.shape[1] != 0:
293 raise PatsyError("encountered outcome variables for a model "
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\highlevel.py in _do_highlevel_design(formula_like, data, eval_env, NA_action, return_type)
167 return build_design_matrices(design_infos, data,
168 NA_action=NA_action,
--> 169 return_type=return_type)
170 else:
171 # No builders, but maybe we can still get matrices
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\build.py in build_design_matrices(design_infos, data, NA_action, return_type, dtype)
886 for factor_info in six.itervalues(design_info.factor_infos):
887 if factor_info not in factor_info_to_values:
--> 888 value, is_NA = _eval_factor(factor_info, data, NA_action)
889 factor_info_to_isNAs[factor_info] = is_NA
890 # value may now be a Series, DataFrame, or ndarray
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\build.py in _eval_factor(factor_info, data, NA_action)
61 def _eval_factor(factor_info, data, NA_action):
62 factor = factor_info.factor
---> 63 result = factor.eval(factor_info.state, data)
64 # Returns either a 2d ndarray, or a DataFrame, plus is_NA mask
65 if factor_info.type == "numerical":
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\eval.py in eval(self, memorize_state, data)
564 return self._eval(memorize_state["eval_code"],
565 memorize_state,
--> 566 data)
567
568 __getstate__ = no_pickling
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\eval.py in _eval(self, code, memorize_state, data)
549 memorize_state["eval_env"].eval,
550 code,
--> 551 inner_namespace=inner_namespace)
552
553 def memorize_chunk(self, state, which_pass, data):
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\compat.py in call_and_wrap_exc(msg, origin, f, *args, **kwargs)
122 origin)
123 # Use 'exec' to hide this syntax from the Python 2 parser:
--> 124 exec("raise new_exc from e")
125 else:
126 # In python 2, we just let the original exception escape -- better
~\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\patsy\compat.py in <module>()
PatsyError: Error evaluating factor: NameError: name 'onset_to_death' is not defined
age ~ onset_to_hospital + onset_to_death + sex
^^^^^^^^^^^^^^
I'm using a library called pysd to translate vensim files to Python, but when I try to do it (library functions) I get a parse error but don't understand what it means.
This is my log.
ParseError Traceback (most recent call last)
<ipython-input-1-9b0f6b9bac1f> in <module>()
1 get_ipython().magic(u'pylab inline')
2 import pysd
----> 3 model = pysd.read_vensim('201520_1A_Volare_Ev.Tecnica.itmx')
/Library/Python/2.7/site-packages/pysd/pysd.pyc in read_vensim(mdl_file)
45 """
46 from .vensim2py import translate_vensim
---> 47 py_model_file = translate_vensim(mdl_file)
48 model = PySD(py_model_file)
49 model.mdl_file = mdl_file
/Library/Python/2.7/site-packages/pysd/vensim2py.pyc in translate_vensim(mdl_file)
651 for section in file_sections:
652 if section['name'] == 'main':
--> 653 model_elements += get_model_elements(section['string'])
654
655 # extract equation components
/Library/Python/2.7/site-packages/pysd/vensim2py.pyc in get_model_elements(model_str)
158 """
159 parser = parsimonious.Grammar(model_structure_grammar)
--> 160 tree = parser.parse(model_str)
161
162 class ModelParser(parsimonious.NodeVisitor):
/Library/Python/2.7/site-packages/parsimonious/grammar.pyc in parse(self, text, pos)
121 """
122 self._check_default_rule()
--> 123 return self.default_rule.parse(text, pos=pos)
124
125 def match(self, text, pos=0):
/Library/Python/2.7/site-packages/parsimonious/expressions.pyc in parse(self, text, pos)
108
109 """
--> 110 node = self.match(text, pos=pos)
111 if node.end < len(text):
112 raise IncompleteParseError(text, node.end, self)
/Library/Python/2.7/site-packages/parsimonious/expressions.pyc in match(self, text, pos)
125 node = self.match_core(text, pos, {}, error)
126 if node is None:
--> 127 raise error
128 return node
129
ParseError: Rule 'escape_group' didn't match at '' (line 1, column 20243).
.itmx is an iThink extension, which unfortunately PySD doesn't support (yet). In the future, we'll work out a conversion pathway that lets you bring these in.
I have a 2000 rows data frame and I'm trying to slice the same data frame into two and combine them together.
t1 = test[:10, :]
t2 = test[20:, :]
temp = t1.rbind(t2)
temp.show()
Then I got this error:
---------------------------------------------------------------------------
EnvironmentError Traceback (most recent call last)
<ipython-input-37-8daeb3375743> in <module>()
2 t2 = test[20:, :]
3 temp = t1.rbind(t2)
----> 4 temp.show()
5 print len(temp)
6 print len(test)
/usr/local/lib/python2.7/dist-packages/h2o/frame.pyc in show(self, use_pandas)
383 print("This H2OFrame has been removed.")
384 return
--> 385 if not self._ex._cache.is_valid(): self._frame()._ex._cache.fill()
386 if H2ODisplay._in_ipy():
387 import IPython.display
/usr/local/lib/python2.7/dist-packages/h2o/frame.pyc in _frame(self, fill_cache)
423
424 def _frame(self, fill_cache=False):
--> 425 self._ex._eager_frame()
426 if fill_cache:
427 self._ex._cache.fill()
/usr/local/lib/python2.7/dist-packages/h2o/expr.pyc in _eager_frame(self)
67 if not self._cache.is_empty(): return self
68 if self._cache._id is not None: return self # Data already computed under ID, but not cached locally
---> 69 return self._eval_driver(True)
70
71 def _eager_scalar(self): # returns a scalar (or a list of scalars)
/usr/local/lib/python2.7/dist-packages/h2o/expr.pyc in _eval_driver(self, top)
81 def _eval_driver(self, top):
82 exec_str = self._do_it(top)
---> 83 res = ExprNode.rapids(exec_str)
84 if 'scalar' in res:
85 if isinstance(res['scalar'], list): self._cache._data = [float(x) for x in res['scalar']]
/usr/local/lib/python2.7/dist-packages/h2o/expr.pyc in rapids(expr)
163 The JSON response (as a python dictionary) of the Rapids execution
164 """
--> 165 return H2OConnection.post_json("Rapids", ast=expr,session_id=H2OConnection.session_id(), _rest_version=99)
166
167 class ASTId:
/usr/local/lib/python2.7/dist-packages/h2o/connection.pyc in post_json(url_suffix, file_upload_info, **kwargs)
515 if __H2OCONN__ is None:
516 raise ValueError("No h2o connection. Did you run `h2o.init()` ?")
--> 517 return __H2OCONN__._rest_json(url_suffix, "POST", file_upload_info, **kwargs)
518
519 def _rest_json(self, url_suffix, method, file_upload_info, **kwargs):
/usr/local/lib/python2.7/dist-packages/h2o/connection.pyc in _rest_json(self, url_suffix, method, file_upload_info, **kwargs)
518
519 def _rest_json(self, url_suffix, method, file_upload_info, **kwargs):
--> 520 raw_txt = self._do_raw_rest(url_suffix, method, file_upload_info, **kwargs)
521 return self._process_tables(raw_txt.json())
522
/usr/local/lib/python2.7/dist-packages/h2o/connection.pyc in _do_raw_rest(self, url_suffix, method, file_upload_info, **kwargs)
592 raise EnvironmentError(("h2o-py got an unexpected HTTP status code:\n {} {} (method = {}; url = {}). \n"+ \
593 "detailed error messages: {}")
--> 594 .format(http_result.status_code,http_result.reason,method,url,detailed_error_msgs))
595
596
EnvironmentError: h2o-py got an unexpected HTTP status code:
500 Server Error (method = POST; url = http://localhost:54321/99/Rapids).
detailed error messages: []
If I count rows (len(temp)), it works find. Also if I change the slicing index a little bit, it works find too. For example, if I change to this, it shows the data frame.
t1 = test[:10, :]
t2 = test[:5, :]
Do I miss something here? Thanks.
Unclear what happened without more information (logs would probably say why the rbind did not take).
What version are you using? I tried your code with iris on the bleeding edge and it all worked as expected.
By the way, rbind is typically going to be expensive, especially since what you're semantically after is a subset:
test[range(10) + range(20,test.nrow),:]
should also give you the desired subset (with caveat that you make the full list of row indices in python and pass it over REST to h2o).
I'm trying to reproduce coal mining example with deterministic function for switchpoint instead of using theano's switch function. Code:
%matplotlib inline
import matplotlib.pyplot as plt
import pymc3
import numpy as np
import theano.tensor as t
import theano
data = np.hstack((np.random.poisson(15,1000),np.random.poisson(2,100)))
plt.plot(data)
#theano.compile.ops.as_op(itypes=[t.lscalar, t.dscalar,t.dscalar],otypes=[t.dvector])
def rate1(sw,mu1,mu2):
n = len(data)
out = np.empty(n)
out[:sw] = mu1
out[sw:] = mu2
return out
with pymc3.Model() as dis:
switchpoint = pymc3.DiscreteUniform('switchpoint',lower=0, upper=len(data)-1)
mu1 = pymc3.Exponential('mu1', lam=1.)
mu2 = pymc3.Exponential('mu2',lam=1.)
disasters=pymc3.Poisson('disasters', mu=rate1, observed = data)
But this code rise an error:
--------------------------------------------------------------------------- KeyError Traceback (most recent call
last) c:\program files\git\theano\theano\tensor\type.py in
dtype_specs(self)
266 'complex64': (complex, 'theano_complex64', 'NPY_COMPLEX64')
--> 267 }[self.dtype]
268 except KeyError:
KeyError: 'object'
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call
last) c:\program files\git\theano\theano\tensor\basic.py in
constant_or_value(x, rtype, name, ndim, dtype)
407 rval = rtype(
--> 408 TensorType(dtype=x_.dtype, broadcastable=bcastable),
409 x_.copy(),
c:\program files\git\theano\theano\tensor\type.py in init(self,
dtype, broadcastable, name, sparse_grad)
49 self.broadcastable = tuple(bool(b) for b in broadcastable)
---> 50 self.dtype_specs() # error checking is done there
51 self.name = name
c:\program files\git\theano\theano\tensor\type.py in dtype_specs(self)
269 raise TypeError("Unsupported dtype for %s: %s"
--> 270 % (self.class.name, self.dtype))
271
TypeError: Unsupported dtype for TensorType: object
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call
last) c:\program files\git\theano\theano\tensor\basic.py in
as_tensor_variable(x, name, ndim)
201 try:
--> 202 return constant(x, name=name, ndim=ndim)
203 except TypeError:
c:\program files\git\theano\theano\tensor\basic.py in constant(x,
name, ndim, dtype)
421 ret = constant_or_value(x, rtype=TensorConstant, name=name, ndim=ndim,
--> 422 dtype=dtype)
423
c:\program files\git\theano\theano\tensor\basic.py in
constant_or_value(x, rtype, name, ndim, dtype)
416 except Exception:
--> 417 raise TypeError("Could not convert %s to TensorType" % x, type(x))
418
TypeError: ('Could not convert FromFunctionOp{rate1} to TensorType',
)
During handling of the above exception, another exception occurred:
AsTensorError Traceback (most recent call
last) in ()
14 mu2 = pymc3.Exponential('mu2',lam=1.)
15 #rate1 = pymc3.switch(switchpoint >= np.arange(len(data)), mu1,mu2)
---> 16 disasters=pymc3.Poisson('disasters', mu=rate1, observed = data)
C:\Users\User\Anaconda3\lib\site-packages\pymc3\distributions\distribution.py
in new(cls, name, *args, **kwargs)
19 if isinstance(name, str):
20 data = kwargs.pop('observed', None)
---> 21 dist = cls.dist(*args, **kwargs)
22 return model.Var(name, dist, data)
23 elif name is None:
C:\Users\User\Anaconda3\lib\site-packages\pymc3\distributions\distribution.py
in dist(cls, *args, **kwargs)
32 def dist(cls, *args, **kwargs):
33 dist = object.new(cls)
---> 34 dist.init(*args, **kwargs)
35 return dist
36
C:\Users\User\Anaconda3\lib\site-packages\pymc3\distributions\discrete.py
in init(self, mu, *args, **kwargs)
185 super(Poisson, self).init(*args, **kwargs)
186 self.mu = mu
--> 187 self.mode = floor(mu).astype('int32')
188
189 def random(self, point=None, size=None, repeat=None):
c:\program files\git\theano\theano\gof\op.py in call(self,
*inputs, **kwargs)
598 """
599 return_list = kwargs.pop('return_list', False)
--> 600 node = self.make_node(*inputs, **kwargs)
601
602 if config.compute_test_value != 'off':
c:\program files\git\theano\theano\tensor\elemwise.py in
make_node(self, *inputs)
540 using DimShuffle.
541 """
--> 542 inputs = list(map(as_tensor_variable, inputs))
543 shadow = self.scalar_op.make_node(
544 *[get_scalar_type(dtype=i.type.dtype).make_variable()
c:\program files\git\theano\theano\tensor\basic.py in
as_tensor_variable(x, name, ndim)
206 except Exception:
207 str_x = repr(x)
--> 208 raise AsTensorError("Cannot convert %s to TensorType" % str_x, type(x))
209
210 # this has a different name, because _as_tensor_variable is the
AsTensorError: ('Cannot convert FromFunctionOp{rate1} to TensorType',
)
How i handle this?
The second thing - when i'm using the pymc3.switch function like this:
with pymc3.Model() as dis:
switchpoint = pymc3.DiscreteUniform('switchpoint',lower=0, upper=len(data)-1)
mu1 = pymc3.Exponential('mu1', lam=1.)
mu2 = pymc3.Exponential('mu2',lam=1.)
rate1 = pymc3.switch(switchpoint >= np.arange(len(data)), mu1,mu2)
disasters=pymc3.Poisson('disasters', mu=rate1, observed = data)
And next try to sample:
with dis:
step1 = pymc3.NUTS([mu1, mu2])
step2 = pymc3.Metropolis([switchpoint])
trace = pymc3.sample(10000, step = [step1,step2])
I get an error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
c:\program files\git\theano\theano\compile\function_module.py in __call__(self, *args, **kwargs)
858 try:
--> 859 outputs = self.fn()
860 except Exception:
TypeError: expected type_num 9 (NPY_INT64) got 7
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call last)
<ipython-input-4-3247d908f897> in <module>()
2 step1 = pymc3.NUTS([mu1, mu2])
3 step2 = pymc3.Metropolis([switchpoint])
----> 4 trace = pymc3.sample(10000, step = [step1,step2])
C:\Users\User\Anaconda3\lib\site-packages\pymc3\sampling.py in sample(draws, step, start, trace, chain, njobs, tune, progressbar, model, random_seed)
153 sample_args = [draws, step, start, trace, chain,
154 tune, progressbar, model, random_seed]
--> 155 return sample_func(*sample_args)
156
157
C:\Users\User\Anaconda3\lib\site-packages\pymc3\sampling.py in _sample(draws, step, start, trace, chain, tune, progressbar, model, random_seed)
162 progress = progress_bar(draws)
163 try:
--> 164 for i, strace in enumerate(sampling):
165 if progressbar:
166 progress.update(i)
C:\Users\User\Anaconda3\lib\site-packages\pymc3\sampling.py in _iter_sample(draws, step, start, trace, chain, tune, model, random_seed)
244 if i == tune:
245 step = stop_tuning(step)
--> 246 point = step.step(point)
247 strace.record(point)
248 yield strace
C:\Users\User\Anaconda3\lib\site-packages\pymc3\step_methods\compound.py in step(self, point)
11 def step(self, point):
12 for method in self.methods:
---> 13 point = method.step(point)
14 return point
C:\Users\User\Anaconda3\lib\site-packages\pymc3\step_methods\arraystep.py in step(self, point)
116 bij = DictToArrayBijection(self.ordering, point)
117
--> 118 apoint = self.astep(bij.map(point))
119 return bij.rmap(apoint)
120
C:\Users\User\Anaconda3\lib\site-packages\pymc3\step_methods\metropolis.py in astep(self, q0)
123
124
--> 125 q_new = metrop_select(self.delta_logp(q,q0), q, q0)
126
127 if q_new is q:
c:\program files\git\theano\theano\compile\function_module.py in __call__(self, *args, **kwargs)
869 node=self.fn.nodes[self.fn.position_of_error],
870 thunk=thunk,
--> 871 storage_map=getattr(self.fn, 'storage_map', None))
872 else:
873 # old-style linkers raise their own exceptions
c:\program files\git\theano\theano\gof\link.py in raise_with_op(node, thunk, exc_info, storage_map)
312 # extra long error message in that case.
313 pass
--> 314 reraise(exc_type, exc_value, exc_trace)
315
316
C:\Users\User\Anaconda3\lib\site-packages\six.py in reraise(tp, value, tb)
656 value = tp()
657 if value.__traceback__ is not tb:
--> 658 raise value.with_traceback(tb)
659 raise value
660
c:\program files\git\theano\theano\compile\function_module.py in __call__(self, *args, **kwargs)
857 t0_fn = time.time()
858 try:
--> 859 outputs = self.fn()
860 except Exception:
861 if hasattr(self.fn, 'position_of_error'):
TypeError: expected type_num 9 (NPY_INT64) got 7
Apply node that caused the error: Elemwise{Composite{Switch(GE(i0, i1), i2, i3)}}(InplaceDimShuffle{x}.0, TensorConstant{[ 0 1..1098 1099]}, InplaceDimShuffle{x}.0, InplaceDimShuffle{x}.0)
Toposort index: 11
Inputs types: [TensorType(int64, (True,)), TensorType(int32, vector), TensorType(float64, (True,)), TensorType(float64, (True,))]
Inputs shapes: [(1,), (1100,), (1,), (1,)]
Inputs strides: [(4,), (4,), (8,), (8,)]
Inputs values: [array([549]), 'not shown', array([ 1.07762995]), array([ 1.01502801])]
Outputs clients: [[Elemwise{eq,no_inplace}(Elemwise{Composite{Switch(GE(i0, i1), i2, i3)}}.0, TensorConstant{(1,) of 0}), Elemwise{Composite{Switch(GE(i0, i1), ((Switch(i2, i3, (i4 * log(i0))) - i5) - i0), i3)}}[(0, 0)](Elemwise{Composite{Switch(GE(i0, i1), i2, i3)}}.0, TensorConstant{(1,) of 0}, InplaceDimShuffle{x}.0, TensorConstant{(1,) of -inf}, TensorConstant{[ 13. 13... 0. 1.]}, TensorConstant{[ 22.55216... ]})]]
HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
Being simple analyst, should i learn all this stuff about theano to being able to work with my statistical problems? Is a new mcmc sampler with gradient feature is only one thing that should motivates me to switch from pymc2 to pymc3?
For your first question, it looks like you're trying to pass a theano function as a variable. You need to call the function with the other variables as arguments, which will then return a theano variable. Try changing your line to
disasters=pymc3.Poisson('disasters', mu=rate1(switchpoint, mu1, mu2), observed = data)
I couldn't reproduce the error in your second part; the sampling worked just fine for me.