I have a python snippet which works just fine to run GLMNET on np.array X and y. However, when X is a column sparse matrix from scipy, the code fails as rpy2 is not able to convert X. Am I making an obvious mistake?
A MCVE is:
import numpy as np
from scipy import sparse
from rpy2 import robjects
import rpy2.robjects.packages as rpackages
from rpy2.robjects import numpy2ri
from rpy2.robjects import pandas2ri
if __name__ == "__main__":
X = sparse.rand(5, 20, density=0.1)
y = np.random.randn(5)
numpy2ri.activate()
pandas2ri.activate()
utils = rpackages.importr('utils')
utils.chooseCRANmirror(ind=1)
if not rpackages.isinstalled('glmnet'):
utils.install_packages("glmnet")
glmnet = rpackages.importr('glmnet')
glmnet = robjects.r['glmnet']
glmnet_fit = glmnet(X, y, intercept=False, standardize=False)
And when I run it I get a NotImplementedError:
Conversion 'py2ri' not defined for objects of type '<class 'scipy.sparse.csc.csc_matrix'>'
Could I provide X in a different way? I'd be surprised if rpy2 could not handle sparse matrices.
You can create a sparse matrix with rpy2 as follows:
import numpy as np
import rpy2.robjects as ro
from rpy2.robjects.packages import importr
from scipy import sparse
X = sparse.rand(5, 20, density=0.1).tocoo()
r_Matrix = importr("Matrix")
r_Matrix.sparseMatrix(
i=ro.IntVector(X.row + 1),
j=ro.IntVector(X.col + 1),
x=ro.FloatVector(X.data),
dims=ro.IntVector(X.shape))
There is indeed no converter Python -> R for your object type included in rpy2. Your Python object is not a conventional arrays but a sparse matrix as you note it (scipy.sparse.csc.csc_matrix to be specific), implemented as one of the numerical extensions available for numpy. As numpy itself is not even required to use rpy2 the support for extension of numpy is rather sparse, at the notable exception of pandas since data tables are ubiquitous.
You may want to write your own converter from css_matrix to gcCMatrix in the R package Matrix (https://stat.ethz.ch/R-manual/R-devel/library/Matrix/html/dgCMatrix-class.html) as the package glmnet appears to be able to handle them.
Writing a custom converter will require how to map or copy the content of the Python object to its chosen R counterpart, but once done plugging the code into rpy2 should be quite easy:
https://rpy2.github.io/doc/v2.9.x/html/generated_rst/s4class.html#custom-conversion
Consider opening an issue as a "feature request" on the rpy2 issue tracker, and reporting progress and outcome, with the hope to see this turn into a pull request complete with unit tests
Also a quick solution that might work would be to save the sparse matrix file temporarily.
import numpy as np
import rpy2.robjects as ro
import warnings
from rpy2.rinterface import RRuntimeWarning
import rpy2.robjects.numpy2ri as numpy2ri
from scipy.io import mmwrite
mmwrite('temp.mtx',matrix)
ro.r('X <- readMM("temp.mtx")')
I would be very interested though, if someone comes with a custom converter for avoiding that copy to disk.
Related
I'm translating a R code to Python code and I need to implement a hierarchical time series clustering with dtw distance for time series of different lengths. The only solution that I find is to use the same library that I used in R (dtwclust, with tsclust) through rpy2 package of Python:
import rpy2
import rpy2.robjects.packages as rpackages
from rpy2.robjects.vectors import StrVector
from rpy2.robjects.packages import importr
from mRMRr import *
from datasets import *
import rpy2.robjects.packages as packages
packa = packages.importr('dtwclust')
Z = packa.tsclust(list_t_matrix_prof,type = "hierarchical", distance = "dtw_basic")
Now I have to access to the cluster assigned to each element and in R i can get this by Z#cluster.
Can anyone tell me which is the corresponding command in Python?
My attempt to use rpy2 in a Jupyter (iPython) notebook fails at the point where I wish to use tidyr::nest() to make a dataframe that has some elements which are a series. (This cannot be avoided, it is necessary for the next step of the analysis.)
After trying many things (including advice from the package home I've managed to get most of the way to the end, failing to understand rpy2 with tidyr::nest at the last step. (Some of the following example may be superfluous.)
How can I fix this?
% Import packages
from rpy2 import robjects
from rpy2.robjects import Formula, Environment
from rpy2.robjects.vectors import IntVector, FloatVector
from rpy2.robjects.lib import grid
from rpy2.robjects.packages import importr, data
from rpy2.rinterface_lib.embedded import RRuntimeError
import warnings
base = importr('base')
datasets = importr('datasets')
from functools import partial
import rpy2.robjects.lib.tidyr as tidyr
from collections import OrderedDict
from rpy2.robjects.vectors import (StrVector,
IntVector)
from rpy2.robjects.lib.tidyr import DataFrame
from rpy2.robjects import rl
tidyr = importr('tidyr')
dplyr = importr('dplyr')
% Obtain the R dataset "iris"
iris = data(datasets).fetch('iris')['iris']
% Use only two columns necessary for the "tidyr::nest" trial
dataf = (
DataFrame(iris)
.select(base.c(1,5))
)
print(dataf.head())
% Failure occurs below
irisNested = tidyr.nest(dataf, data=rl('Sepal.Length'))
EDIT: This isn't a question anymore, as it turns out my original effort was correct. A fragment of another attempt to solve the problem caused it to fail. However, I asked the question as I didn't find the various sources of help available provided a clear way to address my issue, I felt I made a lucky educated guess.
import rpy2.robjects as ro
from rpy2.robjects import pandas2ri
pandas2ri.activate()
Removing this solves the problem.
I have this code
import scipy.sparse as sparse
import numpy as np
id = np.eye(13)
vals, vecs = sparse.linalg.eigsh(id, k=6)
vals
which is just the example code from the documentation here.
I am running it in a Python 2.7 console and I get the following error message:
AttributeError: 'module' object has no attribute 'linalg'
Does anyone know why this happens?
Try this code
import scipy.sparse.linalg as sp
import numpy as np
id = np.eye(13)
vals, vecs = sp.eigsh(id, k=6)
vals
This happens because linalg is a directory and not source code i.e it is a sub-package. And I guess this causes the issue because some of the Scipy sub modules do not have __init__.py, Maybe the devs did this to reduce loading times of top-level packages. You can find this information in Scipy Organization section in this link
I am using rpy2-2.2.2 with the new free Enthought python distribution that includes numpy 1.6.0 and python 2.7.2. I easy_installed rpy2 which resulted in v. 2.2.2 being installed and all tests were successful.
The problem I'm having is with code I wrote that worked fine with rpy2 2.1.8 and python 2.6. The issue is in converting from numpy to R for arrays.
Here is a snippet of the relevant code:
import rpy2
import rpy2.rinterface as rinterface
import rpy2.robjects as rob
import rpy2.rlike.container as rlc
import numpy as np
import rpy2.robjects.numpy2ri
r = rob.r
...
HGr = rob.conversion.py2ri(HG_reg)
RHSr = rob.conversion.py2ri(RHS)
#
CalData = rlc.TaggedList([HGr,RHSr],tags=('hg','rhs'))
CalData = rob.DataFrame(CalData)
r('''library(pls)''')
#rob.globalEnv["HGr"] = HGr
#rob.globalEnv["RHSr"] = RHSr
rob.globalenv["CalData"] = CalData
# perform the PLS regression
if wetlflag:
HGresults = r.plsr(r("hg ~ rhs.1 + rhs.2 + rhs.3 + rhs.4"),data=CalData,validation="LOO")
I will gladly admit it's not the most elegant way to do things, but it worked before and now when I need to provide results all is broken (!). The error I get is the following:
Traceback (most recent call last):
File "Mercury_PLS_WL_DF.py", line 224, in <module>
HGr = rob.conversion.py2ri(HG_reg)
File "/Library/Frameworks/Python.framework/Versions/7.1/lib/python2.7/site-packages/rpy2-2.2.2dev_20110726-py2.7-macosx-10.5-i386.egg/rpy2/robjects/__init__.py", line 134, in default_py2ri
raise(ValueError("Nothing can be done for the type %s at the moment." %(type(o))))
ValueError: Nothing can be done for the type <type 'numpy.ndarray'> at the moment.
I found the discussion here and got the impression that numpy arrays are now automatically converted to R arrays, but commenting out the rob.conversion.py2ri(HG_reg) statements and using the numpy arrays directly also seems to fail. Am I missing something obvious? Why would this break between 2.1.8 and 2.2.2?
From http://rpy.sourceforge.net/rpy2/doc-2.2/html/numpy.html#from-numpy-to-rpy2:
Warning
In earlier versions of rpy2, the import was all that was needed to have the conversion. A side-effect when importing a module can lead to problems, and there is now an extra step to make the conversion active: call the function rpy2.robjects.activate().
So put rpy2.robjects.activate() after the import and you should be fine.
The following code is supposed to create a heatmap in rpy2
import numpy as np
from rpy2.robjects import r
data = np.random.random((10,10))
r.heatmap(data)
However, it results in the following error
Traceback (most recent call last):
File "z.py", line 8, in <module>
labRow=rowNames, labCol=colNames)
File "C:\Python25\lib\site-packages\rpy2\robjects\__init__.py", line 418, in __call__
new_args = [conversion.py2ri(a) for a in args]
File "C:\Python25\lib\site-packages\rpy2\robjects\__init__.py", line 93, in default_py2ri
raise(ValueError("Nothing can be done for the type %s at the moment." %(type(o))))
ValueError: Nothing can be done for the type <type 'numpy.ndarray'> at the moment.
From the documentation I learn that r.heatmap expects "a numeric matrix". How do I convert np.array to the required data type?
You need to add
import rpy2.robjects.numpy2ri
rpy2.robjects.numpy2ri.activate()
See more in rpy2 documentation numpy section (here for the older 2.x version)
Prior to 2.2.x the import alone was sufficient.
That import alone is sufficient to
switch an automatic conversion of
numpy objects into rpy2 objects.
Why make this an optional import,
while it could have been included in
the function py2ri() (as done in the
original patch submitted for that
function) ?
Although both are valid and reasonable
options, the design decision was taken
in order to decouple rpy2 from numpy
the most, and do not assume that
having numpy installed automatically
meant that a programmer wanted to use
it.
For rpy2 2.2.4 I had to add:
import rpy2.robjects.numpy2ri
rpy2.robjects.numpy2ri.activate()
For me (2.2.1) the following also worked (as documented on http://rpy.sourceforge.net/rpy2/doc-2.2/html/numpy.html):
import rpy2.robjects as ro
from rpy2.robjects.numpy2ri import numpy2ri
ro.conversion.py2ri = numpy2ri