My specific issue is exactly the title. I have a large raster processing script in python and need to perform a clump function which I cannot find in gdal / python nor have I figured out how to 'write it' myself.
I am becoming better with python all the time just still newish, but am learning R for this task. (installed R version 3.4.1 (2017-06-30))
I am able to get rpy2 installed within python after spending a little time learning R and through help on Stackoverflow I have been able to perform several 'tests' of rpy2.
The most helpful info in getting rpy2 to respond was to establish where your R is within your python session or script. from another Stack answer. As below:
import os
os.environ['PYTHONHOME'] = r'C:\Python27\ArcGIS10.3\Scripts\new_ve_folder\Scripts'
os.environ['PYTHONPATH'] = r'C:\Python27\ArcGIS10.3\Scripts\new_ve_folder\Lib\site-packages'
os.environ['R_HOME'] = r'C:\Program Files\R\R-3.4.1'
os.environ['R_USER'] = r'C:\Python27\ArcGIS10.3\Scripts\new_ve_folder\Lib\site-packages\rpy2'
However, the main tests listed in the documentation http://rpy.sourceforge.net/rpy2/doc-2.1/html/overview.html I cannot get to work.
import rpy2.robjects.tests
import unittest
# the verbosity level can be increased if needed
tr = unittest.TextTestRunner(verbosity = 1)
suite = rpy2.robjects.tests.suite()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'suite'
However:
import rpy2.robjects as robjects
pi = robjects.r['pi']
pi[0]
works just fine. as do a few other rpy2.robjects tests I have found. I can create string = ''' f <- functions ect ''' and call those from python.
If i use:
python -m 'rpy2.tests'
I get the following error.
r\Scripts>python -m 'rpy2.tests'
r\Scripts\python.exe: No module named 'rpy2
Documentation states: On Python 2.6, this should return that all tests were successful. I am using Python 2.7 and I also tried this in Python 3.3.
My script for clump starts as below:
I do not want to have to actually install the package names each time I run the script as they are already installed in my R Home.
I would like to use my python variables if possible.
I need to figure out why rpy2 does not respond as the documentation indicates, or why I am getting errors. And then after that figure out the correct way to write my clump portion of my python script.
packageNames = ('raster', 'rgdal')
if all(rpackages.isinstalled(x) for x in packageNames):
have_packages = True
else:
have_packages = False
if not have_packages:
utils = rpackages.importr('utils')
utils.chooseCRANmirror(ind=1)
packnames_to_install = [x for x in packageNames if not rpackages.isinstalled(x)]
if len(packnames_to_install) > 0:
utils.install_packages(StrVector(packnames_to_install))
from rpy2.robjects.packages import importr
import rpy2.robjects as robjects
There are several ways I have found to call the raster and clump options from R, however, if I cannot get rpy2 to respond correctly, I am not going to get these to work at all But since several other tests work I am not positive.
raster = robjects.r['raster']
raster = importr('raster')
clump = raster.clump
clump = robjects.r.clump
type(raster.clump)
tempDIR = r"C:\Users\script_out\temp"
slope_recode = os.path.join(tempDIR, "step2b_input.img")
outfile = os.path.join(tempDIR, "Rclumpfile.img")
raster.clump(slope_recode, filename=outfile, direction=4, gaps=True, format='HFA', overwrite=True)
Which results in a large amount of errors.
Traceback (most recent call last):
File "C:/Python27/ArcGIS10.3/Scripts/new_ve_folder/Scripts/rpy2_practice.py", line 97, in <module>
raster.clump(slope_recode, filename=outfile, direction=4, gaps=True, format='HFA', overwrite=True)
File "C:\Python27\ArcGIS10.3\Scripts\new_ve_folder\lib\site-packages\rpy2\robjects\functions.py", line 178, in __call__
return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
File "C:\Python27\ArcGIS10.3\Scripts\new_ve_folder\lib\site-packages\rpy2\robjects\functions.py", line 106, in __call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in (function (classes, fdef, mtable) :
unable to find an inherited method for function 'clump' for signature '"character"'
Issues:
testing rpy2 in command line and script (both produce errors, but I am still able to use basic rpy2
importing the R packages so as not to install them each time
finally getting my clump script called correctly
If I have missed something basic, please point me in the right direction. Thanks all.
For your first problem, replace suite = rpy2.robjects.tests.suite() with suite = rpy2.tests.suite().
For your third problem (getting clump to work correctly), you need to create a RasterLayer object in R using the image. I'm not familiar with the raster package, so I can't give you the exact steps.
I will point out the arcpy module is not "pythonic". Normally, strings of filenames are just strings in Python. arcpy is weird in using plain strings to represent objects like map layers.
In your example, slope_recode is just a string. That's why you got the error unable to find an inherited method for function 'clump' for signature '"character"'. It means slope_recode was passed to R as a character value (which it is), and the clump function expects a RasterLayer object. It doesn't know how to handle character values.
I got this all to work with the below code.
import warnings
os.environ['PATH'] = os.path.join(scriptPath, 'path\\my_VE\\R\\R-3.4.2\\bin\\x64')
os.environ['PYTHONHOME'] = os.path.join(scriptPath, 'path\\my_VE\\Scripts\\64bit')
os.environ['PYTHONPATH'] = os.path.join(scriptPath, 'path\\my_VE\\Lib\\site-packages')
os.environ['R_HOME'] = os.path.join(scriptPath, 'path\\my_VE\\R\\R-3.4.2')
os.environ['R_USER'] = os.path.join(scriptPath, 'path\\my_VE\\Scripts\\new_ve_folder\\Scripts\\rpy2')
#
import platform
z = platform.architecture()
print(z)
## above will confirm you are working on 64 bit
gc.collect()
## this code snippit will tell you which library is being Read
command = 'Rscript'
cmd = [command, '-e', ".libPaths()"]
print(cmd)
x = subprocess.Popen(cmd, shell=True)
x.wait()
import rpy2.robjects.packages as rpackages
import rpy2.robjects as robjects
from rpy2.robjects import r
import rpy2.interactive.packages
from rpy2.robjects import lib
from rpy2.robjects.lib import grid
# # grab r packages
print("loading packages from R")
## fails at this point with the following error
## Error: cannot allocate vector of size 232.6 Mb when working with large rasters
rpy2.robjects.packages.importr('raster')
rpy2.robjects.packages.importr('rgdal')
rpy2.robjects.packages.importr('sp')
rpy2.robjects.packages.importr('utils')
# rpy2.robjects.packages.importr('memory')
# rpy2.robjects.packages.importr('dplyr')
rpy2.robjects.packages.importr('data.table')
grid.activate()
# set python variables for R code names
raster = robjects.r['raster']
writeRaster = robjects.r['writeRaster']
# setwd = robjects.r['setwd']
clump = robjects.r['clump']
# head = robjects.r['head']
crs = robjects.r['crs']
dim = robjects.r['dim']
projInfo = robjects.r['projInfo']
slope_recode = os.path.join(tempDIR, "_lope_recode.img")
outfile = os.path.join(tempDIR, "Rclumpfile.img")
recode = raster(slope_recode) # this is taking the image and reading it into R raster package
## https://stackoverflow.com/questions/47399682/clear-r-memory-using-rpy2
gc.collect() # No noticeable effect on memory usage
time.sleep(2)
gc.collect() # Finally, memory usage drops
R = robjects.r
R('memory.limit()')
R('memory.limit(size = 65535)')
R('memory.limit()')
print"starting Clump with rpy2"
clump(recode, filename=outfile, direction=4, gaps="True", format="HFA")
final = raster(outfile)
final = crs("+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0,-0,-0,-0,0 +no_defs")
print ("clump file created, CRS accurate, next step")
Related
#!/bin/bash/python3
from scapy3k.all import *``
import subprocess
import os
def ifac111():
pkts = sniff(filter="tcp", iface="tun0", count=100)
inp = input('want to see a \'pdfdump?\' \n Y/N--> ')
fag = pkts.summary()
print('-' * 60)
what_df = pkts.show()
print("^^^ Here you got {} packets {}.".format("100", "scanned"))
print("The {} ones are {} and second ones are just {} command".format("first", "summary", "show" ))
print(inp)
if inp == 'Y':
pkts[0].pdfdump()
else:
print("got ya \f hex0")
while 1 > 0:
SSS = input('enter your command\'s here:-> \t ') #\t moves 4 spaces
if SSS == 'packets':
ifac111()
elif SSS == 'nworkscan':
os.system('sudo nmap localhost/24')
elif SSS == 'Virusscan':
os.system('sudo chkrootkit')
elif SSS == 'clear':
subprocess.call('clear')
when i run the pdfdump i get this error
Traceback (most recent call last):
File "scapy2.py", line 27, in <module>
ifac111()
File "scapy2.py", line 16, in ifac111
pkts[0].pdfdump()
File "/usr/local/lib/python3.6/dist-packages/scapy3k/packet.py", line 418, in pdfdump
canvas = self.canvas_dump(**kargs)
File "/usr/local/lib/python3.6/dist-packages/scapy3k/packet.py", line 428, in canvas_dump
canvas = pyx.canvas.canvas()
NameError: name 'pyx' is not defined
sorry if the question is stupid I'm new with coding and been trying to do some research with no result I used ICMP instead of TCP also before on my old os but its not working after changing to parrot os and when I run pdfdump I get that error above
This is a bug in scapy3k.packet indeed - it tries to import pyx and silently continue if there's any import error, which leads to your problem:
try:
import pyx
except ImportError:
pass
You should fill a bug report in the project's github - the package should properly declare it's dependencies on 3rd part packages so they get installed alongside, and it should definitly not silence the fact that a required import failed.
In the meantime you can try installing pyx yourself - it might just work, or it might break elsewhere depending on versions compat.
You should use scapy instead of scapy3k, it contains those fixes and a better interception of PyX
FTR:
scapy3k = fork based on scapy 2.2.0 which used to be the only one supporting python 3
scapy (from secdev) = the original one, up-to-date that also works with python 3 since scapy 2.4.0+
Hard to believe this issue is still around after 3yrs. This is to inform anyone else who lands here wanting to know how to fix this problem.
It is due to "Intended" lazy import placed on the installation as it would require GB/s of downloaded files to support the TeX backend. Thus, this is an intentional error by the scapy project itself.
You need to install MikTeX and LiveTex as it is not evident.
LiveTex can be downloaded from here
Please let me know the procedure, what settings needed to call R packages from Python.
I use Spyder (python 2.7)
I am trying to call apriori package from Python.it fails in arules.
Any help would be appreciated.
I already tried the following
import rpy2
from rpy2 import *
import rpy2.interactive as r
arules = r.packages.importr("arules")
from rpy2.robjects.vectors import ListVector
od = OrderedDict()
od["supp"] = 0.0005
od["conf"] = 0.7
od["target"] = 'rules'
result = ListVector(od)
dataset = 'c:/Apriori/testcase.txt'
my_rules = arules.apriori(dataset, parameter=result)
print('my_rules',my_rules)
File setup:
...\Project_Folder
...\Project_Folder\Project.py
...\Project_folder\Script\TestScript.py
I'm attempting to have Project.py import modules from the folder Script based on user input.
Python Version: 3.4.2
Ideally, the script would look something like
q = str(input("Input: "))
from Script import q
However, python does not recognize q as a variable when using import.
I've tried using importlib, however I cannot figure out how to import from the Script folder mentioned above.
import importlib
q = str(input("Input: "))
module = importlib.import_module(q, package=None)
I'm not certain where I would implement the file path.
Repeat of my answer originally posted at How to import a module given the full path?
as this is a Python 3.4 specific question:
This area of Python 3.4 seems to be extremely tortuous to understand, mainly because the documentation doesn't give good examples! This was my attempt using non-deprecated modules. It will import a module given the path to the .py file. I'm using it to load "plugins" at runtime.
def import_module_from_file(full_path_to_module):
"""
Import a module given the full path/filename of the .py file
Python 3.4
"""
module = None
try:
# Get module name and path from full path
module_dir, module_file = os.path.split(full_path_to_module)
module_name, module_ext = os.path.splitext(module_file)
# Get module "spec" from filename
spec = importlib.util.spec_from_file_location(module_name,full_path_to_module)
module = spec.loader.load_module()
except Exception as ec:
# Simple error printing
# Insert "sophisticated" stuff here
print(ec)
finally:
return module
# load module dynamically
path = "<enter your path here>"
module = import_module_from_file(path)
# Now use the module
# e.g. module.myFunction()
I did this by defining the entire import line as a string, formatting the string with q and then using the exec command:
imp = 'from Script import %s' %q
exec imp
I'm using Rpy2 on windows 7 64 and having trouble loading a package:
in R:
using(mi)
in python:
from rpy2.robjects.packages import importr
mi=importr('mi')
---------------------------------------------------------------------------
RRuntimeError Traceback (most recent call last)
<ipython-input-30-2d393a6df544> in <module>()
----> 1 mi=importr('mi')
C:\Anaconda\lib\site-packages\rpy2\robjects\packages.pyc in importr(name, lib_loc, robject_translations, signature_translation, suppress_messages, on_conflict, data)
397 if _package_has_namespace(rname,
398 _system_file(package = rname)):
--> 399 env = _get_namespace(rname)
400 version = _get_namespace_version(rname)[0]
401 exported_names = set(_get_namespace_exports(rname))
RRuntimeError: Error in loadNamespace(name) : there is no package called 'm
Any suggestions?
I had a similar problem:
rpy2.rinterface.RRuntimeError: Error in loadNamespace(name) : there is no package called speedglm
I noticed that the issue is that rpy2 does not know the location of all R libraries. In my case, typing (in R)
.libPaths()
gave me
[1] "/home/nbarjest/R/x86_64-redhat-linux-gnu-library/3.4"
[2] "/usr/lib64/R/library"
[3] "/usr/share/R/library"
While, typing (in Python 3)
import rpy2.rinterface
rpy2.rinterface.set_initoptions((b'rpy2', b'--no-save', b'--no-restore', b'--quiet'))
from rpy2.robjects.packages import importr
base = importr('base')
print(base._libPaths())
gave me only
[1] "/home/nbarjest/R/x86_64-redhat-linux-gnu-library/3.4"
I couldn't find a way to append the other two paths to base._libpath(). If you find a way to do it, please let me know. I used another workaround:
import rpy2
import rpy2.robjects as RObjects
from rpy2.robjects.packages import importr
utils = importr("utils")
d = {'print.me': 'print_dot_me', 'print_me': 'print_uscore_me'}
try:
thatpackage = importr('speedglm', robject_translations = d, lib_loc = "/home/nbarjest/R/x86_64-redhat-linux-gnu-library/3.4")
except:
try:
thatpackage = importr('speedglm', robject_translations = d, lib_loc = "/usr/lib64/R/library")
except:
thatpackage = importr('speedglm', robject_translations = d, lib_loc = "/usr/share/R/library")
This works. I hope other people who have the same problem find this useful.
For me, in importr, the argument lib_loc inside it worked, putting the first path that appears in the output of .libPaths() in R, like:
importr('name package', lib_loc="/home/nbarjest/R/x86_64-redhat-linux-gnu-library/3.4"),
where the path is the path in the output example of the #Nbarjest answer.
In python: Check the version of R being used by rpy2
import rpy2.robjects as robjects
robjects.r['version']
Check your rpy2 library location
base = importr('base')
print(base._libPaths())
In R: Check your R library location for this version of r
.libPaths()
copy the library installed in your version of r to the folder used by rpy2.
I also have this problem,and i copy the package i need to base._libPaths() ,here , and it works.
import rpy2.robjects as objects
from rpy2.robjects.packages import importer
base = importr('base')
base._libPaths()[0]
I had a similar problem. I had to uninstall R and reinstall it with admin rights, then reinstall the R package while running R with admin rights, so it would install to the standard library location (not a personal library). Then add R to the PATH variable, and reinstall rpy2.
This is was cross-posted, and answered, on the issue tracker for rpy2: https://bitbucket.org/rpy2/rpy2/issue/265/windows-error-in-loadnamespace
I am trying to use my R9 280 GPU with pyopencl but i cant get it to work as my python and pyopencl knowledge is a bit on the dry side. I wonder anyone help me out or at least direct me to the right direction. Below simple python script reads upload.txt to memory and and tries to match randomly created numbers after using basic multiply function. Basicly i cannot write a kernel for this to work. As you see it is only opening the job for gpu and needs a kernel to work on reading the file, checking randomly created numbers and writing the match to a file. Thanks in advance.
#! /usr/bin/env python
import random
import pyopencl as cl
from os import environ, system
def Multiply(x,y):
return 4*y/x
environ["PYOPENCL_CTX"]="0"
ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)
run = True
target = open("upload.txt", 'rb')
readit = target.read()
while run:
r = 8;
p = random.randrange(1,5000);
q = Multiply(r,p);
z = str(q);
print z;
if z in readit:
file = open('found.txt', 'ab')
file.write(str(z))
file.close()
print "Found ",z
run = False
I don't have comment privileges, but please see the following
import random
import pyopencl as cl
from os import environ, system
def Multiply(x,y):
return 4*y/x
environ["PYOPENCL_CTX"]="0"
ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)
# run = True -- no need for this
with open("upload.txt", 'rb') as target: # pythonic & safe way of opening files
readit = target.read()
while True:
r = 8;
p = random.randrange(1,5000);
q = Multiply(r,p);
z = str(q);
print z;
if z in readit:
with open('found.txt', 'ab') as File: # can't use file as variable name, hence the caps
File.write(z)
print("Found ", z)
break
I can not help you anymore until you be specific about your question - please tell more about the error/glitches you are getting & what output do you expect
Edit: Since you are dealing with graphics processing & stuff, data source files may become extremely large - for which I suggest using lazy (buffered) read, see this for a very simple implementation