I have a Python subprocess to call R:
cmd = ['Rscript', 'Rcode.R', 'file_to_process.txt']
out = subprocess.run(cmd, universal_newlines = True, stdout = subprocess.PIPE)
lines = out.stdout.splitlines() #split stdout
My R code first checks if the 'ape' package is installed before proceeding:
if (!require("ape")) install.packages("ape")
library(ape)
do_R_stuff.......
return_output_to_Python
Previously, the whole process from Python to R worked perfectly - R was called and processed output was returned to Python - until I added the first line (if (!require("ape")) install.packages("ape")). Now Python reports: "there is no package called 'ape'" (i.e. when I uninstall ape in R). I have tried wait instructions in both the R and Python scripts but I can't get it working. When checking, the R code works in isolation.
The full error output from Python is:
Traceback (most recent call last):
File ~\Documents\GitHub\wolPredictor\wolPredictor_MANUAL_parallel.py:347 in <module>
if __name__ == '__main__': main()
File ~\Documents\GitHub\wolPredictor\wolPredictor_MANUAL_parallel.py:127 in main
cophen, purge, pge_incr, z, _ = R_cophen('{}/{}'.format(dat_dir, tree), path2script) #get Dist Mat from phylogeny in R
File ~\Documents\GitHub\wolPredictor\wolPredictor_MANUAL_parallel.py:214 in R_cophen
purge = int(np.max(cophen) * 100) + 1 #max tree pw distance
File <__array_function__ internals>:5 in amax
File ~\anaconda3\lib\site-packages\numpy\core\fromnumeric.py:2754 in amax
return _wrapreduction(a, np.maximum, 'max', axis, None, out,
File ~\anaconda3\lib\site-packages\numpy\core\fromnumeric.py:86 in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity
Loading required package: ape
Installing package into 'C:/Users/Windows/Documents/R/win-library/4.1'
(as 'lib' is unspecified)
Error in contrib.url(repos, "source") :
trying to use CRAN without setting a mirror
Calls: install.packages -> contrib.url
In addition: Warning message:
In library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called 'ape'
Execution halted
Point 1: The path to R libraries when using R in a standalone mode may not be the same when using Rscript.
Point 2: The error says there was difficulty in finding the CRAN repository, so perhaps the options that set the repos were not set for the Rscript environment. They can be set in the call to install packages or with a Sys.setenv() call.
Ther OP wrote: "Thanks #IRTFM I had to set a new sub process in a new .R file specifically for the install, but the CRAN mirror was key, I never realised it would be an issue as it's not a requirement on my local machine (not sure why it becomes an issue thru subprocess)."
The places to find more information are the ?Startup help page and the ?Rscript page. Rscript has many fewer defaults. Even the usual set of recommended packages may not get loaded by default. The Rscript help page includes these flags which could be used for debugging and setting a proper path to the libraries desired.:
--verbose
gives details of what Rscript is doing.
--default-packages=list
where list is a comma-separated list of package names or NULL. Sets the environment variable R_DEFAULT_PACKAGES which determines the packages loaded on startup.
Here is a previous similar SO question with an answer that includes some options for construction of a proper working environment: Rscript: There is no package called ...?
There are a few R packages on CRAN that aid in overcoming some of the differences between programming for standalone R and Rsript.
getopt: https://cran.r-project.org/web/packages/getopt/index.html
optparse: https://cran.r-project.org/web/packages/optparse/index.html (styled after a similar Python package.)
argparse: https://cran.r-project.org/web/packages/argparse/index.html
I solved the issue (thanks to #IRTFM) by placing the if-then-install.packages code in a separate Rscript (including the CRAN mirror):
if (!require("ape")) install.packages("ape", repos='http://cran.us.r-project.org')
which I then called using a separate Python subprocess in my Python routine
Related
I am trying to get my first Pyomo model running on my Ubuntu VM (Azure). I have Python3 and the COIN-OR solvers installed on this machine. No matter what solver I try, I get the same result.
Edit: changing the solver to couenne (it's a nonlinear problem) the Jupyter output looks like this. When I open the log files in the tmp directory, there is nothing in the couenne.log file and the pyomo files are the problem formulation. So I assume that Pyomo isn't communicating with the Couenne solver at all?
Solver log file: '/tmp/tmpezw0sov2_couenne.log'
Solver solution file: '/tmp/tmpq6afa7e8.pyomo.sol'
Solver problem files: ('/tmp/tmpq6afa7e8.pyomo.nl',)
ERROR: Solver (asl) returned non-zero return code (-1)
ERROR: See the solver log above for diagnostic information.
---------------------------------------------------------------------------
ApplicationError Traceback (most recent call last)
<ipython-input-6-486e3a9173f4> in <module>()
20 #instance = model.create_instance()
21 opt = SolverFactory('couenne', executable = solverpath_exe)
---> 22 opt.solve(model,tee=True,keepfiles=True)
23 #solver=SolverFactory(solvername,executable=solverpath_exe)
/home/ralphasher/.local/lib/python3.6/site-packages/pyomo/opt/base/solvers.py in solve(self, *args, **kwds)
598 logger.error("Solver log:\n" + str(_status.log))
599 raise pyutilib.common.ApplicationError(
--> 600 "Solver (%s) did not exit normally" % self.name)
601 solve_completion_time = time.time()
602 if self._report_timing:
ApplicationError: Solver (asl) did not exit normally
The "catch-all" exception is raised because the solver runs as a separate, non-Python process, so Python really can't tell what exactly went wrong with it, it just sees that the process has quit abnormally.
As such, solver log is the thing to go to as this is where the solver itself writes its status updates, so the specific error, whatever it is, should be reflected there.
If solver log is empty, this most probably means that the solver has failed to start at all (if solver process is run with stream redirection, the log file is opened -- thus created -- before the solver command is exec'd, so this is a common symptom when there's a problem with a program's startup). Since pyomo is the thing that starts the solver, getting the details of what exactly happens at the time of solver's start is where the answer lies.
According to pyomo solve command — Pyomo 5.6.6 documentation, you can use --info or --verbose command line options to increase the verbosity of the pyomo log.
If that still doesn't produce anything revealing, time to bring out the big guns:
run pyomo under pdb (pyomo is just a script so you can pass it to python -m pdb like any other; make sure to use the same python executable as in the script's shebang) and step through the code in pyomo machinery to see what exactly it does with the solver process (what info it passes, how it invokes it)
you'll be able to see defects in this process if there are (e.g. no info is actually passed) or repeat the same operations by hand to see the result firsthand; and/or
run the command under strace -f (to also monitor the solver's child process) and see if there are any obvious errors like an error to exec the solver command or any errors opening files and such.
Edit: So apparantly my install wasn't working. This pointed me to a mailing list Here where I figured out which commands I was missing. I have the answer for the update below. Now that I think about it, it does make sense. I just wish they'd put this somewhere simple on the dev pages.
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.update(name='aws-cli')
yb.buildTransaction()
yb.processTransaction()
I'm trying to perform an update using yumbase when a server first boots with my kickstart script. At the moment I have a rather crude python subprocess to do "yum update" and would like to make this better.
I'm trying to hook into Yumbase, but the documentation is quite scarce. I have had a look at both the source code and documentation on this page: http://yum.baseurl.org/wiki/5MinuteExamples
I've figured out how to list all packages but not the ones that need updating using an SO answer from 2008: Given an rpm package name, query the yum database for updates
I've also figured out it's a very simple 3-line process to install a new package:
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.install(name='aws-cli')
However the following doesn't work to "update" the package:
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.update(name='aws-cli')
So what I need is:
1: A way to list the packages that need updating, much like "yum check-update"
2: Install the packages above using "yum update"
From what I can see in the yum code, it doesn't seem to be written to be used as a library. The code you gave is not the right way to do it, there's much else happening behind the scenes.
Basically, as of yum-3.4.3, the process looks like this:
->yummain.__main__
<trap KeyboardInterrupt>
->yummain.user_main(sys.argv[1:], exit_code=True)
<check YUM_PROF,YUM_PDB envvars, wrap the following into debugger/profiler if set>
->yummain.main(args)
<set up locale, set up logging>
-><create a YumBaseCli (child of YumBase & YumOutput)>
<incl. fill a list field with YumCommand instances of known commands>
->cli.YumBaseCli.getOptionsConfig()
<parse args into the YumBaseCli instance, includes initializing plugins>
<obtain global yum lock>
<check write permissions for current dir>
->cli.YumBaseCli.doCommands()
<select a YumCommand from the list>
->YumCommand.needTs/needTsRemove if needed
->YumCommand.doCommand(self, self.basecmd, self.extcmds)
<handle errors & set error code if any>
'Resolving Dependencies'
->cli.YumBaseCli.buildTransaction()
<check for an unfinished transaction>
<resolve deps using the info written by the YumCommand into the object>
<honor clean_requirements_on_remove, protected_packages,
protected_multilib, perform some checks>
<handle errors & set error code if any>
'Dependencies Resolved'
->cli.YumBaseCli.doTransaction()
<download, transaction check, transaction test, transaction
using the info in the object>
<handle errors & set error code if any>
'Complete!'
<release global yum lock>
sys.exit(error_code)
As you can see, the main working sequence is embedded directly into main so you can only replicate this logic in-process by running it directly:
yummain.main(<sequence of cmdline arguments>)
Which is just the same as running a separate process minus process isolation.
This question concerns Matlab 2014b, Python 3.4 and Mac OS 10.10.
I have the following Python file tmp.py:
from statsmodels.tsa.arima_process import ArmaProcess
import numpy as np
def generate_AR_time_series():
arparams = np.array([-0.8])
maparams = np.array([])
ar = np.r_[1, -arparams]
ma = np.r_[1, maparams]
arma_process = ArmaProcess(ar, ma)
return arma_process.generate_sample(100)
I want to call generate_AR_time_series from Matlab so I used:
py.tmp.generate_AR_time_series()
which gave a vague error message
Undefined variable "py" or class "py.tmp.generate_AR_time_series".
To look into the problem further, I tried
tmp = py.eval('__import__(''tmp'')', struct);
which gave me a detailed but still obscured error message:
Python Error:
dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/scipy/special/_ufuncs.so, 2): Symbol
not found: __gfortran_stop_numeric_f08
Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/scipy/special/_ufuncs.so
Expected in: /Applications/MATLAB_R2014b.app/sys/os/maci64/libgfortran.3.dylib
in /opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/scipy/special/_ufuncs.so
I can call the function within Python just fine, so I guess the problem is with Matlab. From the detailed message, it seems that the problem lies in something is expected in the Matlab installation path, but of course Matlab installation path does not contain those things since they are third-party libraries for Python.
How to solve this problem?
Edit 1:
libgfortran.3.dylib can be found in a lot of places:
/Applications/MATLAB_R2014a.app/sys/os/maci64/libgfortran.3.dylib
/Applications/MATLAB_R2014b.app/sys/os/maci64/libgfortran.3.dylib
/opt/local/lib/gcc48/libgfortran.3.dylib
/opt/local/lib/gcc49/libgfortran.3.dylib
/opt/local/lib/libgcc/libgfortran.3.dylib
/Users/wdg/Documents/MATLAB/mcode/nativelibs/macosx/bin/libgfortran.3.dylib
Try:
setenv('DYLD_LIBRARY_PATH', '/usr/local/bin/');
For me, using the setenv approach from within MATLAB did not work. Also, MATLAB modifies the DYLD_LIBRARY_PATH variable during startup to include necessary libraries.
First, you have to make sure which version of gfortran scipy was linked against: in Terminal.app, enter otool -L /opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/scipy/special/_ufuncs.so and look for 'libgfortran' in the output.
It worked for me to copy $(MATLABROOT)/bin/.matlab7rc.sh to my home directory and change the line LDPATH_PREFIX='' in the mac section (around line 195 in my case) to LDPATH_PREFIX='/opt/local/lib/gcc49', or whatever path to libgfortran you found above.
This ensures that /opt/local/lib/gcc49/libgfortran.3.dylib is found before the MATLAB version, but leaves other paths intact.
Installed julia v0.3.2 into MacOSX 10.9.5
Did
Pkg.add("IJulia")
Pkg.add("PyPlot")
initiate ijulia with:
ipython notebook --profile=julia
This starts an IJulia notebook (it says IJ in the top left.
I enter using pyplot into the first line of iJulia, hit shift enter, and get this:
objc[21233]: Class TKApplication is implemented in both /Library/Frameworks/Tk.framework/Versions/8.5/Tk and /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk. One of the two will be used. Which one is undefined.
objc[21233]: Class TKMenu is implemented in both /Library/Frameworks/Tk.framework/Versions/8.5/Tk and /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk. One of the two will be used. Which one is undefined.
objc[21233]: Class TKContentView is implemented in both /Library/Frameworks/Tk.framework/Versions/8.5/Tk and /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk. One of the two will be used. Which one is undefined.
objc[21233]: Class TKWindow is implemented in both /Library/Frameworks/Tk.framework/Versions/8.5/Tk and /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk. One of the two will be used. Which one is undefined.
INFO: Loading help data...
Warning: requiring "pyplot" did not define a corresponding module.
OK, so my mac has the tk package installed in two places. If i remove the TK and TCL frameworks from Library/Frameworks, as suggested here:
http://michaelwelburn.com/2013/06/07/python-error-conflicting-tk-libraries-and-activetcl/
and here:
Python tk framework
then I get the following error when I try to start iJulia:
INFO: Loading help data...
Warning: requiring "pyplot" did not define a corresponding module.
The other authors said this worked so I am confused.
My /usr/local/bin directory includes these) executables:
tclselect tclsh tclsh8.5 tclsh8.6 tclvfse wish wish8.5 wish8.6
I hesitate to remove the system framework. I have the native mac python as well as the anaconda version (which has its own lib/tk) I am at a loss as to the next step.
EDIT: My julia code is this:
using PyPlot
# julia set
# (the familiar mandelbrot set is obtained by setting c==z initially)
function julia(z, c; maxiter=200)
for n = 1:maxiter
if abs2(z) > 4
return n-1
end
z = z*z + c
end
return maxiter
end
# varying the second argument to julia() tiny amounts results in a stunning variety of forms
#time m = [ uint8(julia(complex(r,i), complex(-.06,.67))) for i=1:-.002:-1, r=-1.5:.002:1.5 ];
# the notebook is able to display ColorMaps
get_cmap("RdGy")
imshow(m, cmap="RdGy", extent=[-1.5,1.5,-1,1])
each line executes fine in iJulia except the last line starting with imshow which gives this error:
PyError (PyObject_Call) <class '_tkinter.TclError'>
TclError('Can\'t find a usable tk.tcl in the following directories: \n /System/Library/Frameworks/Tk.framework/Versions/8.5/Resources/Scripts\n\n/System/Library/Frameworks/Tk.framework/Versions/8.5/Resources/Scripts/tk.tcl: version conflict for package "Tk": have 8.5.15, need exactly 8.5.9\nversion conflict for package "Tk": have 8.5.15, need exactly 8.5.9\n while executing\n"package require -exact Tk 8.5.9"\n (file "/System/Library/Frameworks/Tk.framework/Versions/8.5/Resources/Scripts/tk.tcl" line 20)\n invoked from within\n"source /System/Library/Frameworks/Tk.framework/Versions/8.5/Resources/Scripts/tk.tcl"\n ("uplevel" body line 1)\n invoked from within\n"uplevel #0 [list source $file]"\n\n\nThis probably means that tk wasn\'t installed properly.\n',)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/pyplot.py", line 2370, in imshow
ax = gca()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/pyplot.py", line 701, in gca
ax = gcf().gca(**kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/pyplot.py", line 343, in figure
**kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/matplotlib/backends/backend_tkagg.py", line 80, in new_figure_manager
window = Tk.Tk()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk/Tkinter.py", line 1764, in __init__
self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
while loading In[7], in expression starting on line 1
in pyerr_check at /Users/userme/.julia/v0.3/PyCall/src/exception.jl:58
in pycall at /Users/userme/.julia/v0.3/PyCall/src/PyCall.jl:85
in imshow at /Users/userme/.julia/v0.3/PyPlot/src/PyPlot.jl:370
I feel so stupid. My problem was the .bash_profile I had these lines
PATH="/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH}"
export PATH
leftover from a previous python install. I also had a bunch of executables in my /usr/local/bin folder from this old python install. When I deleted the python 2.7 folder before putting in anaconda, I forgot to delete these.
Now the error is gone. I hope this helps someone.
For the Warning:
Warning: requiring "*" did not define a corresponding module.
In some cases it helps to check if we use a stable version.
Ran into same issues with 0.38+pre-versions.
After switching back to 0.37 stable the issue was gone.
Run into this issue with PyCall:
Warning: requiring "PyCall" did not define a corresponding module.
Minimal code example, run on OSX and Python 3:
#! /usr/bin/env python3
from dialog import Dialog
d = Dialog(dialog="dialog")
sel = d.menu("Test",
choices=[ ("FooTag", "Foo", False, "FooHelp"),
("BarTag", "Bar", True, "BarHelp")]
)
I get the error message:
Traceback (most recent call last):
File "./test.py", line 5, in <module>
d = Dialog(dialog="dialog")
File "/usr/local/lib/python3.3/site-packages/dialog.py", line 1371, in __init__
self.backend_version())
File "/usr/local/lib/python3.3/site-packages/dialog.py", line 2012, in backend_version
"{1!r}".format(self._dialog_prg, output))
dialog.UnableToRetrieveBackendVersion: Unable to retrieve the version of the dialog-like backend: unable to parse the output of '/usr/local/bin/dialog --print-version': ''
However, dialog reports correctly:
> dialog --print-version
Version: 1.1-20100428
>
Nothing changes, if I use the absolut path /usr/local/bin/dialog. I guess, it is an install problem, but I've no clue.
As you have found yourself, the problem is caused by the fact that your dialog program prints its version on stdout whereas more recent versions print it on stderr. So, the string pythondialog tries to parse as a version number is empty in such a case.
pythondialog 3.2.2rc1 (latest release at the time of this writing) detects this situation and automatically reruns dialog --stdout --print-version, making sure this time to read from dialog's stdout stream. Since this is only done once at Dialog instance creation (the backend version being cached), the impact of this workaround is minimal.
It is better than your solution (passing use_stdout=True to the Dialog constructor), because it allows the rest of pythondialog to use dialog's stderr stream normally: the use of --stdout is only limited to retrieving the version of dialog. On the contrary, the solution you found will pass --stdout to all dialog calls which, according to the dialog(1) manual page, may fail "depending on the platform and your environment".
Had you reported this problem on the pythondialog issue tracker or mailing list, the workaround I mentioned here would certainly have been implemented earlier...
Your example with little fixes for on my computer (i changed False to 'False'). Your issue is that version is not matched against regex. This is code from pythondialog:
mo = self._print_version_cre.match(output)
if mo:
return mo.group("version")
else:
raise UnableToRetrieveBackendVersion(
"unable to parse the output of '{0} --print-version': "
"{1!r}".format(self._dialog_prg, output))
#this is located upper
_print_version_cre = re.compile(r"^Version:[ \t]+(?P<version>.+?)[ \t]*$", re.MULTILINE)
I've tested your version with this regexp and it matched on my computer, but I guess you have other version of pythondialog. So check _print_version_cre in your dialog.py and try to much with dialog version.
I found the solution myself:
dialog in OSX reports to stdout, not stderr.
By using
d = Dialog(dialog="dialog", use_stdout=True)
I get the expected result.