I'm having a problem running two successive python scripts in a single R script that calls the reticulate function. When I run them separately, it works perfectly. Am I missing something? Do I need some sort of function that closes the previous py_run before executing the second one? Note that I am working with the ArcGIS 'arcpy' tools.
Here is an example.
If I run this in an Rstudio session:
use_python("C:/Python27/ArcGISx6410.7/python.exe", required = T)
import("arcpy")
import("os")
import("glob")
import("re")
import("math")
pythonRun1 = paste0("modelName = ", "'", modelName, "'")
py_run_string(pythonRun1)
py_run_file("C:/GIS/Bathymetry_UdeM/Radial_Linear_Mean_R/Output/Workflow/InterpolateResults.py")
and then this in a second Rstudio session:
use_python("C:/Python27/ArcGISx6410.7/python.exe", required = T)
import("arcpy")
import("arcpy.sa")
import("os")
import("glob")
import("re")
import("math")
pythonRun2 = paste0("modelName = ", "'", modelName, "'")
py_run_string(pythonRun2)
py_run_file("C:/GIS/Bathymetry_UdeM/Radial_Linear_Mean_R/Output/Workflow/FocalMean.py")
it works perfectly. But when I'm trying to successively execute these two code blocks:
use_python("C:/Python27/ArcGISx6410.7/python.exe", required = T)
import("arcpy")
import("os")
import("glob")
import("re")
import("math")
pythonRun1 = paste0("modelName = ", "'", modelName, "'")
py_run_string(pythonRun1)
py_run_file("C:/GIS/Bathymetry_UdeM/Radial_Linear_Mean_R/Output/Workflow/InterpolateResults.py")
#########################################################################################################
use_python("C:/Python27/ArcGISx6410.7/python.exe", required = T)
import("arcpy")
import("arcpy.sa")
import("os")
import("glob")
import("re")
import("math")
pythonRun2 = paste0("modelName = ", "'", modelName, "'")
py_run_string(pythonRun2)
py_run_file("C:/GIS/Bathymetry_UdeM/Radial_Linear_Mean_R/Output/Workflow/FocalMean.py")
I get noData as my final output. The second portion of the script does not callback any error technically speaking, but the code does not execute what it is supposed to.
Does anyone have an idea as to what may cause the difference between these two methods (i.e., separate and successive py_run execution)?
Thanks everyone!
Try constructing a MRWE. It's hard to answer a question like this without being able to run the code. Here's a couple of starting points for troubleshooting, though.
I would not advise running reticulate::use_python twice. It is my understanding that once the Python interpreter is embedded into the R session it cannot be changed, so when this is called a second time at best it can have no effect, and at worst it could be causing issues.
You don't show any code where a value is bound to model_name, is it supposed to be changed in-between the two blocks of code?
Is there any state that is being modified in the first code block that is affecting the evaluation of the second code block? For example, if a variable is created in the first code block that is shadowing a variable being used in the second code block.
Try running the analogous code directly in Python and ensure that that works correctly. If not then it will be easier to debug in that environment.
Good luck!
Related
I recently wrote a simple function that gets the present value of an asset, the function PV works properly and I have tested it. The function prints well in my pv file. However, when I run this code in the main file it does not print the output to the terminal and just closes after taking the inputs. Is there a reason for this? For reference the functions just perform some simple mathematics problems.
Below is a minimal reproduction, both files are in the same folder.
in file_x you have a function that works like this:
def func(w,x,y,z):
z1 = w/(1 + (x/100))**(y*z)
print(z1)
return 0
this function is then imported into another file which is written like so
from file_x import func
w = int(input("Future Value: "))
x = float(input("ROR: "))
y = float(input("# of periods: "))
z = float(input("# of payment per anum: "))
func(w,x,y,z)
My problem is that when I run the 2nd file it takes the inputs properly but does not print the result from the function. Hope it was explained properly
In interactive mode an unused expression (here PV(w,x,y,z) which does returns a value and that value is not used) is printed to the terminal, because it is a common way to just display expression values.
But when you run a script in a non interactive mode (python script.py) then those unused expression are just discarded. As a general rule, you should never have such an unused expression in a script, because the behaviour will depend on the interactive mode of the interpretor.
So you should be explicit:
_ = PV(w,x,y,z) # value will be discarded even in interactive mode
or
print(PV(w,x,y,z)) # value will be printed even in non interactive mode
This is an old problem as is demonstrated as in https://community.intel.com/t5/Analyzers/Unable-to-view-source-code-when-analyzing-results/td-p/1153210. I have tried all the listed methods, none of them works, and I cannot find any more solutions on the internet. Basically vtune cannot find the custom python source file no matter what is tried. I am using the most recently version as of speaking. Please let me whether there is a solution.
For example, if you run the following program.
def myfunc(*args):
# Do a lot of things.
if __name__ = '__main__':
# Do something and call myfunc
Call this script main.py. Now use the newest vtune version (I have using Ubuntu 18.04), run the vtune-gui and basic hotspot analysis. You will not found any information on this file. However, a huge pile of information on Python and its other codes are found (related to your python environment). In theory, you should be able to find the source of main.py as well as cost on each line in that script. However, that is simply not happening.
Desired behavior: I would really like to find the source file and function in the top-down manual (or any really). Any advice is welcome.
VTune offer full support for profiling python code and the tool should be able to display the source code in your python file as you expected. Could you please check if the function you are expecting to see in the VTune results, ran long enough?
Just to confirm that everything is working fine, I wrote a matrix multiplication code as shown below (don't worry about the accuracy of the code itself):
def matrix_mul(X, Y):
result_matrix = [ [ 1 for i in range(len(X)) ] for j in range(len(Y[0])) ]
# iterate through rows of X
for i in range(len(X)):
# iterate through columns of Y
for j in range(len(Y[0])):
# iterate through rows of Y
for k in range(len(Y)):
result_matrix[i][j] += X[i][k] * Y[k][j]
return result_matrix
Then I called this function (matrix_mul) on my Ubuntu machine with large enough matrices so that the overall execution time was in the order of few seconds.
I used the below command to start profiling (you can also see the VTune version I used):
/opt/intel/oneapi/vtune/2021.1.1/bin64/vtune -collect hotspots -knob enable-stack-collection=true -data-limit=500 -ring-buffer=10 -app-working-dir /usr/bin -- python3 /home/johnypau/MyIntel/temp/Python_matrix_mul/mat_mul_method.py
Now open the VTune results in the GUI and under the bottom-up tab, order by "Module / Function / Call-stack" (or whatever preferred grouping is).
You should be able to see the the module (mat_mul_method.py in my case) and the function "matrix_mul". If you double click, VTune should be able to load the sources too.
I'm getting used to VSCode in my daily Data Science remote workflow due to LiveShare feature.
So, upon executing functions it just executes the first line of code; if I mark the whole region then it does work, but it's cumbersome way of dealing with the issue.
I tried number of extensions, but none of them seem to solve the problem.
def gini_normalized(test, pred):
"""Simple normalized Gini based on Scikit-Learn's roc_auc_score"""
gini = lambda a, p: 2 * roc_auc_score(a, p) - 1
return gini(test, pred)
Executing the beginning of the function results in error:
def gini_normalized(test, pred):...
File "", line 1
def gini_normalized(test, pred):
^
SyntaxError: unexpected EOF while parsing
There's a solution for PyCharm: Python Smart Execute - https://plugins.jetbrains.com/plugin/11945-python-smart-execute. Also Atom's Hydrogen doesn't have such issue either.
Any ideas regarding VSCode?
Thanks!
I'm a developer on the VSCode DataScience features. Just to make sure that I'm understanding correctly. You would like the shift-enter command to send the entire function to the Interactive Window if you run it on the definition of the function?
If so, then yes, we don't currently support that. Shift-enter can run line by line or run a section of code that you manually highlight. If you want, you can use #%% lines in your code to put functions into code cells. Then when you are in a cell shift-enter will run that entire cell, might be the best current approach for you.
That smart execute does look interesting, if you would like to file that as a suggestion you can use our GitHub here to get it on our backlog to look at.
https://github.com/Microsoft/vscode-python
Hi you could click the symbol before each line and turn it into > (the indented codes of the function was hidden now). Then if you select the whole line and the next line, shift+enter could run them together.
enter image description here
I'd like to run ipython script in python, ie:
code='''a=1
b=a+1
b
c'''
from Ipython import executor
for l in code.split("\n"):
print(executor(l))
that whould print
None
None
2
NameError: name 'c' is not defined
does it exists ? I searched the doc, but it does not seems to be (well) documented.
In short, depending on what you want to do and how much IPython features you want to include, you will need to do more.
First thing you need to know is that IPython separates its code into blocks.
Each block has its own result.
If you use blocks use this advice
If you don't any magic IPython provides you with and don't want any results given by each block, then you could just try to use exec(compile(script, "exec"), {}, {}).
If you want more than that, you will need to actually spawn an InteractiveShell-instance as features like %magic and %%magic will need a working InteractiveShell.
In one of my projects I have this function to execute code in an InteractiveShell-instance:
https://github.com/Irrational-Encoding-Wizardry/yuuno/blob/master/yuuno_ipython/ipython/utils.py#L28
If you want to just get the result of each expression,
then you should parse the code using the ast-Module and add code to return each result.
You will see this in the function linked above from line 34 onwards.
Here is the relevant except:
if isinstance(expr_ast.body[-1], ast.Expr):
last_expr = expr_ast.body[-1]
assign = ast.Assign( # _yuuno_exec_last_ = <LAST_EXPR>
targets=[ast.Name(
id=RESULT_VAR,
ctx=ast.Store()
)],
value=last_expr.value
)
expr_ast.body[-1] = assign
else:
assign = ast.Assign( # _yuuno_exec_last_ = None
targets=[ast.Name(
id=RESULT_VAR,
ctx=ast.Store(),
)],
value=ast.NameConstant(
value=None
)
)
expr_ast.body.append(assign)
ast.fix_missing_locations(expr_ast)
Instead doing this for every statement in the body instead of the last one and replacing it with some "printResult"-transformation will do the same for you.
I have the following code in a function:
phi = fourier_matrix(y, fs)
N = np.size(phi,axis=1)
x = np.ones(N)
for i in range(136):
W_pk = np.diag(x)
temp = pinv(np.dot(phi, W_pk))
q_k = np.dot(temp, y)
x = np.dot(W_pk, q_k)
where phi is (96,750), W_pk is (750,750) and q_k is (750,).
This throws the following error:
Python(12001,0x7fff99d8b380) malloc: * error for object 0x7fc71aa37200: incorrect checksum for freed object - object was probably modified after being freed.
* set a breakpoint in malloc_error_break to debug
If I comment the last dot product the error does not appear.
I think I need to free memory in some way or maybe do the dot product in a different way?
Also, this only happens when I run it from a mac. On windows or linux it does not throw the error.
Python is 3.6 (tried with 3.7), and numpy is 1.14.5, also tried with 1.15
Any help would be greatly appreciated, since I really need to make this work!
Thanks in advance.
EDIT I:
I tried this portion of the code on a jupyter notebook, and it didn't fail. This confused me even more! It fails when I run it in Visual Studio Code on a mac. The rest of my code, an algorithm to remove artifacts from a signal, works as it should until I add that last piece of code x = np.dot(W_pk, q_k). Maybe it works on jupyter because I don't run the rest of the algorithm there? but as I said, it only crashes on that last dot product.
EDIT II: I added the piece of code above the for loop to this question, because I found that the problem is somehow related to how x is being used. You see it's declared above as a float64 ndarray. When it reaches the last line of the for loop, the dot product returns a complex128 (should be complex64, don't know what's happening there) and overwrites the x array. The first time works, but the second time it crashes when trying to overwrite. If I use a new variable for the dot product, say z, then it does not crash! not sure why... but I need to overwrite x in each iteration.
Furthermore, if I do something like this:
z = np.dot(W_pk, q_k)
x = abs(z) #I don't need complex numbers at this point
Then it crashes with the same error on the first dot product (presumably):
temp = pinv(np.dot(phi, W_pk))
Also, the memory consumption is not that bad, around 110M according some measurements, and the same algorithm does not crash on iPython with twice the memory usage. This is what I find the most obscure, why doesn't it crash on iPython??