I am trying to call a python script as a child process within a node script. The output of the script is to be used within a callback. The code looks like this:
//myFunction.js
const myFunction = callback => {
let py = process.spawn('python', ['../folder/pyscript.py'], {
cwd: '../folder/'
});
let str = '';
py.stdout.on('data', data => {
str += data.toString();
}
py.stdout.on('end', () => {
callback(str);
}
}
exports.myFunction = myFunction;
This code works as expected when I directly run node myFunction.js (with an instance of myFunction within the script) and it works fine when I require the module in any other files within the same directory as myFunction.js.
But it fails with the following error when the module is required in a different higher level directory:
error: spawn python ENOENT
I'm guessing this has something to do with paths (value of cwd maybe?) but I can't seem to fix this. I've looked up similar questions but the answers aren't helping.
Any help will be appreciated. :)
Apparently, the issue is with the cwd. Everything in the script is relative to the path of the directory from where the script is invoked. So basically, running node myFunction.js from the project root directory (say ~/projects/myProject would set the cwd to ~/projects/myProject/../folder which would evaluate to ~/projects/folder. This is obviously incorrect, since in all probability, no directory named folder exists on the system, and thus this would lead to an ENOENT error.
The solution would be to construct the absolute path of your script in the code, perhaps by using the __dirname__ property in combination with the functionalities provided by the path module.
I struggled with this issue for days, before realizing that my script file was not getting picked up and spawned by nodeJS, because of some filepath issue.
Although I don't guarantee this will work for everyone, depending on their setup, this is what I did in my nodejs file:
let py = process.spawn('python', [__dirname + '../folder/pyscript.py']);
As you can see, I didn't have to use the {cwd: '../folder/'} option.
If your script is in the current directory as your javascript file, just do
let py = process.spawn('python', [__dirname + './pyscript.py']);
I should also point out that:
process.spawn('python', ['./pyscript.py']);
never worked for me and I spent days wondering why. Could find an answer until I tried this technique. Hope someone having this issue find this answer useful.
Using ${process.cwd()} worked for me... you can write it like this
let py = process.spawn('python', [`${process.cwd()}/folder/pyscript.py`]});
Related
I am trying to integrate python in iOS. I tried same thing as mentioned here - https://github.com/beeware/Python-Apple-support/tree/3.9
Here is my python script in Xcode projects
func RunPythonScript() -> PythonObject {
if let path = Bundle.main.path(forResource:"/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/Resources/",
ofType: nil) {
setenv("PYTHONPATH", path, 1)
setenv("PYTHONHOME", path, 1)
}
let sys = Python.import("sys")
sys.path.append("/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/PAS_10_11_v3/")
let file = Python.import("pythonscript")
let response = file.hello_world()
print(response)
return response
}
It builds successfully but when I call python program it end up saying
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'python3'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = '/Users/karimkhan/Library/Developer/CoreSimulator/Devices/C444D135-C393-4631-AFE2-FF5F86935EF6/data/Containers/Bundle/Application/642E9540-CBAF-448C-8E8D-856B8E5D03EC/PAS_10_11_v3.app/PAS_10_11_v3'
sys.base_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.base_exec_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.platlibdir = 'lib'
sys.executable = '/Users/karimkhan/Library/Developer/CoreSimulator/Devices/C444D135-C393-4631-AFE2-FF5F86935EF6/data/Containers/Bundle/Application/642E9540-CBAF-448C-8E8D-856B8E5D03EC/PAS_10_11_v3.app/PAS_10_11_v3'
sys.prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.exec_prefix = '/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14'
sys.path = [
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/python39.zip',
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/python3.9',
'/Users/runner/work/Python-Apple-support/Python-Apple-support/install/iOS/iphonesimulator.x86_64/python-3.9.14/lib/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000000108fd4600 (most recent call first):
<no Python frame>
I just got this working after days of trying! Using Python 3.11.
For anyone wondering, we're using a version of Python patched for iOS (from Beeware's Python Apple Support), and PythonKit (Swift pkg) to embed Python in an iOS app using Xcode. The goal (for me at least) is to do my UI in SwiftUI, but use my own Python logic on some of the app's data.
OP, I don't know what all of your problems might be (there could be several), but I am immediately noticing at least one thing wrong. You're looking on your mac for Python when your goal is to put Python itself inside of your app. You should be looking in the app for the directory containing the Python standard library ('python-stdlib'), as well as the 'lib-dynload' subdir, provided to you by Beeware's Python Apple Support repo. You want to set PYTHONPATH and PYTHONHOME to this combo of paths.
This post is intended to supplement to "How to embed a Python interpreter in an iOS app - presented by Łukasz Langa." on YT.
In the video the OP linked to, Lukasz's 'python-stdlib' has a different name and a very different directory structure than the latest version of Py for iOS provides. What you'll need to figure out is the path to the 'python-stdlib' dir, AND [probably] its subdir, 'lib-dynload'. Counterintuitively, you do need to specify both, even though the latter is a subdir of the former.
Your line:
if let path = Bundle.main.path(forResource:"/Users/projects/extra/python_apple_support/PAS_10_11_v3/PAS_10_11_v3/Resources/",
ofType: nil) {
Should read something like:
if let libPath = Bundle.main.path(forResource: "python-stdlib", ofType: nil),
let libPath2 = Bundle.main.path(forResource: "python-stdlib/lib-dynload", ofType: nil) {
let mergedPaths = "\(libPath):\(libPath2)"
...where 'python-stdlib' is the PATH of the python libraries directory--not just the directory name. (If that's confusing to a newbie, this is the same: let mergedPaths = libPath + ":" + libPath2.
To get an idea of the path, go to Product > Show Build Folder in Finder, then find Products/Debug-iphonesimulator, right click .app (the greyed / X'd out icon) > Show package contents.
Assuming you have properly copied this stuff into the project, you should be able to find 'python-stdlib' inside the .app. For me, it was right in the root of the .app (since that's where I effectively put it, by adding it to the GROUP (not dir!) called 'Resources'. (Because Resources is a group, whatever is in there will be in the root level of the .app, NOT in a dir called Resources. It seems there can be groups that are backed by folders, and groups that are not. This is an important distinction.)
Now, set PYTHONPATH and PYTHONHOME to mergedPaths like you were doing:
setenv("PYTHONHOME", mergedPaths, 1)
setenv("PYTHONPATH", mergedPaths, 1)
If you don't include the path to 'lib-dynload', then this will only work in the simulator. Why would it work at all? Lovely question... 🤷🏽
IMPORTANT: The script.py file Lukasz created for his custom Python code must go into the 'python-stdlib' directory for this to work. I imagine you could stick it anywhere, as long as you append its new path to mergedPaths, as can be seen in a project generated by Beeware's Briefcase. To append that path to the merged paths, you're just concatenating strings with a ":" between each path. I have not tested this.
COPYING THE LIBRARY PROPERLY
Here's something that screwed me up for a while:
When you copy over Python.xcframework and the std-lib, make sure you're creating folders and not groups (in the prompt after you drag and drop), else you will end up with hundreds of errors due to the flattening of directories resulting from creating groups and not dirs (think of every main.py from the lib ending up in the same folder--no bueno). Make sure your prompt looks exactly like this:
screenshot of said prompt ☝🏽
SIGNING THE PYTHON LIBRARY
Follow 'The Manual Way' > Step 6 # Python Apple Support > USAGE.md.
All I had to change was this line segment:
"$CODESIGNING_FOLDER_PATH/Contents/Resources/python-stdlib/lib-dynload"
Due to my directory structure ('python-stdlib' in root of .app), I changed it to:
"$CODESIGNING_FOLDER_PATH/python-stdlib/lib-dynload"
...omitting the extra directories in the path to 'lib-dynload'.
ALMOST FORGOT--module.modulemap!
Assuming you have created 'module.modulemap', and copied it to all three locations (in the Xcode project and in both Headers dirs in the framework), you'll want it to read like this:
module Python {
umbrella header "Python.h"
export *
link "python-stdlib"
}
...where Lukasz's read link Python, ours must read link python-stdlib, as above. I wouldn't be surprised this is actually the path to 'python-stdlib'. Again, mine is right in the root of the .app, so under that assumption, the name is also the path, here. I have not tested this theory.
NOT NECESSARILY RELEVANT TO THE SOLUTION BUT MAYBE ENTERTAINING
This part is pretty amazing. I've had this working in the simulator only for the last several days. For most of that time, on the device, it would import Random, but not Math (which was being called from Random). Then I figured how to point the app to both 'python-stdlib' and 'python-stdlib/lib-dynload'. Got a new error -- code signature invalid for '.../math.cpython-311-iphoneos.so' (that Python math module!) in 'lib-dynload'! Wow! It was finally SEEING the math module.
PARTIAL CONSOLE OUTPUT:
...(code signature in <45B34416-425D-3E01-BC39-CB7A8C170A0A> '/private/var/containers/Bundle/Application/1465B572-3399-4B76-B017-4EE168637AF5/SIXTH_try.app/python-stdlib/lib-dynload/math.cpython-311-iphoneos.so' not valid for use in process: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.)
Here's the amazing part... YESTERDAY, I told Github to tell me about any changes to Beeware's Python Apple Support. Last night at 7pm, I get a notification that dude updated the USAGE.md to add instructions for code signing! Ha! What are the chances that this would happen on exactly the day that I need it to happen. Anyway, I followed those instructions, ran on my phone, and WHUHBAM! MY IPHONE IS USING THE RANDOM MODULE TO GENERATE INTS UPON A BUTTON TAP! THE WHOLE GUI WAS MADE IN SWIFTUI. HOLY MOTHER#(#)!#$ IT'S WORKING! (I realize enthusiasm and emotion are not always welcome on SO, but I'm going to celebrate, and no one will stop me!)
I am fairly new to Python and .NET in general, but decided to ask more competent people, since I have been struggling with the issue of executing python script from Windows Forms.
The basic idea of my project is a desktop applicaton and the overall logic would be to read from a couple of selected check boxes, pass the values of those selections to my python script, from there I generate an excell table based on those results, and display this table back into the Windows Forms application.
Creating the table and managing to display it in the Desktop App is already done, but I am having serious issues with the communication between the two platforms, when it came to executing the script itself.
I have tried using IronPython and it worked perfectly, untill the fact that I found that Iron Python does not support CPython packages, like Pandas, which is build on numpy, and numpy apparantly is one of those packages. I looked over a lot of articles about this issue and the answers did not seem promising and most of the suggestions were to use pythonnet.
I tried to implement pythonnet, following numerous articles and all I managed to do, besides creating a bigger mess, is nothing as a result.
Finally, I decided to use C# Process class, but did not succeed also.
Would appreciate if there are any comments and suggestions on how to remedy this issue.
Python version: 3.7
Windows 10 (64 bit)
.NET Framework 4.7.2
Here is some of my code attempts in Windows Forms:
Implementation with the usage of the Process Class
Issue here is that I am not able to run this script due to the error messages that it cannot find the packages for the python script
var processStartInfo = new ProcessStartInfo
{
Arguments = "C:\\Users\\Dobromir\\PycharmProjects\\pythonProject\\main.py",
FileName = "C:\\Python27\\python.exe",
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = true
};
Process.Start(processStartInfo);
Implementation using IronPython (which was working before the usage of pandas package)
Issue here is the CPython packages limitation and errors.
For IronPython I had to downgrade to Python 2.7 in order to work with it. For the rest of the examples I am using Python 3.7
ScriptEngine pythonEngine = Python.CreateEngine();
var searchPaths = pythonEngine.GetSearchPaths();
searchPaths.Add(#"C:\Python27\Lib");
searchPaths.Add(#"C:\Users\Dobromir\PycharmProjects\pythonProject\venv\Lib\site-packages");
pythonEngine.SetSearchPaths(searchPaths);
List<String> argv = new List<String>();
argv.Add("Some Value1");
argv.Add("Some Value2");
ScriptSource pythonScript = pythonEngine.CreateScriptSourceFromFile("C:\\Users\\Dobromir\\PycharmProjects\\pythonProject\\main.py");
pythonEngine.GetSysModule().SetVariable("argv", argv);
pythonEngine.SetSearchPaths(searchPaths);
ScriptScope scope = pythonEngine.CreateScope();
pythonScript.Execute(scope);
Implementation of pythonnet
The issue that I got here is on the line using Py.GIL(). I believe it is having trouble finding the python files, and also tried giving the python37.dll in the variable pathToPython.
I received the error that Python.Runtime, Version=2.5.2.0, Culture=neutral....missmatch"
string pathToPython = #"C:\Users\Dobromir\AppData\Local\Programs\Python\Python37";
string path = pathToPython + "; " + Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine);
Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONHOME", pathToPython, EnvironmentVariableTarget.Process);
Console.WriteLine(path);
var lib = new[]
{
#"C:\\Users\\Dobromir\\PycharmProjects\\App37\\main.py",
Path.Combine(pathToPython, "Lib"),
Path.Combine(pathToPython, "DLLs")
};
string paths = string.Join("; ", lib);
Environment.SetEnvironmentVariable("PYTHONPATH", paths, EnvironmentVariableTarget.Process);
using (Py.GIL()) //Initialize the Python engine and acquire the interpreter lock
{
try
{
Console.WriteLine("I am working");
}
catch (PythonException error)
{
Console.WriteLine("Error occured: ", error.Message);
}
}
I Also tried creating a bash script to execute the python script and got the no module found error as well
I know that these are not the best implementations out there, but do the job.
My question is if someone has any idea on how to make this simple operation work I would be very grateful, thank you for your time and understanding
P.S - Apologies for the long post, wanted to write what I have tried before asking for help, but if someone is more interested I will provide additional information.
I did a project like this recently; a couple of things I would suggest to make it easy:
Confirm that the instance of python set in your env variables (WIN+R, sysdm.cpl, Advanced, env variables) is that of the instance of python you wish to use (do this for your python search path too!)
Remove any lines attempting to set these in code; and instead handle errors if they are not found
Then, when you call you script from within your program; it only needs to look like this:
var processStartInfo = new ProcessStartInfo
{
Arguments = "main.py",
FileName = "Python",
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = true
};
Process.Start(processStartInfo);
After some struggle, I found a solution to fit my needs.
Firstly, I completely removed python 2.7 and installed back 3.10.
I tried running the script file inside the shell command line and got the same error that the modules could not be found. What I did is try to import these modules and it gave an error, specifically for bs4 that I am using packages for python 2.x instead of 3.x packages.
After futher investigation I discovered that the packages that I have for my script are treated as "local" packages, meaning I installed them from the IDE (PyCharm) and they work for that project only I guess.
I also found that to "globally" access these packages I had to install them through the command line using the pip3 install <package_name>. After doing this the problem was gone and was left with running the script from the Windows Forms.
NOTE: I did not manage to start the script using python.exe, so I used bash script for the job.
Here is my code and I hope it helps someone down the line...
Code in C#
string myApp = string.Format("{0} {1}", #"C:\testing1.sh", "Hello");
var processStartInfo = new ProcessStartInfo
{
Arguments = myApp,
FileName = "C:\\Program Files\\Git\\git-bash.exe",
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = false
};
Process.Start(processStartInfo)
Code in Bash Script File
#!/bin/bash
arg1="$1"
python C:/Users/Dobromir/PycharmProjects/testing/main.py "$arg1"
Inside the Python file I am using sys.argv[] and accessing the arguments.
NOTE: Passing arguments from the bash script to the python script, in this case, you will receive 2 arguments - first one is the path to the python file and the second is the variable arg1.
Another important thing to mention is you need to have comas around the $1 - this is the property that is being send from the C# file, else it will show as empty.
Articles that were useful:
Installed BeautifulSoup but still get no module named bs4
Passing arguments to Python from Shell Script
https://unix.stackexchange.com/questions/31414/how-can-i-pass-a-command-line-argument-into-a-shell-script
https://gist.github.com/creativcoder/6df4d349447ff1416e68
Thank you to everyone who contributed and tried to help, I managed to learned new things with your suggestions!
I'm trying to run an R Script through Python using subprocess but unfortunately, I'm continually getting the following error:
WindowsError: [Error 2] The system cannot find the file specified
This is extremely frustrating as I've checked the path over everything multiple times (done it without C:/, moved to different directories, changed the R script I'm trying to run, etc). I've also checked the CompSec Environment Variable, and it is also correct (through Windows Systems, although I never checked it directly in my IDE, Spyder). I've tried doing it in Python 2.7 and Python 3.5, and neither version works.
The code goes as follows:
import subprocess
def sort_ByInputColumn(inputPath, inputFileTermination, sortColumn, outputPath, outputFileTermination):
scriptPath = "C:/Users/Kyle/Documents/Rscript_SortByInputColumn.R"
subprocess.call(["Rscript", scriptPath, inputPath, inputFileTermination, sortColumn, outputPath, outputFileTermination])
fileName = 'Alabama'
outputPath = "C:/Users/Kyle/Documents/HillData/Data/Output/Module2/"
sortColumn = str(16)
inputTermination = fileName + 'Module2NN_WorkCounty_Work.csv'
outputFileTermination = fileName + 'Module2NN_SortedWorkCounty.csv'
sort_ByInputColumn(outputPath, inputTermination, sortColumn, outputPath, outputFileTermination)
The fact that I get this error no matter what code I try to run (even blatantly copy-pasting this tutorial to try and make it work) makes me feel like something deeper (or something extremely obvious) is going on that I'm not seeing.
Would appreciate any feedback on the matter.
To solve this most annoying and horrendous problem, I reinstalled R into a directory with no spaces and called Rscript with its full pathing (that is "C:/R/R-3.3.1/bin/Rscript.exe" from C:/Program Files/R/R-3.3.1/bin/Rscript.exe" because Program Files has a space and this kills subprocess because it's command line based, I guess). This time, it worked.
See this similar question for a hint as to where I got the inspiration for this.
I want to call python script from node.js
Here is my script : my.py
def printme( str ):
# print str;
return str;
printme("I'm first call to user defined function!");
printme("Again second call to the same function");
My node script : testpy.js
var PythonShell = require('python-shell');
var pyshell = new PythonShell('my.py');
pyshell.on('message', function(message) {
console.log(message);
});
but getting error
events.js:85
throw er; // Unhandled 'error' event
Error: spawn python ENOENT
at exports._errnoException (util.js:746:11)
at Process.ChildProcess._handle.onexit (child_process.js:1046:32)
at child_process.js:1137:20
at process._tickCallback (node.js:355:11)
at Function.Module.runMain (module.js:503:11)
at startup (node.js:129:16)
at node.js:814:3
P.S I have install Python shell
Also if I want to execute individual function from node.js to python script. can I do that ?Help
You can simply write the 'my.py' file like this-
def printme(str):
return str;
print(printme("I'm first call to user defined function!"));
Check if the path given is correct and check for indentation errors.
Your print statement (my.py line 2) is commented out so nothing will be output and the message event will therefore never fire. Uncomment your print statement, the Node PythonShell object will redirect the stdout (which print writes to) and fire a message event with the output.
As for your error, it looks like the python script isn't being found in the current directory. See https://docs.python.org/2/library/errno.html for error codes and what they mean. Make sure your script is in the right directory or set your python shell to the correct directory using os.chdir.
I think that you need to set up the python script to take in standard input like this
import sys
for v in sys.argv[1:]:
print v
Also when setting up the code the way you have it you need to do a PyhtonShell.send('message'), but I would need to see more of your code because I don't see how you are sending data to the python shell through Node.js.
You can simply import Npm Pythonshell using let keyword instead of const Keyword.
let {PythonShell} = require('python-shell')
this works for me
I've got a bash file that I normally execute using Cygwin.
I need to run this file from my Python code.
I tried this:
for bashfile in files:
p = Popen(bashfile, cwd=dname) #dname is the current directory of the script
stdout, stderr = p.communicate()
I've also seen a similar question here, but when trying to run it that way it says that it can't find the directory of my bash file...
Any ideas? Thanks! :-)
Edit: bashfile has a full path.
Do you need its output to get it directly to Python? If not this may be very fast and easy solution:
os.system("""here some code you use to execute in Terminal""")
You can also try this, though it does (and will no matter what you try) matter where the directory is. This, as far as the output goes, may be a little bit cleaner than the os method.
import commands
cmd="bash ./script.sh"
commands.getoutput(cmd)
If the case is that you need to change the directory:
cmd = "/path/to/your/script/script.sh"
The added benefit of using this method, versus say, os is that you can assign the output to a variable...
fun_times = commands.getoutput("bash ./script.sh")
whereas...
not_fun_times = os.system("./script.sh")
will throw an error.
etc, etc.