Overview:
I have some C++ code (myProgram) that I want to execute from a Lambda function. I have built the code on an Amazon Linux EC2 instance, and it runs there without problem.
When I try to run it from a Python script in Lambda, I receive the following error message:
Command 'LD_LIBRARY_PATH=/var/task/lib; /var/task/myProgram -d' returned non-zero exit status 126.: CalledProcessError
I see from the Bash Reference Manual that 126 means the command is found but is not executable. It is executable before I upload it to Lambda, so I'm not sure how to resolve this.
Edit: Thanks to the comment below, I double checked the permissions on the Lambda instance. Indeed, the file is not marked as executable after it has been uploaded. When I try changing it with chmod, I get an error Read-only file system.
Details:
When I package this and upload it to my lambda function, the directory structure looks like this:
handler.py
myProgram
lib
(required libraries)
Here is the code for handler.py
import os
import subprocess
exepath = os.path.join(os.getcwd(), 'myProgram')
libdir = os.path.join(os.getcwd(), 'lib')
def handler(event, context):
command = 'LD_LIBRARY_PATH={}; {} -d'.format(libdir, exepath)
return subprocess.check_output(command, shell=True, stderr=subprocess.STDOUT)
If I change shell=True to shell=False then I receive a different error
[Errno 2] No such file or directory: 'LD_LIBRARY_PATH=/var/task/lib; /var/task/myProgram -d': FileNotFoundError
Question:
How can I make myProgram executable? I don't see any mention of this process in the Lambda blog on executables
I figured out the solution to this thanks to the tip from Yakk and the answer to Can't run binary from within python aws lambda function.
I had to copy myProgram to /tmp and then chmod to be able to execute the it.
command = 'cp ./myProgram /tmp/myProgram; chmod 755 /tmp/myProgram; LD_LIBRARY_PATH={}; /tmp/myProgram -d '.format(libdir)
return subprocess.check_output(command, shell=True, stderr=subprocess.STDOUT)
Related
I'm trying to use the following "subprocess" command in Python to attempt to change the user and group of a directory of files:
output = subprocess.run(["sudo", "chown", "-R", "user:group", "{}/data/*".format(workingDirectory)],
encoding="utf-8",
capture_output=True)
print(output)
However, I'm getting an error that the directory cannot be accessed:
CompletedProcess(args=['sudo', 'chown', '-R', 'user:group', '/var/lib/method_transform/data/*'], returncode=1, stdout='', stderr='chown: cannot access ‘/var/lib/method_transform/data/*’: No such file or directory\n')
I tried running the following command from a bash shell script and it worked fine.
sudo chown -R user:group /var/lib/method_transform/data/*
I expect the ownership of all the files to be changed to "user:group"
I have a bash script that I can run flawlessly in my Rpi terminal in its folder:
./veye_mipi_i2c.sh -r -f mirrormode -b 10
it works like this: Usage: ./veye_mipi_i2c.sh [-r/w] [-f] function name -p1 param1 -p2 param2 -b bus
options:
-r read
-w write
-f [function name] function name
-p1 [param1] param1 of each function
-p2 [param1] param2 of each function
-b [i2c bus num] i2c bus number
When I try to run it in Python (2) via my Spyder editor with os.system, I get a "0" return which I interpret as "succesfully executed" but in fact the script has not been executed and the functions have not been performed. I know this because the script is suppose to change the camera functioning and by checking the images I take afterwards, I can see that nothing has changed.
import os
status = os.system('/home/pi/VeyeMipi/Camera_Folder/veye_mipi_i2c.sh -w -f mirrormode -p1 0x04 -b 10')
print status
Any idea, what causes this? The bash script uses two other scripts that lie in the same folder location (read and write). Could it be, that it cannot execute these additional scripts when startet through Python? It does not make sense to me, but so do a lot of things....
Many thanks
Ok, I understand that my question was not exemplary because of the lack of a minimal reproducible example, but as I did not understand what the problem was, I was not able to create one.
I have found out, what the problem was. The script I am calling in bash requires two more scripts that are in the same folder. Namely the "write" script and "read" script. When executing in terminal in the folder, no problem, because the folder was the working directory.
I tried to execute the script within Spyder editor and added the file location to the PATH in the user interface. But still it would not be able to execute the "write" script in the folder.
Simply executing it in the terminal did the trick.
It would help if you fix your scripts so they don't depend on the current working directory (that's a very bad practice).
In the meantime, running
import subprocess
p = subprocess.run(['./veye_mipi_i2c.sh', '-r', '-f', 'mirrormode', '-b', '10'], cwd='/home/pi/VeyeMipi/Camera_Folder')
print(p.returncode)
which changes the directory would help.
Use subprocess and capture the output:
import subprocess
output = subprocess.run(stuff, capture_output=True)
Check output.stderr and output.stdout
The goal is to open a particular excel file using a python shell subprocess. The code cannot be more simple yet I cannot figure out what is wrong:
import subprocess
arg1 = "C:\\Program Files (x86)\\Microsoft Office\\root\\Office16\\EXCEL.EXE"
arg2 = "C:\\Users\\user\\Documents\\test.xlsm"
p = subprocess.Popen(["start", arg1, arg2], shell=False)
The command works perfectly directly on the shell but when done through subprocess throws the following error:
FileNotFoundError: [WinError 2] The system cannot find the file specified
I have also tried the below which works equally fine directly on the shell but the behavior is different:
p = subprocess.Popen([arg1, arg2], shell=False)
The following failed assertion pops up from Excel:
snapshopt error
My next try was:
import os
os.system("C:\\Users\\user\\Documents\\test.xlsm")
Which replicates the same assertion error as above plus returns a code 3 which based on
System Error Codes (0-499)
is a path not found.
Again same path works on shell, at this stage I run out of ideas, any help?
os.startfile is actually a better way to do it. As specified in the documentation -
os.startfile(path[, operation])
Start a file with its associated application.
When operation is not specified or 'open', this acts like
double-clicking the file in Windows Explorer, or giving the file name
as an argument to the start command from the interactive command
shell: the file is opened with whatever application (if any) its
extension is associated.
When another operation is given, it must be a “command verb” that
specifies what should be done with the file. Common verbs documented
by Microsoft are 'print' and 'edit' (to be used on files) as well as
'explore' and 'find' (to be used on directories)
You can just do the following -
import os
os.startfile("C:\\Users\\user\\Documents\\test.xlsm")
I'm trying to run code from this repo: https://github.com/tylin/coco-caption, specifically from https://github.com/tylin/coco-caption/blob/master/pycocoevalcap/tokenizer/ptbtokenizer.py, line 51-52:
p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, \
stdout=subprocess.PIPE)
The error I get running this is
OSError: [Errno 2] No such file or directory
I can't figure out why the file can't be found.
The jar I'm trying to run is:
stanford-corenlp-3.4.1.jar
You can see the structure of directory by going to https://github.com/tylin/coco-caption/tree/master/pycocoevalcap/tokenizer. For more specificity into what my actual arguments are when I run the line of code:
cmd= ['java', '-cp', 'stanford-corenlp-3.4.1.jar', 'edu.stanford.nlp.process.PTBTokenizer', '-preserveLines', '-lowerCase', 'tmpWS5p0Z'],
and
path_to_dirname =abs_path_to_folder/tokenizer
I can see the jar that needs to be run, and it looks to be in the right place, so why can't python find it. (Note: I'm using python2.7.) And the temporary File 'tmpWS5p0Z' is where it should be.
Edit: I'm using Ubuntu
try an absolute path ( meaning the path beginning from root / )
https://en.wikipedia.org/wiki/Path_(computing)#Absolute_and_relative_paths
for relative paths in python see i.e. Relative paths in Python , How to refer to relative paths of resources when working with a code repository in Python
UPDATE:
As a test try subprocess.Popen() with the shell=True option and give an absolute path for any involved file, including tmpWS5p0Z
in this subprocess.Popen() call are involved two paths :
1) the python path, python has to find the java executable and the stanford-corenlp-3.4.1.jar which is essentially a java program with its own path
2) the java path of stanford-corenlp-3.4.1.jar
as this is all too complicated try
p_tokenizer = subprocess.Popen(['/absolute_path_to/java -cp /absolute_path_to/stanford-corenlp-3.4.1.jar /absolute_path_to/edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase /absolute_path_to/tmpWS5p0Z' ], shell=True)
Python specify popen working directory via argument
Python subprocess.Popen() error (No such file or directory)
Just in case it might help someone:
I was struggling with the same problem (same https://github.com/tylin/coco-caption code). Might be relevant to say that I was running the code with python 3.7 on CentOS using qsub. So I changed
cmd = ['java', '-cp', 'stanford-corenlp-3.4.1.jar', 'edu.stanford.nlp.process.PTBTokenizer', '-preserveLines', '-lowerCase', 'tmpWS5p0Z']
to
cmd = ['/abs/path/to/java -cp /abs/path/to/stanford-corenlp-3.4.1.jar edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase ', ' /abs/path/to/temporary_file']
Using absolute paths fixed the OSError: [Errno 2] No such file or directory. Notice that I still put '/abs/path/to/temporary_file' as second element in the cmd list, because it got added later on. But then something went wrong in the tokenizer java subprocess, I don't know why or what, just observing because:
p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, stdout=subprocess.PIPE, shell=True)
token_lines = p_tokenizer.communicate(input=sentences.rstrip())[0]
Here token_lines was an empty list (which is not the wanted behavior). Executing this in IPython resulted in the following (just the subprocess.Popen(..., not the communicate).
Exception in thread "main" edu.stanford.nlp.io.RuntimeIOException: java.io.IOException: Input/output error
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:278)
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:163)
at edu.stanford.nlp.process.AbstractTokenizer.hasNext(AbstractTokenizer.java:55)
at edu.stanford.nlp.process.PTBTokenizer.tokReader(PTBTokenizer.java:444)
at edu.stanford.nlp.process.PTBTokenizer.tok(PTBTokenizer.java:416)
at edu.stanford.nlp.process.PTBTokenizer.main(PTBTokenizer.java:760)
Caused by: java.io.IOException: Input/output error
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:279)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:290)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
at java.base/java.io.BufferedReader.read1(BufferedReader.java:210)
at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
at edu.stanford.nlp.process.PTBLexer.zzRefill(PTBLexer.java:24511)
at edu.stanford.nlp.process.PTBLexer.next(PTBLexer.java:24718)
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:276)
... 5 more
Again, I don't know why or what, but I just wanted to share that doing this fixed it:
cmd = ['/abs/path/to/java -cp /abs/path/to/stanford-corenlp-3.4.1.jar edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase /abs/path/to/temporary_file']
And changing cmd.append(os.path.join(path_to_jar_dirname, os.path.basename(tmp_file.name))) into cmd[0] += os.path.join(path_to_jar_dirname, os.path.basename(tmp_file.name)).
So making cmd into a list with only 1 element, containing the entire command with absolute paths at once. Thanks for your help!
As #Lars mentioned above the issue I had was that I Java wasn't installed. Solved it with:
sudo apt update
sudo apt install default-jdk
sudo apt install default-jre
Making this post since I had this issue twice (due to reinstallation problems) and forgot about it.
I'm using the following python script on Windows Server 2008:
import cgitb
import subprocess
cgitb.enable()
print "Content-Type: text/plain;charset=utf-8"
print
cmd = "git tag"
process = subprocess.Popen(cmd.split(), shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
git_output = process.communicate()[0]
print "git output = %s" % git_output
There are, in fact, some git tags. Running this script through the shell works perfectly. However, when run through IIS (7), the output seems to be empty.
I've tried directing the Popen output to a file instead of PIPE. Again, worked when running from the command line, didn't work when running through IIS.
Any ideas?
EDIT:
Following #Wooble's advice, I removed the [0] from the call to communicate to see git errors, and indeed found the enigmatic error "'git' is not recognized as an internal or external command, operable program or batch file." Of course git is installed on the system, and as I said the script works when run directly through the command line.
To no avail, I tried:
Setting the command to use the full path of the git executable
Adding the full path of the git executable directory to python's sys.path
Copying the actual git executable to the working directory - this removed the "git not recognized" error but still yielded an empty result!
Please help!!
I don't know why the path was partial when running from IIS (explanations welcome!), but adding this finally solved the problem:
git_path = "C:\\Program Files (x86)\\Git\\bin\\"
os.environ["PATH"} += git_path + ';'