We Keep Coding

Why can't the Python command "subprocess.Popen" find the jar file to run?

I'm trying to run code from this repo: https://github.com/tylin/coco-caption, specifically from https://github.com/tylin/coco-caption/blob/master/pycocoevalcap/tokenizer/ptbtokenizer.py, line 51-52:
p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, \
stdout=subprocess.PIPE)
The error I get running this is
OSError: [Errno 2] No such file or directory
I can't figure out why the file can't be found.
The jar I'm trying to run is:
stanford-corenlp-3.4.1.jar
You can see the structure of directory by going to https://github.com/tylin/coco-caption/tree/master/pycocoevalcap/tokenizer. For more specificity into what my actual arguments are when I run the line of code:
cmd= ['java', '-cp', 'stanford-corenlp-3.4.1.jar', 'edu.stanford.nlp.process.PTBTokenizer', '-preserveLines', '-lowerCase', 'tmpWS5p0Z'],
and
path_to_dirname =abs_path_to_folder/tokenizer
I can see the jar that needs to be run, and it looks to be in the right place, so why can't python find it. (Note: I'm using python2.7.) And the temporary File 'tmpWS5p0Z' is where it should be.
Edit: I'm using Ubuntu

try an absolute path ( meaning the path beginning from root / )
https://en.wikipedia.org/wiki/Path_(computing)#Absolute_and_relative_paths
for relative paths in python see i.e. Relative paths in Python , How to refer to relative paths of resources when working with a code repository in Python
UPDATE:
As a test try subprocess.Popen() with the shell=True option and give an absolute path for any involved file, including tmpWS5p0Z
in this subprocess.Popen() call are involved two paths :
1) the python path, python has to find the java executable and the stanford-corenlp-3.4.1.jar which is essentially a java program with its own path
2) the java path of stanford-corenlp-3.4.1.jar
as this is all too complicated try
p_tokenizer = subprocess.Popen(['/absolute_path_to/java -cp /absolute_path_to/stanford-corenlp-3.4.1.jar /absolute_path_to/edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase /absolute_path_to/tmpWS5p0Z' ], shell=True)
Python specify popen working directory via argument
Python subprocess.Popen() error (No such file or directory)

Just in case it might help someone:
I was struggling with the same problem (same https://github.com/tylin/coco-caption code). Might be relevant to say that I was running the code with python 3.7 on CentOS using qsub. So I changed
cmd = ['java', '-cp', 'stanford-corenlp-3.4.1.jar', 'edu.stanford.nlp.process.PTBTokenizer', '-preserveLines', '-lowerCase', 'tmpWS5p0Z']
to
cmd = ['/abs/path/to/java -cp /abs/path/to/stanford-corenlp-3.4.1.jar edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase ', ' /abs/path/to/temporary_file']
Using absolute paths fixed the OSError: [Errno 2] No such file or directory. Notice that I still put '/abs/path/to/temporary_file' as second element in the cmd list, because it got added later on. But then something went wrong in the tokenizer java subprocess, I don't know why or what, just observing because:
p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, stdout=subprocess.PIPE, shell=True)
token_lines = p_tokenizer.communicate(input=sentences.rstrip())[0]
Here token_lines was an empty list (which is not the wanted behavior). Executing this in IPython resulted in the following (just the subprocess.Popen(..., not the communicate).
Exception in thread "main" edu.stanford.nlp.io.RuntimeIOException: java.io.IOException: Input/output error
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:278)
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:163)
at edu.stanford.nlp.process.AbstractTokenizer.hasNext(AbstractTokenizer.java:55)
at edu.stanford.nlp.process.PTBTokenizer.tokReader(PTBTokenizer.java:444)
at edu.stanford.nlp.process.PTBTokenizer.tok(PTBTokenizer.java:416)
at edu.stanford.nlp.process.PTBTokenizer.main(PTBTokenizer.java:760)
Caused by: java.io.IOException: Input/output error
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:279)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:290)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
at java.base/java.io.BufferedReader.read1(BufferedReader.java:210)
at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
at edu.stanford.nlp.process.PTBLexer.zzRefill(PTBLexer.java:24511)
at edu.stanford.nlp.process.PTBLexer.next(PTBLexer.java:24718)
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:276)
... 5 more
Again, I don't know why or what, but I just wanted to share that doing this fixed it:
cmd = ['/abs/path/to/java -cp /abs/path/to/stanford-corenlp-3.4.1.jar edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase /abs/path/to/temporary_file']
And changing cmd.append(os.path.join(path_to_jar_dirname, os.path.basename(tmp_file.name))) into cmd[0] += os.path.join(path_to_jar_dirname, os.path.basename(tmp_file.name)).
So making cmd into a list with only 1 element, containing the entire command with absolute paths at once. Thanks for your help!

As #Lars mentioned above the issue I had was that I Java wasn't installed. Solved it with:
sudo apt update
sudo apt install default-jdk
sudo apt install default-jre
Making this post since I had this issue twice (due to reinstallation problems) and forgot about it.

Calling curl from python script

I am trying to use curl to connect to the Splunk API and run a search and store the results into a .csv file. I am able to make this work with the subprocess module and powershell.exe, shown in the code block below
powershell_path = "C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe"
search_string = '<search to be used>'
curl_command = '\ncurl -k -u <other args>'
subprocess.call([powershell_path, search_string, curl_command])
However, I would like to be able to do this without using the powershell.exe. I tried just substituting the path to the powershell.exe with the path to the curl.exe, but then I get errors telling me I have illegal characters in my search_string and curl_command variables (escaping them with a \ doesn't help), and when I try to directly use the paths in the subprocess.call:
subprocess.call(['<path to curl.exe>', '<full search string>, 'curl -k -u <other args>'])
I get the error WindowsError: [Error 2] The system cannot find the file specified (I checked path to file and tried os.join, got nowhere). How can I use curl in Python to connect to the Splunk API, run a search, save the results to a csv, and do all this without calling external applications like Powershell?

Running external command using python 2.7 in Windows

I am a novice user in python and I am having a problem in executing external command with command-line switch (using python 2.7 in Windows 8 64-bit):
lammpsExe = 'C:/Program Files (x86)/LAMMPS 32-bit 20150403/bin/lmp_serial.exe'
os.system(lammpsExe + " -in in.lmps")
It gives the following error message:
'C Program' is not recognized as an internal or external command, operable program or batch file
It seems that os.system cannot understand the string path for lammpsExe.
Then I tried subprocess.call and replace '/' with '\\' in the path:
lammpsExe = 'C:\\Program Files\\LAMMPS 64-bit 20150330\\bin\\lmp_serial.exe'
subprocess.call([lammpsExe,'-in in.lmps'], shell=True)
But it still doesn't work as the command prompt gives the following warning:
IndentationError: unexpected indent
I suspect the command-line switch '-in' is the problem. I have tried various combination of ", ', \, and /, and I still get error messages.

The subprocess docs has a section titled Replacing os.system():
status = os.system("mycmd" + " myarg")
# becomes
status = subprocess.call("mycmd" + " myarg", shell=True)
I would suggest:
lammpsExe = 'C:\\Program Files\\LAMMPS 64-bit 20150330\\bin\\lmp_serial.exe'
subprocess.call(lammpsExe + ' -in in.lmps', shell=True)
The docs do note that "Calling the program through the shell is usually not required."

I think I have found the solution. Instead of trying to define the path in the python script for the external command, I just need to put the path in the Windows system variables, then calling the command directly in the script:
subprocess.call(['lmp_serial.exe','-in','in.lmps'],shell=True)

How can I run command in Microsoft Exchange Server Powershell through Python script?

I want to check the number of mailbox in Microsoft Exchange Server. This command works fine in standard cmd.exe:
"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -command ". 'C:\Program Files\Microsoft\Exchange Server\V14\bin\RemoteExchange.ps1'; Connect-ExchangeServer -auto ; Get-Mailbox | Measure-Object"
Output is
...
Count : 3
Average :
Sum :
Maximum :
Minimum :
Property :
Then I am going to code it in Python, using "-ExecutionPolicy RemoteSigned":
cmd = "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe
-ExecutionPolicy RemoteSigned
-command \". 'C:\\Program Files\\Microsoft\\Exchange Server\\V14\\bin\\RemoteExchange.ps1'; Connect-ExchangeServer -auto; Get-Mailbox | Measure-Object\""
os.system(cmd)
There is lots of error about loading RemoteExchange.ps1 file.
Get-ItemProperty : Cannot find path 'HKLM:\SOFTWARE\Microsoft\ExchangeServer\v14\Setup' because it does not exist.
At C:\Program Files\Microsoft\Exchange Server\V14\bin\RemoteExchange.ps1:46 char:34
+ $global:exbin = (get-itemproperty <<<< HKLM:\SOFTWARE\Microsoft\ExchangeServer\v14\Setup).MsiInstallPath + "bin\"
+ CategoryInfo : ObjectNotFound: (HKLM:\SOFTWARE\...erver\v14\Setup:String) [Get-ItemProperty], ItemNotFo
undException
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetItemPropertyCommand
...
The Exchange types file wasn't loaded because not all of the required files could be found.
Update-TypeData : Cannot find path 'C:\Users\administrator.SCCM01\bin\Exchange.partial.Types.ps1xml' because it does no
t exist.
At C:\Program Files\Microsoft\Exchange Server\V14\bin\RemoteExchange.ps1:104 char:16
+ Update-TypeData <<<< -PrependPath $partialTypeFile
+ CategoryInfo : InvalidOperation: (bin\Exchange.partial.Types.ps1xml:String) [Update-TypeData], ItemNotF
oundException
+ FullyQualifiedErrorId : TypesPrependPathException,Microsoft.PowerShell.Commands.UpdateTypeDataCommand
Although the welcome screen of Exchange Management Shell appears, it failed to load RemoteExchange.ps1, and "Get-Mailbox" command is not working at all.
I guess I must have missed something important. How can I solve this problem? Please help.
Edit: Why should I add -ExecutionPolicy RemoteSigned in Python script? If I do not do that, it will result in a different error:
File C:\Program Files\Microsoft\Exchange Server\V14\bin\RemoteExchange.ps1 cannot be loaded because the execution of scripts is disabled on this system. Please see "get-help about_signing" for more details.
Refer to this thread, and RemoteSigned is better than Unrestricted. Both of them work in cmd.exe, but not work in Python script.

I ran into the exact same problem under slightly different circumstances but the error message was identical. I was trying to provision MS Exchange via Puppet and Powershell scripts. You are suffering from the side effects of the File System Redirector (Windows) which runs silently. Apparently 64-bit registry keys can't be accessed by 32-bit programs and the File System Redirector will change things up on you without you even knowing it.
I found this to be an elegant solution for my situation:
32-bit applications can access the native system directory by
substituting %windir%\Sysnative for %windir%\System32. WOW64
recognizes Sysnative as a special alias used to indicate that the file
system should not redirect the access. This mechanism is flexible and
easy to use, therefore, it is the recommended mechanism to bypass file
system redirection. Note that 64-bit applications cannot use the
Sysnative alias as it is a virtual directory not a real one.

Python: executing a complex command on windows

I am having a problem when running a command on Windows whereas it works perfectly on Linux.
I give you the context, but this is not necessary to understand my issue: I am using gimp in batch mode.
I have a Python script callPythonFuScript.py which calls another Python script, pythonFu.py, which executes a python-fu call.
In callPythonFuScript.py, I construct the command line when I call the function inside pythonFu.py to be executed. This is the command line:
gimp-console-2.8 -idf --batch-interpreter python-fu-eval -b 'import sys;sys.path=['.']+sys.path;import pythonFu;pythonFu.myFunction("arg1","arg2","arg3") ' -b 'pdb.gimp_quit(1)'
This command works perfectly on Linux but when I try to run it on Windows, it does not work.
Error messages are:
The opening of C:\Users\myRep\sys; failed : no such file or directory
The opening of C:\Users\myRep\sys.path=['.']+sys.path; failed : no such file or directory
The opening of C:\Users\myRep\"arg1","arg2","arg3")' failed no such file or directory
I am assuming that Windows interprets characters differently than Linux. Is this correct? How can I fix this problem?

As mentioned in the comments, you are having an escaping issue between what the command prompt sees as arguments, and what is being passed as a literal string for python to eval:
-b 'import sys;sys.path=["."]+sys.path;import pythonFu;pythonFu.myFunction("arg1","arg2","arg3")'
If that still gives you errors, it is possible you might need to escape the double quotes:
-b 'import sys;sys.path=[\".\"]+sys.path;import pythonFu;pythonFu.myFunction(\"arg1\",\"arg2\",\"arg3\")'