We Keep Coding

Python subprocess.checkoutput() exception using 'svn log'

I have run in to an issue that I havent been able to resolve.
I am trying to run a subversion command through python to process the results. I had this code running my dev system but when I run it on target system I get an exception in the subprocess module.
Why would this code produce different results on my two systems?
command = "svn log --username " + __SVNUSERNAME + " --password " + __SVNPASSWORD + " --no-auth-cache --non-interactive -v -r {2022-09-26}:{2022-09-27} --xml " + __SVNTRUNK
bytes = subprocess.check_output([command], stderr=subprocess.STDOUT,shell=True)
I get the following exception:
File "E:\5_scripts\svn.py", line 89, in __run
bytes = subprocess.check_output([command], stderr=subprocess.STDOUT,shell=True)
File "C:\tools\Python3\lib\subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "C:\tools\Python3\lib\subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['{command}']' returned non-zero exit status 1.
I tried to run the same subversion command on the command line and it executes as I expect.
When I echo %ERRORLEVEL% it returns a status of 0. So Im lost as to what is happening differently on my target system.
I tried investigating the python docs for the checkoutput() function and there appears to be no major updates between 3.6 and 3.9
I tried investigating the Apache subversion release notes and didnt find any changes to the 'log' subcommand
Dev System:
Linux 3.10
python 3.6.8
svn 1.9.9
Target System:
Windows Server
python 3.9.6
svn 1.7.4

This is because your target system is Windows. Specifically, the shell=True doesn't work, or doesn't work the same way, on Windows as on Linux. This makes some sense because Windows and Linux have different shells (historically).
You can try the following code, which should work on both platforms:
command = [
"svn", "log",
"--username", __SVNUSERNAME,
"--password", __SVNPASSWORD,
"--no-auth-cache", "--non-interactive", "-v",
"-r", "{2022-09-26}:{2022-09-27}",
"--xml",
__SVNTRUNK
]
bytes = subprocess.check_output(command, stderr=subprocess.STDOUT)
In short, this version no longer relies on the shell to tokenize the command string into a command array, but instead explicitly builds that array. Notice that shell=True has been removed from the call to check_output().
This change can have gotchas, notably that bash or command.com won't do any shell-unique processing of the string (e.g transliterating environmental variables). It doesn't look like your previous code was relying on shell behavior (other than tokenizing the command line) So this is a good change since you are supporting two different environments.

WinError 6 - The handle is invalid with executable but code works on debug mode

I'm using a software that handle video processing in a powershell subprocess.
With PyCharm, when I run my software (in debug mode) everything works as expected.
When I use pyinstaller and inno setup to make an executable and install it on windows I got this error when Sub process start :
[WinError 6] The handle is invalid
I suppose it’s due to an error in a subprocess like this piece of code :
try:
psa1_path = EnvValues.powershell_path().format(project=project)
# using this powershell : C:/Users/${USERNAME}\projects\demo\cmd\powershell.ps1 -m -v 'CC 2018' -wait windowstyle hidden
dc_logger.info(f'using this powershell : {psa1_path}')
if project:
dc_logger.info("PowerShell Rendering Started")
with open(EnvValues.frame_path(), 'a') as f:
p = subprocess.Popen(['C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe',
'-ExecutionPolicy',
'Unrestricted',
psa1_path],
stdout=f)
p.communicate()
dc_logger.info("PowerShell Done Rendering")
return True
else:
dc_logger.info("no project to render")
return False
Is the bug related to arguments passed to the subprocess ? Why executable version of the code is the only one not working ? Why I don't have the bug in the development version ?
Here's my pyinstaller cmds :
pyinstaller --onefile -w -F -i "C:\Users\my_project\icon.ico" Project.py
pyinstaller --onefile -w -F -i "C:\Users\my_project\icon.ico" Project.spec
Then I put this in InnoSetup and I install the output to my windows machine.

The problem is with psa1_path variable
C:/Users/${USERNAME}\projects\demo\cmd\powershell.ps1 -m -v 'CC 2018' -wait windowstyle hidden
This variable has parameters. And the subprocess.Popen use it as a string, then you have to set shell=True so Popen will use this string as a complete shell cmd.
p = subprocess.Popen(['C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe',
'-ExecutionPolicy',
'Unrestricted',
psa1_path],
stdout=f, shell=True, stdin=subprocess.DEVNULL)
Don't forget to add stdin arg because it's also throw the [WinError 6] The handle is invalid
So why the code works with debug code and not with executable :
It's mainly because PyCharm does additionnal configuration and setup behind the scenes when it comes to running a program.
Because when you’re going from IDE to runtime, you need additional hooks to get things going. pyinstaller doesn’t do the subprocess the same way as pyCharm for the shell part

Why can't the Python command "subprocess.Popen" find the jar file to run?

I'm trying to run code from this repo: https://github.com/tylin/coco-caption, specifically from https://github.com/tylin/coco-caption/blob/master/pycocoevalcap/tokenizer/ptbtokenizer.py, line 51-52:
p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, \
stdout=subprocess.PIPE)
The error I get running this is
OSError: [Errno 2] No such file or directory
I can't figure out why the file can't be found.
The jar I'm trying to run is:
stanford-corenlp-3.4.1.jar
You can see the structure of directory by going to https://github.com/tylin/coco-caption/tree/master/pycocoevalcap/tokenizer. For more specificity into what my actual arguments are when I run the line of code:
cmd= ['java', '-cp', 'stanford-corenlp-3.4.1.jar', 'edu.stanford.nlp.process.PTBTokenizer', '-preserveLines', '-lowerCase', 'tmpWS5p0Z'],
and
path_to_dirname =abs_path_to_folder/tokenizer
I can see the jar that needs to be run, and it looks to be in the right place, so why can't python find it. (Note: I'm using python2.7.) And the temporary File 'tmpWS5p0Z' is where it should be.
Edit: I'm using Ubuntu

try an absolute path ( meaning the path beginning from root / )
https://en.wikipedia.org/wiki/Path_(computing)#Absolute_and_relative_paths
for relative paths in python see i.e. Relative paths in Python , How to refer to relative paths of resources when working with a code repository in Python
UPDATE:
As a test try subprocess.Popen() with the shell=True option and give an absolute path for any involved file, including tmpWS5p0Z
in this subprocess.Popen() call are involved two paths :
1) the python path, python has to find the java executable and the stanford-corenlp-3.4.1.jar which is essentially a java program with its own path
2) the java path of stanford-corenlp-3.4.1.jar
as this is all too complicated try
p_tokenizer = subprocess.Popen(['/absolute_path_to/java -cp /absolute_path_to/stanford-corenlp-3.4.1.jar /absolute_path_to/edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase /absolute_path_to/tmpWS5p0Z' ], shell=True)
Python specify popen working directory via argument
Python subprocess.Popen() error (No such file or directory)

Just in case it might help someone:
I was struggling with the same problem (same https://github.com/tylin/coco-caption code). Might be relevant to say that I was running the code with python 3.7 on CentOS using qsub. So I changed
cmd = ['java', '-cp', 'stanford-corenlp-3.4.1.jar', 'edu.stanford.nlp.process.PTBTokenizer', '-preserveLines', '-lowerCase', 'tmpWS5p0Z']
to
cmd = ['/abs/path/to/java -cp /abs/path/to/stanford-corenlp-3.4.1.jar edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase ', ' /abs/path/to/temporary_file']
Using absolute paths fixed the OSError: [Errno 2] No such file or directory. Notice that I still put '/abs/path/to/temporary_file' as second element in the cmd list, because it got added later on. But then something went wrong in the tokenizer java subprocess, I don't know why or what, just observing because:
p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, stdout=subprocess.PIPE, shell=True)
token_lines = p_tokenizer.communicate(input=sentences.rstrip())[0]
Here token_lines was an empty list (which is not the wanted behavior). Executing this in IPython resulted in the following (just the subprocess.Popen(..., not the communicate).
Exception in thread "main" edu.stanford.nlp.io.RuntimeIOException: java.io.IOException: Input/output error
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:278)
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:163)
at edu.stanford.nlp.process.AbstractTokenizer.hasNext(AbstractTokenizer.java:55)
at edu.stanford.nlp.process.PTBTokenizer.tokReader(PTBTokenizer.java:444)
at edu.stanford.nlp.process.PTBTokenizer.tok(PTBTokenizer.java:416)
at edu.stanford.nlp.process.PTBTokenizer.main(PTBTokenizer.java:760)
Caused by: java.io.IOException: Input/output error
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:279)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:290)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
at java.base/java.io.BufferedReader.read1(BufferedReader.java:210)
at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
at edu.stanford.nlp.process.PTBLexer.zzRefill(PTBLexer.java:24511)
at edu.stanford.nlp.process.PTBLexer.next(PTBLexer.java:24718)
at edu.stanford.nlp.process.PTBTokenizer.getNext(PTBTokenizer.java:276)
... 5 more
Again, I don't know why or what, but I just wanted to share that doing this fixed it:
cmd = ['/abs/path/to/java -cp /abs/path/to/stanford-corenlp-3.4.1.jar edu.stanford.nlp.process.PTBTokenizer -preserveLines -lowerCase /abs/path/to/temporary_file']
And changing cmd.append(os.path.join(path_to_jar_dirname, os.path.basename(tmp_file.name))) into cmd[0] += os.path.join(path_to_jar_dirname, os.path.basename(tmp_file.name)).
So making cmd into a list with only 1 element, containing the entire command with absolute paths at once. Thanks for your help!

As #Lars mentioned above the issue I had was that I Java wasn't installed. Solved it with:
sudo apt update
sudo apt install default-jdk
sudo apt install default-jre
Making this post since I had this issue twice (due to reinstallation problems) and forgot about it.

Python extension debugging

I'm trying to debug an extension module for python that I wrote in C. I compiled it using the following:
python setup.py build -g install --user
I then debug with:
gdb python
...
b py_node_make
run test.py
It breaks at py_node_make (one of the functions I defined), but then I try:
(gdb) print node
No symbol "node" in current context.
The function I'm trying to debug is:
static Python_node_t* py_node_make(
node_t* node)
{
Python_node_t* pyNode;
pyNode = PyObject_New(Python_node_t, &t_node);
pyNode->node = node;
pyNode->borrowed = true;
return pyNode;
}

For source debugging to work, your C extensions must be built with debug info (gcc -g). Since you're driving the compilation process with distutils, you can specify the compiler flags used through the CFLAGS environment variable (Installing Python Modules: Tweaking compiler/linker flags):
CFLAGS='-Wall -O0 -g' python setup.py build
Note that even if distutils defaults to a higher optimization level than -O0, you really shouldn't get that No symbol "node" in current context error as long as -g is passed and most Python builds do pass -g by default.

The problem is the optimization. I'm not sure how to do it from the command line, but in the setup.py script I just added extra_compile_args=['-O0'], to the Extension constructor and everything worked.
Would still like (and would accept) an answer that involved a command line arg (something after python setup.py build) that would accomplish the same thing so I don't have to have a compiler specific line in my setup.py file.

Building zxing's cpp/lib Fails under Fedora14 even when Following instructions Exactly

That is, following the instructions in zxing/cpp/README, which say 'To build the library only:
- Run "scons lib" in this folder (cpp)'
Well, that is exactly what I did. But I get:
scons lib
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
o build/core/src/zxing/BarcodeFormat.o -c -O0 -g3 -ggdb -Wall -Ibuild/core/src build/core/src/zxing/BarcodeFormat.cpp
sh: o: command not found
Withs this "O: command not found" repeated many times.
I thought the problem might be gcc not found, so I checked for that: it is installed. I took only a brief look at the python of scons before I gave up on trying to figure ouw why it is looking for a command 'o'. Of course there is none.
BTW: I got my copy of zxing 1.7 using wget only three days ago and the yum installation of 'scons' today. So they are up-to-date.

It is likely that you are correct, and that SCons isn't finding GCC. Your best bet is to add a call to display the contents of some (or all) of the environment. As shown below, you can either pull out specific variable, or show the whole environment. The best place would probably near a call to the builder (SharedLibrary, StaticLibrary or Program). For an environment named 'env':
print env.Dump()
print env['CC']
print env['CXX']
Ensuring that an appropriate default environment is being used initially (probably something like):
env = DefaultEnvironment( ... )
Or that the environment variables on your system (including path) are being propagated through to SCons. One way to do this is by:
import os
# ...
env = Environment( ENV = os.environ, ... )
In extreme cases you can resolve this problem by providing an explicit path to the compiler:
env['CC'] = '/usr/bin/gcc'
Edit:
These changes need to be made in an appropriate SConstruct or SConscript file. Which depends on the exact project, and what you're trying to achieve - in the case of the current version of zxing on google code, it would be reasonable to make the changes on or near line 40 of the SConscript file