I have a python script that prints some 5-digit numbers. I call this script through a Qprocess and read its output which I use later on.
On windows the script is packaged into an executable with py2exe and the executable is called.
When I ran either the executable, or the script from a terminal, the output is fine, but when I call the executable through the Qprocess, I get garbage.
The part of my script that prints is like this
print number
sys.stdout.flush()
I tried replacing print with sys.stoud.write(number) and also with sys.stoud.write(number + '\n') but with no success.
my py2exe script is :
setup(
console=['number.py'],
options = {
'py2exe' : {
'bundle_files': 1,
'compressed': True
}
},
zipfile = None
)
I also tried changing console=['number.py'] to windows=['number.py'],
also with no success.
I know of -u switch of python, but I need to have it as an executable and I'm also not sure if this will work after all.
I have searched both on SO and google but everyone suggests using sys.stdout.flush() which is not enough for my case.
When I checked the output using a logger, I could see that the size of the output was different with a range from 6 characters to 11 incremented by 1 and then again from 6 to 11 and so on.
I' using Python 2.7 and the Qt is 4.8, I tested on Windows 7 and 8. On Linux everything runs smoothly.
I've wasted almost a whole day on this, I would appreciate any help
I also tried using \r\n when using sys.stoud.write(), but this didn't work too.
Trying to get this to work, I replaced the call to the executable with the call to the python script using the -u switch <python -u myscript.py> and it seems to work fine and the size of the output is always 6.
Why is this ?
Looking at the options provided by py2exe, I saw that there is the unbuffered option which says that when True unbuffered stdout and stderr(which I thought is unbuffered by default) are used.
unbuffered: if true, use unbuffered binary stdout and stderr
I set 'unbuffered': True in setup.py and this didn't work either.
You might need to add os.fsync(sys.stdout.fileno()).
Reference: os.fsync(fd)
Force write of file with filedescriptor fd to disk. On Unix, this calls the native fsync() function; on Windows, the MS _commit() function.
If you’re starting with a Python file object f, first do f.flush(), and then do os.fsync(f.fileno()), to ensure that all internal buffers associated with f are written to disk.
Related
I'm trying to run a Ruby project from a Python interface, but in its most current form it only works when run from a terminal, not from a Subprocess in Python. The Ruby project consists of an executable file (let's call it exeFile) which runs a command-line interface tool (let's call it cli.rb), which requires and calls various structures from various other Ruby files in the project. The CLI file takes command-line arguments via Thor. This works in a terminal, but fails in a subprocess call when the cli.rb file has been modified.
I've made sure I'm passing all the right arguments to subprocess.Popen. For example:
popen = subprocess.Popen(['/home/daveshere/.rbenv/shims/ruby', '/media/daveshere/Data/exeFile', '--b', '1.1.1', '-p'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, universal_newlines=True, shell=False)
The p argument hasn't changed in the cli.rb file, and works in both the terminal and the Subprocess. But days ago, I changed the type of b from numeric to string, which allows the program to run properly. Again, this works in the terminal, but not when running from the Subprocess. Instead, printing the Subprocess output shows me the error:
Expected numeric value for '--b'; got "1.1.1"
This is despite the cli.rb file requiring a string type for b now, and the fact that the same call works in the terminal. It seems as if there is some locally-cached outdated version of the cli.rb file being called by ruby in the Subprocess, but not the terminal. I even tried adding this to the cli.rb file to see if it registers:
puts "TESTING MODIFICATION"
That string prints when running from the terminal, but not from the Subprocess (although other, older output does).
I also made sure ruby -v returns the same version in both cases, and it does. I'm really not sure what's causing the disconnect here. I'm also running the Python Subprocess via Pycharm on Ubuntu, if that has any relevance. Any ideas?
Fixed it thanks to engineersmnky's help. For whatever reason, the Subprocess, regardless of whether shell=True or shell=False was used, whether bash was or wasn't used, whether env=os.environ was used, etc., could not read the ruby environment, including the local changes to the gem. It could only read the git-published changes. I'm a ruby/gem newbie, so the reason wasn't obvious to me.
I didn't have ownership of the repo for the gem I was editing, so I created a private fork of the repo, moved my local changes to that repo, edited the *.gemspec file to reflect the changes, and built and installed a new gem locally with the same name. Now everything works. All I have to do now is remember to add my local changes to the private git and reinstall as needed when new changes are made. Technically, maybe I could have just added the local changes to the public repo without committing anything, but I'd prefer playing it safe with the private repo.
Is it possible to use a "more complex" shell than just a single command shell? We have written a python shell that is a command loop, and it works fine in /etc/passwd like this:
user:x:1000:1000::/home/user:/usr/bin/ourshell.py
Of course the Python file has the shebang line for /usr/bin/python in it. However, we'd like to compile the Python shell into a .pyc file to save a bit of time on execution in login. So, after compiling, I've been trying to "quote" the shell line in /etc/passwd as "python ourshell.pyc", and I even tried making the shell a bash script which simply executes that same command (with the initial arguments).
Of course none of this has worked. When we SSH in, there is always some kind of error. Is there any special trick to what I am trying to do?
CPython's .pyc files are not text, and do not allow use of a shebang line. The traditional method is to have your called script be tiny; it would simply import a module with the rest of the program, which can then be precompiled. For instance, here is the main script of xonsh:
#!/usr/bin/env python3 -u
from xonsh.main import main
main()
This script takes negligible time to compile. It is also possible to run installed modules using -m, but that takes module names, not filenames, so is not suitable for a shebang script.
I suggest to code a small C wrapper program running your python shell.
(notice that execve(2) forbids nested shebang interpreters; I don't know if that applies for your case)
Look into your log files, probably /var/log/messages and /var/log/auth.log
You may also need to explicitly add (the compiled C executable for the wrapper) to /etc/shells; see shells(5)
Look also into scsh.
Your sshd daemon is probably using Linux Plugin Authentification Modules. So read more about PAM.
Create a file /usr/bin/shell_wrapper that contains this one line:
#!/usr/bin/python /usr/bin/ourshell.pyc
The compiled bytecode ourshell.pyc has to live in /usr/bin, or else change the path accordingly. The python path should go to the same version that compiled the bytecode.
Then make sure to have your /etc/passwd use /usr/bin/shell_wrapper for the shell executable:
user:x:1000:1000::/home/user:/usr/bin/shell_wrapper
I have a python script that involves multiple subprocess.call commands. I wrote the script on a Mac, and it runs perfectly. I just tried running it on Windows and I am baffled by an error.
The following command to call ImageMagick returns "exit status 4":
file1 = "D:/Temp/OCR_test/sample/images/crops/time_0011.png"
subprocess.call(['convert', file1, '-resize', '200%', file1])
Changing the command to the following works:
subprocess.call(['convert', file1, '-resize', '200%', file1], shell=True)
I'm a little wary of using shell=True because of the warnings in the documentation.
I also need the command to work on both Mac and Windows, and I am confused as to why it wouldn't work on Windows (I checked and the command does work using Windows CMD).
Interestingly, the following line worked earlier in the script (where file, lat_crop1, and croplat are defined variables):
subprocess.call(['ffmpeg', '-loglevel', 'panic', '-i', file, '-vf', lat_crop1, '-n', croplat])
I read this SO question and tried all the suggestions (shlex, variations of my command, etc...), but I still get the same result.
Anyone have any idea how I could modify that line so it can work without shell=True?
Also, what does "exit status 4" mean? I Googled and read so much documentation but found nothing about it.
EDIT: Based on the information provided in the answer, I changed to the command that was not working to subprocess.call(['mogrify', file1, '-resize', '200%', file1]) and that runs successfully in Python on Windows. Luckily ImageMagick provides mogrify as an alternative to convert.
I suspect you're calling C:\Windows\System32\convert.exe (NTFS/FAT partition converter) instead of imagemagick.
When shell=True, the path finds the convert.bat or convert.cmd script from imagemagick, but without it, the path can only find the .exe file, which is a completely different program, and you get error 4: invalid parameter.
In that particular case, it doesn't work even with an executable, because the "wrong" convert is located in a system path. shell=False only searches in system paths (python subprocess Popen environment PATH?). So that's bad luck that a program named convert is located in the system path.
Try to explicitly add .bat extension like this:
subprocess.call(['convert.bat', file1, '-resize', '200%', file1])
To know which executables are likely to be run you can type:
where convert
in a command prompt.
In your case (an executable), that could be workarounded by passing the absolute path of the executable you want to run.
Another way would be to copy/rename ImageMagick convert to imconvert. Which program calls itself convert and doesn't expect conflicts anyway ?
Or in that case, it's legitimate to leave shell=True, with a nice comment explaining that Microsoft left a confusing (and seldom used convert program in a system path for us to trip into)
Solutions are not pretty, at least there are some.
I'm trying to call a process from python using subprocess.call as shown below:
from subprocess import call
exePath = 'C:\\Test\\EXE.exe'
inPath = 'C:\\Test\\IN.in'
outPath = 'C:\\Test\\OUT.out'
call([exePath, inPath, outPath])
This prints a few lines from EXE.exe followed by "The handle is invalid" -- but as a string, not as an error, which makes me think it might be a message from the EXE.exe:
Unzipping Solution...
0.0% The handle is invalid.
However when I open cmd.exe and paste in:
C:\Test\EXE.exe C:\Test\IN.in C:\Test\OUT.out
it works fine.
Can someone point me in the right direction?
Thanks!
I'm running Python 2.7 64-bit on Windows 7.
EDIT:
It looks now like a problem in PyDev where the console cannot handle the the stdout from the process overwriting lines. The code runs fine from IDLE. Still looking for a fix for PyDev...
I think you're having this issue because PyDev is not a real terminal (i.e.: in Python, os.isatty() will return False when run from PyDev).
If the exe really relies on having a terminal, currently there's not much that PyDev can do...
For now, you can make your call from Python as:
In windows:
popen = subprocess.Popen(['myexe', 'params'], creationflags=subprocess.CREATE_NEW_CONSOLE)
popen.wait()
In Linux (as the CREATE_NEW_CONSOLE is not available):
args = ['xterm', '-e'] + ['myexe', 'params']
popen = subprocess.Popen(args)
popen.wait()
so that it works regardless of who's calling it :)
I think Aptana Studio does have an actual terminal replacement, but there's no PyDev integration to launch things on it...
I am trying to call a perl script from my python program with the following line:
subprocess.call(r'/path/to/compute_lexrank.pl /path/to/11sent',shell=True)
when I run the same perl script from the shell (just typing /path/to/compute_lexrank.pl /path/to/11sent) it works fine as expected, but when I run it from the python program, the perl script is executed, but gives a mysterious error:
Math::MatrixReal::new(): number of rows must be integer > 0 at /Users/filippo/Downloads/clairlib-core-1.08/lib//Clair/Network.pm line 1628
now because I havent wrote the perl script myself I dont know how to fix this, but why the same script behaves differently when I run it from the shell or from subprocess.call?
I am using MacOsX, python 2.6 (but I have tried also with 2.5, same stuff) and perl 5.10.
Anyone can help?
In Perl, there's a good rule: if possible, use list forms of popen and system. Python seems to have those, too. I wonder what happens if you try this out:
helper = "/path/to/compute_lexrank.pl"
helper_input = "/path/to/11sent"
subprocess.call([helper, helper_input])
You may need to pass the working directory to the subprocess.call
subprocess.call(r'/path/to/compute_lexrank.pl /path/to/11sent',shell=True,cwd="/path/to")
If cwd is not None, the child’s current directory will be changed to cwd before it is executed. Note that this directory is not considered when searching the executable, so you can’t specify the program’s path relative to cwd.