I am trying to integrate a Python script into a bash script. However when I use the input() function, I am getting an EOFError. How can I fix this problem?
#!/bin/bash
python3 <<END
print(input(">>> "))
END
You cannot source both the script and the user input through the program's standard input. (That's in effect what you're trying to do. << redirects the standard input.)
Ideally, you would provide the script as command line argument instead of stdin using -c SCRIPT instead of <<EOF heredoc EOF:
#!/bin/bash
python3 -c 'print(input(">>> "))'
Note that you may need to mind your quoting and escaping in case you have a more complicated Python script with nested quotes.
You can still let the script run over multiple lines, if you need to:
#!/bin/bash
python3 -c '
import os.path
path_name = input("enter a path name >>> ")
file_exists = os.path.exists(path_name)
print("file " + path_name + " " +
("exists" if file_exists else "does not exist"))
'
Note that you will get into trouble when you want to use single quotes in your Python script, as happens when you want to print doesn't instead of does not.
You can work around that using several approaches. The one I consider most flexible (apart from putting you into quoting hell) is surrounding the Python script with double quotes instead and properly escape all inner double quotes and other characters that the shell interprets:
#!/bin/bash
python3 -c "
print(\"It doesn't slice your bread.\")
print('But it can', 'unsliced'[2:7], 'your strings.')
print(\"It's only about \$0. Neat, right?\")
"
Note that I also escaped $, as the shell would otherwise interpret it inside the surrounding double quotes and the result may not be what you wanted.
I am trying to split a file into a number of parts via a python script:
Here is my snippet:
def bashCommandFunc(commandToRun):
process = subprocess.Popen(commandToRun.split(), stdout=subprocess.PIPE)
output = process.communicate()
return output
filepath = "/Users/user/Desktop/TempDel/part-00000"
numParts = "5"
splitCommand = "split -l$((`wc -l < " + filepath + "/" + numParts + ")) " + filepath
splitCommand:
'split -l$((`wc -l < /Users/user/Desktop/TempDel/part-00000`/5)) /Users/user/Desktop/TempDel/part-00000'
If I run this command on a terminal, it splits the file as it's supposed to, but it fails for the above defined subprocess function.
I have tested the function for other generic commands and it works fine.
I believe the character " ` " (tilde) might be an issue,
What is the work around to getting this command to work?
Are there some better ways to split a file from python into "n" parts.
Thanks
You'll have to let Python run this line via a full shell, rather than trying to run it as a command. You can do that by adding shell=True option and not splitting your command. But you really shouldn't do that if any part of the command may be influenced by users (huge security risk).
You could do this in a safer way by first calling wc, getting the result and then calling split. Or even implement the whole thing in pure Python instead of calling out to other commands.
What happens now is that you're calling split with first parameter -l$((``wc, second parameter -l, etc.
I have a .jar archive that loads a file and then does some things with it and writes it to the disk again.
If I call this .jar directly from the command prompt, everything works. But when I try to do it from within python, I get the following error:
Input file ("C:\xxx.txt") was not found or was not readable.
This is my python code:
import sys, os, subprocess
if os.path.isdir(sys.argv[1]):
for file in os.listdir("."):
print (" ".join(['java', '-jar', sys.argv[2], 'd', "\"" + os.path.abspath(file) + "\"", "\""+os.path.join(os.path.join(os.path.abspath(os.path.dirname(file)), "output"), file) + "\""]))
subprocess.call(['java', '-jar', sys.argv[2], 'd', "\"" + os.path.abspath(file) + "\"", "\""+os.path.join(os.path.join(os.path.abspath(os.path.dirname(file)), "output"), file) + "\""])
When I copy the printed statement into the commandline, the jar executes perfectly; everything works. I tried running cmd as an admin, but that didn't help.
The problem is the extra quotes you're adding. When you pass subprocess a list of args, it already quotes them appropriately; if you quote them yourself, it'll end up quoting your quotes, so instead of passing an argument that, when unquoted, means C:\xxx.txt, you'll be passing an argument that, when unquoted, means "C:\xxx.txt", which is not a valid pathname.
The rule of thumb for Windows* is: If you know exactly what each argument should be, pass them as a list, and don't try to quote them yourself; if you know exactly what the final command-line string should be, pass it as a string, and don't try to break it into a list of separate arguments yourself.
* Note that this is only for Windows. On POSIX, unless you're using shell=True, you should basically never use a string.
I am working on a program that will find some files and provide the file information to a NSIS script. The NSIS script accepts the command line as follows
makensis.exe /DON="This is one" /DOD="c:\path1\path2 to dir\path 3" scriptfile.nsi
The values of the switches will change on each execution of the program. I have tried to get this to execute using subprocess.call and subprocess.Popen. The issue I am having has to do with quoting.
First of all the subprocess calls seem to put the entire argument statement between double quotes making NSIS see them as one argument. Second I am having some difficulty getting the individual switches properly quoted on the command line. Here is a snippet of what my program currently looks like.
subprocess.Popen([setup.profile['NSISExe'], ' /DON="' + setup.profile['DESC'] + '" /DOD="' + setup.profile['InstallDir'] + \
'" /DMT="' + app.machine_type.get() + '" /DSD="' + os.path.join(WinShellVar.LOCAL_APPDATA, 'MLC CAD', appname) + \
'" /DXV=X6 ' + setup.profile['NSISScript']])
And here is the output from NSIS
Can't open script " /DON="Mastercam X6 Standard" /DOD="C:\Users\John\Desktop" /D
MT="mill" /DSD="C:\Users\John\AppData\Local\MLC CAD\mcdeftool" /DXV=X6 bin\packa
ge.002.nsi"
As you can see I am using a mixed bag of data, getting some bits for dicts and some from class calls (be easy on me if my terms are somewhat incorrect, I have been learning python for about 4 days now, correct me please just nicely). If using this data like this is "unpythonic" let me know.
Looking forward to your input
disclaimer -- I don't use windows
I think you probably want something like:
subprocess.Popen([setup.profile['NSISExe'], '/DON=' + setup.profile['DESC'],
'/DOD=' + setup.profile['InstallDir'],
'/DMT=' + app.machine_type.get(),
'/DSD=' + os.path.join(WinShellVar.LOCAL_APPDATA, 'MLC CAD', appname),
'/DXV=X6',
setup.profile['NSISScript']])
When the shell reads the commandline, it splits on non-quoted, non-escaped whitespace. When you pass a list to Popen, it expects the list elements to be the way it would look after the shell split the arguments. The other option is to pass a string (instead of a list) exactly as you would put it into the windows shell and pass shell=True to Popen. But that method isn't preferred as it is much more vulnerable to shell-injection insecurities.
I'm trying to transfer and rename some files in a while loop using the subprocess.Popen(['cp', etc..]) and wait(). Unfortunately it appears that wait() command is not properly working, i.e. not waiting for the file to completely copy to the new directory. Most of the time the files copy over fine, however, a small subset of random files do not (not the same files each time I run the script) and are thus zero byte files or incomplete files. I have also tried using subprocess.check_call() but this does not work either. When I print the poll() value it's always zero which should mean the process has finished. Note all files I'm dealing with are in the range of 150KBs. My python script is being run in pyraf utilising python 2.7, python version of iraf (image reduction and analysis facility) since I'm using iraf routines. Is there any way to force Popen() or check_call() to wait for the file transfer to complete?
while count <= ncross_correlate and skip_flag != 's':
...more stuff
try:
iraf.rv.fxcor (object_fits, template_fits, apertures="*", cursor="",
continuum="both", filter="both", rebin="smallest", pixcorr="no",
osample=osample_reg, rsample=osample_reg, apodize=0.1, function="gaussian",
width="INDEF", height=0., peak="no", minwidth=3., maxwidth=21., weights=1.,
background=0., window=300., wincenter="INDEF", output=output_name, verbose="long",
imupdate="no", graphics="stdgraph", interactive="no", autowrite="yes",
autodraw="yes", ccftype="image", observatory="aao", continpars="", filtpars="",
keywpars="")
# Create a eps file of the cross_correlation file.
gki_output_name = output_name + '.gki'
iraf.plot.sgikern (gki_output_name, device='eps', generic='no', debug='no',
verbose='no', gkiunit='no')
Unfortunately the only way to convert the .gki file created in fxcor to some readable
format outside of iraf is to call the iraf task sgikern which dumps an .eps file in my
iraf/iraf/ directory without giving the option to change the file name or directory placement. In fact the filename is randomly generated. Very frustrating!!! Also note that nothing is wrong with any of the eps files created using iraf.plot.sgikern (i.e. no 0 KB files to begin with). Copying and renaming is where I have issues.
# Find the eps file in /iraf/iraf/, rename it, and move to proper output location.
iraf_dir = '/iraf/iraf/'
eps_file_list = glob.glob(iraf_dir + 'sgi' + '*.eps')
...more code
At this point I have tried using check_call() or Popen():
subprocess.check_call(['cp', eps_file_list[0], ccf_output_dir + object_name_sub +
'.eps'], stdout=subprocess.PIPE)
subprocess.check_call(['rm', eps_file_list[0]], stdout=subprocess.PIPE)
or
process1 = subprocess.Popen(['cp', eps_file_list[0], ccf_output_dir +
object_name_sub + '.eps'], stdout=subprocess.PIPE)
process1.wait()
process2 = subprocess.Popen(['rm', eps_file_list[0]], stdout=subprocess.PIPE)
process2.wait()
...more stuff
# end of try statement
#end of while statement
I reckon if I could somehow combine the two Popen statement in to a single Popen statement and also include a shell sleep time of maybe 0.01s to force the other two processes to finish before returning a completed process, that would probably fix it. Maybe something like this, though I'm not sure of the exact sentax:
process1 = subprocess.Popen(['cp', eps_file_list[0], ccf_output_dir +
object_name_sub + '.eps']; ['rm', eps_file_list[0]]; ['sleep', 0.01],
stdout=subprocess.PIPE)
process1.wait()
Hopefully this gives you an idea of what I'm trying to do. I've been trying lots of different things and looking all over for a solution to this problem and I'm truly stuck.
Cheers,
Brett
Perhaps the following would be sufficient:
subprocess.check_call(['mv', eps_file_list[0], ccf_output_dir + object_name_sub +
'.eps'], stdout=subprocess.PIPE)
and
process1 = subprocess.Popen(['mv', eps_file_list[0], ccf_output_dir +
object_name_sub + '.eps'], stdout=subprocess.PIPE)
process1.wait()
Have you considered using shutil.copyfile for the copy and os.remove for the deletion?
If you really want to use Subprocess, I believe the syntax is something like this:
process1 = subprocess.Popen('cp ' + eps_file_list[0] + ' ' + ccf_output_dir +
object_name_sub + '.eps; rm ' + eps_file_list[0] ' + '; sleep 0.01',
stdout=subprocess.PIPE)
This way, the command you're calling is all in one string: 'cp whatever foo/bar.eps; rm whatever; sleep 0.01'
You can also format the string with triple quotes and have the commands on separate lines:
'''
cp %s %s%s
rm %s
sleep %s
''' % (eps_file_list[0], ccf_output_dir, object_name_sub, eps_file_list[0], 0.01)
This isn't a complete solution nor a satisfying one but it's the best I have come up with and works ~99.9% of the time (out of 4000+ eps I create, 5 files were I either 0 byte or incomplete). This is an improvement over the original way I was doing this which was successful about 95% of the time. I have pasted the code below:
try:
iraf.rv.fxcor (object_fits, template_fits, apertures="*", cursor="",
continuum="both", filter="both", rebin="smallest", pixcorr="no", osample=osample_reg,
rsample=osample_reg, apodize=0.1, function="gaussian", width="INDEF", height=0., peak="no",
minwidth=3., maxwidth=21., weights=1., background=0., window=300.,
wincenter="INDEF", output=output_name, verbose="long", imupdate="no",
graphics="stdgraph", interactive="no", autowrite="yes", autodraw="yes",
ccftype="image", observatory="aao", continpars="", filtpars="", keywpars="")
# Create a eps file of the cross_correlation file.
gki_output_name = output_name + '.gki'
iraf.plot.sgikern (gki_output_name, device='eps', generic='no', debug='no',
verbose='no', gkiunit='no')
time.sleep(0.25)
I put a time sleeper here as I discovered that some of the ps files being created in my iraf directory had not been fully written by the time the my code tried to move the file to another directory.
# Find the eps file in /iraf/iraf/, rename it, move to proper output location, and delete the old eps file.
iraf_dir = '/iraf/iraf/'
eps_file_list = glob.glob(iraf_dir + 'sgi' + '*.eps')
...continuation of code
if len(eps_file_list) == 1:
eps_file_sub = os.path.basename(eps_file_list[0])
cmd1 = 'cp {0} {1}'.format(eps_file_list[0], ccf_output_dir + object_name_sub + '.eps')
cmd2 = 'rm {0}'.format(eps_file_list[0])
cmd3 = 'sleep 0.05'
process1 = subprocess.Popen("{}; {}; {}".format(cmd1, cmd2, cmd3), shell=True, stdout=subprocess.PIPE)
process1.wait()
With process1 I'm sending three subprocess shell commands. The first is to copy the eps files from my /iraf directory to another directory (the iraf function which creates them in the first place does not allow me to give these files a proper name nor location for output). The second is to remove the eps file from my /iraf directory. The third command for forces the kernal to sleep. By doing this, Python does not receive a completed signal until the sleep command has been reached. This part I believe works perfectly. The only issue is that very rarely the iraf routine used to create the eps file doesn't create them fast enough when I reach this command.
#endif
num_cross = num_cross + 1
#Endtry
...more code
This is a very clunky solution and not satisfy to the least bit but it does work 99.9% of the time. If anyone has a better solution, please let me know. This has been a very frustrating problem and everyone I've asked hasn't been able to come up with anything better (including people who program in python regularly in my astro department).