I am trying to write a python script to execute a command line program with parameters imported from another file. The command line interface for the program works as follows:
./executable.x parameter(a) parameter(b) parameter(c) ...
My code is:
#program to pass parameters to softsusy
import subprocess
#open parameter file
f = open('test.dat', 'r')
program = './executable.x'
#select line from file and pass to program
for line in f:
subprocess.Popen([program, line])
The test.dat file looks like this:
param(a) param(b) param(c)...
The script calls the program however it does not pass the variables. What am I missing?
You want:
line=f.readline()
subprocess.Popen([program]+line.split())
What you currently have will pass the entire line to the program as a single argument. (like calling it in the shell as program "arg1 arg2 arg3"
Of course, if you want to call the program once for each line in the file:
with open('test.dat','r') as f:
for line in f:
#you could use shlex.split(line) as well -- that will preserve quotes, etc.
subprocess.Popen([program]+line.split())
To start with, for you case, use subprocess.call() not subprocess.popen()
As for the "params not being passed" there is nothing obviously wrong in your script. Try concatenating the whole thing into long string and giving the string to .call() instead of list.
subprocess.call( program + " " + " ".join(line) )
Are you sure that line contains the data you expect it to contain?
To make sure, (if source file is short) try turning the file into list explicitly and making sure there is data in "line":
for line in file.readlines():
if len(line.trim().split(" ")) < 2:
raise Exception("Where are my params?")
Related
I'm trying to read a log file, written line by line, via readline.
I'm surprised to observe the following behaviour (code executed in the interpreter, but same happens when variations are executed from a file):
f = open('myfile.log')
line = readline()
while line:
print(line)
line = f.readline()
# --> This displays all lines the file contains so far, as expected
# At this point, I open the log file with a text editor (Vim),
# add a line, save and close the editor.
line = f.readline()
print(line)
# --> I'm expecting to see the new line, but this does not print anything!
Is this behaviour standard? Am I missing something?
Note: I know there are better way to deal with an updated file for instance with generators as pointed here: Reading from a frequently updated file. I'm just interested in understanding the issue with this precise use case.
For your specific use case, the explanation is that Vim uses a write-to-temp strategy. This means that all writing operations are performed on a temporary file.
On the contrary, your scripts reads from the original file, so it does not see any change on it.
To further test, instead of Vim, you can try to directly write on the file using:
echo "Hello World" >> myfile.log
You should see the new line from python.
for following your file, you can use this code:
f = open('myfile.log')
while True:
line = readline()
if not line:
print(line)
I want to run my script on the command line and put as argument the name of the file on which I want to run my python script.
This is what I want to enter on the command line:
./pavage_dataframe.py 4.A.2.txt
I would like to get the filename (4.A.2.txt) and open it with open().
I think it's possible to use "sys" but I haven't quite figured out how to use it.
It could be done as
import sys
file = sys.argv[1]
The first element of sys.argv list is a name of the script itself, followed by command line arguments you have passed
I am attempting to cat a CSV file into stdout and then pipe the printed output as input into a python program that also takes a system argument vector with 1 argument. I ran into an issue I think directly relates to how Python's fileinput.input() function reacts with regards to occupying the stdin file descriptor.
generic_user% cat my_data.csv | python3 my_script.py myarg1
Here is a sample Python program:
import sys, fileinput
def main(argv):
print("The program doesn't even print this")
data_list = []
for line in fileinput.input():
data_list.append(line)
if __name__ == "__main__":
main(sys.argv)
If I attempt to run this sample program with the above terminal command and no argument myarg1, the program is able to evaluate and parse the stdin for the data output from the CSV file.
If I run the program with the argument myarg1, it will end up throwing a FileNotFoundError directly related to myarg1 not existing as a file.
FileNotFoundError: [Errno 2] No such file or directory: 'myarg1'
Would someone be able to explain in detail why this behavior takes place in Python and how to handle the logic such that a Python program can first handle stdin data before argv overwrites the stdin descriptor?
You can read from the stdin directly:
import sys
def main(argv):
print("The program doesn't even print this")
data_list = []
for line in iter(sys.stdin):
data_list.append(line)
if __name__ == "__main__":
main(sys.argv)
You are trying to access a file which has not been yet created, hence fileinput cannot open it, but since you are piping the data you have no need for it.
This is by design. The conceptors of fileinput thought that there were use cases where reading from stdin would be non sense and just provided a way to specifically add stdin to the list of files. According to the reference documentation:
import fileinput
for line in fileinput.input():
process(line)
This iterates over the lines of all files listed in sys.argv[1:], defaulting to sys.stdin if the list is empty. If a filename is '-', it is also replaced by sys.stdin.
Just keep your code and use: generic_user% cat my_data.csv | python3 my_script.py - myarg1
to read stdin before myarg1 file or if you want to read it after : ... python3 my_script.py myarg1 -
fileinput implements a pattern common for Unix utilities:
If the utility is called with commandline arguments, they are files to read from.
If it is called with no arguments, read from standard input.
So fileinput works exactly as intended. It is not clear what you are using commandline arguments for, but if you don't want to stop using fileinput, you should modify sys.argv before you invoke it.
some_keyword = sys.argv[1]
sys.argv = sys.argv[:1] # Retain only argument 0, the command name
for line in fileinput.input():
...
I couldn't find this anywhere, so sorry if I missed it. It seems like it should be simple but somehow isn't. I have a simple program that opens a log (log1.lg let's say) and strips any lines that don't contain keywords. It then tosses them into a 2nd file that is renamed to Log1.lg.clean.
The way I've implemented this is by using os.rename so the code looks like this:
#define source and key words
source_log = 'Log1.lg'
bad_words = ['word', 'bad']
#clean up the log
with open(source_log) as orig_log, open('cleanlog.lg', 'w') as cleanlog:
for line in orig_log:
if not any9bad_word in line for bad_word in bad_words):
cleanlog.write(line)
#rename file and open in Notepad
rename = orig_log + '.clean'
new_log = os.rename("cleanlog.lg", rename)
prog = "notepad.exe"
subprocess.Popen(prog, new_log)
Error I'm getting is this:
File "C:\Users\me\Downloads\PythonStuff\stripMmax.py", line 23, in cleanLog
subprocess.Popen(prog, new_log)
File "C:\Python27\lib\subprocess.py", line 339, in __init__
raise TypeError("bufsize must be an integer")
TypeError: bufsize must be an integer
I'm using Python 2.7 if that's relevant. I don't get why this isn't working or why it's requiring a bufsize. I've seen other examples where this works this way so I'm thinking maybe this command doesn't work in 2.7 the way I'm typing it?
The documentation shows how to use this properly using the actual file name in quotes, but as you can see, mine here is contained in a variable which seems to cause issues. Thanks in advance!
See the Popen constructor here: subprocess.Popen. The second argument to Popen is bufsize. That explains your error. Also note that os.rename does not return anything so new_log will be None. Use your rename variable instead. Your call should look like this:
subprocess.Popen([prog, rename])
You likely also want to wait on the created Popen object:
proc = subprocess.Popen([prog, rename])
proc.wait()
Or something like that.
Using Python in NetBeans and having some trouble to set up file arguments as input/output. For instance:
import re, sys
for line in sys.stdin:
for token in re.split("\s+", line.strip()):
print(token)
Command line usage python splitprog.py < input.txt > output.txt works great. But in NetBeans the output window just waits, with nothing happening even if one give a file name (tested many combinations).
The Application Arguments row in project properties (where one would enter these files for a Java project) doesn’t seem to be used either, as the behaviour is the same regardless of whether there are file names/paths there or not. Is there some trick to get this to work or are file args currently unusable when it comes to Python in NetBeans?
ADD: As per suggestion by #John Zwinck, an example solution:
import re, sys
with open(sys.argv[1]) as infile:
with open(sys.argv[2], "w") as outfile:
for line in infile:
for token in re.split("\s+", line.strip()):
print(token, file = outfile)
Argument files are set in NB project properties. In command prompt, the programme is now simply run by python splitprog.py input.txt output.txt.
When you do this:
python splitprog.py < input.txt > output.txt
You are redirecting input.txt to stdin of python, and stdout of python to output.txt. You aren't using command line arguments to splitprog.py at all.
NetBeans does not support this.
Instead, you should pass the filenames as arguments, like this:
python splitprog.py input.txt output.txt
Then in NetBeans you just set the command line arguments to input.txt output.txt and it will work the same as the above command line in the shell. You'll need to modify your program slightly, perhaps like this:
with open(sys.argv[1]) as infile:
for line in infile:
# ...
If you still want to support stdin and stdout, one convention is to use - to mean those standard streams, so you could code your program to support this:
python splitprog.py - - < input.txt > output.txt
That is, you can write your program to understand - as "use the standard stream from the shell", if you need to support the old way of doing things. Or just default to this behavior if no command line arguments are given.