I couldn't find this anywhere, so sorry if I missed it. It seems like it should be simple but somehow isn't. I have a simple program that opens a log (log1.lg let's say) and strips any lines that don't contain keywords. It then tosses them into a 2nd file that is renamed to Log1.lg.clean.
The way I've implemented this is by using os.rename so the code looks like this:
#define source and key words
source_log = 'Log1.lg'
bad_words = ['word', 'bad']
#clean up the log
with open(source_log) as orig_log, open('cleanlog.lg', 'w') as cleanlog:
for line in orig_log:
if not any9bad_word in line for bad_word in bad_words):
cleanlog.write(line)
#rename file and open in Notepad
rename = orig_log + '.clean'
new_log = os.rename("cleanlog.lg", rename)
prog = "notepad.exe"
subprocess.Popen(prog, new_log)
Error I'm getting is this:
File "C:\Users\me\Downloads\PythonStuff\stripMmax.py", line 23, in cleanLog
subprocess.Popen(prog, new_log)
File "C:\Python27\lib\subprocess.py", line 339, in __init__
raise TypeError("bufsize must be an integer")
TypeError: bufsize must be an integer
I'm using Python 2.7 if that's relevant. I don't get why this isn't working or why it's requiring a bufsize. I've seen other examples where this works this way so I'm thinking maybe this command doesn't work in 2.7 the way I'm typing it?
The documentation shows how to use this properly using the actual file name in quotes, but as you can see, mine here is contained in a variable which seems to cause issues. Thanks in advance!
See the Popen constructor here: subprocess.Popen. The second argument to Popen is bufsize. That explains your error. Also note that os.rename does not return anything so new_log will be None. Use your rename variable instead. Your call should look like this:
subprocess.Popen([prog, rename])
You likely also want to wait on the created Popen object:
proc = subprocess.Popen([prog, rename])
proc.wait()
Or something like that.
Related
I'm trying to read a log file, written line by line, via readline.
I'm surprised to observe the following behaviour (code executed in the interpreter, but same happens when variations are executed from a file):
f = open('myfile.log')
line = readline()
while line:
print(line)
line = f.readline()
# --> This displays all lines the file contains so far, as expected
# At this point, I open the log file with a text editor (Vim),
# add a line, save and close the editor.
line = f.readline()
print(line)
# --> I'm expecting to see the new line, but this does not print anything!
Is this behaviour standard? Am I missing something?
Note: I know there are better way to deal with an updated file for instance with generators as pointed here: Reading from a frequently updated file. I'm just interested in understanding the issue with this precise use case.
For your specific use case, the explanation is that Vim uses a write-to-temp strategy. This means that all writing operations are performed on a temporary file.
On the contrary, your scripts reads from the original file, so it does not see any change on it.
To further test, instead of Vim, you can try to directly write on the file using:
echo "Hello World" >> myfile.log
You should see the new line from python.
for following your file, you can use this code:
f = open('myfile.log')
while True:
line = readline()
if not line:
print(line)
Hello I'm trying to run multiple OS command using Python in Mac OS
I want to convert image to tif file by using convert command in Terminal. However I have 100 images to convert, I make Python program to make it easier.
import os
import subprocess
files = os.listdir("/Users/woonie/Downloads/test")
i=1
for file in files:
args=['convert',file,'-resize','100%','-type',"Grayscale","/Users/woonie/Downloads/test/kor.",i,"test.exp0.tif"]
subprocess.Popen(args)
i = i+1
convert filename -...- output_filename.exp0.tif is the form of convert command so I need to change file name and output_filename every time. I have filename list in files. And I want to put number of image after "test" so it becomes kor.test1.exp0.tif, kor.test2.exp0.tif, etc.
Traceback (most recent call last):
File "/Users/woonie/PycharmProjects/image_chage/change.py", line 9, in <module>
subprocess.Popen(args)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1637, in _execute_child
self.pid = _posixsubprocess.fork_exec(
TypeError: expected str, bytes or os.PathLike object, not int
But this error came out.
So I have changed that code to
args=['convert'+str(file)+'-resize','100%','-type',"Grayscale","/Users/woonie/Downloads/test/kor."+str(i)+"test.exp0.tif"]
But same error came out...
It was work when I put that command directly into Terminal.
Did I use subprocess wrong? or I can't make program what I want to
Use
subprocess.run(args)
instead of
subprocess.Popen(args)
That being said, your args array should probably look like
args=['convert', str(file), '-resize','100%','-type',"Grayscale","/Users/woonie/Downloads/test/kor."+str(i)+"test.exp0.tif"]
because the filename and the resize options are probably separate arguments.
There are quite a few issues with your code.
Rather than carry around long pathnames multi-levels deep in your input and output files, it is often better to just change directory to where the files are and deal with simple filenames - it is less error-prone and less of a maintenance nightmare.
So, consider starting your code with something more like:
import os
# Go to where the images are
os.chdir('/Users/woonie/Downloads/test')
Rather than use use os.listdir(), you get much more flexibility if you use globbing exactly the same as you would in the shell. So, if you want to process just JPEGs rather than TIFFs you produced on the previous run, use:
import glob
# Get list of JPEGs to process
JPEGs = glob.glob('*jpg')
I am unsure why you are using subprocess.Popen() - you would normally do that when you want to capture the output, you might as well use subprocess.run()
I don't know why you use -resize 100%. Where did that come from? The image should be 100% anyway.
When you want to formulate your output filename, you can more simply use f-strings. So, having got rid of all the long paths, your last parameter becomes:
f'kor.{i}test.exp0.tif'
I am creating a simple file in python to reorganize some text data I grabbed from a website. I put the data in a .txt file and then want to use the "tail" command to get rid of the first 5 lines. I'm able to make this work for a simple filename shown below, but when I try to change the filename (to what I'd actually like it to be) I get an error. My code:
start = 2010
end = 2010
for i in range(start,end+1)
year = str(i)
...write data to a file called file...
teamname=open(file).readline() # want to use this in the new filename
teamfname=teamname.replace(" ","") #getting rid of spaces
file2 = "gotdata2_"+year+".txt"
os.system("tail -n +5 gotdata_"+year+".txt > "+file2)
The above code works as intended, creating file, then creating file2 that excludes the first 5 lines of file. However, when I change the name of file2 to be:
file2 = teamfname+"_"+year+".txt"
I get the error:
sh: line 1: _2010.txt: command not found
It's as if the end of my file2 statement is getting chopped off and the .txt part isn't being recognized. In this case, my code outputs a file but is missing the _2010.txt at the end. I've double checked that both year and teamfname are strings. I've also tried it with and without spaces in the teamfname string. I get the same error when I try to include a os.system mv statement that would rename the file to what I want it to be, so there must be something wrong with my understanding of how to specify the string here.
Does anyone have any ideas about what causes this? I haven't been able to find a solution, but I've found this problem difficult to search for.
Without knowing what your actual strings are, it's impossible to be sure what the problem is. However, it's almost certainly something to do with failing to properly quote and/or escape arguments for the command line.
My first guess would be that you have a newline in the middle of your filename, and the shell is truncating the command at the newline. But I wouldn't bet too heavily on that. If you actually printed out the repr of the pathname, I could tell you for sure. But why go through all this headache?
The solution to almost any problem with os.system is to not use os.system.
If you look at the docs, they even tell you this:
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.
If you use subprocess instead of os.system, you can avoid the shell entirely. You can also pass arguments as a list instead of trying to figure out how to quote them and escape them properly. Which would completely avoid the exact problem you're having.
For example, if you do this:
file2 = "gotdata2_"+year+".txt"
with open(file2, 'wb') as f:
subprocess.check_call(['tail', '-n', '+5', "gotdata_"+year+".txt"], stdout=f)
Then, if you change that first line to this:
file2 = teamfname+"_"+year+".txt"
It will still work even if teamfname has a space or a quote or another special character in it.
That being said, I'm not sure why you want to use tail in the first place. You can skip the first 5 lines just as easily directly in Python.
I am trying to write a python script to execute a command line program with parameters imported from another file. The command line interface for the program works as follows:
./executable.x parameter(a) parameter(b) parameter(c) ...
My code is:
#program to pass parameters to softsusy
import subprocess
#open parameter file
f = open('test.dat', 'r')
program = './executable.x'
#select line from file and pass to program
for line in f:
subprocess.Popen([program, line])
The test.dat file looks like this:
param(a) param(b) param(c)...
The script calls the program however it does not pass the variables. What am I missing?
You want:
line=f.readline()
subprocess.Popen([program]+line.split())
What you currently have will pass the entire line to the program as a single argument. (like calling it in the shell as program "arg1 arg2 arg3"
Of course, if you want to call the program once for each line in the file:
with open('test.dat','r') as f:
for line in f:
#you could use shlex.split(line) as well -- that will preserve quotes, etc.
subprocess.Popen([program]+line.split())
To start with, for you case, use subprocess.call() not subprocess.popen()
As for the "params not being passed" there is nothing obviously wrong in your script. Try concatenating the whole thing into long string and giving the string to .call() instead of list.
subprocess.call( program + " " + " ".join(line) )
Are you sure that line contains the data you expect it to contain?
To make sure, (if source file is short) try turning the file into list explicitly and making sure there is data in "line":
for line in file.readlines():
if len(line.trim().split(" ")) < 2:
raise Exception("Where are my params?")
I have a lot of Perl scripts that looks something like the following. What it does is that it will automatically open any file given as a command line argument and in this case print the content of that file. If no file is given it will instead read from standard input.
while ( <> ) {
print $_;
}
Is there a way to do something similar in Python without having to explicitly open each file?
The fileinput module in Python's standard library is designed exactly for this purpose, and I quote a bit of code from the URL I just gave:
import fileinput
for line in fileinput.input():
process(line)
Use print in lieu of process and you have the exact equivalent of your Perl code.
You could look into sys.argv. It may help.