my python script is the following code:
1 import subprocess
2
3 # initial 'output' to make
4 r0 = 21
5 # number of runs to make
6 rn = 10
7
8 rf = r0+rn
9
10 for i in range(r0, rf):
11 #output directory
12 opt_dir = 'output'+str(i)
13 #put it in this output directory
14 popt_dir = './output'+str(i)
15
16 subprocess.call(['mkdir', opt_dir])
17 subprocess.call(['./exp_fit', 'efit.inp'])
18 subprocess.call(['mv', 'at*', popt_dir])
the intention is this:
i have a program called "exp_fit" which takes an input file "efit.inp". one call to ./exp_fit efit.inp will create output files called 'at0_l0_l0', 'at0_l1_l-1', ... etc (total 475 files starting with 'at').
now, i have been generating data files by running 'exp_fit', then creating output directories and moving them into the output directories with the following bash commands:
(for example, with the 20th run of my code)
mkdir output20
mv at* ./output20
so i would think that my script should do the same thing. however, it only does the following:
(1) it correctly generates all output files (475 files starting with 'at')
(2) it correctly creates the desired directories (output21 - output30)
(3) it DOES NOT, however, correctly move all the output files starting with 'at' into the desired directories. why is this? shouldn't the call to line 18 correctly execute the command to move all my files starting with 'at' into the desired directory?
should i be writing this script with bash instead of python? what is wrong with this?
Don't issue subprocess calls for things you can do natively from Python. To move files/dirs around, just use os.rename.
To create a directory, use os.mkdir.
To execute an external program, using subprocess is the right tool.
The problem is that this subprocess command
subprocess.call(['mv', 'at*', './output20'])
is not the same as typing this at a prompt
$ mv at* ./output20
In the latter case, the bash glob expansion converts the single at* argument to a list of arguments of matching filenames for the mv command. So the kernel sees the second as
['mv', 'at0_l0_l0', 'at0_l1_l-1', './output20']
kev's answer tells Python to pass the command through the shell, so the escaping will occur.
But the better solution is to use the glob module and os.rename libraries and not call the subprocess. Creating subprocesses is expensive, and using shell=True could lead to security holes, so it's best to avoid that habit.
(Actually, I suggest making the output directory, switching into it, and then running the exp_fit program from within that directory. Then you won't have to move the output. Try that first.)
If shell=True, the executable argument specifies which shell to use.
On Unix, the default shell is /bin/sh.
subprocess.call(['mv', 'at*', popt_dir], shell=True)
Related
I am using python and i want to get listing of all of the files /directories (not nested) at a given path. Meaning i need exact equivalent out put of "ls -l" command using python.
For e.g. at path /opt/test/ ls -l out put is shown below.
-rw-r--r-- 1 user qa-others 16715 Jan 16 13:38 file_2001161337
-rw-r--r-- 1 user qa-others 16715 Jan 16 13:46 file_2001161346
-rw-r--r-- 1 user qa-others 16715 Jan 16 13:54 file_2001161353
My python code is shown below.
print(subprocess.check_output(['ls', '-l']))
How can i pass the path value i.e. "/opt/temp" and get the same put of "ls -l" as shown above?
You can use pathlib.Path() for this (Python >=3.4):
from pathlib import Path
source = Path('/opt/temp')
# Get all children
content = source.glob('*')
By default this will return an iterator of pathlib.Path objects (cast iterator to list if you need a visual check).
Then you can programmatically access file attributes using pahtlib.Path.stat().
You are far, far better off using os.listdir, which essentially exactly what you want.
You can also use os.scandir if you need other information about these folders, though you'll need to filter to ensure you're only picking up folders:
[e for e in os.scandir() if e.isdir]
Each of these functions takes a path argument if you want to explicitly specify it, otherwise they run on the current directory.
I am attempting to run this .PY file from Command Prompt:
# Merge two .BSG files
# Starting block and world position are taken from the first file
# Example: "bsgmerge input.bsg output.bsg merged.bsg"
import io, sys
one = open(sys.argv[1]).readlines()
two = open(sys.argv[2]).readlines()
for n in [1,3,5,7,9,11,17,19,21,23]:
one[n] = one[n][:-1]+"|"+two[n].partition("|")[2]
open(sys.argv[3],"w").write("".join(one))
It is a program that takes a creation from the game Beseige and merges it with another saved creation so that opening the merged file results in both creations being present. If you want more details, you can read up on that here.
I am having trouble figuring out how to call this program from the command line. At first I thought the problem was me having Python 2 (it requires Python 3), so I uninstalled 2 and installed 3. This did not help.
What I am doing is entering the "python" command to pull up the Python environment within CMD, then entering the command to call the program based on the third comment in the file ("bsgmerge input.bsg output.bsg merged.bsg").
I tried using full file paths or simply changing to the correct directory before typing the "python" command and using only the file names, but so far I've had no luck.
When I am in the correct directory, then enter the Python environment, typing the command "bsgmerge 1.bsg 2.bsg M.bsg" (my existing files to be merged are 1.bsg and 2.bsg), this error occurs:
File "<stdin>", line 1
bsgmerge 1.bsg 2.bsg M.bsg
^
SyntaxError: invalid syntax
I took a Python course (which is why I used to have Python 2 on my machine) last fall, so I noticed that there is no "def" defining a function in the above code, which is something I've never encountered, so I'm thinking that is the root of my problems.
Thanks in advance for the help.
I was probably same problem with python launcher.
If you use Linux, first line shoud be:
#! /path/to/your/python/3
In Windows it some more complicated:
In registry by regedit change
HKEY_CLASSES_ROOT\Python.File\shell\open\command from "C:\Python27\python.exe" "%1" %* to "C:\Windows\py.exe" "%1" %*.
And first line of script shoud be:
#! python3
Now it shoud work properly.
So I've got a python script that, at it's core, makes .7z archives of selected directories for the purpose of backing up data. For simplicty sake I've simply invoked 7-zip through the windows command line, like so:
def runcompressor(target, contents):
print("Compressing {}...".format(contents))
archive = currentmodule
archive += "{}\\{}.7z".format(target, target)
os.system('7z u "{}" "{}" -mx=9 -mmt=on -ssw -up1q0r2x2y2z1w2'.format(archive, contents))
print("Done!")
Which creates a new archive if one doesn't exist and updates the old one if it does, but if something goes wrong the archive will be corrupted, and if this command hits an existing, corrupted archive, it just gives up. Now 7zip has a command for testing the integrity of an archive, but the documentation says nothing about giving an output, and then comes the trouble of capturing that output in python.
Is there a way I can test the archives first, to determine if they've been corrupted?
The 7z executable returns a value of two or greater if it encounters a problem. In a batch script, you would generally use errorlevel to detect this. Unfortunately, os.system() under Windows gives the return value of the command interpreter used to run your program, not the exit value of your program itself.
If you want the latter, you'll probably going to have to get your hands a little dirtier with the subprocess module, rather than using the os.system() call.
If you have version 3.5 (or better), this is as simple as:
import subprocess as sp
x = sp.run(['7z', 'a', 'junk.7z', 'junk.txt'], stdout=sp.PIPE, stderr=sp.STDOUT)
print(x.returncode)
That junk.txt in my case is a real file but junk.7z is just a copy of one of my text files, hence an invalid archive. The output from the program is 2 so it's easily detectable if something went wrong.
If you print out x rather than just x.returncode, you'll see something like (reformatted and with \r\n sequences removed for readability):
CompletedProcess(
args=['7z', 'a', 'junk.7z', 'junk.txt'],
returncode=2,
stdout=b'
7-Zip [64] 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
Error: junk.7z is not supported archive
System error:
Incorrect function.
'
)
I am trying to execute a python script on all text files in a folder:
for fi in sys.argv[1:]:
And I get the following error
-bash: /usr/bin/python: Argument list too long
The way I call this Python function is the following:
python functionName.py *.txt
The folder has around 9000 files. Is there some way to run this function without having to split my data in more folders etc? Splitting the files would not be very practical because I will have to execute the function in even more files in the future... Thanks
EDIT: Based on the selected correct reply and the comments of the replier (Charles Duffy), what worked for me is the following:
printf '%s\0' *.txt | xargs -0 python ./functionName.py
because I don't have a valid shebang..
This is an OS-level problem (limit on command line length), and is conventionally solved with an OS-level (or, at least, outside-your-Python-process) solution:
find . -maxdepth 1 -type f -name '*.txt' -exec ./your-python-program '{}' +
...or...
printf '%s\0' *.txt | xargs -0 ./your-python-program
Note that this runs your-python-program once per batch of files found, where the batch size is dependent on the number of names that can fit in ARG_MAX; see the excellent answer by Marcus Müller if this is unsuitable.
No. That is a kernel limitation for the length (in bytes) of a command line.
Typically, you can determine that limit by doing
getconf ARG_MAX
which, at least for me, yields 2097152 (bytes), which means about 2MB.
I recommend using python to work through a folder yourself, i.e. giving your python program the ability to work with directories instead of individidual files, or to read file names from a file.
The former can easily be done using os.walk(...), whereas the second option is (in my opinion) the more flexible one. Use the argparse module to give your python program an easy-to-use command line syntax, then add an argument of a file type (see reference documentation), and python will automatically be able to understand special filenames like -, meaning you could instead of
for fi in sys.argv[1:]
do
for fi in opts.file_to_read_filenames_from.read().split(chr(0))
which would even allow you to do something like
find -iname '*.txt' -type f -print0|my_python_program.py -file-to-read-filenames-from -
Don't do it this way. Pass mask to your python script (e.g. call it as python functionName.py "*.txt") and expand it using glob (https://docs.python.org/2/library/glob.html).
I think about using glob module. With this module you invoke your program like:
python functionName.py "*.txt"
then shell will not expand *.txt into file names. You Python program will receive *.txt in argumens list and you can pass it into glob.glob():
for fi in glob.glob(sys.argv[1]):
...
I am trying to make a python script that will open a directory, apply a perl script to every file in that directory and get its out put in either multiple text files or just one.
I currently have:
import shlex, subprocess
arg_str = "perl tilt.pl *.pdb > final.txt"
arg = shlex.split(arg_str)
import os
framespdb = os.listdir("prac_frames")
for frames in framespdb:
subprocess.Popen(arg, stdout=True)
I keep getting *.pdb not found. I am very new to all of this so any help trying to complete this script would help.
*.pdb not found means exactly that - there won't be a *.pdb in whatever directory you're running the script... and as I read the code - I don't see anything to imply it's within 'frames' when it runs the perl script.
you probably need os.chdir(path) before the Popen.
How do I "cd" in Python?
...using a python script to run somewhat dubious syscalls to perl may offend some people but everyone's done it.. aside from that I'd point out:
always specify full paths (this becomes a problem if you will later say, want to run your job automatically from cron or an environment that doesn't have your PATH).
i.e. 'which perl' - put that full path in.
./.pdb would be better but not as good as the fullpath/.pdb (which you could use instead of the os.chdir option).
subprocess.Popen(arg, stdout=True)
does not expand filename wildcards. To handle your *.pdb, use shell=True.