We Keep Coding

for fi in sys.argv[1:]: argument list too long

I am trying to execute a python script on all text files in a folder:
for fi in sys.argv[1:]:
And I get the following error
-bash: /usr/bin/python: Argument list too long
The way I call this Python function is the following:
python functionName.py *.txt
The folder has around 9000 files. Is there some way to run this function without having to split my data in more folders etc? Splitting the files would not be very practical because I will have to execute the function in even more files in the future... Thanks
EDIT: Based on the selected correct reply and the comments of the replier (Charles Duffy), what worked for me is the following:
printf '%s\0' *.txt | xargs -0 python ./functionName.py
because I don't have a valid shebang..

This is an OS-level problem (limit on command line length), and is conventionally solved with an OS-level (or, at least, outside-your-Python-process) solution:
find . -maxdepth 1 -type f -name '*.txt' -exec ./your-python-program '{}' +
...or...
printf '%s\0' *.txt | xargs -0 ./your-python-program
Note that this runs your-python-program once per batch of files found, where the batch size is dependent on the number of names that can fit in ARG_MAX; see the excellent answer by Marcus Müller if this is unsuitable.

No. That is a kernel limitation for the length (in bytes) of a command line.
Typically, you can determine that limit by doing
getconf ARG_MAX
which, at least for me, yields 2097152 (bytes), which means about 2MB.
I recommend using python to work through a folder yourself, i.e. giving your python program the ability to work with directories instead of individidual files, or to read file names from a file.
The former can easily be done using os.walk(...), whereas the second option is (in my opinion) the more flexible one. Use the argparse module to give your python program an easy-to-use command line syntax, then add an argument of a file type (see reference documentation), and python will automatically be able to understand special filenames like -, meaning you could instead of
for fi in sys.argv[1:]
do
for fi in opts.file_to_read_filenames_from.read().split(chr(0))
which would even allow you to do something like
find -iname '*.txt' -type f -print0|my_python_program.py -file-to-read-filenames-from -

Don't do it this way. Pass mask to your python script (e.g. call it as python functionName.py "*.txt") and expand it using glob (https://docs.python.org/2/library/glob.html).

I think about using glob module. With this module you invoke your program like:
python functionName.py "*.txt"
then shell will not expand *.txt into file names. You Python program will receive *.txt in argumens list and you can pass it into glob.glob():
for fi in glob.glob(sys.argv[1]):
...

Renaming sequential image files with gaps

I'm looking for a way to rename a list of image files with gaps to be sequential. Also I want to give them a padding of 4. I'm currently using Python 2.7 and Linux bash to program this.
Example:
1.png
2.png
3.png
20.png
21.png
50.png
Should turn into:
0001.png
0002.png
0003.png
0004.png
0005.png
0006.png
I also would like for the files name to be the same as the directory that they are currently in.
Example:
c_users_johnny_desktop_images.0001.png
c_users_johnny_desktop_images.0002.png
c_users_johnny_desktop_images.0003.png
c_users_johnny_desktop_images.0004.png
c_users_johnny_desktop_images.0005.png
c_users_johnny_desktop_images.0006.png
Any help would be greatly appreciated! :)
Cheers

this is python
#first collect all files that start with a number and end with .png
my_files = [f for f in os.listdir(some_directory) if f[0].isdigit() and f.endswith(".png")]
#sort them based on the number
sorted_files = sorted(my_files,key=lambda x:int(x.split(".")[0])) # sort the file names by starting number
#rename them sequentially
for i,fn in enumerate(sorted_files,1): #thanks wim
os.rename(sorted_files[i],"{0:04d}.png".format(i))
I could have used list.sort(key=...) to sort in place but I figured this would be marginally more verbose and readable ...

Try doing this in a shell :
rename -n '
$s = substr(join("_", split("/", $ENV{PWD})), 1) . ".";
s/(\d+)\.png/$s . sprintf("%04d", ++$c) . ".png"/e
' *.png
Output :
1.png -> c_users_johnny_desktop_images.0001.png
2.png -> c_users_johnny_desktop_images.0002.png
3.png -> c_users_johnny_desktop_images.0003.png
20.png -> c_users_johnny_desktop_images.0004.png
21.png -> c_users_johnny_desktop_images.0005.png
50.png -> c_users_johnny_desktop_images.0006.png
rename is http://search.cpan.org/~pederst/rename/ and is the defalut rename command on many distros.
When the command is tested as well, you can remove the -n switch to do it for real.

Blah Blah Blah. CSH is bad. BASH is good. Python is better. Bah humbug. I still use TCSH...
% set i = 1
% foreach FILE ( `ls *[0-9].png | sort -n` )
echo mv $FILE `printf %04d $i`.png ; # i ++
end
Output:
mv 1.png 0001.png
mv 2.png 0002.png
mv 3.png 0003.png
mv 20.png 0004.png
mv 21.png 0005.png
mv 50.png 0006.png
Responding to comments:
Still need c_users_johnny_desktop_images.
Ok, so use:
echo mv $FILE c_users_johnny_desktop_images.`printf %04d $i`.png ; # i ++
It's not like my example was hard to read.
Correction: Perhaps you meant to automatically extract the current directory name and incorporate it. E.g.:
echo mv $FILE `echo $cwd | sed -e 's|^/||' -e 's|/|_|g'`.`printf %04d $i`.png ; # i ++
-
are globs not present in tcsh ? Your parsing of ls seems scary
Of course globs are present. That's what we are passing into ls. But globbing gives us a list that is sorted alphabetically, as in 1,2,20,21,3,50. We want a numerical sort, as in 1,2,3,20,21,50. Standard problem when we don't have leading zeros in the numbers.
sort -n does a numeric sort. ls gives us a newline after each filename. We could just as easily write:
foreach FILE ( `echo *[0-9].png | tr ' ' '\012' | sort -n` )
But I'm lazy and ls does the newline for me. What's so scary about it?

shell script to find a string and print the current line and next line recursively

I have written a small script which searches the string and prints the current line. But m little confused to print the next line. I am ok with bash/perl/python
#!/bin/bash
CURRENT_DIR=`pwd`
cnt=0
for dir in $(find $CURRENT_DIR -type d)
do
for myFile in $dir/*
do
if [ -f "$myFile" ]; then
cat $myFile | while myLine=`line`
do
allFile="$myLine"
if echo "$myLine" | grep -q $1 ; then
echo "$myFile" "$allFile" ""
fi
#echo 'expr $count+1'
#echo "$allFile" ""
done #LINE
fi
done #FILE
done # DIRECTORY

If your grep is GNU:
grep -A1 pattern file

I am giving an example to you here in bash, This one considers a bunch of text files in a directory. You can manipulate it as you need .
Within one dir
grep "search string" *.txt
Search or go to sub-dir
find /full/path/to/dir -name "*.txt" -exec grep "search string" {} ;
Hope this helps you .

you can do it using awk:
awk '/Message/{print;getline;print}' your_file
Above is for one single file.This command will show you the matched pattern line and the next line in the file.
If you want to do it recursive in all the files in a directory structure then :
find . -name -type f|xargs awk '/Message/{print;getline;print}'

Add line on top of each Python file in current and sub directories

I'm on an Ubuntu platform and have a directory containing many .py files and subdirectories (also containing .py files). I would like to add a line of text to the top of each .py file. What's the easiest way to do that using Perl, Python, or shell script?

find . -name \*.py | xargs sed -i '1a Line of text here'
Edit: from tchrist's comment, handle filenames with spaces.
Assuming you have GNU find and xargs (as you specified the linux tag on the question)
find . -name \*.py -print0 | xargs -0 sed -i '1a Line of text here'
Without GNU tools, you'd do something like:
while IFS= read -r filename; do
{ echo "new line"; cat "$filename"; } > tmpfile && mv tmpfile "$filename"
done < <(find . -name \*.py -print)

for a in `find . -name '*.py'` ; do cp "$a" "$a.cp" ; echo "Added line" > "$a" ; cat "$a.cp" >> "$a" ; rm "$a.cp" ; done

#!/usr/bin/perl
use Tie::File;
for (#ARGV) {
tie my #array, 'Tie::File', $_ or die $!;
unshift #array, "A new line";
}
To process all .py files in a directory recursively run this command in your shell:
find . -name '*.py' | xargs perl script.pl

This will
recursively walk all directories starting with the current working
directory
modify only those files whose filename end with '.py'
preserve file permissions (unlike
open(filename,'w').)
fileinput also gives you the option of backing up your original files before modifying them.
import fileinput
import os
import sys
for root, dirs, files in os.walk('.'):
for line in fileinput.input(
(os.path.join(root,name) for name in files if name.endswith('.py')),
inplace=True,
# backup='.bak' # uncomment this if you want backups
):
if fileinput.isfirstline():
sys.stdout.write('Add line\n{l}'.format(l=line))
else:
sys.stdout.write(line)

import os
for root, dirs, files in os.walk(directory):
for file in files:
if file.endswith('.py')
file_ptr = open(file, 'r')
old_content = file_ptr.read()
file_ptr = open(file, 'w')
file_ptr.write(your_new_line)
file_ptr.write(old_content)
As far as I know you can't insert in begining or end of file in python. Only re-write or append.

What's the easiest way to do that using Perl, Python, or shell script?
I'd use Perl, but that's because I know Perl much better than I know Python. Heck, maybe I'd do this in Python just to learn it a bit better.
The easiest way is to use the language that you're familiar with and can work with. And, that's also probably the best way too.
If these are all Python scripts, I take it you know Python or have access to a bunch of people who know Python. So, you're probably better off doing the project in Python.
However, it's possible with shell scripts too, and if you know shell the best, be my guest. Here's a little, completely untested shell script right off the top of my head:
find . -type f -name "*.py" | while read file
do
sed 'i\
I want to insert this line
' $file > $file.temp
mv $file.temp $file
done

removing extensions in subdirectories

I need to remove the extension ".tex":
./1-aoeeu/1.tex
./2-thst/2.tex
./3-oeu/3.tex
./4-uoueou/4.tex
./5-aaa/5.tex
./6-oeua/6.tex
./7-oue/7.tex
Please, do it with some tools below:
Sed and find
Ruby
Python
My Poor Try:
$find . -maxdepth 2 -name "*.tex" -ok mv `sed 's#.tex##g' {}` {} +

A Python script to do the same:
import os.path, shutil
def remove_ext(arg, dirname, fnames):
argfiles = (os.path.join(dirname, f) for f in fnames if f.endswith(arg))
for f in argfiles:
shutil.move(f, f[:-len(arg)])
os.path.walk('/some/path', remove_ext, '.tex')

One way, not necessarily the fastest (but at least the quickest developed):
pax> for i in *.c */*.c */*/*.c ; do
...> j=$(echo "$i" | sed 's/\.c$//')
...> echo mv "$i" "$j"
...> done
It's equivalent since your maxdepth is 2. The script is just echoing the mv command at the moment (for test purposes) and working on C files (since I had no tex files to test with).
Or, you can use find with all its power thus:
pax> find . -maxdepth 2 -name '*.tex' | while read line ; do
...> j=$(echo "$line" | sed 's/\.tex$//')
...> mv "$line" "$j"
...> done

Using "for i in" may cause "too many parameters" errrors
A better approach is to pipe find onto the next process.
Example:
find . -type f -name "*.tex" | while read file
do
mv $file ${file%%tex}g
done
(Note: Wont handle files with spaces)

Using bash, find and mv from your base directory.
for i in $(find . -type f -maxdepth 2 -name "*.tex");
do
mv $i $(echo "$i" | sed 's|.tex$||');
done
Variation 2 based on other answers here.
find . -type f -maxdepth 2 -name "*.tex" | while read line;
do
mv "$line" "${line%%.tex}";
done
PS: I did not get the part about escaping '.' by pax...

There's an excellent Perl rename script that ships with some distributions, and otherwise you can find it on the web. (I'm not sure where it resides officially, but this is it). Check if your rename was written by Larry Wall (AUTHOR section of man rename). It will let you do something like:
find . [-maxdepth 2] -name "*.tex" -exec rename 's/\.tex//' '{}' \;
Using -exec is simplest here because there's only one action to perform, and it's not too expensive to invoke rename multiple times. If you need to do multiple things, use the "while read" form:
find . [-maxdepth 2] -name "*.tex" | while read texfile; do rename 's/\.tex//' $texfile; done
If you have something you want to invoke only once:
find . [-maxdepth 2] -name "*.tex" | xargs rename 's/\.tex//'
That last one makes clear how useful rename is - if everything's already in the same place, you've got a quick regexp renamer.