We Keep Coding

Errors with Glob while outputting file names

I am combining two questions here because they are related to each other.
Question 1: I am trying to use glob to open all the files in a folder but it is giving me "Syntax Error". I am using Python 3.xx. Has the syntax changed for Python 3.xx?
Error Message:
File "multiple_files.py", line 29
files = glob.glob(/src/xyz/rte/folder/)
SyntaxError: invalid syntax
Code:
import csv
import os
import glob
from pandas import DataFrame, read_csv
#extracting
files = glob.glob(/src/xyz/rte/folder/)
for fle in files:
with open (fle) as f:
print("output" + fle)
f_read.close()
Question 2: I want to read input files, append "output" to the names and print out the names of the files. How can I do that?
Example: Input file name would be - xyz.csv and the code should print output_xyz.csv .
Your help is appreciated.

Your first problem is that strings, including pathnames, need to be in quotes. This:
files = glob.glob(/src/xyz/rte/folder/)
… is trying to divide a bunch of variables together, but the leftmost and rightmost divisions are missing operands, so you've confused the parser. What you want is this:
files = glob.glob('/src/xyz/rte/folder/')
Your next problem is that this glob pattern doesn't have any globs in it, so the only thing it's going to match is the directory itself.
That's perfectly legal, but kind of useless.
And then you try to open each match as a text file. Which you can't do with a directory, hence the IsADirectoryError.
The answer here is less obvious, because it's not clear what you want.
Maybe you just wanted all of the files in that directory? In that case, you don't want glob.glob, you want listdir (or maybe scandir): os.listdir('/src/xyz/rte/folder/').
Maybe you wanted all of the files in that directory or any of its subdirectories? In that case, you could do it with rglob, but os.walk is probably clearer.
Maybe you did want all the files in that directory that match some pattern, so glob.glob is right—but in that case, you need to specify what that pattern is. For example, if you wanted all .csv files, that would be glob.glob('/src/xyz/rte/folder/*.csv').
Finally, you say "I want to read input files, append "output" to the names and print out the names of the files". Why do you want to read the files if you're not doing anything with the contents? You can do that, of course, but it seems pretty wasteful. If you just want to print out the filenames with output appended, that's easy:
for filename in os.listdir('/src/xyz/rte/folder/'):
print('output'+filename)

This works in http://pyfiddle.io:
Doku: https://docs.python.org/3/library/glob.html
import csv
import os
import glob
# create some files
for n in ["a","b","c","d"]:
with open('{}.txt'.format(n),"w") as f:
f.write(n)
print("\nFiles before")
# get all files
files = glob.glob("./*.*")
for fle in files:
print(fle) # print file
path,fileName = os.path.split(fle) # split name from path
# open file for read and second one for write with modified name
with open (fle) as f,open('{}{}output_{}'.format(path,os.sep, fileName),"w") as w:
content = f.read() # read all
w.write(content.upper()) # write all modified
# check files afterwards
print("\nFiles after")
files = glob.glob("./*.*") # pattern for all files
for fle in files:
print(fle)
Output:
Files before
./d.txt
./main.py
./c.txt
./b.txt
./a.txt
Files after
./d.txt
./output_c.txt
./output_d.txt
./main.py
./output_main.py
./c.txt
./b.txt
./output_b.txt
./a.txt
./output_a.txt
I am on windows and would use os.walk (Doku) instead.
for d,subdirs,files in os.walk("./"): # deconstruct returned aktDir, all subdirs, files
print("AktDir:", d)
print("Subdirs:", subdirs)
print("Files:", files)
Output:
AktDir: ./
Subdirs: []
Files: ['d.txt', 'output_c.txt', 'output_d.txt', 'main.py', 'output_main.py',
'c.txt', 'b.txt', 'output_b.txt', 'a.txt', 'output_a.txt']
It also recurses into subdirs.

Iterating over a text files in a subdirectory

How do I iterate over text files only within a directory? What I have thus far is;
for file in glob.glob('*'):
f = open(file)
text = f.read()
f.close()
This works, however I am having to store my .py file in the same directory (folder) to get it to run, and as a result the iteration is including the .py file itself. Ideally what I want to command is either;
"Look in this subdirectory/folder, and iterate over all the files in there"
OR...
"Look through all files in this directory and iterate over those with .txt extension"
I'm sure I'm asking for something fairly straight forward, but I do not know how to proceed. Its probably worth me highlighting that I got the glob module through trial and error, so if this is the wrong way to go around this particular method feel free to correct me! Thanks.

The glob.glob function actually takes a globbing pattern as its parameter.
For instance, "*.txt" while match the files whose name ends with .txt.
Here is how you can use it:
for file in glob.glob("*.txt"):
f = open(file)
text = f.read()
f.close()
If however you want to exclude some specific files, say .py files, this is not directly supported by globbing's syntax, as explained here.
In that case, you'll need to get those files, and manually exclude them:
pythonFiles = glob.glob("*.py")
otherFiles = [f for f in glob.glob("*") if f not in pythonFiles]

glob.glob() uses the same wildcard pattern matching as your standard unix-like shell. The pattern can be used to filter on extensions of course:
# this will list all ".py" files in the current directory
# (
>>> glob.glob("*.py")
['__init__.py', 'manage.py', 'fabfile.py', 'fixmig.py']
but it can also be used to explore a given path, relative:
>>> glob.glob("../*")
['../etc', '../docs', '../setup.sh', '../tools', '../project', '../bin', '../pylint.html', '../sql']
or absolute:
>>> glob.glob("/home/bruno/Bureau/mailgun/*")
['/home/bruno/Bureau/mailgun/Domains_ Verify - Mailgun.html', '/home/bruno/Bureau/mailgun/Domains_ Verify - Mailgun_files']
And you can of course do both at once:
>>> glob.glob("/home/bruno/Bureau/*.pdf")
['/home/bruno/Bureau/marvin.pdf', '/home/bruno/Bureau/24-pages.pdf', '/home/bruno/Bureau/alice-in-wonderland.pdf']

The solution is very simple.
for file in glob.glob('*'):
if not file.endswith('.txt'):
continue
f = open(file)
text = f.read()
f.close()

Python script to strip new lines from .txt files contained in a directory

I have a list of .txt files contained in a directory. Each file may have multiple lines. What could be the Python script to strip off all newlines from all the files contained in that directory? The resulting files should have exactly one line containing all the texts.
import os
os.chdir("/home/Pavyel/Desktop/Python Programs")
for i in os.listdir(os.getcwd()):
if i.endswith(".txt") :
f = open(i)
contents = f.read()
new_contents = contents.replace('\n', '')
print i
continue
else:
continue

Use glob.glob() to find the files of interest, i.e. those that end with .txt. glob() returns a list of matching filenames and also leaves the path intact, so you don't need to change directories.
Process the files in place with fileinput.input():
import fileinput
from glob import glob
pattern = '/home/Pavyel/Desktop/Python Programs/*.txt'
files = glob(pattern)
if files:
with fileinput.input(files, inplace=True) as f:
for line in f:
print(line.rstrip('\n'), end='')
If you are using Python 2 it might be worth passing mode='U' to fileinput.input() to ensure that universal newline processing is enabled, as is the default for Python 3. With that enabled you can be sure that \n will match newlines regardless of the platform on which your code runs.

import os
import sys
import fileinput
dir = "." #Directory to scan for files
file_list = os.listdir(dir)
for file in file_list:
if file.endswith(".txt"):
with fileinput.FileInput(file, inplace=True, backup=".bak") as f:
for line in f:
sys.stdout.write(line.replace("\n", ""))
This will also create a backup of all files it edited, just in case.
If you don't want the backup remove , backup=".bak" from the 7th.

Python: Find and replace strings in batch csv files

I have several hundred csv files that I would like to search for the string "Keyed,Bet" and change it to "KeyedBet". The string may or may not be within the file, and may be in different columns in different files.
I cobbled together the script below, but it doesn't work. I am definitely using replace() incorrectly, but can't quite figure out how, and am creating a new file when I don't really need to- if it simply updated the current file and saved under the same name, that would be fine (but beyond my beginner capabilities).
Where did I go wrong here? Thanks for the help!
import os
import csv
path='.'
filenames = os.listdir(path)
for filename in filenames:
if filename.endswith('.csv'):
r=csv.reader(open(filename))
new_data = []
for row in r:
replace("Keyed,Bet","KeyedBet")
new_data.append(row)
newfilename = "".join(filename.split(".csv")) + "_edited3.csv"
with open(newfilename, "w") as f:
writer = csv.writer(f)
writer.writerows(new_data)

Why reinvent the wheel? Just download sed + its dependencies, then
sed -i 's/Keyed,Bet/KeyedBet/ig' *.csv
Edit: The command above should work fine in Linux. Windows sed requires its quoted tokens to be double-quoted, rather than single.
sed -i "s/Keyed,Bet,KeyedBet/ig" *.csv

If you want to change the original files you can use fileinput.input with inplace=True to actually modify the original file, glob will find all the csv files for you in the given directory:
from glob import iglob
import fileinput
path = '.'
for line in fileinput.input(iglob(os.path.join(path, "*.csv")),inplace=True):
print(line.replace("Keyed,Bet", "KeyedBet"),end="")
Not quite one line but a lot less than 15.
To create new files:
path='.'
from glob import iglob
for filename in iglob(os.path.join(path,"*.csv")):
with open(os.path.join(path,filename)) as f,open(os.path.join(path, os.path.splitext(filename)[0]+ "_edited3.csv"), "w") as f2:
for line in f:
f2.write(line.replace("Keyed,Bet", "KeyedBet"))
Considering you are replacing strings it is easier to just open the files without the csv module and use str.replace, if you knew the string always appeared in the same row then the csv module would be a better option but it seems that substring can appear anywhere.

How to write tag deleter script in python

I want to implement a file reader (folders and subfolders) script which detects some tags and delete those tags from the files.
The files are .cpp, .h .txt and .xml And they are hundreds of files under same folder.
I have no idea about python, but people told me that I can do it easily.
EXAMPLE:
My main folder is A: C:\A
Inside A, I have folders (B,C,D) and some files A.cpp A.h A.txt and A.xml. In B i have folders B1, B2,B3 and some of them have more subfolders, and files .cpp, .xml and .h....
xml files, contains some tags like <!-- $Mytag: some text$ -->
.h and .cpp files contains another kind of tags like //$TAG some text$
.txt has different format tags: #$This is my tag$
It always starts and ends with $ symbol but it always have a comment character (//,
The idea is to run one script and delete all tags from all files so the script must:
Read folders and subfolders
Open files and find tags
If they are there, delete and save files with changes
WHAT I HAVE:
import os
for root, dirs, files in os.walk(os.curdir):
if files.endswith('.cpp'):
%Find //$ and delete until next $
if files.endswith('.h'):
%Find //$ and delete until next $
if files.endswith('.txt'):
%Find #$ and delete until next $
if files.endswith('.xml'):
%Find <!-- $ and delete until next $ and -->

The general solution would be to:
use the os.walk() function to traverse the directory tree.
Iterate over the filenames and use fn_name.endswith('.cpp') with if/elseif to determine which file you're working with
Use the re module to create a regular expression you can use to determine if a line contains your tag
Open the target file and a temporary file (use the tempfile module). Iterate over the source file line by line and output the filtered lines to your tempfile.
If any lines were replaced, use os.unlink() plus os.rename() to replace your original file
It's a trivial excercise for a Python adept but for someone new to the language, it'll probably take a few hours to get working. You probably couldn't ask for a better task to get introduced to the language though. Good Luck!
----- Update -----
The files attribute returned by os.walk is a list so you'll need to iterate over it as well. Also, the files attribute will only contain the base name of the file. You'll need to use the root value in conjunction with os.path.join() to convert this to a full path name. Try doing just this:
for root, d, files in os.walk('.'):
for base_filename in files:
full_name = os.path.join(root, base_filename)
if full_name.endswith('.h'):
print full_name, 'is a header!'
elif full_name.endswith('.cpp'):
print full_name, 'is a C++ source file!'
If you're using Python 3, the print statements will need to be function calls but the general idea remains the same.

Try something like this:
import os
import re
CPP_TAG_RE = re.compile(r'(?<=// *)\$[^$]+\$')
tag_REs = {
'.h': CPP_TAG_RE,
'.cpp': CPP_TAG_RE,
'.xml': re.compile(r'(?<=<!-- *)\$[^$]+\$(?= *-->)'),
'.txt': re.compile(r'(?<=# *)\$[^$]+\$'),
}
def process_file(filename, regex):
# Set up.
tempfilename = filename + '.tmp'
infile = open(filename, 'r')
outfile = open(tempfilename, 'w')
# Filter the file.
for line in infile:
outfile.write(regex.sub("", line))
# Clean up.
infile.close()
outfile.close()
# Enable only one of the two following lines.
os.rename(filename, filename + '.orig')
#os.remove(filename)
os.rename(tempfilename, filename)
def process_tree(starting_point=os.curdir):
for root, d, files in os.walk(starting_point):
for filename in files:
# Get rid of `.lower()` in the following if case matters.
ext = os.path.splitext(filename)[1].lower()
if ext in tag_REs:
process_file(os.path.join(root, base_filename), tag_REs[ext])
Nice thing about os.splitext is that it does the right thing for filenames that start with a ..