We Keep Coding

Filter Directory using Regex and output filtered files to another directory

I am simply trying to create a python 3 program that runs through all .sql files in a specific directory and then apply my regex that adds ; after a certain instance and write the changes made to the file to a separate directory with their respective file names as the same.
So, if I had file1.sql and file2.sql in "/home/files" directory, after I run the program, the output should write those two files to "/home/new_files" without changes the content of the original files.
Here is my code:
import glob
import re
folder_path = "/home/files/d_d"
file_pattern = "/*sql"
folder_contents = glob.glob(folder_path + file_pattern)
for file in folder_contents:
print("Checking", file)
for file in folder_contents:
read_file = open(file, 'rt',encoding='latin-1').read()
#words=read_file.split()
with open(read_file,"w") as output:
output.write(re.sub(r'(TBLPROPERTIES \(.*?\))', r'\1;', f, flags=re.DOTALL))
I receive an error of File name too long:"CREATE EXTERNAL TABLe" and also I am not too sure where I would put my output path (/home/files/new_dd)in my code.
Any ideas or suggestions?

With read_file = open(file, 'rt',encoding='latin-1').read() the whole content of the file was being used as the file descriptor. The code provided here iterate over the files names found with glob.glob pattern open to read, process data, and open to write (assuming that a folder newfile_sqls already exist,
if not, an error would rise FileNotFoundError: [Errno 2] No such file or directory).
import glob
import os
import re
folder_path = "original_sqls"
#original_sqls\file1.sql, original_sqls\file2.sql, original_sqls\file3.sql
file_pattern = "*sql"
# new/modified files folder
output_path = "newfile_sqls"
folder_contents = glob.glob(os.path.join(folder_path,file_pattern))
# iterate over file names
for file_ in [os.path.basename(f) for f in folder_contents]:
# open to read
with open(os.path.join(folder_path,file_), "r") as inputf:
read_file = inputf.read()
# use variable 'read_file' here
tmp = re.sub(r'(TBLPROPERTIES \(.*?\))', r'\1;', read_file, flags=re.DOTALL)
# open to write to (previouly created) new folder
with open(os.path.join(output_path,file_), "w") as output:
output.writelines(tmp)

text file cannot be made a folder

test.txt cannot be made. I want to make a folder&text file when I write a directory in html's form like:
.
The code:
os.makedirs(directory, exist_ok=True)
f = open(directory, 'w')
f.write("testtesttesttest")
f.close()
I want to make 0422 folder in /Users/xxx/Downloads,and test.txt in 0422 folder.
But when I run the file and I put directory in the form, 0422 folder was made and test.txt "folder" was made in 0422 folder.
I want to make test.txt (text file) in the folders. What is the problem in the code? Directory's variable can be gotten /Users/xxx/Downloads/0422/test.txt ,so it is ok.
How should I fix this?

You need to first create the directories, then create the file:
import os
directory = '/Users/xxx/Downloads/free.txt'
os.makedirs(os.path.dirname(directory), exist_ok=True)
with open(directory, 'w') as f:
f.write('blah blah')

You need to make one addition of the open statement.
os.makedirs(directory, exist_ok=True)
f = open(directory +'/test.txt', 'w')
f.write("testtesttesttest")
f.close()
We need to mention the text.txt somewhere

Opening a file and creating a new file in the same folder

def Function222(inF):
inF = open("C:\\Users\\Dell\\Desktop\\FF1\\txttt.txt")
outputF=open("output.txt", "w")
lines=inF.readlines()
for line in lines:
outputF.write('\n')
outputF.write(line*4)
inF.close()
outputF.close()
I need to create a new file called outputF and it should show up in the same folder that the inF is in, the problem is that it doesn't appear in the folder and I searched for the file on my computer but didn't find it

Get the Path:
import os
path= os.path.abspath("C:/example/cwd/mydir/myfile.txt")
open new file in path and write to it

Because the current working directory isn't the directory of the input file. Use os.getcwd() to get the current working directory, if it doesnt't match the directory of the input file, then you need to change your working directory first:
import os
def Function222(inF):
inF = open("C:\\Users\\Dell\\Desktop\\FF1\\txttt.txt")
#change the working directory
os.chdir("C:\\Users\\Dell\\Desktop\\FF1")
outputF=open("output.txt", "w")
lines=inF.readlines()
for line in lines:
outputF.write('\n')
outputF.write(line*4)
inF.close()
outputF.close()

Python: Looping through files in a different directory and scanning data

I am having a hard time looping through files in a directory that is different from the directory where the script was written. I also ideally would want my script through go to through all files that start with sasa. There are a couple of files in the folder such as sasa.1, sasa.2 etc... as well as other files such as doc1.pdf, doc2.pdf
I use Python Version 2.7 with windows Powershell
Locations of Everything
1) Python Script Location ex: C:Users\user\python_project
2) Main_Directory ex: C:Users\user\Desktop\Data
3) Current_Working_Directory ex: C:Users\user\python_project
Main directory contains 100 folders (folder A, B, C, D etc..)
Each of these folders contains many files including the sasa files of interest.
Attempts at running script
For 1 file the following works:
Script is run the following way: python script1.py
file_path = 'C:Users\user\Desktop\Data\A\sasa.1
def writing_function(file_path):
with open(file_path) as file_object:
lines = file_object.readlines()
for line in lines:
print(lines)
writing_function(file_path)
However, the following does not work
Script is run the following way: python script1.py A sasa.1
import os
import sys
from os.path import join
dr = sys.argv[1]
file_name = sys.argv[2]
file_path = 'C:Users\user\Desktop\Data'
new_file_path = os.path.join(file_path, dr)
new_file_path2 = os.path.join(new_file_path, file_name)
def writing_function(paths):
with open(paths) as file_object:
lines = file_object.readlines()
for line in lines:
print(line)
writing_function(new_file_path2)
I get the following error:
with open(paths) as file_object:
IO Error: [Errno 2] No such file or directory:
'C:Users\\user\\Desktop\\A\\sasa.1'
Please note right now I am just working on one file, I want to be able to loop through all of the sasa files in the folder.

It can be something in the line of:
import os
from os.path import join
def function_exec(file):
code to execute on each file
for root, dirs, files in os.walk('path/to/your/files'): # from your argv[1]
for f in files:
filename = join(root, f)
function_exec(filename)
Avoid using the variable dir. it is a python keyword. Try print(dir(os))
dir_ = argv[1] # is preferable

No one mentioned glob so far, so:
https://docs.python.org/3/library/glob.html
I think you can solve your problem using its ** magic:
If recursive is true, the pattern “**” will match any files and zero
or more directories and subdirectories. If the pattern is followed by
an os.sep, only directories and subdirectories match.

Also note you can change directory location using
os.chdir(path)

Python packaging-How to read files in a relative path

I use setuptools to distribute a python package. My directory structure is like following.
Mypackage
--setup.py
--my_package.py
Data
--some_name.log
I want users to put data files in this folder, and name can be anything with the extension .log.
log_list = []
for file in glob.glob('/home/ginger/Mypackage/Data/*.log'):
with open(file,'r') as f:
for line in f:
try:
data = p.parse(line)
except:
pass
log_list.append(data)
This code works fine. But when I try to get the absolute path from the relative path it does not read the data file. Why is this?
path = os.path.abspath("Data/*.log")
log = []
for file in glob.glob(path):
with open(file,'r') as f:
for line in f:
log.append(line)

Found the solution. Path has to be defined as following with the immediate directory.
path = os.path.abspath("Mypackage/Data/*.log")