I have a primary folder filled with subfolders, and each contains files with a particular naming scheme. I have unit-tested a function for creating and editing a text document within a single directory based on information in these files, but am now running into issues trying to get this function to iterate over every subdirectory.
Problem:
I am getting a "KeyError" for line 38 if (row["r_id"]) in filters:. This is because the file br_ids.csv is not being created. In the unit test this was functioning fine, so I can only assume it is some issue with how I am using os.walk.
Code:
import csv
import os
with open('hasf.txt','w') as hf:
for root, subFolders, files in os.walk('/path/to/topdir/'):
#if folder contains 'f_r.csv', list the path in 'hasf.txt'
if 'f_r.csv' in files:
hf.write("%s\n" % root)
if 'r.csv' in files:
with open(os.path.join(root, "r.csv")) as inf, open(os.path.join(root, "br_ids.csv"), "w") as output:
reader = csv.DictReader(inf, quotechar='"')
headers = ["r_id"]
writer_br = csv.DictWriter(output, headers, extrasaction='ignore')
writer_br.writeheader()
for row in reader:
if int(row["r_type"]) == 3:
writer_br.writerow(row)
# End creating br_ids
# parse the data you're about to filter with
with open(os.path.join(root, 'br_ids.csv'), 'r') as f:
filters = {(row["r_id"]) for row in csv.DictReader(f, delimiter=',', quotechar='"')}
with open(os.path.join(root, 'bt_ids.csv'), 'w') as out_f:
headers = ["t_id"]
out = csv.DictWriter(out_f, headers, extrasaction='ignore')
out.writeheader()
# go thru your rows and see if the matching(row[r_id]) is
# found in the previously parsed set of filters; if yes, skip the row
with open(os.path.join(root, 't.csv'), 'r') as f:
for row in csv.DictReader(f, delimiter=','):
if (row["r_id"]) in filters:
out.writerow(row)
I have gone through a few similar questions here, but none of them have directly hit on creating, editing, and using a file inside of each location of an os.walk. This is my first time using Python, and I am at somewhat of a loss. Also, if there is any way to make my other code more pythonic, I am all ears.
Thanks!
It turns out the issue was directly the KeyError - in some of the folders, br_id.csv had zero entries, and was throwing a KeyError because of this. The way I solved it was with try, like so:
# parse the data you're about to filter with
with open(os.path.join(root, 'br_ids.csv'), 'r') as f:
filters = {(row["r_id"]) for row in csv.DictReader(f, delimiter=',', quotechar='"')}
with open(os.path.join(root, 'bt_ids.csv'), 'w') as out_f:
headers = ["t_id"]
out = csv.DictWriter(out_f, headers, extrasaction='ignore')
out.writeheader()
# go thru your rows and see if the matching(row[r_id]) is
# found in the previously parsed set of filters; if yes, skip the row
with open(os.path.join(root, 't.csv'), 'r') as f:
for row in csv.DictReader(f, delimiter=','):
try:
if (row["r_id"]) in filters:
out.writerow(row)
except KeyError:
continue
In another case I had a if (row["r_id"]) not in filters: and bypassed this using the same method, except that if it returned a KeyError, then it went ahead and did out.writerow(row).
Related
I have a csv file with a list of image names and I want to filter the corresponding images into a new folder. Here is what I hoped could work but it doesn't. I get no error message, so I guess it iterates through the for loops but never returns a True at the if-section but I can't figure out why.
I already tried out str() but it still doesn't work.
Any ideas?
Thank you!
with open(csvFilePath, 'r', encoding='utf-8') as inp:
# run through every row of the file
for row in csv.reader(inp):
# search for filename in folder
for file in os.walk(imgFilePath):
if file == row[column]:
shutil.copy2(file, newImgPath)
Found a solution:
Little bit different approach. First we generate a list with all items of the certain column we are interested in. Then we check if the filenames are listed in the list. If True we copy the file to the new folder.
import os
import shutil
import csv
def test(csvFilePath, imgFilePath, newImgPath):
img_list = []
with open(csvFilePath, "r") as csv_file:
csv_reader = csv.reader(csv_file, delimiter = ',')
for rows in csv_reader:
img_list.append(rows[0])
for root, dirs, files in os.walk(imgFilePath):
for file in files:
if file in img_list:
shutil.copy(os.path.join(root,file), newImgPath)
I am looking for some assistance with writing API results to a .CSV file using Python.
I have my source as CSV file. It contains the below urls in a column as separate rows.
https://webapi.nhtsa.gov/api/SafetyRatings/modelyear/2013/make/Acura/model/rdx?format=csv
https://webapi.nhtsa.gov/api/SafetyRatings/modelyear/2017/make/Chevrolet/model/Corvette?format=csv
I can call the Web API and get the printed results. Please find attached 'Web API results' snapshot.
When I try to export these results into a csv, I am getting them as per the attached 'API results csv'. It is not transferring all the records. Right now, It is only sending the last record to csv.
My final output should be as per the attached 'My final output should be' for all the given inputs.
Please find the below python code that I have used. I appreciate your help on this. Please find attached image for my code.My Code
import csv, requests
with open('C:/Desktop/iva.csv',newline ='') as f:
reader = csv.reader(f)
for row in reader:
urls = row[0]
print(urls)
r = requests.get(urls)
print (r.text)
with open('C:/Desktop/ivan.csv', 'w') as csvfile:
csvfile.write(r.text)
You'll have to create a writer object of the csvfile(to be created). and use the writerow() method you could write to the csvfile.
import csv,requests
with open('C:/Desktop/iva.csv',newline ='') as f:
reader = csv.reader(f)
for row in reader:
urls = row[0]
print(urls)
r = requests.get(urls)
print (r.text)
with open('C:/Desktop/ivan.csv', 'w') as csvfile:
writerobj=csv.writer(r.text)
for line in reader:
writerobj.writerow(line)
One problem in your code is that every time you open a file using open and mode w, any existing content in that file will be lost. You could prevent that by using append mode open(filename, 'a') instead.
But even better. Just open the output file once, outside the for loop.
import csv, requests
with open('iva.csv') as infile, open('ivan.csv', 'w') as outfile:
reader = csv.reader(infile)
for row in reader:
r = requests.get(urls[0])
outfile.write(r.text)
I need to write something into files, which I am passing through command line in python. I am using the below code mycode.py
import csv
import sys
path = sys.argv[1]
row = ['4', ' Danny', ' New York']
with open(r"path" , 'w') as csvFile:
writer = csv.writer(csvFile)
writer.writerow(row)
When I execute it, the file is not written, but when I hardcode path as
with open(r"C:\Users\venkat\Desktop\python\sam.csv", 'w') as
csvFile:
the file is being written, Please let me know if I am missing anything.
One more requirement is I have to pass only the directory in open, and append some file name.
For example: I can pass
C:\Users\venkat\Desktop\python, sam.csv
I have to append to the directory in code.
You should use the path variable's value.
Replace
with open(r"path" , 'w') as csvFile:
with
with open(path , 'w') as csvFile:
^^^^
If you want to append one file to a directory path, you could use os package.
file_path = os.path.join(path, file)
Well this worked
import csv
import sys
path = sys.argv[1]
row = ['4', ' Danny', ' New York']
with open(path, 'w') as csvFile:
writer = csv.writer(csvFile)
writer.writerow(row)
If you want to append(or write) an existing file,this worked too using format:
path="insert\\pathOf\\file.txt"
with open("{}".format(path),'a') as file:
file.write("excellent\n")
The 'a' is for append,so it will add the 'excellent' string to file.txt.
If you want to write a new file just put 'w' instead of 'a'.
Using 'w' will overwrite the file.txt if already exists.
The \n is for ending in new line so if you run the same code 2 times it will add 'excellent' in two different lines and not side by side.
You should add curly braces if you want to convert it to raw format
with open(rf"{path}" , 'w') as csvFile:
I am compiling a load of CSVs into one. The first CSV contains the headers, which I am opening in write mode (maincsv). I am then making a list of all the others which live in a different folder and attempting to append them to the main one.
It works, however it just writes over the headings. I just want to start appending from line 2. I'm sure it's pretty simple but all the next(), etc. things I try just throw errors. The headings and data are aligned if that helps.
import os, csv
maincsv = open(r"C:\Data\OSdata\codepo_gb\CodepointUK.csv", 'w', newline='')
maincsvwriter = csv.writer(maincsv)
curdir = os.chdir(r"C:\Data\OSdata\codepo_gb\Data\CSV")
csvlist = os.listdir()
csvfiles = []
for file in csvlist:
path = os.path.abspath(file)
csvfiles.append(path)
for incsv in csvfiles:
opencsv = open(incsv)
csvreader = csv.reader(opencsv)
for row in csvreader:
maincsvwriter.writerow(row)
maincsv.close()
To simplify things I have the code load all the files in the directory the python code is run in. This will get the first line of the first .csv file and use it as the header.
import os
count=0
collection=open('collection.csv', 'a')
files=[f for f in os.listdir('.') if os.path.isfile(f)]
for f in files:
if ('.csv' in f):
solecsv=open(f,'r')
if count==0:
# assuming header is 1 line
header=solecsv.readline()
collection.write(header)
for x in solecsv:
if not (header in x):
collection.write(x)
collection.close()
I am trying to write to a .tsv file using python's CSV module, this is my code so far
file_name = "test.tsv"
TEMPLATE = "template.tsv"
fil = open(file_name, "w")
# Added suggested change
template = csv.DictReader(open(TEMPLATE, 'r'), delimiter='\t')
new_file = csv.DictWriter(fil, fieldnames=template.fieldnames, delimiter='\t')
new_file.writeheader()
basically TEMPLATE is a file that will contain the headers for the file, so i read the headers using DictReader and pass the fieldnames to DictWriter, as far as i know the code is fine, the file test.tsv is being created but for some reason the headers are not being written.
Any help as to why this is happening is appreciated, thanks.
DictReader's first argument should be a file object (create with open()), cf. http://docs.python.org/py3k/library/csv.html#csv.DictReader
You forgot open() for the TEMPLATE file.
import csv
file_name = "test.tsv"
TEMPLATE = "template.tsv"
fil = open(file_name, "w")
# you forgot this line, which will open the file
template_file = open(TEMPLATE, 'r')
template = csv.DictReader(template_file, delimiter='\t')
new_file = csv.DictWriter(fil, fieldnames=template.fieldnames, delimiter='\t')
new_file.writeheader()
Try to give DictReader opened file instead of file name:
csv.DictReader(open(TEMPLATE, 'r'), delimiter='\t')
Same for the writer, but opened for writing.