i am having troubles on this python error.
I want to save changing variables to an csv file, however while the code runs again with an different variable it overwrites the previous one. I do not have the variables predetermined, they are generated while the code runs, so every time the loop will loop the program there will a different email passed.
Here is my code:
import csv
def hello(hme):
header = ['email']
data = [hme]
with open('countries.csv', 'w', encoding='UTF8', newline='') as f:
writer = csv.writer(f)
writer.writerow(header)
writer.writerows(data)
hello(["test#icloud.com"])
Thank you!
you should open the file as append, instead of write:
'a' instead of 'w'
import csv
def hello(hme):
header = ['email']
data = [hme]
with open('countries.csv', 'a', encoding='UTF8', newline='') as f:
writer = csv.writer(f)
writer.writerow(header)
writer.writerows(data)
hello(["test#icloud.com"])
Just replace 'w' by 'a' where 'w' writes in file (override) while 'a' appends the file whenever you write in it.
with open('countries.csv', 'a', encoding='UTF8', newline='') as f:
For the header "email" just write it before you add the loop of emails to do not duplicate it
Read the file contents first; add the new data; write the data to a file.
def hello(hme):
try:
with open('countries.csv', encoding='UTF8', newline='') as f:
stuff = list(csv.reader(f))
except FileNotFoundError:
# this must be the first time the function was called
stuff = [['email']]
stuff.append([hme])
with open('countries.csv', 'w', encoding='UTF8', newline='') as f:
writer = csv.writer(f)
writer.writerows(stuff)
If your file really only has one column you don't really need to use the csv module. Just append the new line to the file.
# assumes header is present
def hello(hme):
with open('countries.csv', 'a', encoding='UTF8') as f:
f.write(hme + '\n')
I currently have code that is writing to a csv file, however, it keesp replacing the first row of the file and overwriting it instead of writing of it.
NOTE: I have this function inside a for loop inside main().
def write_csv_report(filename, region, data, current, server1, server2=False):
if not os.path.exists(os.path.dirname(filename)):
try:
dir = os.makedirs(os.path.dirname(filename))
header = ['region','old_dns_server', 'proposed_dns_server1', 'proposed_dns_server2']
data = [region, current, server1, server2]
with open(filename, 'a') as file:
writer = csv.writer(file)
writer.writerow(header)
writer.writerow(data)
except OSError as exc:
if exc.errno != errno.EEXIST:
raise
When I run this function in my script, it writes to the file but only one entry is added.
Is there any suggestions or solution where I can add multiple entries? I have looked at other sage questions and googled but it looks like I already have the necessary changes needed to achieve this outcome, not sure what I'm missing.
For csv.writer (and csv.reader), you need to open the file with newline=''.
You also write the header every time, which is a bit odd.
Here's a minimal example that does more-or-less what you want:
import csv
header = ['region','old_dns_server', 'proposed_dns_server1', 'proposed_dns_server2']
data1 = ['foo', 'bar', 'bar', 'quux']
data2 = ['foo1', 'bar1', 'bar1', 'quux2']
filename = 'test.csv'
with open(filename, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(header)
writer.writerow(data1)
with open(filename, 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow(data2)
giving:
region,old_dns_server,proposed_dns_server1,proposed_dns_server2
foo,bar,bar,quux
foo1,bar1,bar1,quux2
There are two CSV files. I need to convert to JSON. Code is below
import csv
import json
import os
import glob
os.chdir(r'C:\Users\user\Desktop\test' )
result = glob.glob( '*.csv' )
print (result)
def make_json(csvFile, jsonFile):
csvFile, jsonFile = '',''
for i in result:
data = {}
with open(csvFile, encoding='utf-8') as csvf:
csvReader = csv.DictReader(csvf)
for rows in csvReader:
key = rows['id']
data[key] = rows
with open(jsonFile, 'w', encoding='utf-8') as jsonf:
jsonf.write(json.dumps(data, indent=4))
csvFilePath =f"{i}"
jsonFilePath =f"{i.split('.')[-2]}.json"
make_json(csvFile, jsonFile)
I got error > csvFile is not mentioned. But the third line from the end mentions the CSV file.
Disclaimer. Please find the error in the code. I already know of the working code which is in pandas
Below is the correct code, but I would recommend you learn to use the python debugger so you can resolve any logic flaws in your code next time. Documentation on the python debugger can be found here:
https://docs.python.org/3/library/pdb.html
Your code was structured in a way that meant for each csv file, you were not setting the file name until after you attempted to open it. The immediate error you saw was caused because you tried to call make_json() before you defined the values for csvFile and jsonFile.
I would recommend changing the code to:
import csv
import json
import glob
def make_json(csvList):
for csvFile in csvList:
data = {}
with open(csvFile, encoding='utf-8') as csvf:
csvReader = csv.DictReader(csvf)
for rows in csvReader:
key = rows['id']
data[key] = rows
jsonFile =f"{csvFile.split('.')[-2]}.json"
with open(jsonFile, 'w', encoding='utf-8') as jsonf:
jsonf.write(json.dumps(data, indent=4))
make_json(glob.glob('*.csv'))
You should try this
import csv, json, os, glob
os.chdir(r'C:\Users\user\Desktop\test' )
result = glob.glob( '*.csv' )
print(result)
def make_json():
for i in result:
with open(i, encoding='utf-8') as csvf:
data = [row for row in csv.DictReader(csvf)]
with open(f"{i.split('.')[-2]}.json", 'w', encoding='utf-8') as jsonf:
json.dump(data, jsonf)
make_json()
You did not initialize both the arguments of make_json() - (csvFilePath & jsonFilePath)
I'm new to Python and I'm trying to scrape some data and save them in a csv.
I'm trying to loop a csv with a list of URLs, read the data from each URL and write that information in another csv file
The following code is writing roughly half of the data in the cvs but is printing everything fine while it's writing
df_link = pd.read_csv('url_list')
with open('url_list.csv', newline='') as urls, open('output.csv', 'w', newline='') as output:
csv_urls = csv.reader(urls)
csv_output = csv.writer(output)
csv_output.writerow(['details','date'])
for link in df_link.iterrows():
url = link[1]['url']
browser.get(url)
soup = BeautifulSoup(browser.page_source)
csv_file = open('output.csv', 'w')
csv_writer = csv.writer(csv_file)
csv_writer.writerow(['details'])
details=[i.text for i in soup.find_all(class_='product-info-content-
block product-info')]
print('details :', details)
dt = date.today()
print('date :', dt)
csv_output.writerow([str(details).strip('[]'), dt])
csv_file.close()
Everything is being printed fine when the code is running, but not all the rows of data are being written in the output csv.
I hope someone can help.
Thank you!
It looks like you are opening output.csv twice, once in the beginning and then in the for loop. Since you are opening with the option w like csv_file = open('output.csv', 'w') it will overwrite the file every loop.
So if you move the below part out of the loop it might work better
csv_file = open('output.csv', 'w')
csv_writer = csv.writer(csv_file)
csv_writer.writerow(['details'])
So I have a program that creates CSV from .Json.
First I load the json file.
f = open('Data.json')
data = json.load(f)
f.close()
Then I go through it, looking for a specific keyword, if I find that keyword. I'll write everything related to that in a .csv file.
for item in data:
if "light" in item:
write_light_csv('light.csv', item)
This is my write_light_csv function :
def write_light_csv(filename,dic):
with open (filename,'a') as csvfile:
headers = ['TimeStamp', 'light','Proximity']
writer = csv.DictWriter(csvfile, delimiter=',', lineterminator='\n',fieldnames=headers)
writer.writeheader()
writer.writerow({'TimeStamp': dic['ts'], 'light' : dic['light'],'Proximity' : dic['prox']})
I initially had wb+ as the mode, but that cleared everything each time the file was opened for writing. I replaced that with a and now every time it writes, it adds a header. How do I make sure that header is only written once?.
You could check if file is already exists and then don't call writeheader() since you're opening the file with an append option.
Something like that:
import os.path
file_exists = os.path.isfile(filename)
with open (filename, 'a') as csvfile:
headers = ['TimeStamp', 'light', 'Proximity']
writer = csv.DictWriter(csvfile, delimiter=',', lineterminator='\n',fieldnames=headers)
if not file_exists:
writer.writeheader() # file doesn't exist yet, write a header
writer.writerow({'TimeStamp': dic['ts'], 'light': dic['light'], 'Proximity': dic['prox']})
Just another way:
with open(file_path, 'a') as file:
w = csv.DictWriter(file, my_dict.keys())
if file.tell() == 0:
w.writeheader()
w.writerow(my_dict)
You can check if the file is empty
import csv
import os
headers = ['head1', 'head2']
for row in interator:
with open('file.csv', 'a') as f:
file_is_empty = os.stat('file.csv').st_size == 0
writer = csv.writer(f, lineterminator='\n')
if file_is_empty:
writer.writerow(headers)
writer.writerow(row)
I would use some flag and run a check before writing headers! e.g.
flag=0
def get_data(lst):
for i in lst:#say list of url
global flag
respons = requests.get(i)
respons= respons.content.encode('utf-8')
respons=respons.replace('\\','')
print respons
data = json.loads(respons)
fl = codecs.open(r"C:\Users\TEST\Desktop\data1.txt",'ab',encoding='utf-8')
writer = csv.DictWriter(fl,data.keys())
if flag==0:
writer.writeheader()
writer.writerow(data)
flag+=1
print "You have written % times"%(str(flag))
fl.close()
get_data(urls)
Can you change the structure of your code and export the whole file at once?
def write_light_csv(filename, data):
with open (filename, 'w') as csvfile:
headers = ['TimeStamp', 'light','Proximity']
writer = csv.DictWriter(csvfile, delimiter=',', lineterminator='\n',fieldnames=headers)
writer.writeheader()
for item in data:
if "light" in item:
writer.writerow({'TimeStamp': item['ts'], 'light' : item['light'],'Proximity' : item['prox']})
write_light_csv('light.csv', data)
You can use the csv.Sniffer Class and
with open('my.csv', newline='') as csvfile:
if csv.Sniffer().has_header(csvfile.read(1024))
# skip writing headers
While using Pandas: (for storing Dataframe data to CSV file)
just add this check before setting header property if you are using an index to iterate over API calls to add data in CSV file.
if i > 0:
dataset.to_csv('file_name.csv',index=False, mode='a', header=False)
else:
dataset.to_csv('file_name.csv',index=False, mode='a', header=True)
Here's another example that only depends on Python's builtin csv package. This method checks that the header is what's expected or it throws an error. It also handles the case where the file doesn't exist or does exist but is empty by writing the header. Hope this helps:
import csv
import os
def append_to_csv(path, fieldnames, rows):
is_write_header = not os.path.exists(path) or _is_empty_file(path)
if not is_write_header:
_assert_field_names_match(path, fieldnames)
_append_to_csv(path, fieldnames, rows, is_write_header)
def _is_empty_file(path):
return os.stat(path).st_size == 0
def _assert_field_names_match(path, fieldnames):
with open(path, 'r') as f:
reader = csv.reader(f)
header = next(reader)
if header != fieldnames:
raise ValueError(f'Incompatible header: expected {fieldnames}, '
f'but existing file has {header}')
def _append_to_csv(path, fieldnames, rows, is_write_header: bool):
with open(path, 'a') as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
if is_write_header:
writer.writeheader()
writer.writerows(rows)
You can test this with the following code:
file_ = 'countries.csv'
fieldnames_ = ['name', 'area', 'country_code2', 'country_code3']
rows_ = [
{'name': 'Albania', 'area': 28748, 'country_code2': 'AL', 'country_code3': 'ALB'},
{'name': 'Algeria', 'area': 2381741, 'country_code2': 'DZ', 'country_code3': 'DZA'},
{'name': 'American Samoa', 'area': 199, 'country_code2': 'AS', 'country_code3': 'ASM'}
]
append_to_csv(file_, fieldnames_, rows_)
If you run this once you get the following in countries.csv:
name,area,country_code2,country_code3
Albania,28748,AL,ALB
Algeria,2381741,DZ,DZA
American Samoa,199,AS,ASM
And if you run it twice you get the following (note, no second header):
name,area,country_code2,country_code3
Albania,28748,AL,ALB
Algeria,2381741,DZ,DZA
American Samoa,199,AS,ASM
Albania,28748,AL,ALB
Algeria,2381741,DZ,DZA
American Samoa,199,AS,ASM
If you then change the header in countries.csv and run the program again, you'll get a value error, like this:
ValueError: Incompatible header: expected ['name', 'area', 'country_code2', 'country_code3'], but existing file has ['not', 'right', 'fieldnames']