I am looking for some assistance with writing API results to a .CSV file using Python.
I have my source as CSV file. It contains the below urls in a column as separate rows.
https://webapi.nhtsa.gov/api/SafetyRatings/modelyear/2013/make/Acura/model/rdx?format=csv
https://webapi.nhtsa.gov/api/SafetyRatings/modelyear/2017/make/Chevrolet/model/Corvette?format=csv
I can call the Web API and get the printed results. Please find attached 'Web API results' snapshot.
When I try to export these results into a csv, I am getting them as per the attached 'API results csv'. It is not transferring all the records. Right now, It is only sending the last record to csv.
My final output should be as per the attached 'My final output should be' for all the given inputs.
Please find the below python code that I have used. I appreciate your help on this. Please find attached image for my code.My Code
import csv, requests
with open('C:/Desktop/iva.csv',newline ='') as f:
reader = csv.reader(f)
for row in reader:
urls = row[0]
print(urls)
r = requests.get(urls)
print (r.text)
with open('C:/Desktop/ivan.csv', 'w') as csvfile:
csvfile.write(r.text)
You'll have to create a writer object of the csvfile(to be created). and use the writerow() method you could write to the csvfile.
import csv,requests
with open('C:/Desktop/iva.csv',newline ='') as f:
reader = csv.reader(f)
for row in reader:
urls = row[0]
print(urls)
r = requests.get(urls)
print (r.text)
with open('C:/Desktop/ivan.csv', 'w') as csvfile:
writerobj=csv.writer(r.text)
for line in reader:
writerobj.writerow(line)
One problem in your code is that every time you open a file using open and mode w, any existing content in that file will be lost. You could prevent that by using append mode open(filename, 'a') instead.
But even better. Just open the output file once, outside the for loop.
import csv, requests
with open('iva.csv') as infile, open('ivan.csv', 'w') as outfile:
reader = csv.reader(infile)
for row in reader:
r = requests.get(urls[0])
outfile.write(r.text)
Related
I'm trying to read sentences in a csv file, convert them to lowercase and save in other csv file.
import csv
import pprint
with open('dataset_elec_4000.csv') as f:
with open('output.csv', 'w') as ff:
data = f.read()
data = data.lower
writer = csv.writer(ff)
writer.writerow(data)
but I got error "_csv.Error: sequence expected". What should I do?
*I'm a beginner. Please be nice to me:)
You need to read over your input CSV row-by-row, and for each row, transform it, then write it out:
import csv
with open('output.csv', 'w', newline='') as f_out:
writer = csv.writer(f_out)
with open('dataset_elec_4000.csv', newline='') as f_in:
reader = csv.reader(f_in)
# comment these two lines if no input header
header = next(reader)
writer.writerow(header)
for row in reader:
# row is sequence/list of cells, so...
# select the cell with your sentence, I'm presuming it's the first cell (row[0])
data = row[0]
data = data.lower()
# need to put data back into a "row"
out_row = [data]
writer.writerow(out_row)
Python contains a module called csv for the handling of CSV files. The reader class from the module is used for reading data from a CSV file. At first, the CSV file is opened using the open() method in ‘r’ mode(specifies read mode while opening a file) which returns the file object then it is read by using the reader() method of CSV module that returns the reader object that iterates throughout the lines in the specified CSV document.
import csv
# opening the CSV file
with open('Giants.csv', mode ='r')as file:
# reading the CSV file
csvFile = csv.reader(file)
# displaying the contents of the CSV file
for lines in csvFile:
print(lines)
So I'm trying to run a scrape of a website. The scraper runs very well. But whenever I try to write the scraped information/rows into the csv file it deletes the previous row. I end up just having the very last scrape result in the file at the end. I'm sure it's just an indentation error? I'm still new to Python so any help would be appreciated!
Code:
# create general field names
fields = ['name', 'about', 'job_title', 'location','company',
'education','accomplishments','linkedin_url']
with open('ScrapeResults.csv', 'w') as f:
# using csv.writer method from CSV package
write = csv.writer(f)
write.writerow(fields)
f.close()
# Loop-through urls to scrape multiple pages at once
for individual,link in contact_dict.items():
## assign ##
the_name = individual
the_link = link
# scrape peoples url:
person = Person(the_link, driver=driver, close_on_complete=False)
# rows to be written
rows = [[person.name, person.about, person.job_title, person.location, person.company,
person.educations, person.accomplishments, person.linkedin_url]]
# write
with open('ScrapeResults.csv', 'w') as f:
# using csv.writer method from CSV package
write = csv.writer(f)
write.writerows(rows)
f.close()
You will need to open the file in append mode.
change
with open('ScrapeResults.csv', 'w') as f:
to
with open('ScrapeResults.csv', 'a') as f:
I think you're overwriting the file instead of appending it. Append mode in the for loop should fix the issue
with open('ScrapeResults.csv', "a")
Please refer python IO documentation: https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files
You have to open the file in append mode.
replace
with open('ScrapeResults.csv', 'w') as f:
to
with open('ScrapeResults.csv', 'a') as f:
everything good?
I need some help to save this script in CSV that reads a CSV and transforms the data through a lib. I've been racking my brain for hours and I can't figure out why I can't save the CSV file.
Can anybody help me? I am a beginner in python and I am learning the tool to use in ETL processes.
import csv
from user_agents import parse
with open('UserAgent.csv', 'r') as csv_file:
csv_reader = csv.reader(csv_file)
idUser = 0
space = ' / '
for line in csv_reader:
user_agent = parse(line[0])
idUser = idUser + 1
with open('data.csv', 'w') as f:
writer = csv.writer(f)
writer.writerow(user_agent)
writer.writerow expects an iterable. Your user_agent must not be an iterable.
Try
writer.writerow( [user_agent] )
instead of
writer.writerow(user_agent)
Check if that's what you want.
I am trying to fetch data from the internet to save it to a csv file.
I am facing a problem with writing to the csv file, the library leaves an empty row in the file
The data is random.org integers in text/plain format.
I'm using urllib.request to fetch the data and I am using this code to get the data and decode it
req = urllib.request.Request(url, data=None, headers={
'User-Agent': '(some text)'})
with urllib.request.urlopen(req) as response:
html = response.read()
encoding = response.headers.get_content_charset('utf-8')
data = html.decode(encoding)
I am using this line of code to open the csv file :csvfile = open('data.csv', "a")
Writing to the file:
writer = csv.writer(csvfile, lineterminator = '\n')
writer.writerows(data)
and of course I close the file at the end
Things I tried and didn't help :
Using (lineterminator = '\n') when writing
Using (newline = "") when opening the file
Defining a "delimiter" and a "quotechar" and "quoting"
Updated Answer
If when you create the list that is being written to the data list, if you add it as a list so that data becomes a list of lists, then set your line delimiter to be '\n', it should work. Below is the working code I used to test.
import csv
import random
csvfile = open('csvTest.csv', 'a')
data = []
for x in range(5):
data.append([str(random.randint(0, 100))])
writer = csv.writer(csvfile, lineterminator = '\n')
writer.writerows(data)
csvfile.close()
and it outputs
I'm trying to read the rows of a CSV file and print each row, but the code I've been using isn't opening the file or running the FOR loop. Any ideas?
import csv
domainFile = 'magtest.csv'
f = open(domainFile, 'ab+')
try:
reader = csv.reader(f)
print "file opened"
for row in reader:
print "Read domain: %s" %row
finally:
f.close()
ab+ mode opens a file for both appending and binary format, so therefore you cannot read in the contents if it is open to be appended to, you want r to read it:
f = open(domainFile, 'r')
For more information about all of the different file modes, please refer to this documentation.