I have a python function that creates a CSV file using a Postgresql copy statement. I need to add a new column to this spreadsheet called 'UAL' with an example value in the first row of say 30,000, but without editing the copy statement. This is the current code:
copy_sql = 'COPY (
SELECT
e.name AS "Employee Name",
e.title AS "Job Title"
e.gross AS "Total Pay",
e.total AS "Total Pay & Benefits",
e.year AS "Year",
e.notes AS "Notes",
j.name AS "Agency",
e.status AS "Status"
FROM employee_employee e
INNER JOIN jurisdiction_jurisdiction j on e.jurisdiction_id = j.id
WHERE
e.year = 2011 AND
j.id = 4479
ORDER BY "Agency" ASC, "Total Pay & Benefits" DESC
)'
with open(path, 'w') as csvfile:
self.cursor.copy_expert(copy_sql, csvfile)
What I am trying to do is use something like csv.writer to add content like this:
with open(path, 'w') as csvfile:
self.cursor.copy_expert(copy_sql, csvfile)
writer = csv.writer(csvfile)
writer.writerow('test123')
But this is adding the text to the last row. I am also unsure how to add a new header column. Any advice?
adding a header is easy: write the header before the call to copy_expert.
with open(path, 'w') as csvfile:
writer = csv.writer(csvfile)
writer.writerow(["my","super","header"])
self.cursor.copy_expert(copy_sql, csvfile)
But adding a column cannot be done without re-reading the file again and add your info on each row, so the above solution doesn't help much.
If the file isn't too big and fits in memory, you could write the sql output to a "fake" file:
import io
fakefile = io.StringIO()
self.cursor.copy_expert(copy_sql, fakefile)
now rewind the file and parse it as csv, add the extra column when writing it back
import csv
fakefile.seek(0)
with open(path, 'w', newline="") as csvfile:
writer = csv.writer(csvfile)
reader = csv.reader(fakefile) # works if copy_expert uses "," as separator, else change it
writer.writerow(["my","super","header","UAL"])
for row in reader:
writer.writerow(row+[30000])
or instead of the inner loop:
writer.writerows(row+[30000] for row in reader)
And if the file is too big, write it in a temp file, and proceed the same way (less performant)
Related
I am working on one program and trying to achieve following functionalities.
add new student
Remove student based on id
here is my code
from csv import writer
import csv
def add(file_name, list_of_elem):
# Open file in append mode
with open(file_name, 'a+', newline='') as write_obj:
# Create a writer object from csv module
csv_writer = writer(write_obj)
# Add contents of list as last row in the csv file
csv_writer.writerow(list_of_elem)
def remove():
id = input("Enter ID : ")
with open('students.csv', 'rb') as inp, open('students.csv', 'wb') as out:
writer = csv.writer(out)
for row in csv.reader(inp):
if row[0] != id:
writer.writerow(row)
# List of strings
row_contents = [11,'mayur','Java','Tokyo','Morning']
# Append a list as new line to an old csv file
add('students.csv', row_contents)
remove()
add function works properly but when i tried remove function it removes all existing entries.Could anyone please help me.
First I will show the code and below I will left some comments about the changes.
from csv import writer
import csv
def add(file_name, list_of_elem):
# Open file in append mode
with open(file_name, 'a+', newline = '') as write_obj:
# Create a writer object from csv module
csv_writer = writer(write_obj)
# Add contents of list as last row in the csv file
csv_writer.writerow(list_of_elem)
def remove():
idt = input("Enter ID : ")
with open('students.csv', 'r') as inp:
newrows = []
data = csv.reader(inp)
for row in data:
if row[0] != idt:
newrows.append(row)
with open('students.csv', 'w') as out:
csv_writer = writer(out)
for row in newrows:
csv_writer.writerow(row)
def display():
with open('students.csv','r') as f:
data = csv.reader(f)
for row in data:
print(row)
# List of strings
row_contents = [10,'mayur','Java','Tokyo','Morning']
add('students.csv', row_contents)
row_contents = [11,'mayur','Java','Tokyo','Morning']
add('students.csv', row_contents)
row_contents = [12,'mayur','Java','Tokyo','Morning']
add('students.csv', row_contents)
# Append a list as new line to an old csv file
display()
remove()
If your file is a CSV, you should use a text file, instead of a binary one.
I changed the name of the variable id to ìdt because id is built-in to return the identity of an object and it's not a good practice overwrite built-in functions.
To remove only rows with an specific idt you should read all the file, store into a var (list), remove what you want to delete and only after that save the result.
You should use a temporary file instead of opening and writing to the same file simultaneously. Checkout this answer: https://stackoverflow.com/a/17646958/14039323
I tried this but it just writes "lagerungskissen kleinkind,44" several times instead of transferring every row.
keyword = []
rank = []
rank = list(map(int, rank))
data = []
with open("keywords.csv", "r") as file:
for line in file:
data = line.strip().replace('"', '').split(",")
keyword = data[0]
rank = data[3]
import csv
with open("mynew.csv", "w", newline="") as f:
thewriter = csv.writer(f)
thewriter.writerow(["Keyword", "Rank"])
for row in keyword:
thewriter.writerow([keyword, rank])
It should look like this
This is writing the same line in your output CSV because the final block is
for row in keyword:
thewriter.writerow([keyword, rank])
Note that the keyword variable doesn't change in the loop, but the row does. You're writing that same [keyword, rank] line len(keyword) times.
I would use the csv package to do the reading and the writing for this. Something like
import csv
input_file = '../keywords.csv'
output_file = '../mynew.csv'
# open the files
fIn = open(input_file, 'r', newline='')
fOut = open(output_file, 'w')
csvIn = csv.reader(fIn, quotechar='"') # check the keyword args in the docs!
csvOut = csv.writer(fOut)
# write a header, then write each row one at a time
csvOut.writerow(['Keyword', 'Rank'])
for row in csvIn:
keyword = row[0]
rank = row[3]
csvOut.writerow([keyword, rank])
# and close the files
fOut.close()
fIn.close()
As as side note, you could write the above using the with context manager (e.g. with open(...) as file:). The answer here shows how to do it with multiple files (in this case fIn and fOut).
i'm new with python and try to modify csv file so i will able to delete specific rows with specific fields according to given list.
in my current code i get the rows which i want to delete but i can't delete it and save the changes on same file (replace).
import os, sys, glob
import time ,csv
# Open a file
path = 'C:\\Users\\tzahi.k\\Desktop\\netzer\\'
dirs = os.listdir( path )
fileslst = []
alertsCode = ("42001", "42003", "42006","51001" , "51002" ,"61001" ,"61002","71001",
"71002","71003","71004","71005","71006","72001","72002","72003","72004",
"82001","82002","82003","82004","82005","82006","82007","83001","84001")
# This would print the unnesscery codes
for file in dirs:
if "ALERTS" in file.upper() :
fileslst.append(file)
fileslst.sort()
with open(fileslst[-1], 'rb') as csvfile:
csvReader = csv.reader(csvfile)
for row in csvReader:
for alert in alertsCode:
if any(alert in row[2] for s in alertsCode) :
print row
any help?
Read all the rows into a list using a list comprehension and excluding the unwanted rows. Then rewrite the rows to the file in mode w (write mode) which overwrites or replaces the content of the file:
with open(fileslst[-1], 'rb') as csvfile:
csvReader = csv.reader(csvfile)
clean_rows = [row for row in csvReader if not any(alert in row[2] for alert in alertsCode)]
# csvfile.truncate()
with open(fileslst[-1], 'wb') as csvfile:
csv_writer = csv.writer(csvfile)
csv_writer.writerows(clean_rows)
In the below script, I cannot figure out how to either rename or "faux-rename" the headers.
import csv,time,string,os
print "rendering report. This will take a few minutes..."
raw_report = "\\\\network\\x\\RAWREPORT.csv"
today = time.strftime("%Y-%m-%d")
fields = ["As of Date", "EB", "Cycle", "Col", "APP Name", "Home Country" ]
with open(raw_report) as infile, open("c:\\upload\\test_" + today + ".csv", "wb") as outfile:
r = csv.DictReader(infile)
w = csv.DictWriter(outfile, fields, extrasaction="ignore")
w.writeheader()
for row in r:
w.writerow(row)
This script works fine, and it takes 6 columns out of a .csv with about 90 columns, but in order to write only those 6 columns in fields to my output file, I need to call them by name.
However, I need them to ultimately be named something different., (e.g. - "order_date", "phone_number"... instead of "As of Date", "EB").
I tried the approach of just skipping the first row and writing my own:
r = csv.DictReader(infile)
w = csv.DictWriter(outfile, fields, extrasaction="ignore")
next(r, None)
w.writerow(["order_date","phone_number",...])
but then python doesn't know which columns to copy into the new file because the names don't match.
How would I go about doing what I'm trying to do? Can I reference the columns I want to copy by number instead of by name, or is there a way to go back and change the value of the first row once everything is copied?
I was thinking about this incorrectly. I can define fields as the columns in the original file to pull from, but I don't need to include those necessarily in the output file as they are two separate files.
This code works:
fields = ["As of Date", "EB", "Cycle", "Col", "APP Name", "Home Country" ]
with open(raw_report) as infile, open("c:\\upload\\test_" + today + ".csv", "wb") as outfile:
r = csv.DictReader(infile)
w = csv.DictWriter(outfile, fields, extrasaction="ignore")
#w.writeheader() #remove the writeheader command
#write our custom header
wtr = csv.writer( outfile )
wtr.writerow(["order_date", "phone_number", etc....])
#then, write the rest of the file
for row in r:
w.writerow(row)
I have a CSV file which has certain columns which I need to extract. One of those columns is a text string from which I need to extract the first and last items. I have a print statement in a for loop which get exactly what I need but cannot figure out how to either get that data into a list or dict. Not sure which is the best to use.
Code so far:
f1 = open ("report.csv","r") # open input file for reading
users_dict = {}
with open('out.csv', 'wb') as f: # output csv file
writer = csv.writer(f)
with open('report.csv','r') as csvfile: # input csv file
reader = csv.DictReader(csvfile, delimiter=',')
for row in reader:
print row['User Name'],row['Address'].split(',')[0],row['Last Login DateTime'],row['Address'].split(',')[7]
users_dict.update(row)
#users_list.append(row['Address'].split(','))
#users_list.append(row['Last Login DateTime'])
#users_list.append(row[5].split(',')[7])
print users_dict
f1.close()
Input from file:
User Name,Display Name,Login Name,Role,Last Login DateTime,Address,Application,AAA,Exchange,Comment
SUPPORT,SUPPORT,SUPPORT,124,2015-05-29 14:32:26,"Test Company,Bond St,London,London,1111 111,GB,test#test.com,IS",,,LSE,
Output on print:
SUPPORT Test Company 2015-05-29 14:32:26 IS
Using this code, I've got the line you need:
import csv
f1 = open ("report.csv","r") # open input file for reading
users_dict = {}
with open('out.csv', 'wb') as f: # output csv file
writer = csv.writer(f)
with open('report.csv','r') as csvfile: # input csv file
reader = csv.DictReader(csvfile, delimiter=',')
for row in reader:
print row['User Name'],row['Address'].split(',')[0],row['Last Login DateTime'],row['Address'].split(',')[7]
users_dict.update(row)
#users_list.append(row['Address'].split(','))
#users_list.append(row['Last Login DateTime'])
#users_list.append(row[5].split(',')[7])
print users_dict
f1.close()
The only changes:
Including the import csv at the top.
Indenting the code just after the with open('out.csv' ......
Does this solve your problem?
With some testing I finally get the line to write the csv file:
for row in reader:
writer.writerow([row['User Name'],row['Address'].split(',')[0],row['Last Login DateTime'],row['Address'].split(',')[7]])