I am attempting to store a csv file on an ftp server using python's ftplib module.
Right now, I have about 30 lines of code which generates probabilities of weather values in a 2-d array. I then write this 2-d array to a csv file.
When I write the csv file onto my local drive, the file displays as expected within excel. However, when I view the file after I uploaded it to an ftp server, I see that a new line character has been added after every row.
I've done some minor testing to see what the problem may be, and I have been able to upload the csv file with coreftp. The csv file displays correctly after I do that. So I am pretty sure the file is fine, its something that is happening when python uploads it onto an ftp server.
I was originally creating a text file with a .csv extension file then reopening it as a binary file and uploading it. I thought that may be the issue so I tried using the csv module, but same issue.
Here is my code at the moment...
TEMPSHEADER = [i-50 for i in range(181)]#upper bounds exclusive
WINDSHEADER = [i for i in range(101)]#upper bounds exclusive
HEADER = TEMPSHEADER + WINDSHEADER
for site in ensmosdic:
ensmos = ensmosdic.get(site)
with open(utcnow.strftime("%Y-%m-%d") + "-" +site+"-prob.csv","w",newline='') as csvfile:
writer = csv.writer(csvfile, delimiter=",")
writer.writerow(["CODE ","F","ForecastDate","HOUR"]+HEADER)
siteTable =[[0 for x in range(286)] for y in range(24,169)]#upper bounds exclusive
###########
#other code here, but not important with regards to post
###########
for i in siteTable:
writer.writerow(i)
csvfile.close()#not sure if you have to close csv file, not in csv module docs
f = open(utcnow.strftime("%Y-%m-%d") + "-" +site+"-prob.csv","rb")
ftpInno.storbinary("STOR " + utcnow.strftime("%Y-%m-%d-") + site +"-prob.csv",f)
f.close()
ftpInno.close()
Thanks in advance
After an hour or so of trouble shooting, the answer is fairly simple, although I am not entirely sure why it works.
what i did was create a text file instead of a csv file which I was doing in my original question
with open(FILELOCATION + utcnow.strftime("%Y-%m-%d") + "-" +site+"-prob.txt","w") as f:
#write to the file below
f.close()
#open file again as a txt file
f = open(FILELOCATION + utcnow.strftime("%Y-%m-%d") + "-" +site+"-prob.txt","rb")
ftp.storlines("STOR " + utcnow.strftime("%Y-%m-%d-") + site +"-prob.csv",f)
f.close()
reading the file as a binary file and then storing it with the storlines method removed the extra lines I was seeing within the file after I uploaded it to an ftp server.
This might shed some light on your issue. I had a project where I was using windows command line and also windows powershell to transfer .csv files with the ftp get and mget commands. And like you said I was getting a extra between each row. It seems like switching to binary transfer mode fixed my issue. For example once you are in the ftp dialog just type "binary" and hit enter and it switches the mode.
Related
I've just managed to run my python code on ubuntu, all seems to be going well. My python script writes out a .csv file every hour and I can't seem to find the .csv file.
Having the .csv file is important as I need to do research on the data. I am also using Filezilla, I would have thought the .csv would have run into there.
import csv
import time
collectionTime= datetime.now().strftime('%Y-%m-%d %H:%M:%S')
mylist= [d['Spaces'] for d in data]
mylist.append(collectionTime)
print(mylist)
with open("CarparkData.csv","a",newline="") as f:
writer = csv.writer(f)
writer.writerow(mylist)
In short, your code is outputting to wherever the file you're opening is in this line:
with open("CarparkData.csv","a",newline="") as f:
You can change this filename to the location of wherever you'd like the file to be read/written from/to. For example, data/CarparkData.csv if you had a folder named data/ within your application dedicated to holding data files.
As written in your code, writer.writerow will write the lines to both python's in-memory object of the file (instantiated with open("filename.csv"...), and the file itself (in this case, CarparkData.csv).
The way your code is structured, it won't be creating a new .csv every hour because it is using a static filename. If a file with this name did not exist at time of opening, it will create one, and if it did, it will continue to append new lines to the existing file.
I have a bunch of CSV files with common columns but different rows. I want to merge them all into one CSV file. Here is the script I wrote to do that
import glob, os
os.chdir("./data")
fout = open("merged.csv", "a")
lout = open("merger_log", "a")
for fname in glob.glob("*.csv*"):
with open(fname) as f:
# exclude header for all but the first csv file.
if os.stat("merged.csv").st_size > 0:
next(f)
fout.writelines(f)
log = "Appended %s \n" % fname
print(log)
lout.write(log)
fout.close()
lout.close()
When I run this script, it successfully appends the first few files but gets stuck on one file every time. And by stuck it seems to be adding bits from said file to the output file without moving on to the next file. There's nothing special about the file it stops on, it's about the same size as the rest of them and is not malformed. In fact, I removed that file from the data set and the program hung on a different file. Not sure what is wrong with this script.
If anyone has a better way to merge a bunch of CSV files, I'm all ears.
Thanks!
EDIT: I should mention this script works perfectly fine with just two files.
I am working on a project that requires me to read a file with a .dif extension. Dif stands for data information exchange. The file opens nicely in Open Office Calc. Then you can easily save as a csv file, however when I open in Python all I get are random characters that don't make sense. Here is the last code that I tried just to see if I could read.
txt = open('C:\myfile.dif', 'rb').read()
print txt
I would even be open to programatically converting the file to csv first. before opening if someone knows how to do that. As always, any help is much appreciated. Below is a partial screenshot of what I get when I run the code.
Hadn't heard of this file format. Went and got a sample here.
I tested your method and it works fine:
>>> content = open(r"E:\sample.dif", 'rb').read()
>>> print (content)
b'TABLE\r\n0,1\r\n"EXCEL"\r\nVECTORS\r\n0,8\r\n""\r\nTUPLES\r\n0,3\r\n""\r\nDATA\r\n0,0\r\n""\r\n-1,0\r\nBOT\r\n1,0\r\n"Welcome to File Extension FYI Center!"\r\n1,0\r\n""\r\n1,0\r\n""\r\n-1,0\r\nBOT\r\n1,0\r\n""\r\n1,0\r\n""\r\n1,0\r\n""\r\n-1,0\r\nBOT\r\n1,0\r\n"ID"\r\n1,0\r\n"Type"\r\n1,0\r\n"Description"\r\n-1,0\r\nBOT\r\n0,1\r\nV\r\n1,0\r\n"ASP"\r\n1,0\r\n"Active Server Pages"\r\n-1,0\r\nBOT\r\n0,2\r\nV\r\n1,0\r\n"JSP"\r\n1,0\r\n"JavaServer Pages"\r\n-1,0\r\nBOT\r\n0,3\r\nV\r\n1,0\r\n"PNG"\r\n1,0\r\n"Portable Network Graphics"\r\n-1,0\r\nBOT\r\n0,4\r\nV\r\n1,0\r\n"GIF"\r\n1,0\r\n"Graphics Interchange Format"\r\n-1,0\r\nBOT\r\n0,5\r\nV\r\n1,0\r\n"WMV"\r\n1,0\r\n"Windows Media Video"\r\n-1,0\r\nEOD\r\n'
>>>
The question is what is in the file and how do you want to handle it. Personally I liked:
with open(r"E:\sample.dif", 'rb') as f:
for line in f:
print (line)
In the first code block, that long line that has a b'' (for bytes!) in front of it can be iterated on \r\n:
b'TABLE\r\n'
b'0,1\r\n'
b'"EXCEL"\r\n'
b'VECTORS\r\n'
b'0,8\r\n'
b'""\r\n'
b'TUPLES\r\n'
b'0,3\r\n'
b'""\r\n'
b'DATA\r\n'
b'0,0\r\n'
.
.
.
b'"Windows Media Video"\r\n'
b'-1,0\r\n'
b'EOD\r\n'
This is some example data from the csv I'm reading/writing to:
ALTIMA_SD,2014,12,Promoter,Promoter,3/5/2015
ALTIMA_SD,2015,3,Promoter,Promoter,3/5/2015
PATHFINDER,2014,12,Promoter,Promoter,3/5/2015
I'm looking to add a column between the first two, so that the end product looks like this:
ALTIMA_SD,Comparo,2014,12,Promoter,Promoter,3/5/2015
SENTRA,Comparo,2015,3,Promoter,Promoter,3/5/2015
PATHFINDER,Pathfinder,2014,12,Promoter,Promoter,3/5/2015
This is my code:
with open('pathfinder query2.csv','r+') as myfile:
for i in myfile:
i=i.rstrip('\n')
i=i.split(',')
if i[0]=='PATHFINDER' or i[0]=='PATHFINDER_H':
myfile.write(str(i[0])+','+'Pathfinder'+','+str(i[1])+','+str(i[2])+','+str(i[3])+','+str(i[4])+','+str(i[5])+'\n')
else:
myfile.write(str(i[0])+','+'Comparo'+','+str(i[1])+','+str(i[2])+','+str(i[3])+','+str(i[4])+','+str(i[5])+'\n')
I'm getting a IOError: [Errno 0] Error in reference to line 8. I've seen in other posts that the myfile.seek() function should be used, but I'm not sure how to apply it. Any help would be greatly appreciated.
Opening a file in Python is similar to fopen in C, see here for an overview of the modes. The problem is you are opening the file with r+ (read + update), while you need to open your file with w+ (write + update). Hence you must change your code to:
with open('pathfinder query2.csv','w+') as myfile:
Also as commented (Thx. Jon.), it's not wise to read and write to the same file. I would recommend you make a copy of the file or read the entire file to a buffer.
I am writing a script to log into a switch, write the config to a file, and then rename the file. I have the parts working separately. The issue is that I cannot figure out how to get all parts with in the same function so that I can use the function on a list of devices. I get a file not open for reading in the for 'line in f' statement. when as far as i can see the file is still open.
I have tried writing a function to rename the file that works on its own, but not when in this script with the other parts.
I have another script that i wrote that has the rename portion outside of the function which works, but will not work to rename the file if multiple hosts are called with the Exscript 'quickstart' module.
Thanks for any help,
from Exscript.util.start import quickstart
import os
import datetime
import time
time = datetime.datetime.now().strftime("%d-%m-%Y")
tm = 'c:/test/tmp.txt'
def do_something(job, host, conn):
f = open(tm, 'w+') #opens File with read and write permissions
conn.execute('term len 0')
conn.execute('sh run')
f.write(conn.response)
conn.execute('quit')
#this is the part where the error comes
for line in f:
if "hostname" in line:
host = line.strip()
test = 'c:/test/' + host[9:] + 'on' + time + '.txt'
os.rename(tm, test)
quickstart('ssh://x.x.x.x', do_something)
According to the manual, mode w+ truncates (removes all the content from) the file. If you want to open the file for both reading and writing without destroying its contents, use mode r+ or a+.
::edit:: Note, I'm not sure how this works on Windows.
You have to test the file pointer at the beginning of the file using f.seek(0). Or first write to the file then close it then reopen it for reading. But you dont need a file at all - you can as well work on a local variable.