Automatic recognize new file and get the path - python

i have written a python script that reads continuously a csv file outputted and updated by a sensor and live plots some data with matplotlib.
Everytime i start recording data from the sensor it creates a new file like:
data-2020_02_27_14_42_29.csv
So everytime i have to update my script to point the correct csv file.
How i can automate this?
Is there a way to recognize the creation of a new file in a specific directory and take the name of it?
with open('/home/matteo/Documents/PlatformIO/Projects/200213-123258-INS/data/data-2020_02_27_14_42_29.csv', 'r') as csv_file:
csv_reader=csv.DictReader(csv_file, delimiter=',')
Thanks

You can get a list of what files there were at the start at end of your script, then get the new item by comparing the two lists.
def read(file):
folder='/home/matteo/Documents/PlatformIO/Projects/200213-123258-INS/data'
start_list=[file for file in os.listdir(folder)]
with open(os.path.join(folder,file), 'r') as csv_file:
csv_reader=csv.DictReader(csv_file, delimiter=',')
#any other code that happens after reading the csv....
end_list=[file for file in os.listdir(folder)]
new_file=list(set(start_list)-set(end_list))
#checks that a new file was found (if nothing new list is empty)
#if it is empty, re-run function on existing file
if new_file!=[]
new_file=new_file[0]
else:
new_file=file
#recursion to call the function on the new file we found
read(new_file)

Related

I have got two (.py) files. When the first program comes to end, it will close itself then open and run the second program file. How can ı do it?

Here is my first file's codes:
from csv import writer
film=[""] #The list contains informations about the film
#İnputs:
filmname=input("Film name: ")
category=input("category: ")
score=input("point: ")
score=str(score)
complement=input("complement: ")
valuablenames=input("İmportant actors: ")
#Appending Inputs
film.append(filmname)
film.append(category)
film.append(score)
film.append(complement)
film.append(valuablenames)
#The Function which appends the list into csv file:
def liste_append(filename,inf_list):
with open("FilmSet.csv", 'a+', newline='') as write_obj: # Open file in append mode
csv_writer = writer(write_obj) # Create a writer object from csv module
csv_writer.writerow(film) # Add contents of list as last row in the csv file
#Calling Function
liste_append("FilmSet.csv",film)
#Jumping to File which converts csv to xlsx
#Jumping to File which converts csv to xlsx
The part i need your help to close this file and jump to the other one (which is a converter program and named as "converter.py").So when i run this program,it will update the csv file and jump to "converter.py" for updating excel file from csv converted.( deletes the old xlsx and creates new xlsx which is converted from csv )
Best solution
My recommendation would be to turn one of the python files into a package and simply import the required functions.
So in this case put your csv conversion from conversion.py into a function like csv_to_xlsx() and then from conversion import csv_to_xlsx() and then run it at the end of your main file.
Other solutions
You could use the subprocess module with subprocess.Popen() or os.system(). These will allow you to execute python code from another file. Also the worst other solution is exec() this allows you to pass python code as a string and execute it. so you could do
exec(open('file.py', 'r').read())

Where can I find my .csv file after running my python code on Ubuntu?

I've just managed to run my python code on ubuntu, all seems to be going well. My python script writes out a .csv file every hour and I can't seem to find the .csv file.
Having the .csv file is important as I need to do research on the data. I am also using Filezilla, I would have thought the .csv would have run into there.
import csv
import time
collectionTime= datetime.now().strftime('%Y-%m-%d %H:%M:%S')
mylist= [d['Spaces'] for d in data]
mylist.append(collectionTime)
print(mylist)
with open("CarparkData.csv","a",newline="") as f:
writer = csv.writer(f)
writer.writerow(mylist)
In short, your code is outputting to wherever the file you're opening is in this line:
with open("CarparkData.csv","a",newline="") as f:
You can change this filename to the location of wherever you'd like the file to be read/written from/to. For example, data/CarparkData.csv if you had a folder named data/ within your application dedicated to holding data files.
As written in your code, writer.writerow will write the lines to both python's in-memory object of the file (instantiated with open("filename.csv"...), and the file itself (in this case, CarparkData.csv).
The way your code is structured, it won't be creating a new .csv every hour because it is using a static filename. If a file with this name did not exist at time of opening, it will create one, and if it did, it will continue to append new lines to the existing file.

delete all .csv file content in python

so, my program will replace the old data with new data which is the program will put in same .csv file. and after i run the program it is not replaced. below is my code.
TEXTFILE = open("data.csv", "w")
for i in book_list:
TEXTFILE.write("{},{},{},{}".format(i[0],i[1],i[2],i[3]))
TEXTFILE.close()
book_list is the list that save new data to stored
the result i got:
k,k,45,c
a,a,65,r
d,s,65,r
as,as,65,r
df,df6,65,r
as,as,6,r
as,as,46,r
as,as,45,r
as,as,56,rk,k,45,r
a,a,65,r
d,s,65,r
as,as,65,r
df,df6,65,r
as,as,6,r
as,as,46,r
as,as,45,r
as,as,56,r
it stored to csv with combining old and new content.
well the origininal file is looks like this:
k,k,45,r
a,a,65,r
d,s,65,r
as,as,65,r
df,df6,65,r
as,as,6,r
as,as,46,r
as,as,45,r
as,as,56,r
idk how to explain. but I expect that the result will change one line with new data in the fourth row. for example, the previous line is k,k,45,r (line 1). and the program will change it become k,k,45,c that way
hope you all can help me :)
Try using .truncate() , if called after opening file it destroys it's content.
TEXTFILE = open("data.csv", "w")
TEXTFILE.truncate()
for i in book_list:
TEXTFILE.write("{},{},{},{}".format(i[0],i[1],i[2],i[3]))
TEXTFILE.close()
I hope I understood you correctly.

Python program for line search and replacement in a CSV file

I am building a small tool in Python, the function of the tool is the following:
Open a master data file (cvs format)
open a log file (cvs format)
Request the user to select the pointer for the field that will need to be compared in both files.
Start comparing record by record
when the field is not found in the record proceed to look forward in the log file until the field can be found, in the meantime keep in memory the pointer for when the comparison will be continued.
Once the field is found proceed to cut the whole record off that line and place it in the right position
Here is an example :
Data file
"1","1234","abc"
"2","5678","def"
"3","9012","ghi"
log file
"1","1234","abc"
"3","9012","ghi"
"2","5678","def"
final log file :
"1","1234","abc"
"2","5678","def"
"3","9012","ghi"
I was looking in the cvs lib in python and the sqlite3 lib but there is nothing that seems to really do a swap in file so i was thinking that maybe i should just create a new file with all the records in order.
What could be done in this regard ? Is there a library or a command that can move records in an existing file ?
I would prefer to modify the existing file instead of creating a new one but if it's not possible i would just move to create a new one.
In addition to that the code i was planning to use to verify the files was this :
import csv
reader1 = csv.reader(open('data.csv', 'rb'), delimiter=',', quotechar='"'))
row1 = reader1.next()
reader2 = csv.reader(open('log.csv', 'rb'), delimiter=',', quotechar='"'))
row2 = reader2.next()
if (row1[0] == row2[0]) and (row1[2:] == row2[2:]):
#here it move to the next record
else:
#here it would run a function that replace the field
Please note that this piece of code was found at this page :
Python: Comparing specific columns in two csv files
(i don't want to take away the glory from another coder).
I just like it for it's simplicity.
Thanks to all for the attention.
Regards
Danilo

python write method append

I'm stuck in a very basic problem of I/O in python. I'd like to insert some line in existing file (called ofe, output file), extracted from an source file (called ife, input file) according to arguments passed by user as stored in an list called lineRange (which has an index idx and values lineNumber).
This is the result:
for ifeidx,ifeline in enumerate(ife,1): #for each line of the input file...
with open(outFile,'r+') as ofe:
for idx,lineNumber in enumerate(lineRange,1): #... check if it's present in desired list of lines...
if (ifeidx == lineNumber): #...if found...
ofeidx = 0
for ofeidx, ofeline in enumerate(ofe,1):
if (ofeidx == idx): #...just scroll the the output file and find which is the exact position in desired list...
ofe.write(ifeline) #...put the desired line in correct order. !!! This is always appending at the end of out file!!!!
break
Problem is, the write() method is always pointing to the end of file, appending the lines instead of inserting them when scrolling the output file.
I really don't understand what's happening since the file is open in read+write (r+) mode, neither append (a) nor read+append (r+a) mode, .
I'm also aware that code will (should) overwrite the output file lines. Additional information are the OS WIndow7, Python version 2.7 and development tool is Eclipse with PyDev 3.7.1.xx
Any suggestion on what I'm doing wrong?
You can start by reading the whole file with readlines(), which will return a list. After that you just need to do list.insert(index, value) and write it again back to the file.
with open(outFile, "r") as f:
data = f.readlines()
data.insert(index, value)
with open(outFile, "w+") as f:
f.write(data)
Of course you should change this approach if you are dealing with a huge file.
By the way, if you are not using the with statement you should close the file in the end.

Categories