I've recently started working with Python and image processing. The HoughLinesP function from CV outputted this "[[[465 391 521 391]]]" and I need to export the values to an excel sheet afterwards. So, I need to access each of those elements individually.
How would I go about accessing those elements and storing them for later use?
Thank-you for your help!
Here is the corresponding documentation. The result of calling the function is an array, with the points you need. Here is how you could access them and store them to a CSV:
lines = cv2.HoughLinesP(...)
with open('tmp.csv', 'w') as f:
for l in lines:
f.write(','.join(str(x) for x in l) + "\n")
The file tmp.csv should contain data that can be opened in Excel.
The easier way, with csv:
import csv
# Assuming lines is already defined and in scope.
with open('tmp.csv', 'w') as f:
writer = csv.writer(f, delimiter=',')
writer.writerow(lines)
Related
i'm rather new to programming and am trying to reduce the time taken to write my data into a file, and i found that the writing part is the main issue.
The following is part of my code for a machine learning program:
filename="data.csv"
f=open(filename,"w")
headers="row,open\n"
f.write(headers)
for i in range (0,55970):
score=rf.predict(edit[i].reshape(1, -1))
score=str(score).replace('[','').replace(']','')
f.write(str(i) +","+ score +"\n")
f.close()
I understand that I should be writing the data only after i have gotten all of it, but i am not sure how to go about doing it - given that i only know f.write(). Do i make a function for my prediction and return score, then create a list to store all the scores and write it in? (if that is possible)
[Edit]
score=rf.predict(edit)
with open('data.csv', 'w',newline='') as f:
writer = csv.writer(f)
writer.writerow(['row', 'open'])
for i in range(55970):
writer.writerow([i,str(score[i])])
^ added based on new suggestion. Found that i should just do the predict and then write the rows which improved the time taken significantly!
Thank you for your help!!
The CSV module is a better tool for this. More specifically, writerows() is what you are looking for.
https://docs.python.org/3/library/csv.html#csv.csvwriter.writerows
Here is an example from the docs:
import csv
with open('some.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerows(someiterable)
import csv
with open('data.csv', 'w') as csv_file:
writer = csv.writer(csv_file)
writer.writerow(['row_id', 'open_flag'])
for i in range(55970):
score = str(rf.predict(edit[i].reshape(1, -1)))
score.replace('[', '').replace(']', '')
writer.writerow([i, score])
Right now I have a small script that writes and read data to a CSV file.
Brief snippet of the write function:
with open(filename,'w') as f1:
writer=csv.writer(f1, delimiter=';',lineterminator='\n',)
for a,b in my_function:
do_things_to_get_data
writer.writerow([tech_link, str(total), str(avg), str(unique_count)])
Then brief snippet of reading the file:
infile = open(filename,"r")
for line in infile:
row = line.split(";")
tech = row[0]
total = row[1]
average = row[2]
days_worked = row[3]
do_things_with_each_row_of_data
I'd like to just skip the CSV part all together and see if I can just hold all that data in a variable but I'm not sure what that looks like. Any help is appreciated.
Thank you.
...no point in me saving data to a csv file just to read it later in the script
Just keep it in a list of lists
data = []
for a,b in my_function:
do_things_to_get_data
data.append([tech_link, str(total), str(avg), str(unique_count)])
...
for tech, total, average, days_worked in data:
do_things_with_each_row_of_data
It might be worth saving each row as a namedtuple or a dictionary
Ok, I couldn't really find an answer to this anywhere else, so I figured I'd ask.
I'm working with some .csv files that have about 74 million lines right now and I'm trying to add columns into one file from another file.
ex.
Week,Sales Depot,Sales Channel,Route,Client,Product,Units Sold,Sales,Units Returned,Returns,Adjusted Demand
3,1110,7,3301,15766,1212,3,25.14,0,0,3
3,1110,7,3301,15766,1216,4,33.52,0,0,4
combined with
Units_cat
0
1
so that
Week,Sales Depot,Sales Channel,Route,Client,Product,Units Sold,Units_cat,Sales,Units Returned,Returns,Adjusted Demand
3,1110,7,3301,15766,1212,3,0,25.14,0,0,3
3,1110,7,3301,15766,1216,4,1,33.52,0,0,4
I've been using pandas to read in and output the .csv files, but the issue I'm coming to is the program keeps crashing because creating the DataFrame overloads my memory. I've tried applying the csv library from Python but I'm not sure how merge the files the way I want (not just append).
Anyone know a more memory efficient method of combining these files?
Something like this might work for you:
Using csv.DictReader()
import csv
from itertools import izip
with open('file1.csv') as file1:
with open('file2.csv') as file2:
with open('result.csv', 'w') as result:
file1 = csv.DictReader(file1)
file2 = csv.DictReader(file2)
# Get the field order correct here:
fieldnames = file1.fieldnames
index = fieldnames.index('Units Sold')+1
fieldnames = fieldnames[:index] + file2.fieldnames + fieldnames[index:]
result = csv.DictWriter(result, fieldnames)
def dict_merge(a,b):
a.update(b)
return a
result.writeheader()
result.writerows(dict_merge(a,b) for a,b in izip(file1, file2))
Using csv.reader()
import csv
from itertools import izip
with open('file1.csv') as file1:
with open('file2.csv') as file2:
with open('result.csv', 'w') as result:
file1 = csv.reader(file1)
file2 = csv.reader(file2)
result = csv.writer(result)
result.writerows(a[:7] + b + a[7:] for a,b in izip(file1, file2))
Notes:
This is for Python2. You can use the normal zip() function in Python3. If the two files are not of equivalent lengths, consider itertools.izip_longest().
The memory efficiency comes from passing a generator expression to .writerows() instead of a list. This way, only the current line is under consideration at any moment in time, not the entire file. If a generator expression isn't appropriate, you'll get the same benefit from a for loop: for a,b in izip(...): result.writerow(...)
The dict_merge function is not required starting from Python3.5. In sufficiently new Pythons, try result.writerows({**a,**b} for a,b in zip(file1, file2)) (See this explanation).
I am writing the output of a function to a file, how ever if the number that is written has an e+01 at the end it writes one of the numbers on to the next line. for example
0.12605695 1.64761066 1.55001798 0.01785089 15.80005801
0.12231149 1.65899943 1.56369755 0.01511221 10.45653689
0.1238494 1.67704048 1.59633667 0.00687468 0.96146016
1.10437450e-01 1.62918081e+00 1.56024230e+00 1.82792879e-03
-2.05519064e+00
1.04326863e-01 1.63545256e+00 1.58687599e+00 -4.50753949e-03
-6.40408013e+00
There should be five numbers per line, here is the part of my code where i write the numbers to a file,
kop=open('new.txt', 'w')
results=PCA(kk)
res=results.mu
print results.mu
kop.write(str(res)+'\n')
kop.close()
how do I get them to write to the one line and not go onto the next?
I need them all on the correct line as when I used numpy.genfromtxt in a different piece of code it requires them to all be in the correct columns.
Assuming results.mu is something that can be iterated like a normal list, this snippet should do what you need by utilizing the csv module:
import csv
with open('new.txt', 'w') as f:
writer = csv.writer(f, delimiter="\t")
writer.writerows(results.mu)
Depending on the type of result.mu you may try to use rstrip():
results=PCA(kk)
res=results.mu
print results.mu
for i,y in enumerate(result.mu):
result[i] = y.rstrip()
kop.write(str(res)+'\n')
kop.close()
This was the answer, I just didnt realize it was a numpy array.
kop=open('new.txt', 'w')
results=PCA(kk)
res=results.mu
res=np.ndarray.tolist(res)
print results.mu
kop.write(str(res)+'\n')
kop.close()
I'm trying to create a csv file that contains the contents of a list of strings in Python, using the script below. However when I check my output file, it turns out that every character is delimited by a comma. How can I instruct csv.writer to delimit every individual string within the list rather than every character?
import csv
RESULTS = ['apple','cherry','orange','pineapple','strawberry']
result_file = open("output.csv",'wb')
wr = csv.writer(result_file, dialect='excel')
for item in RESULTS:
wr.writerow(item)
I checked PEP 305 and couldn't find anything specific.
The csv.writer writerow method takes an iterable as an argument. Your result set has to be a list (rows) of lists (columns).
csvwriter.writerow(row)
Write the row parameter to the writer’s file object, formatted according to the current dialect.
Do either:
import csv
RESULTS = [
['apple','cherry','orange','pineapple','strawberry']
]
with open('output.csv','w') as result_file:
wr = csv.writer(result_file, dialect='excel')
wr.writerows(RESULTS)
or:
import csv
RESULT = ['apple','cherry','orange','pineapple','strawberry']
with open('output.csv','w') as result_file:
wr = csv.writer(result_file, dialect='excel')
wr.writerow(RESULT)
Very simple to fix, you just need to turn the parameter to writerow into a list.
for item in RESULTS:
wr.writerow([item])
I know I'm a little late, but something I found that works (and doesn't require using csv) is to write a for loop that writes to your file for every element in your list.
# Define Data
RESULTS = ['apple','cherry','orange','pineapple','strawberry']
# Open File
resultFyle = open("output.csv",'w')
# Write data to file
for r in RESULTS:
resultFyle.write(r + "\n")
resultFyle.close()
I don't know if this solution is any better than the ones already offered, but it more closely reflects your original logic so I thought I'd share.
A sample - write multiple rows with boolean column (using example above by GaretJax and Eran?).
import csv
RESULT = [['IsBerry','FruitName'],
[False,'apple'],
[True, 'cherry'],
[False,'orange'],
[False,'pineapple'],
[True, 'strawberry']]
with open("../datasets/dashdb.csv", 'wb') as resultFile:
wr = csv.writer(resultFile, dialect='excel')
wr.writerows(RESULT)
Result:
df_data_4 = pd.read_csv('../datasets/dashdb.csv')
df_data_4.head()
Output:
IsBerry FruitName
0 False apple
1 True cherry
2 False orange
3 False pineapple
4 True strawberry