We Keep Coding

Storing different lists in csv file- not overwriting last addition

I am new to Python and somehow can't quite get this simple task to run.
I have generated a randomization of some images I would like to later present to each participant in my experiment. The randomization assigns each participant a particular order for a list of images to be presented. The randomization looks as follows:
all_images_face = ["01_1.jpg","02_1.jpg", "03_1.jpg", "04_1.jpg", "05_1.jpg",
"06_1.jpg", "07_1.jpg", "08_1.jpg", "09_1.jpg","10_1.jpg"]
all_images_scene = ["01_2.jpg", "02_2.jpg", "03_2.jpg", "04_2.jpg", "05_2.jpg",
"06_2.jpg", "07_2.jpg", "08_2.jpg", "09_1.jpg", "10_2.jpg]
ind_scene = range(len(all_images_scene))
len_scene = len(all_images_scene)
for p in range (0,participants): #for each participant
rand_ind_face=random.sample(range(len(all_images_face)),len(all_images_face)/2)
TLS = []
list_scene = []
while True:
for i in range(0,len_scene):
print "loop round: ", i
if ind_scene[i] not in rand_ind_face[:]:
el = ind_scene[i]
TLS.append(el)
list_scene.append(el)
print "list_scene: ", list_scene
print "TLS: ", TLS
if len(TLS) == 0:
EF = "Error"
list_all = TLS + rand_ind_face # scenes + faces
final_scene = [] # list to retrieve elements from index
final_face = []
for i in list_all[:len(all_images_face)/2]: # first half scenes
all_images_scene[i]
final_scene.append(all_images_scene[i])
for i in list_all[len(all_images_face)/2:]: # second half faces
all_images_face[i]
final_face.append(all_images_face[i])
str_all = final_scene + final_face
print str_all
# needed data
random.shuffle(str_all) #shuffle order of scene/face of stimuli
print str_all
# write the str_all into csv
fp = open('list_stim.csv','w')
wr = csv.writer(fp,delimiter = ',')
wr.writerow(str_all)
if p == participants:
fp.close()
I end up with, for example, a list that looks like this for p == 1:
str_all = ['01_2.jpg', '06_2.jpg', '08_1.jpg', '04_2.jpg', '10_1.jpg',
'07_2.jpg', '02_1.jpg', '05_2.jpg', '09_1.jpg', '03_1.jpg']
For each participant this random list of string names is different. I would like to store each new str_all list into a new row of the same cvs file, where each element corresponds to a column (meaning that each added row is for each new participant). I created manually with Excel a csv file called list_stim.csv.
This last code allows me to add my newly created str_all list, but when I run the loop again (for p == 2) it does not add the new list but overwrites the old one.
# write the str_all into csv
fp = open('list_stim.csv','w')
wr = csv.writer(fp,delimiter = ',')
wr.writerow(str_all)

In the code pasted above, I cannot see any loop which iterates through the length str_all, as it can be seen you are calling str_all[i].
The solution which I think should work for you is:
fp = open('list_stim.csv','w')
wr = csv.writer(fp,delimiter = ',')
wr.writerow(str_all)
fp.close()
This will write the str_all into the CSV. Each item of the list str_all will be a column in the CSV file.
From the question, it seems you want to write multiple such lists.
So you will have to define a list which contains all this list. I am showing an example here:
str_all1=['01_2.jpg', '06_2.jpg', '08_1.jpg', '04_2.jpg','10_1.jpg']
str_all2=[ '10_1.jpg','01_2.jpg', '06_2.jpg', '08_1.jpg','03_1.jpg']
str_all3=['06_2.jpg', '08_1.jpg','01_2.jpg', '04_2.jpg', '10_1.jpg']
big_str = [str_all1,str_all2,str_all3]
fp = open('list_stim.csv','w')
wr = csv.writer(fp,delimiter = ',')
for i in range(len(big_str)):
wr.writerow(big_str[i])
fp.close()

This is easily stored in a json-string-file. Here is an example of how:
import json
import os
import random
def func_returns_imagelist():
# Do some magic here
l = ['01_2.jpg', '06_2.jpg', '08_1.jpg', '04_2.jpg', '10_1.jpg',
'07_2.jpg', '02_1.jpg', '05_2.jpg', '09_1.jpg', '03_1.jpg']
random.shuffle(l)
return l
def open_database(s):
# Load database if exist, else set d to empty dict
if os.path.isfile(s):
with open(s) as f:
return json.load(f)
else:
return {}
def save_database(s,d):
# Save to file
with open(s, 'w') as f:
json.dump(d,f,indent=2)
def main():
pathtofile = 'mydb.json'
d = open_database(pathtofile)
d['participant{}'.format(len(d)+1)] = func_returns_imagelist()
save_database(pathtofile,d)
main()
This function will try to open a file called 'mydb.json' and return the data as a dictionary, if it doesn't exist it creates it.
It will add a participant with the function (func_returns_imagelist)
It will save back the file to json.
Try running this multiple times and you can see that the file ('mydb.json') is growing each time.

Matching Outlook Calendar Dates to CSV Dates and Appending Match to List

Disclaimer: I'm relatively new to Python. I am attempting to write a script that will go through a CSV, check if certain columns match Outlook calendar items (by subject, organizer, and date match), and then have the script note that there was a successful match in a new column (I stole heavily from this question). Below is my whole script.
import win32com.client, datetime, re, os, csv, shutil
# set cwd, create copy of original CSV, and access outlook API
os.chdir('C:\\')
shutil.copy('masterCheck.csv', 'inspectCheck.csv')
outlook = win32com.client.Dispatch("Outlook.Application").GetNamespace("MAPI")
inspectors = {'ASMITH': 'Smith, Aaron', 'JDOE': 'Doe, Jane', 'KZEEBOP': 'Zeebop, Kim'}
#access csv and put it into a list
with open('inspectCheck.csv', 'r', newline = '', encoding='utf-8') as csvAppointments:
reader = csv.reader(csvAppointments)
masterList = list(reader)
del masterList[-1] # delete blank nested list
del masterList[1] #delete header
for i in masterList: # switch out names so they can match Outlook descriptors later
for key, value in inspectors.items():
if i[3] in key:
i[3] = value
# create another list for appending later in the script
finalList = []
finalList += masterList
# access the inspectors' calendars
x = 0
try:
for inspector in inspectors.values():
recipient = outlook.createRecipient(inspector)
resolved = recipient.Resolve()
sharedCalendar = outlook.GetSharedDefaultFolder(recipient, 9)
codeAppointments = sharedCalendar.Items
#restrict to items in the next year
begin = datetime.date.today()
end = begin + datetime.timedelta(days = 365);
restriction = "[Start] >= '" + begin.strftime("%m/%d/%Y") + "' AND [End] <= '" +end.strftime("%m/%d/%Y") + "'"
restrictedItems = codeAppointments.Restrict(restriction)
# loop through inspectors' appointments and match
for appointmentItem in restrictedItems:
for i in masterList:
addressSearch = i[1]
if re.search(r'%s' % addressSearch, appointmentItem.Subject, re.IGNORECASE)\
and i[3] in appointmentItem.Organizer\
and i[4] in appointmentItem.Start:
finalList[x].append(appointmentItem.Subject)
x += 1
except IndexError:
pass
# update spreadsheet
with open('inspectCheckFinal.csv', 'w', newline = '') as csvAppointments:
appointmentsWriter = csv.writer(csvAppointments)
appointmentsWriter.writerows(finalList)
I have had success matching columns from my CSV to Outlook items. For example, I can get this to match.
if re.search(r'%s' % addressSearch, appointmentItem.Subject, re.IGNORECASE)
However, as soon as I try to match it to my date column (i[4]), it throws the error: TypeError: argument of type 'pywintypes.datetime' is not iterable. The dates in my CSV look like 2/11/2018 but the dates (when printed) in Outlook look like 2017-11-16 11:00:00+00:00. I'm at a loss on how to match these two.
Moreover, I am having trouble marking the CSV with successful matches. While the script will append a value to the end of each nested list (and then write it to a CSV), it will not append the value to the matched row of the CSV. For example, my output looks like:
Inspector |Address |Date |Success?(print Address)
ASMITH |212 Clark St|11/21/18 |Yes. 33 Blart Ave
ASMITH |33 Blart Ave|11/20/18 |Yes. 212 Clark St
My guess is that my script finds a match in Outlook, and then appends that value to the end of a nested list. What I would like it to do is to match it in the row/nested list where it was actually matched. Apologies for the long post and thank you to those who read it.

Nevermind, I solved the TypeError by converting
and i[4] in appointmentItem.Start:
# to
and i[4] in str(appointmentItem.Start):
Moreover, I reformatted my CSV beforehand so it will now match Outlook's format. As for my matches being appended in the wrong rows, I think I will solve that by appending matches to a separate CSV/dataframe, and then joining that dataframe to the original CSV/dataframe.

python newbie - where is my if/else wrong?

Complete beginner so I'm sorry if this is obvious!
I have a file which is name | +/- or IG_name | 0 in a long list like so -
S1 +
IG_1 0
S2 -
IG_S3 0
S3 +
S4 -
dnaA +
IG_dnaA 0
Everything which starts with IG_ has a corresponding name. I want to add the + or - to the IG_name. e.g. IG_S3 is + like S3 is.
The information is gene names and strand information, IG = intergenic region. Basically I want to know which strand the intergenic region is on.
What I think I want:
open file
for every line, if the line starts with IG_*
find the line with *
print("IG_" and the line it found)
else
print line
What I have:
with open(sys.argv[2]) as geneInfo:
with open(sys.argv[1]) as origin:
for line in origin:
if line.startswith("IG_"):
name = line.split("_")[1]
nname = name[:-3]
for newline in geneInfo:
if re.match(nname, newline):
print("IG_"+newline)
else:
print(line)
where origin is the mixed list and geneInfo has only the names not IG_names.
With this code I end up with a list containing only the else statements.
S1 +
S2 -
S3 +
S4 -
dnaA +
My problem is that I don't know what is wrong to search so I can (attempt) to fix it!

Below is some step-by-step annotated code that hopefully does what you want (though instead of using print I have aggregated the results into a list so you can actually make use of it). I'm not quite sure what happened with your existing code (especially how you're processing two files?)
s_dict = {}
ig_list = []
with open('genes.txt', 'r') as infile: # Simulating reading the file you pass in sys.argv
for line in infile:
if line.startswith('IG_'):
ig_list.append(line.split()[0]) # Collect all our IG values for later
else:
s_name, value = line.split() # Separate out the S value and its operator
s_dict[s_name] = value.strip() # Add to dictionary to map S to operator
# Now you can go back through your list of IG values and append the appropriate operator
pulled_together = []
for item in ig_list:
s_value = item.split('_')[1]
# The following will look for the operator mapped to the S value. If it is
# not found, it will instead give you 'not found'
corresponding_operator = s_dict.get(s_value, 'Not found')
pulled_together.append([item, corresponding_operator])
print ('List structure')
print (pulled_together)
print ('\n')
print('Printout of each item in list')
for item in pulled_together:
print(item[0] + '\t' + item[1])

nname = name[:-3]
Python's slicing through list is very powerful, but can be tricky to understand correctly.
When you write [:-3], you take everything except the last three items. The thing is, if you have less than three element in your list, it does not return you an error, but an empty list.
I think this is where things does not work, as there are not much elements per line, it returns you an empty list. If you could tell what do you exactly want it to return there, with an example or something, it would help a lot, as i don't really know what you're trying to get with your slicing.

Does this do what you want?
from __future__ import print_function
import sys
# Read and store all the gene info lines, keyed by name
gene_info = dict()
with open(sys.argv[2]) as gene_info_file:
for line in gene_info_file:
tokens = line.split()
name = tokens[0].strip()
gene_info[name] = line
# Read the other file and lookup the names
with open(sys.argv[1]) as origin_file:
for line in origin_file:
if line.startswith("IG_"):
name = line.split("_")[1]
nname = name[:-3].strip()
if nname in gene_info:
lookup_line = gene_info[nname]
print("IG_" + lookup_line)
else:
pass # what do you want to do in this case?
else:
print(line)

Reading in data from file using regex in python

I have a data file with tons of data like:
{"Passenger Quarters",27.`,"Cardassian","not injured"},{"Passenger Quarters",9.`,"Cardassian","injured"},{"Passenger Quarters",32.`,"Romulan","not injured"},{"Bridge","Unknown","Romulan","not injured"}
I want to read in the data and save it in a list. I am having trouble getting the exact right code to exact the data between the { }. I don't want the quotes and the ` after the numbers. Also, data is not separated by line so how do I tell re.search where to begin looking for the next set of data?

At first glance, you can break this data into chunks by splitting it on the string },{:
chunks = data.split('},{')
chunks[0] = chunks[0][1:] # first chunk started with '{'
chunks[-1] = chunks[-1][:-1] # last chunk ended with '}'
Now you have chunks like
"Passenger Quarters",27.`,"Cardassian","not injured"
and you can apply a regular expression to them.

You should do this in two passes. One to get the list of items and one to get the contents of each item:
import re
from pprint import pprint
data = '{"Passenger Quarters",27.`,"Cardassian","not injured"},{"Passenger Quarters",9.`,"Cardassian","injured"},{"Passenger Quarters",32.`,"Romulan","not injured"},{"Bridge","Unknown","Romulan","not injured"}'
# This splits up the data into items where each item is the
# contents inside a pair of braces
item_pattern = re.compile("{([^}]+)}")
# This plits up each item into it's parts. Either matching a string
# inside quotation marks or a number followed by some garbage
contents_pattern = re.compile('(?:"([^"]+)"|([0-9]+)[^,]+),?')
rows = []
for item in item_pattern.findall(data):
row = []
for content in contents_pattern.findall(item):
if content[1]: # Number matched, treat it as one
row.append(int(content[1]))
else: # Number not matched, use the string (even if empty)
row.append(content[0])
rows.append(row)
pprint(rows)

The following will produce a list of lists, where each list is an individual record.
import re
data = '{"Passenger Quarters",27.`,"Cardassian","not injured"},{"Passenger Quarters",9.`,"Cardassian","injured"},{"Pssenger Quarters",32.`,"Romulan","not injured"},{"Bridge","Unknown","Romulan","not injured"}'
# remove characters we don't want and split into individual fields
badchars = ['{','}','`','.','"']
newdata = data.translate(None, ''.join(badchars))
fields = newdata.split(',')
# Assemble groups of 4 fields into separate lists and append
# to the parent list. Obvious weakness here is if there are
# records that contain something other than 4 fields
records = []
myrecord = []
recordcount = 1
for field in fields:
myrecord.append(field)
recordcount = recordcount + 1
if (recordcount > 4):
records.append(myrecord)
myrecord = []
recordcount = 1
for record in records:
print record
Output:
['Passenger Quarters', '27', 'Cardassian', 'not injured']
['Passenger Quarters', '9', 'Cardassian', 'injured']
['Pssenger Quarters', '32', 'Romulan', 'not injured']
['Bridge', 'Unknown', 'Romulan', 'not injured']

Read txt file and add a variable to the text - Python

In order to simplify some of my code I have decided to move queries and HTML code to txt files. However, an issue has come up: most of my queries and HTML that I normally keep inside the code have variable in the middle. For example, I have this in my code:
count = 0
for x in reviewers:
query = """select *
from mytable
where reviewer = """ + reviewers[count]
cur.execute(query)
count = count + 1
#do more stuff
The question is, how do I save queries or HTML code in txt files and then add variables in the middle of the strings?
Thanks!!

Ok so here is the solution I came up with I hope it helps
So you can save the Queries in text files in the form
SELECT * from %s where id = %d
And once you get the query you can place your variable in it. I am assuming that I already got the query from file.
query = "SELECT * from %s where id = %d"
completeQuery=query% ('myTable', 21)
print completeQuery
The output will be
SELECT * from myTable where id = 21
Reference

I'm still not sure what you want, Here's a way to read a file and add a variable name in the text
query = ""
f = open("query_file",'r')
query = f.read() # read the query file into a string
f.close()
for x in reviewers:
query = query+reviewers[count] # add variable name in the string assuming reviewers[count] gives a string
cur.execute(query)
count = count + 1
#do more stuff
EDIT
An important point strings in Python are immutable
if you want to modify string then you'd have to create a new string
for e.g
query = "Select from Table"
you want to make it Select Col from Table
here is what you do:-
add_me = "Col"
new_string = query[:-10] + add_me + query[6:]
now new_string string will have Select Col from Table