So here is my issue, I have created a Dictionary for my lists and am trying to add the data I find and append it to each row but instead it is just appending to the same column with the same data type.
How could I get it so each new append add to new row.
data_dict = {'contact name': [], 'name': [], 'telephone': [], 'email': [],
'mobile': [], 'feedback average': []}
try:
data_dict['telephone'].append(soup.find('span',itemprop='telephone').text)
except AttributeError:
data_dict['telephone'].append('No telephone')
print data_dict
field_names = fn = data_dict.keys()
with open('./file.csv','w') as csvfile:
f = csv.DictWriter(csvfile, fieldnames=fn)
f.writeheader()
f.writerow(data_dict)
Try something like this:
data_dict = {'contact name': [], 'name': [], 'telephone': [], 'email': [],
'mobile': [], 'feedback average': []}
try:
data_dict['telephone'].append(soup.find('span',itemprop='telephone').text)
except AttributeError:
data_dict['telephone'].append('No telephone')
print data_dict
fn = data_dict.keys()
with open('./file.csv','w') as csvfile:
f = csv.reader(csvfile)
for row in f:
for i in len(fn):
data_dict[fn[i]].append(row[i])
This should work for you, if I got you right.
But care, this requires that one row in the csv contains exactly the elements of your dictionary, in the correct order.
If this isn't the case, you will need to find out which value is written in which column, and then add the value of this column to the list in your dictionary.
So you would need to replace
for i in len(fn):
data_dict[fn[i]].append(row[i])
by
for k in fn:
data_dict[k].append(row[columns[k]])
where columns is a dictionary that contains the same keys as data_dict, and as the values the columns in which the data of the specific key is stored in the csv-file. For an example, columns could look like this:
columns = {'contact name': 1, 'name': 3, 'telephone' : 6, 'email': 7, 'mobile':8, 'feedback average': 2}
Related
I want to get all fieldnames from the union of two lists to later export as csv, but I'm only getting fildname from just one list.
I want to get all fieldnames because when I go to export to csv I get the following error:
ValueError: dict contains fields not in fieldnames: 'amzlink', 'url', 'asin'
amazondata = [{'amzlink': 'https://www.amazon.com/dp/B084ZZ7VY3', 'asin': 'B084ZZ7VY3', 'url': 'https://www.amazon.com/s?k=712145360504&s=review-rank'}]
amazonPage = [{'price': '$14.95', 'image': 'https://m.media-amazon.com/images/I/81D1P4QqLfL._AC_SX425_.jpg', 'rating': '4.7 out of 5'}]
result = []
amazonPage.extend(amazondata)
for myDict in amazonPage:
if myDict not in result:
result.append(myDict)
print (result[0])
If you are just looking to get a list of all field names in the dictionaries:
Extract the keys from the dictionaries, convert to set, and take union of sets.
Borrowed #Baramr's amazondata list to demonstrate this below:
amazondata = [{'amzlink': 'https://www.amazon.com/dp/B084ZZ7VY3', 'asin': 'B084ZZ7VY3', 'url': 'https://www.amazon.com/s?k=712145360504&s=review-rank'}]
amazonPage = [{'price': '$14.95', 'image': 'https://m.media-amazon.com/images/I/81D1P4QqLfL._AC_SX425_.jpg', 'rating': '4.7 out of 5'}]
amazondata_fields = set(amazondata[0].keys())
amazonPage_fields = set(amazonPage[0].keys())
all_fields = amazondata_fields.union(amazonPage_fields)
print(all_fields)
> {'price', 'rating', 'asin', 'image', 'amzlink', 'url'}
If you are looking to fuse two dictionaries: Use the update method.
amazondata[0].update(amazonPage[0])
print(amazondata[0])
> {'amzlink': 'https://www.amazon.com/dp/B084ZZ7VY3', 'asin':
> 'B084ZZ7VY3', 'url':
> 'https://www.amazon.com/s?k=712145360504&s=review-rank', 'price':
> '$14.95', 'image':
> 'https://m.media-amazon.com/images/I/81D1P4QqLfL._AC_SX425_.jpg',
> 'rating': '4.7 out of 5'}
Loop over all the dictionaries, adding the keys to a set.
amazondata = [{'amzlink': 'https://www.amazon.com/dp/B084ZZ7VY3', 'asin': 'B084ZZ7VY3', 'url': 'https://www.amazon.com/s?k=712145360504&s=review-rank'}]
amazonPage = [{'price': '$14.95', 'image': 'https://m.media-amazon.com/images/I/81D1P4QqLfL._AC_SX425_.jpg', 'rating': '4.7 out of 5'}]
result = []
amazonPage.extend(amazondata)
all_fields = set()
for myDict in amazonPage:
all_fields |= myDict.keys()
print(all_fields)
I have a list of dictionaries and need to create a new one that contains new keys and also keys&values from my original dictionary. One of the keys will have to contain a list of dictionaries (those would be the values from original dictionary)
My data looks like the following:
data = [{'CloseDate': '2020-05-01',
'OpportunityID': '1',
'CustomerID': '10'},
{'CloseDate': '2020-07-31',
'OpportunityID': '2',
'CustomerID': '11'}]
I want my new list of dicts look like this:
new_data = [{'id': '39',
'Query': [{'records': '40', 'Order Name': '1', 'CustomerID': '10'}]},
{'id': '39',
'Query': [{'records': '40', 'Order Name': '2', 'CustomerID': '11'}]}]
I have tried the following:
new_data = []
for item in data:
params_dict = {}
params_dict["id"] = "39"
params_dict["Query"] = []
# push new_dicts in params_dict
new_dict = {}
new_dict["records"] = "40"
new_dict["Order Name"] = data["OpportunityID"]
params_dict.append(new_dict)
new_data.append(params_dict)
Error: TypeError: list indices must be integers or slices, not str
datas_list=[]
for get_dict in data:
new_dict={}
new_dict["id"] = 39
new_dict['Query']=[]
other_dictionary={}
other_dictionary['records']=40
for values in get_dict:
if values == "OpportunityID":
other_dictionary['Order Name'] = get_dict[values]
if values == "CustomerID" :
other_dictionary[values] = get_dict[values]
new_dict["Query"].append(other_dictionary)
datas_list.append(new_dict)
You were trying to iterate through item and not data inside the loop.
Also you need to append to Query.
Try:
new_data = []
for item in data:
params_dict = {}
params_dict["id"] = "39"
params_dict["Query"] = []
new_dict = {} # defined new_dict
new_dict["records"] = "40"
new_dict["Order Name"] = item["OpportunityID"] # here it should be item
params_dict["Query"].append(new_dict)
new_data.append(params_dict)
Also:
new_data = []
for item in data:
params_dict = {}
params_dict["id"] = "39"
params_dict["Query"] = [{"records" : "40","Order Name" :item["OpportunityID"] }]
new_data.append(params_dict)
I have a csv file and Im trying to create a nested dictionary that looks like this:
contacts = {"Tom": {"name": "Tom Techie",
"phone": "123 123546",
"email": "tom#tom.fi",
"skype": "skypenick"},
"Mike": {"name": "Mike Mechanic",
"phone": "000 123546",
"email": "mike#mike.fi",
"skype": "-Mike-M-"}}
etc
And this is what I have written:
file = open("csv","r")
d = {}
for i in file:
f = i.strip()
x = f.split(";")
if x[4] != "":
d.update({x[0] : {"name":x[1],
"phone":x[2],
"email":x[3],
"skype":x[4]}})
else:
d.update ({x[0] : {"name": x[1],
"phone": x[2],
"email": x[3]}})
However it prints the dict as a normal dictionary with the updates as keys when they should be like stated above.
EDIT:
First lines of the csv:
key;name;phone;email;skype
Tom;Tom Techie;123 123546;tom#tom.fi;skypenick
Mike;Mike Mechanic;000 123456;mike#mike.fi;-Mike-M-
Archie;Archie Architect;050 987654;archie#archie
You can use pd.read_csv() and to_dict():
import pandas as pd
contacts = pd.read_csv('test.csv', sep=';').set_index('key').to_dict(orient='index')
Yields:
{'Tom': {'name': 'Tom Techie', 'phone': '123 123546', 'email': 'tom#tom.fi', 'skype': 'skypenick'}, 'Mike': {'name': 'Mike Mechanic', 'phone': '000 123456', 'email': 'mike#mike.fi', 'skype': '-Mike-M-'}, 'Archie': {'name': 'Archie Architect', 'phone': '050 987654', 'email': 'archie#archie', 'skype': nan}}
I like the pandas answer, but if you don't want a 3rd party library, use the built-in csv module:
import csv
from pprint import pprint
D = {}
with open('csv',newline='') as f:
r = csv.DictReader(f,delimiter=';')
for line in r:
name = line['key']
del line['key']
D[name] = dict(line)
pprint(D)
Output:
{'Archie': {'email': 'archie#archie',
'name': 'Archie Architect',
'phone': '050 987654',
'skype': None},
'Mike': {'email': 'mike#mike.fi',
'name': 'Mike Mechanic',
'phone': '000 123456',
'skype': '-Mike-M-'},
'Tom': {'email': 'tom#tom.fi',
'name': 'Tom Techie',
'phone': '123 123546',
'skype': 'skypenick'}}
You can use zip() to achieve your goal:
file = """key;name;phone;email;skype
Tom;Tom Techie;123 123546;tom#tom.fi;skypenick
Mike;Mike Mechanic;000 123456;mike#mike.fi;-Mike-M-
Archie;Archie Architect;050 987654;archie#archie""".splitlines()
d = {}
h = None
for i in file: # works the same for your csv-file
# first row == header, store in h
if h is None:
h = i.strip().split(";")[1:]
continue # done for first row
x = i.strip().split(";")
# zip pairs the read in line with the header line to get tuples
# wich are fed into the dict constructor that creates the inner dict
d[x[0]] = dict(zip(h,x[1:]+[""])) # no default for skype
# use this instead if you want the skype key always present with empty default
# d[x[0]] = dict(zip(h,x[1:]+[""]))
print(d)
zip() discards the elements of the longer list - you won't need any checks for that.
Output:
{'Tom': {'name': 'Tom Techie', 'phone': '123 123546',
'email': 'tom#tom.fi', 'skype': 'skypenick'},
'Mike': {'name': 'Mike Mechanic', 'phone': '000 123456',
'email': 'mike#mike.fi', 'skype': '-Mike-M-'},
'Archie': {'name': 'Archie Architect', 'phone': '050 987654',
'email': 'archie#archie'}}
If you use the commented line, the data will get a default value of '' for the skype - works only b/c skype is the last element of the splitted line
You can use a dict comprehension! Assuming the data is something like
with open("df.csv", "r") as file:
d = {x.split(";")[0]:{
"name": x.split(";")[2],
"phone": x.split(";")[3],
"email": x.split(";")[1],
"skype": x.split(";")[4][:-1] # Slice off trailing newline
} for x in file}
d.pop("")
We want to open files using with whenever possible to benefit from Python's context management. See https://www.python.org/dev/peps/pep-0343/ for fundamental understanding of the with statement.
Since the key "" only appears once at the head of the csv, we can pop it at the end and avoid performing a comparison at every iteration. A dict comprehension accomplishes the same thing you wanted to achieve with d.update.
More about comprehensions:
https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions
Edit: refactoring to remove the repetitive calls to .split can look something like this:
def line_to_dict(x, d):
x = x.split(";")
d[x[0]] = {
"name": x[2],
"phone": x[3],
"email": x[1],
"skype": x[4][:-1] # Slice off trailing newline
}
with open("df.csv", "r") as file:
d = {}
for x in file:
line_to_dict(x, d)
d.pop("")
I am looping through an API to retrieve data for multiple ICO tokens. Now, I would like to save the data to a csv with variables in columns and 1 row for each ICO token. The basic code works, I have 2 problems:
- entries are written only in every second line, which is quite unpractical. How can I specify not to leave rows blank?
- the variable price is a list itself and thus saved in as a single item (with > 1 variables inside). How can I decompose the list to write one variable per column?
See my code here:
ICO_Wallet = '0xe8ff5c9c75deb346acac493c463c8950be03dfba',
'0x7654915a1b82d6d2d0afc37c52af556ea8983c7e',
'0x4DF812F6064def1e5e029f1ca858777CC98D2D81'
for index, Wallet in enumerate(ICO_Wallet) :
Name = ICO_name[index]
Number = ICO_No[index]
try:
URL = 'http://api.ethplorer.io/getTokenInfo/' + Wallet + '?apiKey=freekey'
except:
print(Wallet)
json_obj = urlopen(URL)
data = json.load(json_obj)
with open('token_data_test.csv','a') as f:
w = csv.writer(f, delimiter=";")
w.writerow(data.values())
time.sleep(1)
Sample output:
data Out[59]:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price': {'availableSupply': '11388258.0',
'currency': 'USD',
'diff': -20.71,
'diff30d': -14.155971452386,
'diff7d': -22.52,
'marketCapUsd': '2814942.0',
'rate': '0.2471792958',
'ts': '1542641433',
'volume24h': '2371.62380719'},
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
As mentioned, it's an easy fix for the first problem, just modify the csv.writer line like this:
w = csv.writer(f, delimiter=";", lineterminator='\n')
For your second problem, you can flatten your json before passing into csv:
for k, v in data.pop('price').items():
data['price_{}'.format(k)] = v
This changes all items under price into price_itemname as a flattened key. The .pop() method also helps remove the 'price' key at the same time.
Result:
{'address': '0x8a854288a5976036a725879164ca3e91d30c6a1b',
'countOps': 24207,
'decimals': '18',
'ethTransfersCount': 0,
'holdersCount': 10005,
'issuancesCount': 0,
'lastUpdated': 1542599890,
'name': 'GET',
'owner': '0x9a417e4db28778b6d9a4f42a5d7d01252a3af849',
'price_availableSupply': '11388258.0',
'price_currency': 'USD',
'price_diff': -20.71,
'price_diff30d': -14.155971452386,
'price_diff7d': -22.52,
'price_marketCapUsd': '2814942.0',
'price_rate': '0.2471792958',
'price_ts': '1542641433',
'price_volume24h': '2371.62380719',
'symbol': 'GET',
'totalSupply': '33368773400000170376363910',
'transfersCount': 24207}
Now you can just pass that into your csv.writer().
This question already has answers here:
List of dicts to/from dict of lists
(14 answers)
Closed 6 years ago.
I have a dictionary that stores a list of items for each key as shown:
name_dict = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
I want to print the data in the dictionary as table format:
FirstName,MiddleName,LastName # the keys of the dictionary
Elizabeth,H.,Perkins #the values of the keys in one line (the first item in the list)
Scott,T.,Joseph #the values of the keys in new line (the following item in the list)
How to solve this problem?
I have tried doing the suggested solution by Gareth Latty, but that did not work.
with open('C:/Output.csv',"w+") as file:
w = csv.DictWriter(file,name_dict.keys())
w.writeheader()
w.writerow(name_dict)
It outputs the following:
MiddleName,LastName,FirstName
"['H.', 'T.']","['Perkins', 'Joseph']","['Perkins', 'Joseph']"
Any idea how to output the values (the item in the list) of each of the keys in new row?
csv.DictWriter expects a dictionary with the field:single_line pairs for each row which is unfortunately not what you have, you basically need to convert your data structure to be a list of dicts for the single lines:
[{'MiddleName': 'H.', 'FirstName': 'Elizabeth ', 'LastName': 'Perkins'}, {'MiddleName': 'T.', 'FirstName': 'Scott ', 'LastName': 'Joseph'}]
You can convert it with something like this:
import csv
def seperate_entries(dict_of_lists):
iters = [(k,iter(v)) for k,v in dict_of_lists.items()]
try:
while True:
yield {k:next(v) for k,v in iters}
except StopIteration:
return
name_dict = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
with open('sample.csv',"w+") as file:
w = csv.DictWriter(file,name_dict.keys())
w.writeheader()
w.writerows(seperate_entries(name_dict))
I think you misuse dict. When you have multiple values, you should use a list of dicts and not a dict which the values are lists. Instead of
Dict = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
You should do:
Dict = [{'FirstName': 'Elizabeth', 'MiddleName': 'H.', 'LastName': 'Perkins'}, {'FirstName': 'Joseph', 'MiddleName': 'T. ', 'LastName': 'Scott'}]
or in a more readable version:
Dict = [
{'FirstName': 'Elizabeth', 'MiddleName': 'H.', 'LastName': 'Perkins'},
{'FirstName': 'Joseph', 'MiddleName': 'T. ', 'LastName': 'Scott' }
]
If you want to print one line (one dictionary of the list), you can do something like that:
def printDict(d):
print d["FirstName"] + "," + d["MiddleName"] + "," + d["LastName"]
And if you want to print each of the elements in the list you have:
def printList(l):
for i in l:
printDict(i)
And just use it like that:
printList(Dict)
With your first (original) Dict, accessing Dict["FirstName"] would return a list, and when printed it would print as:
["Elizabeth", "Joesph"]
But with the second (new way I suggested) Dict, accessing Dict[0]["FirstName"] would return a string, and will print like:
Elizabeth
To access the keys in the Dictionary, you just need to do the following:
middleNames=Dict['Middlename']
firstNames=Dict['FirstName']
lastNames=Dict['LastName']
You now have access to the values stored in the inner list information, this can similarly be accessed by the following:
# Find how many items read (optional)
len(middleNames)
# Then iterate through the items
for mName in middleName:
print mName # this will print each list item e.g.
H.
T.
# for brevity, this is the same as doing this...
middleName[0] etc
Hope this helps.
You need define how many row you have.
Just flat it into rows with all keys in Dict.
import csv
Dict = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
len_row = 2
with open('Output.csv', "w+") as file:
w = csv.DictWriter(file, Dict.keys())
w.writeheader()
for i in range(len_row):
row = {}
for k in Dict.keys():
for v in Dict.values():
row[k] = Dict[k][i]
w.writerow(row)
I'm sure there are more efficient ways of achieving what you want but this lays down a simple outline of what you want and shows how you can achieve it.
names = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
output = open('output.txt','w')
#NOTE: this will write headers in alphabetical format. You will need to change that so it follows the FirstName, MiddleName, LastName pattern
for key in sorted(names.keys()):
output.write(key + ',')
output.write('\n')
#assuming we always have same amount of middle, first, and last names
for i in range(len(names['FirstName'])):
personFname = ''
personLname = ''
personMname = ''
for key in names.keys():
if key == 'MiddleName':
personMname = names[key][i]
elif key == 'FirstName':
personFname = names[key][i]
elif key == 'LastName':
personLname = names[key][i]
output.write(personFname + ',' + personMname + ',' + personLname)
output.write('\n')
output.close()