We Keep Coding

Get all fieldnames from the union of two lists

I want to get all fieldnames from the union of two lists to later export as csv, but I'm only getting fildname from just one list.
I want to get all fieldnames because when I go to export to csv I get the following error:
ValueError: dict contains fields not in fieldnames: 'amzlink', 'url', 'asin'
amazondata = [{'amzlink': 'https://www.amazon.com/dp/B084ZZ7VY3', 'asin': 'B084ZZ7VY3', 'url': 'https://www.amazon.com/s?k=712145360504&s=review-rank'}]
amazonPage = [{'price': '$14.95', 'image': 'https://m.media-amazon.com/images/I/81D1P4QqLfL._AC_SX425_.jpg', 'rating': '4.7 out of 5'}]
result = []
amazonPage.extend(amazondata)
for myDict in amazonPage:
if myDict not in result:
result.append(myDict)
print (result[0])

If you are just looking to get a list of all field names in the dictionaries:
Extract the keys from the dictionaries, convert to set, and take union of sets.
Borrowed #Baramr's amazondata list to demonstrate this below:
amazondata = [{'amzlink': 'https://www.amazon.com/dp/B084ZZ7VY3', 'asin': 'B084ZZ7VY3', 'url': 'https://www.amazon.com/s?k=712145360504&s=review-rank'}]
amazonPage = [{'price': '$14.95', 'image': 'https://m.media-amazon.com/images/I/81D1P4QqLfL._AC_SX425_.jpg', 'rating': '4.7 out of 5'}]
amazondata_fields = set(amazondata[0].keys())
amazonPage_fields = set(amazonPage[0].keys())
all_fields = amazondata_fields.union(amazonPage_fields)
print(all_fields)
> {'price', 'rating', 'asin', 'image', 'amzlink', 'url'}
If you are looking to fuse two dictionaries: Use the update method.
amazondata[0].update(amazonPage[0])
print(amazondata[0])
> {'amzlink': 'https://www.amazon.com/dp/B084ZZ7VY3', 'asin':
> 'B084ZZ7VY3', 'url':
> 'https://www.amazon.com/s?k=712145360504&s=review-rank', 'price':
> '$14.95', 'image':
> 'https://m.media-amazon.com/images/I/81D1P4QqLfL._AC_SX425_.jpg',
> 'rating': '4.7 out of 5'}

Loop over all the dictionaries, adding the keys to a set.
amazondata = [{'amzlink': 'https://www.amazon.com/dp/B084ZZ7VY3', 'asin': 'B084ZZ7VY3', 'url': 'https://www.amazon.com/s?k=712145360504&s=review-rank'}]
amazonPage = [{'price': '$14.95', 'image': 'https://m.media-amazon.com/images/I/81D1P4QqLfL._AC_SX425_.jpg', 'rating': '4.7 out of 5'}]
result = []
amazonPage.extend(amazondata)
all_fields = set()
for myDict in amazonPage:
all_fields |= myDict.keys()
print(all_fields)

Not able to convert json data into csv in python while fetching data through api

I read a string containing a json document.
d2 = json.loads(s1)
I am getting data in this format, a list of dictionnaries.
[{'creati_id': 123,
'creativ_id': 234,
'status': 'adsc',
'name': 'seded',
…
'video_75_views': None,
'video_100_views': None,
'estimated': None,
'creative1': 1.0,
'creative': 'Excellent',
'value': 1.023424324}]}
How can I save this data in CSV format?

This can easily be achieved with the csv module:
import csv
data = [
{
"creati_id": 123,
"creativ_id": 234,
"status": "adsc",
"name": "seded",
}
]
with open("data_file.csv", "w") as data_file:
csv_writer = csv.writer(data_file)
header = data[0].keys()
csv_writer.writerow(header)
for line in data:
csv_writer.writerow(line.values())

You can use the standard csv library in Python to write CSV files. From your question, I'm assuming that you have multiple rows, each having the structure you shared. If that's the case, then something like this should do the trick:
import csv
json1 = [
{'creati_id': 123, 'creativ_id': 234, 'status': 'adsc', 'name': 'seded', 'email': None, 'brand': 'adc', 'market': 'dcassca', 'channel': 'dAD'},
{'creati_id': 123, 'creativ_id': 234, 'status': 'adsc', 'name': 'seded', 'email': None, 'brand': 'adc', 'market': 'dcassca', 'channel': 'dAD'}
]
header_names = json1[0].keys() # Extract the header names
data_rows = [row.values() for row in json1] # Extract the values for each
with open('output.csv', 'w', encoding='UTF8', newline='') as file:
writer = csv.writer(file)
writer.writerow(header_names) # Writes the header
writer.writerows(data_rows) # Writes the rows

Dynamically assign obtained results to variables in Python

I have an API response for listing out information of all Volumes. I want to loop through the response and get the value of the name and assign each one of them dynamically to each url.
This is my main API endpoint which returns the following:
[{'source': None, 'serial': '23432', 'created': '2018-11-
12T04:27:14Z', 'name': 'v001', 'size':
456456}, {'source': None, 'serial': '4364576',
'created': '2018-11-12T04:27:16Z', 'name': 'v002',
'size': 345435}, {'source': None, 'serial':
'6445645', 'created': '2018-11-12T04:27:17Z', 'name': 'v003', 'size':
23432}, {'source': None,
'serial': 'we43235', 'created': '2018-11-12T04:27:20Z',
'name': 'v004', 'size': 35435}]
I'm doing this to get the value of 'name'
test_url = 'https://0.0.0.0/api/1.1/volume'
test_data = json.loads(r.get(test_url, headers=headers,
verify=False).content.decode('UTF-8'))
new_data = [{
'name': value['name']
} for value in test_data]
final_data = [val['name'] for val in new_data]
for k in final_data:
print(k)
k prints out all the values in name, but i'm stuck at where i want to be able to use it in assigning different API endpoints. Now, k returns
v001
v002
v003
v004
I want to assign each one of them to different endpoints like below:
url_v001 = test_url + v001
url_v002 = test_url + v002
url_v003 = test_url + v003
url_v004 = test_url + v004
I want this to be dynamically done, because there may be more than 4 volume names returned by my main API.

It wouldn't be good to do that, but the best way is to use a dictionary:
d={}
for k in final_test:
d['url_'+k] = test_url + k
Or much better in a dictionary comprehension:
d={'url_'+k:test_url + k for k in final_test}
And now:
print(d)
Both reproduce:
{'url_v001': 'https://0.0.0.0/api/1.1/volumev001', 'url_v002': 'https://0.0.0.0/api/1.1/volumev002', 'url_v003': 'https://0.0.0.0/api/1.1/volumev003', 'url_v004': 'https://0.0.0.0/api/1.1/volumev004'}
To use d:
for k,v in d.items():
print(k+',',v)
Outputs:
url_v001, https://0.0.0.0/api/1.1/volumev001
url_v002, https://0.0.0.0/api/1.1/volumev002
url_v003, https://0.0.0.0/api/1.1/volumev003
url_v004, https://0.0.0.0/api/1.1/volumev004

Python CSV append new row

So here is my issue, I have created a Dictionary for my lists and am trying to add the data I find and append it to each row but instead it is just appending to the same column with the same data type.
How could I get it so each new append add to new row.
data_dict = {'contact name': [], 'name': [], 'telephone': [], 'email': [],
'mobile': [], 'feedback average': []}
try:
data_dict['telephone'].append(soup.find('span',itemprop='telephone').text)
except AttributeError:
data_dict['telephone'].append('No telephone')
print data_dict
field_names = fn = data_dict.keys()
with open('./file.csv','w') as csvfile:
f = csv.DictWriter(csvfile, fieldnames=fn)
f.writeheader()
f.writerow(data_dict)

Try something like this:
data_dict = {'contact name': [], 'name': [], 'telephone': [], 'email': [],
'mobile': [], 'feedback average': []}
try:
data_dict['telephone'].append(soup.find('span',itemprop='telephone').text)
except AttributeError:
data_dict['telephone'].append('No telephone')
print data_dict
fn = data_dict.keys()
with open('./file.csv','w') as csvfile:
f = csv.reader(csvfile)
for row in f:
for i in len(fn):
data_dict[fn[i]].append(row[i])
This should work for you, if I got you right.
But care, this requires that one row in the csv contains exactly the elements of your dictionary, in the correct order.
If this isn't the case, you will need to find out which value is written in which column, and then add the value of this column to the list in your dictionary.
So you would need to replace
for i in len(fn):
data_dict[fn[i]].append(row[i])
by
for k in fn:
data_dict[k].append(row[columns[k]])
where columns is a dictionary that contains the same keys as data_dict, and as the values the columns in which the data of the specific key is stored in the csv-file. For an example, columns could look like this:
columns = {'contact name': 1, 'name': 3, 'telephone' : 6, 'email': 7, 'mobile':8, 'feedback average': 2}

Python script reading from a csv file [duplicate]

This question already has answers here:
How do I read and write CSV files with Python?
(7 answers)
Closed 3 months ago.
"Type","Name","Description","Designation","First-term assessment","Second-term assessment","Total"
"Subject","Nick","D1234","F4321",10,19,29
"Unit","HTML","D1234-1","F4321",18,,
"Topic","Tags","First Term","F4321",18,,
"Subtopic","Review of representation of HTML",,,,,
All the above are the value from an excel sheet , which is converted to csv and that is the one shown above
The header as you notice contains seven coulmns,the data below them vary,
I have this script to generate these from python script,the script is below
from django.db import transaction
import sys
import csv
import StringIO
file = sys.argv[1]
no_cols_flag=0
flag=0
header_arr=[]
print file
f = open(file, 'r')
while (f.readline() != ""):
for i in [line.split(',') for line in open(file)]: # split on the separator
print "==========================================================="
row_flag=0
row_d=""
for j in i: # for each token in the split string
row_flag=1
print j
if j:
no_cols_flag=no_cols_flag+1
data=j.strip()
print j
break
How to modify the above script to say that this data belongs to a particular column header..
thanks..

You're importing the csv module but never use it. Why?
If you do
import csv
reader = csv.reader(open(file, "rb"), dialect="excel") # Python 2.x
# Python 3: reader = csv.reader(open(file, newline=""), dialect="excel")
you get a reader object that will contain all you need; the first row will contain the headers, and the subsequent rows will contain the data in the corresponding places.
Even better might be (if I understand you correctly):
import csv
reader = csv.DictReader(open(file, "rb"), dialect="excel") # Python 2.x
# Python 3: reader = csv.DictReader(open(file, newline=""), dialect="excel")
This DictReader can be iterated over, returning a sequence of dicts that use the column header as keys and the following data as values, so
for row in reader:
print(row)
will output
{'Name': 'Nick', 'Designation': 'F4321', 'Type': 'Subject', 'Total': '29', 'First-term assessment': '10', 'Second-term assessment': '19', 'Description': 'D1234'}
{'Name': 'HTML', 'Designation': 'F4321', 'Type': 'Unit', 'Total': '', 'First-term assessment': '18', 'Second-term assessment': '', 'Description': 'D1234-1'}
{'Name': 'Tags', 'Designation': 'F4321', 'Type': 'Topic', 'Total': '', 'First-term assessment': '18', 'Second-term assessment': '', 'Description': 'First Term'}
{'Name': 'Review of representation of HTML', 'Designation': '', 'Type': 'Subtopic', 'Total': '', 'First-term assessment': '', 'Second-term assessment': '', 'Description': ''}