I want to put lists of data in a csv file, this is my data
list_info = [('1552150125', '02141592cc00000001', '2', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000200', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '5', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000500', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '6', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000600', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '7', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000700', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '8', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000800', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '9', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000900', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '10', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000a00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '11', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000b00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '12', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000c00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '13', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000d00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '14', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000e00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '15', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000f00', '0', '0', '0', '23')]
This is my code:
print(list_info)
for row in list_info:
print (row)
for column in row:
out.write(column)
out.write('\n')
out.close()
But it does not work, it gives me every list concatenated in the same column, like this:
155215012502141592cc000000012000000000001000000020000023
The expected result is:
time | ID |a|b|c|d|e|f|g|h|i|j|k|l|m|n |o|p|q|r|
1552150|02141592cc00000001|2|0|0|0|0|0|0|0|0|0|0|0|1|0000000200|0|0|0|23|
You could just use the csv module,
import csv
# your list_info here
header = 'time ID a b c d e f g h i j k l m n o p q r'.split()
with open('some_data.csv', 'w') as csvfile:
writer = csv.writer(csvfile, delimiter='|')
writer.writerow(header)
writer.writerows(list_info) # write all the rows at once as suggested by #roganjosh
use csv package:
import csv
list_info = [('1552150125', '02141592cc00000001', '2', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000200', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '5', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000500', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '6', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000600', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '7', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000700', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '8', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000800', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '9', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000900', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '10', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000a00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '11', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000b00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '12', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000c00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '13', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000d00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '14', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000e00', '0', '0', '0', '23'), ('1552150125', '02141592cc00000001', '15', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0000000f00', '0', '0', '0', '23')]
h = 'time ID a b c d e f g h i j k l m n o p q r'.split()
with open('file.csv', 'w',newline='') as myfile:
wr = csv.writer(myfile, quoting=csv.QUOTE_NONE)
wr.writerow(h)
wr.writerows(list_info)
A solution with elementary Python tools.
Using csv module of the standard library would be also a solid solution. This just uses str.format. Note, that I advise to always use an encoding when opening files.
out = open('test.csv', 'w', encoding='utf-8')
for row in list_info:
out.write('|'.join(row))
out.write('\n')
out.close()
Related
ratinglist = open("ratings.txt").readlines()
booklist = open("booklist.txt").readlines()
keys = [key.strip('\n') for key in ratinglist[0::2]]
values = [value.strip(' \n').split(' ') for value in ratinglist[1::2]]
my_dict = dict(zip(keys, values))
for values in my_dict:
value * x = sum
So, I have a giant dictionary that has a list of values for each key.
Here is an example of my dictionary:
'KeeLed': ['0', '0', '0', '5', '0', '0', '0', '5', '0', '5', '5', '0', '0', '0', '0', '0', '5', '-3', '-3', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '5', '0', '0', '5', '0', '5', '5', '5', '5', '0', '0', '0', '0', '0', '3', '1', '0', '0', '0', '0'], 'Megan': ['5', '5', '0', '0', '0', '0', '0', '0', '0', '3', '0', '5', '0', '0', '1', '0', '5', '0', '1', '5', '0', '0', '0', '0', '0', '1', '0', '5', '0', '0', '3', '5', '5', '0', '0', '5', '0', '0', '3', '0', '0', '3', '5', '5', '0', '0', '0', '0', '0', '5', '5', '0', '5', '0', '0']}
What I need to do is do a for loop, that loops through each value, multiplies them, and then adds them.
I just need help getting the for loop to iterate through the values correctly.
For example, for KeeLed and Megan, I want this for loop to iterate through their seperate lists of values, and multiply them, then add them.
Like this: 0 * 5, + 5 * 0, + 0 * 0, + 5 * 0.... etc etc
The issue is I haven't found anything online on how to iterate through lists that are inside a dictionary . I've found some resources on looping through lists, but I need to loop through lists of values that are inside of a dictionary.
I would like the results to be put in a list, but I could do that if I get some help with the for loop.
scores for each person = [325, 450]
"The issue is I haven't found anything online on how to iterate through lists that are inside a dictionary . I've found some resources on looping through lists, but I need to loop through lists of values that are inside of a dictionary."
EXAMPLE
myDict = {
'KeeLed': ['0', '0', '0', '5', '0', '0', '0', '5', '0', '5', '5', '0', '0', '0', '0', '0', '5', '-3', '-3', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '5', '0', '0', '5', '0', '5', '5', '5', '5', '0', '0', '0', '0', '0', '3', '1', '0', '0', '0', '0'],
'Megan': ['5', '5', '0', '0', '0', '0', '0', '0', '0', '3', '0', '5', '0', '0', '1', '0', '5', '0', '1', '5', '0', '0', '0', '0', '0', '1', '0', '5', '0', '0', '3', '5', '5', '0', '0', '5', '0', '0', '3', '0', '0', '3', '5', '5', '0', '0', '0', '0', '0', '5', '5', '0', '5', '0', '0']
}
# For-looping through dict will give the key.
for name in myDict:
total = 0
# applying the key into myDict will give the list associated with the dict
for score in myDict[name]:
total = total + int(score)
print(name,total)
Result
KeeLed 54
Megan 85
I have this giant dictionary, where I have a key for a key that is a person, and a list of values for each person, here is an example of two people inside of the dictionary:
{'NuNu': ['0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '0', '0', '0', '0', '3', '0', '3', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '3', '0', '0', '1', '0', '5', '0', '0', '0', '0', '0', '1', '3', '3', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0'], 'Hamza ALi': ['0', '0', '0', '0', '0', '0', '0', '-5', '0', '5', '1', '0', '0', '0', '1', '0', '5', '0', '-3', '0', '3', '0', '0', '0', '0', '0', '0', '-5', '0', '0', '-3', '5', '0', '5', '0', '3', '0', '0', '1', '0', '3', '1', '3', '5', '0', '0', '0', '0', '0', '1', '-5', '0', '0', '0', '0']}
Now, what I need to do is multiply each value together, like 0 * 0, 0 * 0, etc etc for the entire list of the values, and then add all of the values together.
I have already done this in another piece of code, by just for two lists that I put the values inside for.
I have no idea how I would do it with a dictionary that has lists inside of it, that are values.
list1 = ['0', '0', '0', '5', '0', '0', '0', '5', '0', '5', '5', '0', '0', '0', '0', '0', '5', '-3', '-3', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '5', '0', '0', '5', '0', '5', '5', '5', '5', '0', '0', '0', '0', '0', '3', '1', '0', '0', '0', '0']
list2 = ['5', '5', '0', '0', '0', '0', '0', '0', '0', '3', '0', '5', '0', '0', '1', '0', '5', '0', '1', '5', '0', '0', '0', '0', '0', '1', '0', '5', '0', '0', '3', '5', '5', '0', '0', '5', '0', '0', '3', '0', '0', '3', '5', '5', '0', '0', '0', '0', '0', '5', '5', '0', '5', '0', '0']
int_list_1 = map(int,list1)
int_list_2 = map(int,list2)
third = map(sum, zip(int_list_1, int_list_2))
product = []
for num1, num2 in zip(int_list_1,int_list_2):
product.append(num1 * num2)
x = sum(product)
print(x)
^ This gives me the correct value that I need, but I need help setting this piece of code up for multiple lists that are inside of a dictionary.
I know I need to have a for loop, maybe something like this?
sum = 0
for value in dict:
sum + y
print(sum)
This is my solution in general case. Suppose we have n items in dictionary d. Then I make a collection of lists with integer values like so:
vals = map(lambda x: [int(n) for n in x], d.values())
After that I get all the tuples of items that is going to be multiplied elementwise:
p = zip(*vals)
Finally, this is a way to calculate all the products:
#from functools import reduce
products = [reduce(lambda x,y: x*y, n) for n in p]
output = sum(products)
Alternative way
Since lengths of dictionary values are balanced, we can use numpy actions:
import numpy as np
vals = np.array(list(d.values())).astype(int)
products = np.prod(vals, axis=0)
output = np.sum(products)
You try this.
a={'NuNu': ['0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '0', '0', '0', '0', '3', '0', '3', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '3', '0', '0', '1', '0', '5', '0', '0', '0', '0', '0', '1', '3', '3', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0'], 'Hamza ALi': ['0', '0', '0', '0', '0', '0', '0', '-5', '0', '5', '1', '0', '0', '0', '1', '0', '5', '0', '-3', '0', '3', '0', '0', '0', '0', '0', '0', '-5', '0', '0', '-3', '5', '0', '5', '0', '3', '0', '0', '1', '0', '3', '1', '3', '5', '0', '0', '0', '0', '0', '1', '-5', '0', '0', '0', '0']}
x,y=a.values()
x=map(int,x)
y=map(int,y)
out=[i*j for i,j in zip(x,y)]
print(out)
print(sum(out))
[0, 0, 0, 0, 0, 0, 0, -5, 0, 0, 0, 0, 0, 0, 0, 0, 15, 0, -9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -9, 0, 0, 5, 0, 15, 0, 0, 0, 0, 0, 1, 9, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
37
Another way is to use the reduce function:
from functools import reduce
inDict = {'NuNu': ['0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '0', '0', '0', '0', '3', '0', '3', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '3', '0', '0', '1', '0', '5', '0', '0', '0', '0', '0', '1', '3', '3', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0'], 'Hamza ALi': ['0', '0', '0', '0', '0', '0', '0', '-5', '0', '5', '1', '0', '0', '0', '1', '0', '5', '0', '-3', '0', '3', '0', '0', '0', '0', '0', '0', '-5', '0', '0', '-3', '5', '0', '5', '0', '3', '0', '0', '1', '0', '3', '1', '3', '5', '0', '0', '0', '0', '0', '1', '-5', '0', '0', '0', '0']}
result = sum([reduce(lambda x,y: x*y, map(int,elem)) for elem in zip(*inDict.values())])
print(result)
Output:
37
You've got the right idea. What about something like this for dicts with any number of keys.
def mul(iterable):
t = 1
for i in iterable:
t *= int(i)
return t
z = zip(*a.values()) # => [('0', '0'), ('0', '0'), ...]
m = map(mul, z) # => [0, 0, ...]
print(sum(m)) # => 37
Edit
You could replace the mul function with functools.reduce
from functools import reduce
z = zip(*a.values())
m = map(lambda t: reduce(lambda x, y: int(x) * int(y), t), z)
print(sum(m))
Now all as one line:
print(sum(map(lambda t: reduce(lambda x, y: int(x) * int(y), t), zip(*a.values()))))
I have a dataset like the following:
0000000000000001,0
0000000000000010,0
0000000000000011,1
0000000000000100,1
0000000000000101,1
0000000000000110,0
0000000000000111,1
0000000000001000,0
0000000000001001,0
.
.
.
I want to convert the first column from each row to be a comma seperated something like this:
From 0000000000000001,0 to 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0
The following code has a small issue: full_data is not 2D numpy of integers, yet it is 1D of string. I am not sure how to fix that. Any hint is appreciated.
data = []
my_data = np.genfromtxt('file.csv', delimiter=',', dtype='str')[:,0]
for i in my_data:
x= np.asarray(i)
data.append(x.tolist())
full_data = np.asarray(data)
Thank you
This is one approach.
Demo:
import numpy as np
data = []
my_data = np.genfromtxt(filename, delimiter=',', dtype='str')
data = [list(i) + [j] for i, j in my_data]
print(data)
Output:
[
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0'],
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0', '0'],
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '1', '1'],
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '1'],
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0', '1', '1'],
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '1', '0', '0'],
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '1', '1', '1'],
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0'],
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '1', '0']
]
I need to parse this XML Document and move the date and time into a %Y-%m-%d %H:%M:%S format as well as the variables hourly-qpf and probability-of-precipitation to columns in a tab-delimited .txt file.
All I have managed to do is read in the XML file using this code:
page = urllib2.urlopen('http://forecast.weather.gov/MapClick.php?lat=47.6062&lon=-122.3321&FcstType=digitalDWML')
page_content = page.read()
with open('KBFI.xml', 'w') as fid:
fid.write(page_content)
I am at a loss after this. I've only parsed one XML doc before, and it looked completely different from this.
EDIT
Sorry for not having anything to give you guys before, but I wasn't sure what module to use, as I only have experience with minidom and it didn't seem like the right choice. I've been messing around with Element Tree and I have come up with this:
data = []
import xml.etree.ElementTree as ET
tree = ET.parse('KBFI.xml')
root = tree.getroot()
for data in root.findall('data'):
for time-layout in root.findall('time-layout'):
start-valid-time = time-layout.find('start-valid-time')
time = datetime.datetime.strptime(start-valid-time, '%Y-%m-%dT%H:%M:%S')
for parameters in root.findall('parameters'):
for probability-of-precipitation in root.findall('probability-of-precipitation'):
value = probability-of-precipitation.find('value')
for hourly-qpf in root.findall('hourly-qpf'):
value2 = hourly-qpf.find('value')
data = data.append([time,
value,
value2])
with open('KBFI.txt','w') as file:
file.writelines('\t'.join(map(str,i)) + '\n' for i in data)
However, there is a problem because the variables are hyphenated and I do not know how to change them to underscores or remove them. Also, because of this, I have no idea if my code is any good!
You can use the python xml lib:
https://docs.python.org/2/library/xml.etree.elementtree.html
import urllib2
import xml.etree.ElementTree as ET
page = urllib2.urlopen('http://forecast.weather.gov/MapClick.php?lat=47.6062&lon=-122.3321&FcstType=digitalDWML')
page_content = page.read()
root = ET.fromstring(page_content)
for _f in root.itertext():
***Do your formatting here***
I suggest using xmltodict for parsing and extracting data from XML because it is straightforward and easy to use since it converts XML to Python dicts with the same nesting as the XML source. For those familiar with Python syntax, using it is natural and Python dicts are fully versatile, meaning they are capable of expressing heterogeneous and nested data stuctures. For example the Pickling Tools Library relies on Python dicts for Python, C++ and Java data interoperability and provides tools for converting XML to dict. Advantages of xmltodict are that its small, fast, and a standalone module just for converting XML to dict.
As an example of xmltodict usage, the following script downloads this XML document and extracts its creation-date and lists of probability-of-precipitation and hourly-qpf values:
import requests
url='http://forecast.weather.gov/MapClick.php?lat=47.6062&lon=-122.3321&FcstType=digitalDWML'
r = requests.get(url)
import xmltodict
result = xmltodict.parse(r.text)
cd = result['dwml']['head']['product']['creation-date']['#text']
print("creation-date =",cd)
pop = result['dwml']['data']['parameters']['probability-of-precipitation']['value']
print("\nprobability-of-precipitation =", pop)
hqpf = result['dwml']['data']['parameters']['hourly-qpf']['value']
print("\nhourly-qpf =", hqpf)
Here is the output from running this script (on 20150730):
creation-date = 2015-07-30T08:53:12-07:00
probability-of-precipitation = ['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '7', '7', '7', '7', '7', '7', '7', '7', '7', '7', '7', '7', '5', '5', '5', '5', '5', '5', '5', '5', '5', '5', '5', '5', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '8', '6', '6', '6', '6', '6', '6', '6', '6', '6', '6', '6', '6', '13', '13', '13', '13', '13', '13', '13', '13', '13', '13', '13', '13', '19', '19', '19', '19', '19', '19', '19', '19', '19', '19', '19', '19', '28', '28', '28', '28', '28', '28']
hourly-qpf = ['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0.0067', '0.0067', '0.0067', '0.0067', '0.0067', '0.0067', '0.0033', '0.0033', '0.0033', '0.0033', '0.0033', '0.0033', '0.0067', '0.0067', '0.0067', '0.0067', '0.0067', '0.0067']
xmltodict can be installed with 'pip install xmltodict'. It was developed by Martin Blech and its GitHub project is at https://github.com/martinblech/xmltodict.
In order to access start-valid-time and end-valid-time its helps to know their data structures as well their locations. Since both are a series of values enclosed in identical tags, intuitively each series should be formed in a separate list as the value of a key with their name similar to probability-of-precipitation and hourly-qpf. This can be confirmed by printing the entire result dict and inspecting the format of start-valid-time and end-valid-time in it and that can be facilitated by pretty printing the result dict (with import pprint and then running pprint.pprint(result)). For this XML document, pretty printing its equivilant dict generates over 2000 lines, however start-valid-time begins on line 26 and its value is clearly an list:
{'dwml': {'#version': '1.0',
'#xmlns:xsd': 'http://www.w3.org/2001/XMLSchema',
'#xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
'#xsi:noNamespaceSchemaLocation': 'http://graphical.weather.gov/xml/DWMLgen/schema/DWML.xsd',
'head': {'product': {'#concise-name': 'tabular-digital',
'#operational-mode': 'developmental',
'#srsName': 'WGS 1984',
'creation-date': {'#refresh-frequency': 'PT1H',
'#text': '2015-07-31T14:20:30-07:00'}},
'source': {'production-center': 'Seattle, WA',
'credit': 'http://www.wrh.noaa.gov/sew',
'more-information': 'http://www.nws.noaa.gov/forecasts/xml/'}},
'data': {'location': {'location-key': 'point1',
'description': 'Downtown Seattle WA, WA',
'point': {'#latitude': '47.61',
'#longitude': '-122.32'},
'city': {'#state': 'WA',
'#text': 'Downtown Seattle WA'},
'height': {'#datum': 'mean sea level',
'#text': '240'}},
'moreWeatherInformation': {'#applicable-location': 'point1',
'#text': 'http://forecast.weather.gov/MapClick.php?lat=47.61&lon=-122.32&FcstType=digital'},
'time-layout': {'#time-coordinate': 'local',
'#summarization': 'none',
'layout-key': 'k-p1h-n1-0',
'start-valid-time': ['2015-07-31T16:00:00-07:00',
'2015-07-31T17:00:00-07:00',
'2015-07-31T18:00:00-07:00',
...
Here is a script that extracts and prints creation-date as a scalar value, all start-valid-time values in a list, all end-valid-time values in a list, all probability-of-precipitation values in a list and all hourly-qpf values in a list and prints the length of each extracted list:
import xmltodict
result = xmltodict.parse(r.text)
cd = result['dwml']['head']['product']['creation-date']['#text']
print("creation-date =",cd)
svt = result['dwml']['data']['time-layout']['start-valid-time']
print("\nstart-valid-time =", svt)
print("number of start-valid-time entries =", len(svt))
evt = result['dwml']['data']['time-layout']['end-valid-time']
print("\nend-valid-time =", evt)
print("number of end-valid-time entries =", len(evt))
pop = result['dwml']['data']['parameters']['probability-of-precipitation']['value']
print("\nprobability-of-precipitation =", pop)
print("number of probability-of-precipitation entries =", len(pop))
hqpf = result['dwml']['data']['parameters']['hourly-qpf']['value']
print("\nhourly-qpf =", hqpf)
print("number of hourly-qpf entries =", len(hqpf))
Here is the output from running this script (on 20150731):
creation-date = 2015-07-31T14:20:30-07:00
start-valid-time = ['2015-07-31T16:00:00-07:00', '2015-07-31T17:00:00-07:00', '2015-07-31T18:00:00-07:00', '2015-07-31T19:00:00-07:00', '2015-07-31T20:00:00-07:00', '2015-07-31T21:00:00-07:00', '2015-07-31T22:00:00-07:00', '2015-07-31T23:00:00-07:00', '2015-08-01T00:00:00-07:00', '2015-08-01T01:00:00-07:00', '2015-08-01T02:00:00-07:00', '2015-08-01T03:00:00-07:00', '2015-08-01T04:00:00-07:00', '2015-08-01T05:00:00-07:00', '2015-08-01T06:00:00-07:00', '2015-08-01T07:00:00-07:00', '2015-08-01T08:00:00-07:00', '2015-08-01T09:00:00-07:00', '2015-08-01T10:00:00-07:00', '2015-08-01T11:00:00-07:00', '2015-08-01T12:00:00-07:00', '2015-08-01T13:00:00-07:00', '2015-08-01T14:00:00-07:00', '2015-08-01T15:00:00-07:00', '2015-08-01T16:00:00-07:00', '2015-08-01T17:00:00-07:00', '2015-08-01T18:00:00-07:00', '2015-08-01T19:00:00-07:00', '2015-08-01T20:00:00-07:00', '2015-08-01T21:00:00-07:00', '2015-08-01T22:00:00-07:00', '2015-08-01T23:00:00-07:00', '2015-08-02T00:00:00-07:00', '2015-08-02T01:00:00-07:00', '2015-08-02T02:00:00-07:00', '2015-08-02T03:00:00-07:00', '2015-08-02T04:00:00-07:00', '2015-08-02T05:00:00-07:00', '2015-08-02T06:00:00-07:00', '2015-08-02T07:00:00-07:00', '2015-08-02T08:00:00-07:00', '2015-08-02T09:00:00-07:00', '2015-08-02T10:00:00-07:00', '2015-08-02T11:00:00-07:00', '2015-08-02T12:00:00-07:00', '2015-08-02T13:00:00-07:00', '2015-08-02T14:00:00-07:00', '2015-08-02T15:00:00-07:00', '2015-08-02T16:00:00-07:00', '2015-08-02T17:00:00-07:00', '2015-08-02T18:00:00-07:00', '2015-08-02T19:00:00-07:00', '2015-08-02T20:00:00-07:00', '2015-08-02T21:00:00-07:00', '2015-08-02T22:00:00-07:00', '2015-08-02T23:00:00-07:00', '2015-08-03T00:00:00-07:00', '2015-08-03T01:00:00-07:00', '2015-08-03T02:00:00-07:00', '2015-08-03T03:00:00-07:00', '2015-08-03T04:00:00-07:00', '2015-08-03T05:00:00-07:00', '2015-08-03T06:00:00-07:00', '2015-08-03T07:00:00-07:00', '2015-08-03T08:00:00-07:00', '2015-08-03T09:00:00-07:00', '2015-08-03T10:00:00-07:00', '2015-08-03T11:00:00-07:00', '2015-08-03T12:00:00-07:00', '2015-08-03T13:00:00-07:00', '2015-08-03T14:00:00-07:00', '2015-08-03T15:00:00-07:00', '2015-08-03T16:00:00-07:00', '2015-08-03T17:00:00-07:00', '2015-08-03T18:00:00-07:00', '2015-08-03T19:00:00-07:00', '2015-08-03T20:00:00-07:00', '2015-08-03T21:00:00-07:00', '2015-08-03T22:00:00-07:00', '2015-08-03T23:00:00-07:00', '2015-08-04T00:00:00-07:00', '2015-08-04T01:00:00-07:00', '2015-08-04T02:00:00-07:00', '2015-08-04T03:00:00-07:00', '2015-08-04T04:00:00-07:00', '2015-08-04T05:00:00-07:00', '2015-08-04T06:00:00-07:00', '2015-08-04T07:00:00-07:00', '2015-08-04T08:00:00-07:00', '2015-08-04T09:00:00-07:00', '2015-08-04T10:00:00-07:00', '2015-08-04T11:00:00-07:00', '2015-08-04T12:00:00-07:00', '2015-08-04T13:00:00-07:00', '2015-08-04T14:00:00-07:00', '2015-08-04T15:00:00-07:00', '2015-08-04T16:00:00-07:00', '2015-08-04T17:00:00-07:00', '2015-08-04T18:00:00-07:00', '2015-08-04T19:00:00-07:00', '2015-08-04T20:00:00-07:00', '2015-08-04T21:00:00-07:00', '2015-08-04T22:00:00-07:00', '2015-08-04T23:00:00-07:00', '2015-08-05T00:00:00-07:00', '2015-08-05T01:00:00-07:00', '2015-08-05T02:00:00-07:00', '2015-08-05T03:00:00-07:00', '2015-08-05T04:00:00-07:00', '2015-08-05T05:00:00-07:00', '2015-08-05T06:00:00-07:00', '2015-08-05T07:00:00-07:00', '2015-08-05T08:00:00-07:00', '2015-08-05T09:00:00-07:00', '2015-08-05T10:00:00-07:00', '2015-08-05T11:00:00-07:00', '2015-08-05T12:00:00-07:00', '2015-08-05T13:00:00-07:00', '2015-08-05T14:00:00-07:00', '2015-08-05T15:00:00-07:00', '2015-08-05T16:00:00-07:00', '2015-08-05T17:00:00-07:00', '2015-08-05T18:00:00-07:00', '2015-08-05T19:00:00-07:00', '2015-08-05T20:00:00-07:00', '2015-08-05T21:00:00-07:00', '2015-08-05T22:00:00-07:00', '2015-08-05T23:00:00-07:00', '2015-08-06T00:00:00-07:00', '2015-08-06T01:00:00-07:00', '2015-08-06T02:00:00-07:00', '2015-08-06T03:00:00-07:00', '2015-08-06T04:00:00-07:00', '2015-08-06T05:00:00-07:00', '2015-08-06T06:00:00-07:00', '2015-08-06T07:00:00-07:00', '2015-08-06T08:00:00-07:00', '2015-08-06T09:00:00-07:00', '2015-08-06T10:00:00-07:00', '2015-08-06T11:00:00-07:00', '2015-08-06T12:00:00-07:00', '2015-08-06T13:00:00-07:00', '2015-08-06T14:00:00-07:00', '2015-08-06T15:00:00-07:00', '2015-08-06T16:00:00-07:00', '2015-08-06T17:00:00-07:00', '2015-08-06T18:00:00-07:00', '2015-08-06T19:00:00-07:00', '2015-08-06T20:00:00-07:00', '2015-08-06T21:00:00-07:00', '2015-08-06T22:00:00-07:00', '2015-08-06T23:00:00-07:00', '2015-08-07T00:00:00-07:00', '2015-08-07T01:00:00-07:00', '2015-08-07T02:00:00-07:00', '2015-08-07T03:00:00-07:00', '2015-08-07T04:00:00-07:00', '2015-08-07T05:00:00-07:00', '2015-08-07T06:00:00-07:00', '2015-08-07T07:00:00-07:00', '2015-08-07T08:00:00-07:00', '2015-08-07T09:00:00-07:00', '2015-08-07T10:00:00-07:00', '2015-08-07T11:00:00-07:00', '2015-08-07T12:00:00-07:00', '2015-08-07T13:00:00-07:00', '2015-08-07T14:00:00-07:00', '2015-08-07T15:00:00-07:00']
number of start-valid-time entries = 168
end-valid-time = ['2015-07-31T17:00:00-07:00', '2015-07-31T18:00:00-07:00', '2015-07-31T19:00:00-07:00', '2015-07-31T20:00:00-07:00', '2015-07-31T21:00:00-07:00', '2015-07-31T22:00:00-07:00', '2015-07-31T23:00:00-07:00', '2015-08-01T00:00:00-07:00', '2015-08-01T01:00:00-07:00', '2015-08-01T02:00:00-07:00', '2015-08-01T03:00:00-07:00', '2015-08-01T04:00:00-07:00', '2015-08-01T05:00:00-07:00', '2015-08-01T06:00:00-07:00', '2015-08-01T07:00:00-07:00', '2015-08-01T08:00:00-07:00', '2015-08-01T09:00:00-07:00', '2015-08-01T10:00:00-07:00', '2015-08-01T11:00:00-07:00', '2015-08-01T12:00:00-07:00', '2015-08-01T13:00:00-07:00', '2015-08-01T14:00:00-07:00', '2015-08-01T15:00:00-07:00', '2015-08-01T16:00:00-07:00', '2015-08-01T17:00:00-07:00', '2015-08-01T18:00:00-07:00', '2015-08-01T19:00:00-07:00', '2015-08-01T20:00:00-07:00', '2015-08-01T21:00:00-07:00', '2015-08-01T22:00:00-07:00', '2015-08-01T23:00:00-07:00', '2015-08-02T00:00:00-07:00', '2015-08-02T01:00:00-07:00', '2015-08-02T02:00:00-07:00', '2015-08-02T03:00:00-07:00', '2015-08-02T04:00:00-07:00', '2015-08-02T05:00:00-07:00', '2015-08-02T06:00:00-07:00', '2015-08-02T07:00:00-07:00', '2015-08-02T08:00:00-07:00', '2015-08-02T09:00:00-07:00', '2015-08-02T10:00:00-07:00', '2015-08-02T11:00:00-07:00', '2015-08-02T12:00:00-07:00', '2015-08-02T13:00:00-07:00', '2015-08-02T14:00:00-07:00', '2015-08-02T15:00:00-07:00', '2015-08-02T16:00:00-07:00', '2015-08-02T17:00:00-07:00', '2015-08-02T18:00:00-07:00', '2015-08-02T19:00:00-07:00', '2015-08-02T20:00:00-07:00', '2015-08-02T21:00:00-07:00', '2015-08-02T22:00:00-07:00', '2015-08-02T23:00:00-07:00', '2015-08-03T00:00:00-07:00', '2015-08-03T01:00:00-07:00', '2015-08-03T02:00:00-07:00', '2015-08-03T03:00:00-07:00', '2015-08-03T04:00:00-07:00', '2015-08-03T05:00:00-07:00', '2015-08-03T06:00:00-07:00', '2015-08-03T07:00:00-07:00', '2015-08-03T08:00:00-07:00', '2015-08-03T09:00:00-07:00', '2015-08-03T10:00:00-07:00', '2015-08-03T11:00:00-07:00', '2015-08-03T12:00:00-07:00', '2015-08-03T13:00:00-07:00', '2015-08-03T14:00:00-07:00', '2015-08-03T15:00:00-07:00', '2015-08-03T16:00:00-07:00', '2015-08-03T17:00:00-07:00', '2015-08-03T18:00:00-07:00', '2015-08-03T19:00:00-07:00', '2015-08-03T20:00:00-07:00', '2015-08-03T21:00:00-07:00', '2015-08-03T22:00:00-07:00', '2015-08-03T23:00:00-07:00', '2015-08-04T00:00:00-07:00', '2015-08-04T01:00:00-07:00', '2015-08-04T02:00:00-07:00', '2015-08-04T03:00:00-07:00', '2015-08-04T04:00:00-07:00', '2015-08-04T05:00:00-07:00', '2015-08-04T06:00:00-07:00', '2015-08-04T07:00:00-07:00', '2015-08-04T08:00:00-07:00', '2015-08-04T09:00:00-07:00', '2015-08-04T10:00:00-07:00', '2015-08-04T11:00:00-07:00', '2015-08-04T12:00:00-07:00', '2015-08-04T13:00:00-07:00', '2015-08-04T14:00:00-07:00', '2015-08-04T15:00:00-07:00', '2015-08-04T16:00:00-07:00', '2015-08-04T17:00:00-07:00', '2015-08-04T18:00:00-07:00', '2015-08-04T19:00:00-07:00', '2015-08-04T20:00:00-07:00', '2015-08-04T21:00:00-07:00', '2015-08-04T22:00:00-07:00', '2015-08-04T23:00:00-07:00', '2015-08-05T00:00:00-07:00', '2015-08-05T01:00:00-07:00', '2015-08-05T02:00:00-07:00', '2015-08-05T03:00:00-07:00', '2015-08-05T04:00:00-07:00', '2015-08-05T05:00:00-07:00', '2015-08-05T06:00:00-07:00', '2015-08-05T07:00:00-07:00', '2015-08-05T08:00:00-07:00', '2015-08-05T09:00:00-07:00', '2015-08-05T10:00:00-07:00', '2015-08-05T11:00:00-07:00', '2015-08-05T12:00:00-07:00', '2015-08-05T13:00:00-07:00', '2015-08-05T14:00:00-07:00', '2015-08-05T15:00:00-07:00', '2015-08-05T16:00:00-07:00', '2015-08-05T17:00:00-07:00', '2015-08-05T18:00:00-07:00', '2015-08-05T19:00:00-07:00', '2015-08-05T20:00:00-07:00', '2015-08-05T21:00:00-07:00', '2015-08-05T22:00:00-07:00', '2015-08-05T23:00:00-07:00', '2015-08-06T00:00:00-07:00', '2015-08-06T01:00:00-07:00', '2015-08-06T02:00:00-07:00', '2015-08-06T03:00:00-07:00', '2015-08-06T04:00:00-07:00', '2015-08-06T05:00:00-07:00', '2015-08-06T06:00:00-07:00', '2015-08-06T07:00:00-07:00', '2015-08-06T08:00:00-07:00', '2015-08-06T09:00:00-07:00', '2015-08-06T10:00:00-07:00', '2015-08-06T11:00:00-07:00', '2015-08-06T12:00:00-07:00', '2015-08-06T13:00:00-07:00', '2015-08-06T14:00:00-07:00', '2015-08-06T15:00:00-07:00', '2015-08-06T16:00:00-07:00', '2015-08-06T17:00:00-07:00', '2015-08-06T18:00:00-07:00', '2015-08-06T19:00:00-07:00', '2015-08-06T20:00:00-07:00', '2015-08-06T21:00:00-07:00', '2015-08-06T22:00:00-07:00', '2015-08-06T23:00:00-07:00', '2015-08-07T00:00:00-07:00', '2015-08-07T01:00:00-07:00', '2015-08-07T02:00:00-07:00', '2015-08-07T03:00:00-07:00', '2015-08-07T04:00:00-07:00', '2015-08-07T05:00:00-07:00', '2015-08-07T06:00:00-07:00', '2015-08-07T07:00:00-07:00', '2015-08-07T08:00:00-07:00', '2015-08-07T09:00:00-07:00', '2015-08-07T10:00:00-07:00', '2015-08-07T11:00:00-07:00', '2015-08-07T12:00:00-07:00', '2015-08-07T13:00:00-07:00', '2015-08-07T14:00:00-07:00', '2015-08-07T15:00:00-07:00', '2015-08-07T16:00:00-07:00']
number of end-valid-time entries = 168
probability-of-precipitation = ['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '2', '2', '2', '2', '2', '2', '2', '2', '2', '2', '2', '2', '5', '5', '5', '5', '5', '5', '5', '5', '5', '5', '5', '5', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '9', '10', '10', '10', '10', '10', '10', '10', '10', '10', '10', '10', '10', '13', '13', '13', '13', '13', '13', '13', '13', '13', '13', '13', '13', '23', '23', '23', '23', '23', '23', '23', '23', '23', '23', '23', '23', '12', '12', '12', '12', '12', '12', '12', '12', '12', '12', '12', '12', '34', '34', '34', '34', '34', '34', '34', '34', '34', '34', '34']
number of probability-of-precipitation entries = 168
hourly-qpf = ['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0.0050', '0.0050', '0.0050', '0.0050', '0.0050', '0.0050', '0.0033', '0.0033', '0.0033', '0.0033', '0.0033', '0.0033', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0.0017', '0.0017', '0.0017', '0.0017', '0.0017', '0.0017', '0.0083', '0.0083', '0.0083', '0.0083', '0.0083']
number of hourly-qpf entries = 168
This question already has answers here:
Efficient way to convert delimiter separated string to numpy array
(3 answers)
Closed 8 years ago.
I have string final_line as
0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0130631958731,0.0,0.0,0.0,0.0,0.0,0.0,0.0130631958731,0.0,0.0,0.0,0.0,0.0,0.0,,
0.00507937313707,0.0,0.0,0.0201058520009,0.0,0.0,0.0459562331449,0.0268078026679,0.0,0.0103772139359,0.0,0.0,0.0438673134565,0.0,0.0268078026679,0.0,0.0,0.0,0.0,0.0,0.0224437417685,,
0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.070802847389,0.0,0.0,,
0.0,0.0,0.169140135429,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,,
0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0261263917462,0.0,0.0,0.0,0.0,0.0,0.0,0.0261263917462,0.0,0.0,0.0,0.0,0.0,0.0,,
0.0961428138228,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,,
I want to convert it into numpy array for without transferring to CSV file. Here each line in string should be converted to different row as
[['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0.08364751'], ['0.017944717', '0', '0', '0', '0.009470823', '0', '0', '0', '0', '0', '0'], ['0.012620501', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0'], ['0.012620501', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0'], ['0', '0', '0', '0', '0.01332164', '0', '0', '0', '0', '0', '0'], ['0.012620501', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0'], ['0', '0', '0', '0', '0', '0.097414177', '0.042092545', '0', '0', '0', '0'], ['0', '0', '0', '0', '0.01332164', '0', '0', '0', '0', '0', '0'], ['0.005324215', '0', '0', '0.04598186', '0.028100025', '0', '0', '0', '0', '0', '0.023525603'], ['0', '0', '0', '0', '0', '0', '0', '0', '0.055765006', '0', '0'], ['0', '0', '0.133216404', '0', '0', '0', '0', '0', '0', '0', '0']]
I know here output having different values but I have just shown to get the format
Example
I have string as
a1,b1,c1,d1,e1,,
a2,b2,c2,d2,e2,,
a3,b3,c3,d3,e3,,
then output should be
[['a1','b1','c1','d1','e1'],
['a2','b2','c2','d2','e2'],
['a3','b3','c3','d3','e3']]
I have seen other questions but it converts into only single row of array. Here I want every single line in new row as shown in example.
map can be useful in this case. Also, the last ,, in your final list is not necessary so I just ignored it (look at :-2 before split operation).
import numpy as np
A = final_line[:-2].split(',,')
B = np.array([map(float,a.split(',')) for a in A])
Your data is in the form of lists, with each line separated by an additional comma.
d_in = "1,2,3,,4,5,6,,7,8,9"
first = d_in.split(",,")
final = [line.split(",") for line in first]
first stores a list of strings (each of your columns). final stores your data in the proper format (in my example, [[1,2,3],[4,5,6],[7,8,9]])