I'm pretty new to numpy and I'm trying to vectorize a simple for loop for performance reasons, but I can't seem to come up with a solution. I have a numpy array with unique words and for each of these words i need the number of times they occur in another numpy array, called array_to_compare. The number is passed to a third numpy array, which has the same shape as the unique words array.
Here is the code which contains the for loop:
import numpy as np
unique_words = np.array(['a', 'b', 'c', 'd'])
array_to_compare = np.array(['a', 'b', 'a', 'd'])
vector_array = np.zeros(len(unique_words))
for word in np.nditer(unique_words):
counter = np.count_nonzero(array_to_compare == word)
vector_array[np.where(unique_words == word)] = counter
vector_array = [2. 1. 0. 1.] #the desired output
I tried it with np.where and np.isin, but did not get the desired result. I am thankful for any help!
I'd probably use a Counter and a list comprehension to solve this:
In [1]: import numpy as np
...:
...: unique_words = np.array(['a', 'b', 'c', 'd'])
...: array_to_compare = np.array(['a', 'b', 'a', 'd'])
In [2]: from collections import Counter
In [3]: counter = Counter(array_to_compare)
In [4]: counter
Out[4]: Counter({'a': 2, 'b': 1, 'd': 1})
In [5]: vector_array = np.array([counter[key] for key in unique_words])
In [6]: vector_array
Out[6]: array([2, 1, 0, 1])
Assembling the Counter is done in linear time and iterating through your unique_words is also linear.
A numpy comparison of array values using broadcasting:
In [76]: unique_words[:,None]==array_to_compare
Out[76]:
array([[ True, False, True, False],
[False, True, False, False],
[False, False, False, False],
[False, False, False, True]])
In [77]: (unique_words[:,None]==array_to_compare).sum(1)
Out[77]: array([2, 1, 0, 1])
In [78]: timeit (unique_words[:,None]==array_to_compare).sum(1)
9.5 µs ± 2.79 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
But Counter is also a good choice:
In [72]: %%timeit
...: c=Counter(array_to_compare)
...: [c[key] for key in unique_words]
12.7 µs ± 30.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Your use of count_nonzero can be improved with
In [73]: %%timeit
...: words=unique_words.tolist()
...: vector_array = np.zeros(len(words))
...: for i,word in enumerate(words):
...: counter = np.count_nonzero(array_to_compare == word)
...: vector_array[i] = counter
...:
23.4 µs ± 505 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
Iteration on lists is faster than on arrays (nditer doesn't add much). And enumerate lets us skip the where test.
Similar to #DanielLenz's answer, but using np.unique to create a dict:
import numpy as np
unique_words = np.array(['a', 'b', 'c', 'd'])
array_to_compare = np.array(['a', 'b', 'a', 'd'])
counts = dict(zip(*np.unique(array_to_compare, return_counts=True)))
result = np.array([counts[word] if word in counts else 0 for word in unique_words])
[2 1 0 1]
Related
A critical portion of my script relies on the concatenation of a large number of fixed-length strings. So I would like to use low-level numpy.char.join function instead of the classical python build str.join.
However, I can't get it to work right:
import numpy as np
# Example array.
array = np.array([
['a', 'b', 'c'],
['d', 'e', 'f'],
['g', 'h', 'i'],
], dtype='<U1')
# Now I wish to get:
# array(['abc', 'def', 'ghi'], dtype='<U3')
# But none of these is successful :(
np.char.join('', array)
np.char.join('', array.astype('<U3'))
np.char.join(np.array(''), array.astype('<U3'))
np.char.join(np.array('').astype('<U3'), array.astype('<U3'))
np.char.join(np.array(['', '', '']).astype('<U3'), array.astype('<U3'))
np.char.join(np.char.asarray(['', '', '']).astype('<U3'), np.char.asarray(array))
np.char.asarray(['', '', '']).join(array)
np.char.asarray(['', '', '']).astype('<U3').join(array.astype('<U3'))
.. and my initial array is always left unchanged.
What am I missing here?
What's numpy's most efficient way to concatenate each line of a large 2D <U1 array?
[EDIT]: Since performance is a concern, I have benchmarked proposed solutions. But I still don't know how to call np.char.join properly.
import numpy as np
import numpy.random as rd
from string import ascii_lowercase as letters
from time import time
# Build up an array with many random letters
n_lines = int(1e7)
n_columns = 4
array = np.array(list(letters))[rd.randint(0, len(letters), n_lines * n_columns)]
array = array.reshape((n_lines, n_columns))
# One quick-n-dirty way to benchmark.
class MeasureTime(object):
def __enter__(self):
self.tic = time()
def __exit__(self, type, value, traceback):
toc = time()
print(f"{toc-self.tic:0.3f} seconds")
# And test three concatenations procedures.
with MeasureTime():
# Involves str.join
cat = np.apply_along_axis("".join, 1, array)
with MeasureTime():
# Involves str.join
cat = np.array(["".join(row) for row in array])
with MeasureTime():
# Involve low-level np functions instead.
# Here np.char.add for example.
cat = np.char.add(
np.char.add(np.char.add(array[:, 0], array[:, 1]), array[:, 2]), array[:, 3]
)
outputs
41.722 seconds
19.921 seconds
15.206 seconds
on my machine.
Would np.char.join do better? How to make it work?
On the original (3,3) array (timings may scale differently):
The chained np.char.add:
In [88]: timeit np.char.add(np.char.add(arr[:,0],arr[:,1]),arr[:,2])
29 µs ± 223 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
An equivalent approach, using object dtype. For python strings, '+' is a string join.
In [89]: timeit arr.astype(object).sum(axis=1)
14.1 µs ± 18.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
For a list of strings, ''.join() is supposed to be faster than string sum. Plus it lets you specify a 'delimiter':
In [90]: timeit np.array([''.join(row) for row in arr])
13.8 µs ± 41.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Without the conversion back to array:
In [91]: timeit [''.join(row) for row in arr]
10.2 µs ± 15.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Better yet, use tolist to convert the array to a list of lists of strings:
In [92]: timeit [''.join(row) for row in arr.tolist()]
1.01 µs ± 1.81 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
the list comprehension equivalent of the nested np.char.add:
In [97]: timeit [row[0]+row[1]+row[2] for row in arr.tolist()]
1.19 µs ± 2.68 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
numpy does not have low-level string code, at least not in the same sense that it has low-level compiled numeric code. It still depends on Python string code, even if it calls it from the C-API.
====
Since the strings are U1, we can view them as U3:
In [106]: arr.view('U3')
Out[106]:
array([['abc'],
['def'],
['ghi']], dtype='<U3')
In [107]: arr.view('U3').ravel()
Out[107]: array(['abc', 'def', 'ghi'], dtype='<U3')
In [108]: timeit arr.view('U3').ravel()
1.04 µs ± 9.81 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
===
To use np.char.join we have to collect the rows into some sort of tuple, list, etc. One way to do that is make an object dtype array, and fill it from the array:
In [110]: temp = np.empty(arr.shape[0], object)
In [111]: temp
Out[111]: array([None, None, None], dtype=object)
In [112]: temp[:] = list(arr)
In [113]: temp
Out[113]:
array([array(['a', 'b', 'c'], dtype='<U1'),
array(['d', 'e', 'f'], dtype='<U1'),
array(['g', 'h', 'i'], dtype='<U1')], dtype=object)
In [114]: np.char.join('',temp)
Out[114]: array(['abc', 'def', 'ghi'], dtype='<U3')
or filling it with a list of lists:
In [115]: temp[:] = arr.tolist()
In [116]: temp
Out[116]:
array([list(['a', 'b', 'c']), list(['d', 'e', 'f']),
list(['g', 'h', 'i'])], dtype=object)
In [117]: np.char.join('',temp)
Out[117]: array(['abc', 'def', 'ghi'], dtype='<U3')
In [122]: %%timeit
...: temp = np.empty(arr.shape[0], object)
...: temp[:] = arr.tolist()
...: np.char.join('', temp)
...:
...:
22.1 µs ± 69.1 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
====
To get a better idea of what np.char.join can do, compare it with split:
In [132]: temp
Out[132]:
array([list(['a', 'b', 'c']), list(['d', 'e', 'f']),
list(['g', 'h', 'i'])], dtype=object)
In [133]: b = np.char.join(',',temp)
In [134]: b
Out[134]: array(['a,b,c', 'd,e,f', 'g,h,i'], dtype='<U5')
In [135]: np.char.split(b,',')
Out[135]:
array([list(['a', 'b', 'c']), list(['d', 'e', 'f']),
list(['g', 'h', 'i'])], dtype=object)
Another way to apply ''.join to the elements of the object array:
In [136]: np.frompyfunc(lambda s: ','.join(s), 1,1)(temp)
Out[136]: array(['a,b,c', 'd,e,f', 'g,h,i'], dtype=object)
np.array([''.join(row) for row in array])
is the pythonic way, using list comprehension then treating it as a numpy array.
I'm writing a program in which I want to flatten an array, so I used the following code:
list_of_lists = [["a","b","c"], ["d","e","f"], ["g","h","i"]]
flattened_list = [i for j in list_of_lists for i in j]
This results in ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i'], the desired output.
I then found out that using a numpy array, I could've done the same simply by using np.array(((1,2),(3,4),(5,6))).flatten().
I was wondering if there is any downside to always using numpy arrays in the place of regular Python lists? In other words, is there something that Python lists can do which numpy arrays can't?
With your small example, the list comprehension is faster than the array method, even when taking the array creation out of the timing loop:
In [204]: list_of_lists = [["a","b","c"], ["d","e","f"], ["g","h","i"]]
...: flattened_list = [i for j in list_of_lists for i in j]
In [205]: timeit [i for j in list_of_lists for i in j]
757 ns ± 17.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [206]: np.ravel(list_of_lists)
Out[206]: array(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i'], dtype='<U1')
In [207]: timeit np.ravel(list_of_lists)
8.05 µs ± 12.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [208]: %%timeit x = np.array(list_of_lists)
...: np.ravel(x)
2.33 µs ± 22.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
With a much larger example, I expect [208] to get better.
If the sublists differ in size, the array is not 2d, and flatten does nothing:
In [209]: list_of_lists = [["a","b","c",23], ["d",None,"f"], ["g","h","i"]]
...: flattened_list = [i for j in list_of_lists for i in j]
In [210]: flattened_list
Out[210]: ['a', 'b', 'c', 23, 'd', None, 'f', 'g', 'h', 'i']
In [211]: np.array(list_of_lists)
Out[211]:
array([list(['a', 'b', 'c', 23]), list(['d', None, 'f']),
list(['g', 'h', 'i'])], dtype=object)
Growing lists is more efficient:
In [217]: alist = []
In [218]: for row in list_of_lists:
...: alist.append(row)
...:
In [219]: alist
Out[219]: [['a', 'b', 23], ['d', None, 'f'], ['g', 'h', 'i']]
In [220]: np.array(alist)
Out[220]:
array([['a', 'b', 23],
['d', None, 'f'],
['g', 'h', 'i']], dtype=object)
We strongly discourage iterative concatenation. Collect the sublists or arrays in a list first.
Yes there are. The rule of thumb would be to remember numpy.array is better for data of the same datatype (all integers, all double precision fp, all booleans, strings of the same length etc) instead of a mix bag of things. In the latter case you might just as well using generic list, considering this:
In [93]: a = [b'5', 5, '55', 'ab', 'cde', 'ef', 4, 6]
In [94]: b = np.array(a)
In [95]: %timeit 5 in a
65.6 ns ± 0.79 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
In [96]: %timeit 6 in a # worst case
219 ns ± 5.48 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [97]: %timeit 5 in b
10.9 µs ± 217 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
look at this several magnitudes of performance difference, where numpy.array is slower! Certainly this depends on the dimension of the list, and in this particular case depends on the index of 5 or 6 (worst case of O(n) complexity), but you get the idea.
Numpy arrays and functions are better for the most part. Here is an article if you want to look into it more: https://webcourses.ucf.edu/courses/1249560/pages/python-lists-vs-numpy-arrays-what-is-the-difference
I have an array a and i want to create another array b with a certain string repeated the number of times specified by a
a = np.array([1,2,3])
s = 'a'
i want the b to be np.array(['a','aa','aaa']). What would be the numpy way to do it without loops?
Though my use case does not need it but, in general
a = np.array([1,2,3])
s = np.array(['a','b','c'])
How to get b to be np.array(['a','bb','ccc']) without loops?
There is a built-in method:
output = np.core.defchararray.multiply(s,a)
Let's compare the alternatives:
In [495]: a = np.array([1, 2, 3])
...: s = np.array(['a', 'b', 'c'])
Using the np.char function. Under the covers this applies string multiply to each element of the array (with a loop):
In [496]: np.char.multiply(s,a)
Out[496]: array(['a', 'bb', 'ccc'], dtype='<U3')
An explicit loop. i.item() converts the numpy string to Python string:
In [498]: np.array([i.item()*j for i,j in zip(s,a)])
Out[498]: array(['a', 'bb', 'ccc'], dtype='<U3')
Another way of creating an array of Python strings:
In [499]: s.astype(object)*a
Out[499]: array(['a', 'bb', 'ccc'], dtype=object)
Timings:
In [500]: timeit np.char.multiply(s,a)
21.3 µs ± 975 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [501]: timeit np.array([i.item()*j for i,j in zip(s,a)])
13.4 µs ± 21.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [502]: timeit s.astype(object)*a
9.16 µs ± 226 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
So the explicit loop approach does pretty well.
Another idea - use frompyfunc:
In [504]: np.frompyfunc(lambda i,j: i*j, 2,1)(s,a)
Out[504]: array(['a', 'bb', 'ccc'], dtype=object)
In [505]: timeit np.frompyfunc(lambda i,j: i*j, 2,1)(s,a)
6.28 µs ± 56 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
I thought of frompyfunc because I wondered if we could use broadcasting:
In [508]: np.frompyfunc(lambda i,j: i*j, 2,1)(s,a[:,None])
Out[508]:
array([['a', 'b', 'c'],
['aa', 'bb', 'cc'],
['aaa', 'bbb', 'ccc']], dtype=object)
But that kind of broadcasting works for the other methods as well.
np.vectorize uses np.frompyfunc but does dtype conversions (frompyfunc always returns object dtype), but it tends to be slower.
I don't understand why you insist on them being numpy objects? Maybe I'm misunderstanding the question but you would handle it the same way as a list:
import numpy as np
a = np.array([1, 2, 3])
s = np.array(['a', 'b', 'c'])
new_array = np.array([str(s[i])*a[i] for i in range(len(s))])
print(new_array)
Outputs:
['a', 'bb', 'ccc']
This assumes a and s are of equal length, because it was not specified otherwise
I have a massive array with rows and columns. Some rows are larger than others. I need to get the max length row, that is, the row that has the highest length. I wrote a simple function for this, but I wanted it to be as fas as possible, like numpy fast. Currently, it looks like this:
Example array:
values = [
[1,2,3],
[4,5,6,7,8,9],
[10,11,12,13]
]
def values_max_width(values):
max_width = 1
for row in values:
if len(row) > max_width:
max_width = len(row)
return max_width
Is there any way to accomplish this with numpy?
In [261]: values = [
...: [1,2,3],
...: [4,5,6,7,8,9],
...: [10,11,12,13]
...: ]
...:
In [262]:
In [262]: values
Out[262]: [[1, 2, 3], [4, 5, 6, 7, 8, 9], [10, 11, 12, 13]]
In [263]: def values_max_width(values):
...: max_width = 1
...: for row in values:
...: if len(row) > max_width:
...: max_width = len(row)
...: return max_width
...:
In [264]: values_max_width(values)
Out[264]: 6
In [265]: [len(v) for v in values]
Out[265]: [3, 6, 4]
In [266]: max([len(v) for v in values])
Out[266]: 6
In [267]: np.max([len(v) for v in values])
Out[267]: 6
Your loop and the list comprehension are similar in speed, np.max is much slower - it has to first turn the list into an array.
In [268]: timeit max([len(v) for v in values])
656 ns ± 16.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [269]: timeit np.max([len(v) for v in values])
13.9 µs ± 181 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [271]: timeit values_max_width(values)
555 ns ± 13 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
If you are starting with a list, it's a good idea to thoroughly test the list implementation. numpy is fast when it is doing compiled array stuff, but creating an array from a list is time consuming.
Making an array directly from values isn't much help. The result in a object dtype array:
In [272]: arr = np.array(values)
In [273]: arr
Out[273]:
array([list([1, 2, 3]), list([4, 5, 6, 7, 8, 9]), list([10, 11, 12, 13])],
dtype=object)
Math on such an array is hit-or-miss, and always slower than math on pure numeric arrays. We can iterate on such an array, but that iteration is slower than on a list.
In [275]: values_max_width(arr)
Out[275]: 6
In [276]: timeit values_max_width(arr)
1.3 µs ± 8.27 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
Not sure how you can make it faster. I've tried using np.max over the length of each item, but that will take even longer:
import numpy as np
import time
values = []
for k in range(100000):
values.append(list(np.random.randint(100, size=np.random.randint(1000))))
def timeit(func):
def wrapper(*args, **kwargs):
now = time.time()
retval = func(*args, **kwargs)
print('{} took {:.5f}s'.format(func.__name__, time.time() - now))
return retval
return wrapper
#timeit
def values_max_width(values):
max_width = 1
for row in values:
if len(row) > max_width:
max_width = len(row)
return max_width
#timeit
def value_max_width_len(values):
return np.max([len(l) for l in values])
values_max_width(values)
value_max_width_len(values)
values_max_width took 0.00598s
value_max_width_len took 0.00994s
* Edit *
As #Mstaino suggested, using map does make this code faster:
#timeit
def value_max_width_len(values):
return max(map(len, values))
values_max_width took 0.00598s
value_max_width_len took 0.00499s
This question already has answers here:
Numpy: get values from array where indices are in another array
(2 answers)
Closed 4 years ago.
Given two vectors v=['a','b','c'] and i=np.random.randint(len(v),size=10), I can get the "substitution" vector
vi = [v[i[x]] for x in range(len(i))]
E.g., vi is
['a', 'a', 'c', 'c', 'b', 'a', 'c', 'a', 'c', 'a']
if
i = array([0, 0, 2, 2, 1, 0, 2, 0, 2, 0])
Is there a vectorized operation for this?
You can simply use numpy indexing (note that you need to convert v to a numpy.array for this to work):
v = np.array(['a','b','c'])
i = np.random.randint(len(v),size=10)
>>> v[i]
array(['c', 'b', 'a', 'b', 'c', 'b', 'a', 'a', 'b', 'b'], dtype='<U1')
Timings
In [26]: i = np.random.randint(len(v),size=1000000)
In [27]: %timeit [v[i[x]] for x in range(len(i))]
554 ms ± 6.41 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [28]: %timeit v[i]
4.85 ms ± 12.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [29]: %timeit [v[s] for s in i]
505 ms ± 1.95 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)