I apologize if the question seems straightforward and easy. I tried to look for an answer, but did not find one that could solve my problem.
I have a very simple minimization problem: I need to maximize an expected value (in a second phase the objective function will become more complicated):
def EV(q, P):
return (-1)*np.sum(100 * q * (2*P - 1))
q is a 12 dimensional vector whose elements need to be between 0 and 1 and, clearly, the sum of the elements of q is 1. So I proceed to set the bounds and constraints:
cons = {'type': 'eq', 'fun': lambda q: np.sum(q) - 1}
bds = [(0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1)]
P = array([ 0.32510069, 0.96284943, 0.33966465, 0.61696874, 0.77368336,
0.10127222, 0.47836665, 0.87537657, 0.2086234 , 0.52468426,
0.31931169, 0.86424427]).
Then I call scipy.optimize.minimize:
X0 = np.array([0.5,0,0,0,0,0,0,0,0,0,0.4,0])
qstar = scipy.optimize.minimize(fun = EV, x0 = X0, args = (P), method = 'L-BFGS-B', bounds = bds, constraints = cons).
However, when I print the solution qstar I get the following:
fun: -323.56132559388169
hess_inv: <12x12 LbfgsInvHessProduct with dtype=float64>
jac: array([ 34.97985972, -92.56988847, 32.06706651, -23.39374987,
-54.7366767 , 79.74555274, 4.32666525, -75.0753145 ,
58.27532163, -4.93685093, 36.13766353, -72.84884873])
message: 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
nfev: 26
nit: 1
status: 0
success: True
x: array([ 0., 1., 0., 1., 1., 0., 0., 1., 0., 1., 0., 1.])
Why isn't the solution satisfying the equality constraint? Is it, perhaps, because of the message? Any help is very much appreciated.
Change the solver method to SLSQP, as mentioned in the comment, constraints are only supported in SLSQP and COBYLA. SLSQP solves the problem by sequential least squares quadratic programming.
Note that COBYLA only supports inequality constraints.
import numpy as np
import scipy.optimize
def EV(q, P):
return (-1)*np.sum(100 * q * (2*P - 1))
cons = {'type': 'eq', 'fun': lambda q: np.sum(q) - 1}
bds = [(0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1), (0, 1)]
P = np.array([ 0.32510069, 0.96284943, 0.33966465, 0.61696874, 0.77368336,
0.10127222, 0.47836665, 0.87537657, 0.2086234 , 0.52468426,
0.31931169, 0.86424427])
X0 = np.array([0.5,0,0,0,0,0,0,0,0,0,0.4,0])
qstar = scipy.optimize.minimize(fun = EV, x0 = X0, args = (P), method ='SLSQP', bounds = bds, constraints = cons)
print(qstar)
gives me the following output.
fun: -92.56988588438836
jac: array([ 34.97986126, -92.56988621, 32.06707001, -23.39374828,
-54.7366724 , 79.74555588, 4.32666969, -75.07531452,
58.27532005, -4.93685246, 36.13766193, -72.84885406])
message: 'Optimization terminated successfully.'
nfev: 28
nit: 2
njev: 2
status: 0
success: True
x: array([ 2.07808604e-10, 1.00000000e+00, 1.95365391e-10,
0.00000000e+00, 0.00000000e+00, 4.37596612e-10,
5.51522994e-11, 0.00000000e+00, 3.28030922e-10,
8.07265366e-12, 2.14253171e-10, 0.00000000e+00])
Related
I want a numpy array of different mixed datatypes, basically a combination of float32 and uint32.
The thing is, I don't write the array manually (as all other forums that I've found). Here is a piece of code of what I'm trying to do:
a = np.full((1, 10), 1).astype(np.float32)
b = np.full((1, 10), 2).astype(np.float32)
c = np.full((1, 10), 3).astype(np.float32)
d = np.full((1, 10), 4).astype(np.uint32)
arr = np.dstack([a, b, c, d]) # arr.shape = 1, 10, 4
I want axis 2 of arr to be of mixed data types. Of course a, b, c, and d are read from files, but for simplicity i show them as constant values!
One important note: I want this functionality. Last element of the array have to be represented as a uint32 because I'm dealing with hardware components that expects this order of datatypes (think of it as an API that will throw an error if the data types do not match)
This is what I've tried:
arr.astype("float32, float32, float32, uint1")
but this duplicate each element in axis 2 four times with different data types (same value).
I also tried this (which is basically the same thing):
dt = np.dtype([('floats', np.float32, (3, )), ('ints', np.uint32, (1, ))])
arr = np.dstack((a, b, c, d)).astype(dt)
but I got the same duplication as well.
I know for sure if I construct the array as follows:
arr = np.array([((1, 2, 3), (4)), ((5, 6, 7), (8))], dtype=dt)
where dt is from the code block above, it works nice-ish. but I read those a, b, c, d arrays and I don't know if constructing those tuples (or structures) is the best way to do it because those arrays have length of 850k in practice.
Your dtype:
In [83]: dt = np.dtype([('floats', np.float32, (3, )), ('ints', np.uint32, (1, ))])
and a sample uniform array:
In [84]: x= np.arange(1,9).reshape(2,4);x
Out[84]:
array([[1, 2, 3, 4],
[5, 6, 7, 8]])
the wrong way of making a structured array:
In [85]: x.astype(dt)
Out[85]:
array([[([1., 1., 1.], [1]), ([2., 2., 2.], [2]), ([3., 3., 3.], [3]),
([4., 4., 4.], [4])],
[([5., 5., 5.], [5]), ([6., 6., 6.], [6]), ([7., 7., 7.], [7]),
([8., 8., 8.], [8])]],
dtype=[('floats', '<f4', (3,)), ('ints', '<u4', (1,))])
The right way:
In [86]: import numpy.lib.recfunctions as rf
In [87]: rf.unstructured_to_structured(x,dt)
Out[87]:
array([([1., 2., 3.], [4]), ([5., 6., 7.], [8])],
dtype=[('floats', '<f4', (3,)), ('ints', '<u4', (1,))])
and alternate way:
In [88]: res = np.zeros(2,dt)
In [89]: res['floats'] = x[:,:3]
In [90]: res['ints'] = x[:,-1:]
In [91]: res
Out[91]:
array([([1., 2., 3.], [4]), ([5., 6., 7.], [8])],
dtype=[('floats', '<f4', (3,)), ('ints', '<u4', (1,))])
https://numpy.org/doc/stable/user/basics.rec.html
I have Python-generated data, of the type
fa fb fc
fa1 fb1 [fc01, fc02,..., fc0m]
fa2 fb2 [fc11, fc12,..., fc1m]
... ... ...
fan fbn [fcn1, fcn2,..., fcnm]
I need to create a Python-compatible data structure to store it, maximizing ease of creation, and minimizing memory usage and read/write time. I need to be able to identify columns via field names (i.e. retrieve fa1 with something like data['fa'][0]). fa values are ints, and fb and fc are floats. Neither m nor n are known before runtime, but are known before data is inserted into the data structure, and do not change. m will not exceed 1000, and n won't exceed 10000. Data is generated one row at a time.
Until now, I've used a numpy associative array, asar, of dtype=[('f0,'i2'), ('f1','f8'), ('f2', 'f8', (m))]. However, since I can't just add a new row to a numpy array without deleting and recreating it each time a row is added, I've been using a separate counting variable ind_n, creating asar with asar = numpy.zeroes(n, dtype=dtype), overwriting asar[ind_n]'s zeroes with the data to be added, then incrementing ind_n until it reaches n. This works, but it seems like there must be a better solution (or at least one that allows me to eliminate ind_n). Is there a standard way to create the skeleton of asar (perhaps with something like np.zeroes()), then insert each line of data into the first nonzero row? Or a way to convert a standard python nested list to an associative array, once the nested list has been completely generated? (I know this conversion can definitely be done, but run into issues (e.g. ValueError: setting an array element with a sequence.) when converting the subarray, when I attempt it.)
In [39]: n, m = 5, 3
In [41]: dt=np.dtype([('f0','i2'), ('f1','f8'), ('f2', 'f8', (m))])
In [45]: asar = np.zeros(n, dt)
In [46]: asar
Out[46]:
array([(0, 0., [0., 0., 0.]), (0, 0., [0., 0., 0.]),
(0, 0., [0., 0., 0.]), (0, 0., [0., 0., 0.]),
(0, 0., [0., 0., 0.])],
dtype=[('f0', '<i2'), ('f1', '<f8'), ('f2', '<f8', (3,))])
Filling by field:
In [49]: asar['f0'] = np.arange(5)
In [50]: asar['f1'] = np.random.rand(5)
In [51]: asar['f2'] = np.random.rand(5,3)
In [52]: asar
Out[52]:
array([(0, 0.45120412, [0.86481761, 0.08861093, 0.42212446]),
(1, 0.63926708, [0.43788684, 0.89254029, 0.90637292]),
(2, 0.33844457, [0.80352251, 0.25411018, 0.315124 ]),
(3, 0.24271258, [0.27849709, 0.9905879 , 0.94155558]),
(4, 0.89239324, [0.1580938 , 0.52844036, 0.59092695])],
dtype=[('f0', '<i2'), ('f1', '<f8'), ('f2', '<f8', (3,))])
Generating a list with matching nesting:
In [53]: alist = [(i,i,[10]*3) for i in range(5)]
In [54]: np.array(alist, dt)
Out[54]:
array([(0, 0., [10., 10., 10.]), (1, 1., [10., 10., 10.]),
(2, 2., [10., 10., 10.]), (3, 3., [10., 10., 10.]),
(4, 4., [10., 10., 10.])],
dtype=[('f0', '<i2'), ('f1', '<f8'), ('f2', '<f8', (3,))])
Obviously you could do:
for i, row in enumerate(alist):
asar[i] = row
enumerate is a nice idiomatic way of generating an index along with a value. But then so is range(n).
If you know n at the time you create the first record your solution is essentially correct.
You can use np.empty instead of np.zeros saving a bit (but not much) time.
If you feel bad about ind_n you can create an array iterator instead.
>>> m = 5
>>> n = 7
>>> dt = [('col1', 'i2'), ('col2', float), ('col3', float, (m,))]
>>> data = [(np.random.randint(10), np.random.random(), np.random.random((m,))) for _ in range(n)]
>>>
>>> rec = np.empty((n,), dt)
>>> irec = np.nditer(rec, op_flags=[['readwrite']], flags=['c_index'])
>>>
>>> for src in data:
... # roughly equivalent to list.append:
... next(irec)[()] = src
... print()
... # getting the currently valid part:
... print(irec.operands[0][:irec.index+1])
...
[(9, 0.07368308, [0.44691665, 0.38875103, 0.83522137, 0.39281718, 0.62078615])]
[(9, 0.07368308, [0.44691665, 0.38875103, 0.83522137, 0.39281718, 0.62078615])
(6, 0.82350335, [0.57971597, 0.61270304, 0.05280996, 0.03702404, 0.99159465])]
[(9, 0.07368308, [0.44691665, 0.38875103, 0.83522137, 0.39281718, 0.62078615])
(6, 0.82350335, [0.57971597, 0.61270304, 0.05280996, 0.03702404, 0.99159465])
(3, 0.06565234, [0.88921842, 0.21097122, 0.83276431, 0.01824657, 0.49105466])]
[(9, 0.07368308, [0.44691665, 0.38875103, 0.83522137, 0.39281718, 0.62078615])
(6, 0.82350335, [0.57971597, 0.61270304, 0.05280996, 0.03702404, 0.99159465])
(3, 0.06565234, [0.88921842, 0.21097122, 0.83276431, 0.01824657, 0.49105466])
(2, 0.69806099, [0.87749632, 0.22119474, 0.25623813, 0.26587436, 0.04772489])]
[(9, 0.07368308, [0.44691665, 0.38875103, 0.83522137, 0.39281718, 0.62078615])
(6, 0.82350335, [0.57971597, 0.61270304, 0.05280996, 0.03702404, 0.99159465])
(3, 0.06565234, [0.88921842, 0.21097122, 0.83276431, 0.01824657, 0.49105466])
(2, 0.69806099, [0.87749632, 0.22119474, 0.25623813, 0.26587436, 0.04772489])
(1, 0.77573727, [0.44359522, 0.62471617, 0.65742177, 0.38889958, 0.13901824])]
[(9, 0.07368308, [0.44691665, 0.38875103, 0.83522137, 0.39281718, 0.62078615])
(6, 0.82350335, [0.57971597, 0.61270304, 0.05280996, 0.03702404, 0.99159465])
(3, 0.06565234, [0.88921842, 0.21097122, 0.83276431, 0.01824657, 0.49105466])
(2, 0.69806099, [0.87749632, 0.22119474, 0.25623813, 0.26587436, 0.04772489])
(1, 0.77573727, [0.44359522, 0.62471617, 0.65742177, 0.38889958, 0.13901824])
(0, 0.45797521, [0.79193395, 0.69029592, 0.0541346 , 0.49603146, 0.36146384])]
[(9, 0.07368308, [0.44691665, 0.38875103, 0.83522137, 0.39281718, 0.62078615])
(6, 0.82350335, [0.57971597, 0.61270304, 0.05280996, 0.03702404, 0.99159465])
(3, 0.06565234, [0.88921842, 0.21097122, 0.83276431, 0.01824657, 0.49105466])
(2, 0.69806099, [0.87749632, 0.22119474, 0.25623813, 0.26587436, 0.04772489])
(1, 0.77573727, [0.44359522, 0.62471617, 0.65742177, 0.38889958, 0.13901824])
(0, 0.45797521, [0.79193395, 0.69029592, 0.0541346 , 0.49603146, 0.36146384])
(6, 0.85225039, [0.62028917, 0.4895316 , 0.00922578, 0.66836154, 0.53082779])]
There is an one-dimensional array, for instance, as shown in the following. Are there any functions that can transform this array into another array, which only keeps the top 5 elements of the existing array. These five kept elements are marked as 5, 4,3,2,1 based on their respective numerical values, and other elements are just marked as 0.
9.00E-05
8.74E-05
-6.67E-05
-0.000296984
-0.00016961
-7.49E-06
-0.000102942
-0.000183901
0.000206149
5.62E-05
0.000112588
5.93E-05
9.85E-05
-2.29E-05
5.08E-05
0.00015748
Here is one solution from rank
s=df.rank(ascending=False)
s.mask(s>5,0).astype(int)
Out[74]:
0 5
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 1
9 0
10 3
11 0
12 4
13 0
14 0
15 2
Name: val, dtype: int32
If you want the numbers to remain in the same order and obtain an array of tuples with the original number and rank, you could do this:
numbers = [ 9.00E-05, 8.74E-05, -6.67E-05, -0.000296984, -0.00016961, -7.49E-06, -0.000102942, -0.000183901, 0.000206149, 5.62E-05, 0.000112588, 5.93E-05, 9.85E-05, -2.29E-05, 5.08E-05, 0.00015748]
ranks = { n:max(5-i,0) for (i,n) in enumerate(sorted(numbers)) }
tagged = [ (n,ranks[n]) for n in numbers ]
# tagged will contain : [(9e-05, 0), (8.74e-05, 0), (-6.67e-05, 1), (-0.000296984, 5), (-0.00016961, 3), (-7.49e-06, 0), (-0.000102942, 2), (-0.000183901, 4), (0.000206149, 0), (5.62e-05, 0), (0.000112588, 0), (5.93e-05, 0), (9.85e-05, 0), (-2.29e-05, 0), (5.08e-05, 0), (0.00015748, 0)]
if the original order doesn't matter, you only need this:
tagged = [ (n,max(5-i,0)) for (i,n) in enumerate(sorted(numbers)) ]
# then tagge will be : [(-0.000296984, 5), (-0.000183901, 4), (-0.00016961, 3), (-0.000102942, 2), (-6.67e-05, 1), (-2.29e-05, 0), (-7.49e-06, 0), (5.08e-05, 0), (5.62e-05, 0), (5.93e-05, 0), (8.74e-05, 0), (9e-05, 0), (9.85e-05, 0), (0.000112588, 0), (0.00015748, 0), (0.000206149, 0)]
One way is to use numpy. We assume your array is held in variable arr.
args = arr.argsort()
arr[args[-5:]] = range(5, 0, -1)
arr[args[:-5]] = 0
# array([ 5., 0., 0., 0., 0., 0., 0., 0., 1., 0., 3., 0., 4.,
# 0., 0., 2.])
I have a 2D np.array:
array([[ 1523., 172., 1613.],
[ 3216., 117., 1999.],
[ 85., 1271., 4.]])
I would to extract the sorted indexes of this np.array by value.
The results should be (for example) :
[[2,2],[2,0],[1,1],[0,1],[2,1],[0,0],[0,2],[1,2],[1,0]]
I already saw how to extract the min :
np.unravel_index(np.argmin(act),act.shape) #(2,2)
Thank you
Using numpy.argsort with axis=None (assuming flatten array):
>>> import numpy as np
>>>
>>> act = np.array([[ 1523., 172., 1613.],
... [ 3216., 117., 1999.],
... [ 85., 1271., 4.]])
>>> n = act.shape[1]
>>> zip(*np.argsort(act, axis=None).__divmod__(n))
[(2, 2), (2, 0), (1, 1), (0, 1), (2, 1), (0, 0), (0, 2), (1, 2), (1, 0)]
Either ndarray.reshape or numpy.newaxis can be used to add a new dimension to an array. They both seem to create a view, is there any reason or advantage to use one instead of the other?
>>> b
array([ 1., 1., 1., 1.])
>>> c = b.reshape((1,4))
>>> c *= 2
>>> c
array([[ 2., 2., 2., 2.]])
>>> c.shape
(1, 4)
>>> b
array([ 2., 2., 2., 2.])
>>> d = b[np.newaxis,...]
>>> d
array([[ 2., 2., 2., 2.]])
>>> d.shape
(1, 4)
>>> d *= 2
>>> b
array([ 4., 4., 4., 4.])
>>> c
array([[ 4., 4., 4., 4.]])
>>> d
array([[ 4., 4., 4., 4.]])
>>>
`
One reason to use numpy.newaxis over ndarray.reshape is when you have more than one "unknown" dimension to operate with. So, for example, for the following array:
>>> arr.shape
(10, 5)
This works:
>>> arr[:, np.newaxis, :].shape
(10, 1, 5)
But this does not:
>>> arr.reshape(-1, 1, -1)
...
ValueError: can only specify one unknown dimension
I don't see evidence of much difference. You could do a time test on very large arrays. Basically both fiddle with the shape, and possibly the strides. __array_interface__ is a nice way of accessing this information. For example:
In [94]: b.__array_interface__
Out[94]:
{'data': (162400368, False),
'descr': [('', '<f8')],
'shape': (5,),
'strides': None,
'typestr': '<f8',
'version': 3}
In [95]: b[None,:].__array_interface__
Out[95]:
{'data': (162400368, False),
'descr': [('', '<f8')],
'shape': (1, 5),
'strides': (0, 8),
'typestr': '<f8',
'version': 3}
In [96]: b.reshape(1,5).__array_interface__
Out[96]:
{'data': (162400368, False),
'descr': [('', '<f8')],
'shape': (1, 5),
'strides': None,
'typestr': '<f8',
'version': 3}
Both create a view, using the same data buffer as the original. Same shape, but reshape doesn't change the strides. reshape lets you specify the order.
And .flags shows differences in the C_CONTIGUOUS flag.
reshape may be faster because it is making fewer changes. But either way the operation shouldn't affect the time of larger calculations much.
e.g. for large b
In [123]: timeit np.outer(b.reshape(1,-1),b)
1 loops, best of 3: 288 ms per loop
In [124]: timeit np.outer(b[None,:],b)
1 loops, best of 3: 287 ms per loop
Interesting observation that: b.reshape(1,4).strides -> (32, 8)
Here's my guess. .__array_interface__ is displaying an underlying attribute, and .strides is more like a property (though it may all be buried in C code). The default underlying value is None, and when needed for calculation (or display with .strides) it calculates it from the shape and item size. 32 is the distance to the end of the 1st row (4x8). np.ones((2,4)).strides has the same (32,8) (and None in __array_interface__.
b[None,:] on the other hand is preparing the array for broadcasting. When broadcasted, existing values are used repeatedly. That's what the 0 in (0,8) does.
In [147]: b1=np.broadcast_arrays(b,np.zeros((2,1)))[0]
In [148]: b1.shape
Out[148]: (2, 5000)
In [149]: b1.strides
Out[149]: (0, 8)
In [150]: b1.__array_interface__
Out[150]:
{'data': (3023336880L, False),
'descr': [('', '<f8')],
'shape': (2, 5),
'strides': (0, 8),
'typestr': '<f8',
'version': 3}
b1 displays the same as np.ones((2,5)) but has only 5 items.
np.broadcast_arrays is a function in /numpy/lib/stride_tricks.py. It uses as_strided from the same file. These functions directly play with the shape and strides attributes.