We Keep Coding

Is there a way to use numpy array functions to get this same effect?

I am trying to filter out all non-gray values within a given tolerance with the following code. It gives the expected results but runs too slowly to use in practice. Is there a way to do the following using numpy operations?
for i in range(height):
for j in range(width):
r, g, b = im_arr[i][j]
r = (r + 150) / 2
g = (g + 150) / 2
b = (b + 150) / 2
mean = (r + g + b) / 3
diffr = abs(mean - r)
diffg = abs(mean - g)
diffb = abs(mean - b)
maxdev = 2
if (diffr + diffg + diffb) > maxdev:
im_arr[i][j][0] = 0
im_arr[i][j][1] = 0
im_arr[i][j][2] = 0

Looping in plain Python is slow: one of the advantages of numpy is that
transversing the arrays is highly optimized. Without commenting on the algorithm, you can get the same results using only numpy, which will be much faster
Since im_arr is an image, it is very likely that the dtype is np.uint8.
That is only 8 bits, so you have to be careful of overflows. In you code, when you add 150 to a number, the result will be of type np.int64. But if you add 150 to an 8-bit np.ndarray, the result will still be of type np.uint8 and it can overflow.
You can either change the array type (using astype) or add a float, which will automatically promote the array to float
mod_img = (im_arr + 150.)/2 # the point of "150." is important
signed_dif = mod_img - np.mean(mod_img, axis=2, keepdims=True)
collapsed_dif = np.sum(np.abs(signed_dif), axis=2)
maxdev = 2
im_arr[collapsed_dif > maxdev] = 0

This can be done without any loop. I'll try to break out every step into a dedicated line
import numpy as np
im_arr = np.random.rand(300,400,3) # Assuming this how you image looks like
img_shifted = (im_arr + 15) / 2 # This can be done in one go
mean_v = np.mean(img_shifted, axis=2) # Compute the mean along the channel axis
diff_img = np.abs(mean_v[:,:,None] - img_shifted) # Broadcasting to subtract n x m from n x m x k
maxdev = 2
selection = np.sum(diff_img, axis=2) > maxdev
im_arr[selection] = 0 # Using fancy indexing with booleans

Python operations on tensors without TensorFlow?

Here is my example from my code:
loss += lambda_coord * ((tf.math.sqrt(bb[2]) - tf.math.sqrt(label[2]))**2
+ (tf.math.sqrt(bb[3]) - tf.math.sqrt(label[3]))**2)
bb is a tensor and so is label
Is it possible for me to rewrite this as
loss += lambda_coord * ((math.sqrt(bb[2]) - math.sqrt(label[2]))**2
+ (math.sqrt(bb[3]) - math.sqrt(label[3]))**2)
using only the standard library function?
Another example, if I have a tensor which only contains the value 2, can I do 2 * tensor, or do I have to use the tf.math.multiply function?

how to use scipy.special.expi (Exponential integral) in tensorflow?

I have to code scipy.special.expi in tensorflow but I don't know how!!!!
please, someone, help
since there is no as such direct code in tensorflow so I'm stuck here
please help!!!

I don't really know much about this function, but based on the Fortran implementation in SciPy, the function EIX in scipy/special/specfun/specfun.f, I have put together a TensorFlow implementation following each step there. It is only for positive values, though, as the computation for negative values included a loop harder to vectorize.
import math
import tensorflow as tf
def expi(x):
x = tf.convert_to_tensor(x)
# When X is zero
m_0 = tf.equal(x, 0)
y_0 = -math.inf + tf.zeros_like(x)
# When X is negative
m_neg = x < 0
# This should be -e1xb(-x) according to ScyPy
# (negative exponential integral -1)
# Here it is just left as NaN
y_neg = math.nan + tf.zeros_like(x)
# When X is less or equal to 40 - Power series around x = 0
m_le40 = x <= 40
k = tf.range(1, 101, dtype=x.dtype)
r = tf.cumprod(tf.expand_dims(x, -1) * k / tf.square(k + 1), axis=-1)
ga = tf.constant(0.5772156649015328, dtype=x.dtype)
y_le40 = ga + tf.log(x) + x * (1 + tf.reduce_sum(r, axis=-1))
# Otherwise (X is greater than 40) - Asymptotic expansion (the series is not convergent)
k = tf.range(1, 21, dtype=x.dtype)
r = tf.cumprod(k / tf.expand_dims(x, -1), axis=-1)
y_gt40 = tf.exp(x) / x * (1 + tf.reduce_sum(r, axis=-1))
# Select values
return tf.where(
m_0, y_0, tf.where(
m_neg, y_neg, tf.where(
m_le40, y_le40, y_gt40)))
A small test
import tensorflow as tf
import scipy.special
import numpy as np
# Test
x = np.linspace(0, 100, 20)
y = scipy.special.expi(x)
with tf.Graph().as_default(), tf.Session() as sess:
y_tf = sess.run(expi(x))
print(np.allclose(y, y_tf))
# True
Note however this will take more memory than SciPy, because it is unrolling the approximation loops in memory instead of computing one step at a time.

Translating tensorflows conv2d to numpy/scipy operations?

This is the documentation for tf.nn.conv2d: Given an input tensor of shape [batch, in_height, in_width, in_channels] and a filter / kernel tensor of shape [filter_height, filter_width, in_channels, out_channels], this op performs the following
Flattens the filter to a 2-D matrix with shape [filter_height *
filter_width * in_channels,
Extracts image patches from the input tensor to form a virtual tensor of shape [batch, out_height, out_width, filter_height *
filter_width * in_channels].
For each patch, right-multiplies the filter matrix and the image patch vector.
In other words, it takes in a tensor of n images and does convolution with out_channel filters.
I am trying to translate to code that uses only numpy operations and the code is the following:
def my_conv2d(x, kernel):
nf = kernel.shape[-1] # number of filters
rf = kernel.shape[0] # filter size
w = kernel
s = 1 # stride
h_range = int((x.shape[2] - rf) / s) + 1 # (W - F + 2P) / S
w_range = int((x.shape[1] - rf) / s) + 1 # (W - F + 2P) / S
np_o = np.zeros((1, h_range, w_range, nf))
for i in range(x.shape[0]):
for z in range(nf):
for _h in range(h_range):
for _w in range(w_range):
np_o[0, _h, _w, z] = np.sum(x[i, _h * s:_h * s + rf, _w * s:_w * s
+ rf, * w[:, :, :, z])
return np_o
The problem is that code is extremely slow. Are there any numpy or scipy functions that can replicate what tensorflows' conv2d is doing that is of similar efficiency? I have looked at https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.convolve2d.html and it does convolution ONCE, meaning I have to pass a 2d tensor alongside a 2d kernel (it does not do multiple filters).
None of the previous stackoverflow questions helped much with this.
Thanks
Edit: did some testing and my code is about 44000% slower than doing tf.nn.conv2d!

Things are slow for you because you are using loops. Implementing with vector operations will be much faster but not as efficient as the high-level APIs such as tf.nn.conv2d or tf.nn.convolution. This post should be able to help you with the vectorized implementation of the same in numpy : https://wiseodd.github.io/techblog/2016/07/16/convnet-conv-layer/

Vectorizing python code to numpy

I have the following code snippet (for Hough circle transform):
for r in range(1, 11):
for t in range(0, 360):
trad = np.deg2rad(t)
b = x - r * np.cos(trad)
a = y - r * np.sin(trad)
b = np.floor(b).astype('int')
a = np.floor(a).astype('int')
A[a, b, r-1] += 1
Where A is a 3D array of shape (height, width, 10), and
height and width represent the size of a given image.
My goal is to convert the snippet exclusively to numpy code.
My attempt is this:
arr_r = np.arange(1, 11)
arr_t = np.deg2rad(np.arange(0, 360))
arr_cos_t = np.cos(arr_t)
arr_sin_t = np.sin(arr_t)
arr_rcos = arr_r[..., np.newaxis] * arr_cos_t[np.newaxis, ...]
arr_rsin = arr_r[..., np.newaxis] * arr_sin_t[np.newaxis, ...]
arr_a = (y - arr_rsin).flatten().astype('int')
arr_b = (x - arr_rcos).flatten().astype('int')
Where x and y are two scalar values.
I am having trouble at converting the increment part: A[a,b,r] += 1. I thought of this: A[a,b,r] counts the number of occurrences of the pair (a,b,r), so a clue was to use a Cartesian product (but the arrays are too large).
Any tips or tricks I can use?
Thank you very much!
Edit: after filling A, I need (a,b,r) as argmax(A). The tuple (a,b,r) identifies a circle and its value in A represents the confidence value. So I want that tuple with the highest value in A. This is part of the voting algorithm from Hough circle transform: find circle parameter with unknown radius.

Method #1
Here's one way leveraging broadcasting to get the counts and update A (this assumes the a and b values computed in the intermediate steps are positive ones) -
d0,d1,d2 = A.shape
arr_r = np.arange(1, 11)
arr_t = np.deg2rad(np.arange(0, 360))
arr_b = np.floor(x - arr_r[:,None] * np.cos(arr_t)).astype('int')
arr_a = np.floor(y - arr_r[:,None] * np.sin(arr_t)).astype('int')
idx = (arr_a*d1*d2) + (arr_b * d2) + (arr_r-1)[:,None]
A.flat[:idx.max()+1] += np.bincount(idx.ravel())
# OR A.flat += np.bincount(idx.ravel(), minlength=A.size)
Method #2
Alternatively, we could avoid bincount to replace the last step in approach #1, like so -
idx.ravel().sort()
idx.shape = (-1)
grp_idx = np.flatnonzero(np.concatenate(([True], idx[1:]!=idx[:-1],[True])))
A.flat[idx[grp_idx[:-1]]] += np.diff(grp_idx)
Improvement with numexpr
We could also leverage numexpr module for faster sine, cosine computations, like so -
import numexpr as ne
arr_r2D = arr_r[:,None]
arr_b = ne.evaluate('floor(x - arr_r2D * cos(arr_t))').astype(int)
arr_a = ne.evaluate('floor(y - arr_r2D * sin(arr_t))').astype(int)

np.add(np.array ([arr_a, arr_b, 10]), 1)