We Keep Coding

Sympy : simplification with expression substitution

I have several expressions involving the norm or norm squared of a vector u. I'd like to simplify these expressions by substituting a known value for the norm of u. However, it seems that obvious expressions involving even simple multiples of the norm are not simplified.
As an example, this code does what I would expect :
import sympy as sp
u1,u2,u3 = sp.symbols('u_1, u_2, u_3',real=True,positive=True)
utu = u1**2 + u2**2 + u3**2
print("Ex. 1")
print(utu.subs(utu,1))
This produces the expected output
Ex. 1
1
However, 2*utu does not simplify in the way I would expect :
print("Ex 2")
print((2*utu).subs(utu,1))
Ex 2
2*u_1**2 + 2*u_2**2 + 2*u_3**2
I can explicitly force the substitution with this :
print("Ex 3")
print((2*utu).subs(2*utu,2))
which produces the expected output :
Ex 3
2
Ideally, I'd like to substitute under a norm function, but the run into the same issue.
u = sp.Matrix(3, 1, [u1,u2,u3])
print("Ex 4")
print(u.norm().subs(utu,1))
print("Ex 5")
print((2*u).norm().subs(utu,1))
print("Ex 6")
print((2*u).norm().subs(4*utu,4))
which produces
Ex 4
1
Ex 5
sqrt(4*u_1**2 + 4*u_2**2 + 4*u_3**2)
Ex 6
2
Are there tricks I am missing that will catch these obvious (to me at least - maybe not to Sympy?) simplifications? I've tried factor and expand, without much luck.

Let's analyze this expression:
expr = 2*utu
# out: 2*u_1**2 + 2*u_2**2 + 2*u_3**2
The multiplication has been evaluated. This is SymPy's default behavior: it evaluates things. We can work with the expression manipulation functions to achieve our goal.
For example:
expr = collect_const(expr)
# out: 2*(u_1**2 + u_2**2 + u_3**2)
expr.subs(utu, 1)
# out: 2
Another example:
expr = (2 * u).norm()
# out: sqrt(4*u_1**2 + 4*u_2**2 + 4*u_3**2)
expr = expr.simplify() # Note that expr.factor() produces the same result with this expression
# out: 2*sqrt(u_1**2 + u_2**2 + u_3**2)
expr.subs(utu, 1)
# out: 2
If you play (and modify) with these examples, you will realize that the same result can be achieved with different functions (factor, simplify, collect, collect_const, ...), but even one little change in the expression might prevent one function from "doing its work", while others might be able to. Expression manipulation is kind of an art that one should practice (a lot).
For completeness, I'm going to show you UnevaluatedExpr, which allows a particular expression to remain unevaluated during expression manipulation, though it might not always be the best choice:
n = UnevaluatedExpr(utu)
# out: u_1**2 + u_2**2 + u_3**2
expr = 4 * n
# out: 4*(u_1**2 + u_2**2 + u_3**2)
Note that SymPy didn't proceed with the full evaluation. Now:
expr.subs(utu, 1)
# out: 4*1
Why is there a 4*1 instead of 4? The 1 refers to the UnevaluateExpr object that we created earlier: to evaluate it we can use the doit() method:
expr.subs(utu, 1).doit()
# 4
Keep in mind that while using UnevaluateExpr, the expression becomes non-commutative (I think it's a bug with SymPy), which will prevent other functions to produce the expected results.

Substituting compound expressions is problematic. For the most part you should only expect subs to work if the expression to be replaced is known to always appear literally as part of the expression tree that you are substituting into. When possible then it is better to rearrange for a single symbol like:
In [10]: utu
Out[10]:
2 2 2
u₁ + u₂ + u₃
In [11]: (2*utu).subs(u1**2, 1 - u2**2 - u3**2)
Out[11]: 2
Even here we are substituting for a power of a symbol (u1**2) which is potentially fragile if we can't be sure that exactly that power will always appear in the expression. More generally there are functions that can simplify expressions based on knowing some polynomial relation e.g. ratsimpmodprime:
In [16]: e = (1 - u1**2) / (u1**2 + u2**2 + u3**2)
In [17]: e
Out[17]:
2
1 - u₁
───────────────
2 2 2
u₁ + u₂ + u₃
In [18]: ratsimpmodprime(e, [u1**2 + u2**2 + u3**2 - 1])
Out[18]:
2 2
u₂ + u₃
Other possibilities could be using resultants or Groebner bases to do similar things. Note that u.norm() has a square root which is symbolically awkward so it is better to work with the square of the norm (same as when working on pen and paper):
In [20]: ratsimpmodprime((2*u).norm()**2, [u1**2 + u2**2 + u3**2 - 1])
Out[20]: 4
Also if you just want a more powerful version of subs then you can use replace but with patterns:
In [21]: a = Wild('a')
In [22]: p = a*u1**2 + a*u2**2 + a*u3**2
In [23]: (2*utu).replace(p, a)
Out[23]: 2
In [24]: (2*u).norm().replace(p, a)
Out[24]: 2

Both solid answers already. If you have an arbitrary expression that you expect to be a factor in another, factor_terms is what I try first to make that factor appear. It will collect common factors without doing factoring. But if this doesn't work and you know you have a factor, div is a nice way to check and see the expression with the factor removed:
>>> expr = 2*(x + y)
>>> factor_terms(expr)
2*(x + y)
>>> e2 = expand(expr*(x -y)) # 2*x**2 - y**2
>>> factor_terms(e2)
2*(x**2 - y**2)
>>> div(_,-x-y)
(-2*x + 2*y, 0)
>>> _[0]*z # if you wanted to replace factor -x-y with z
z*(-2*x + 2*y)

SymPy does not recognize equal values

Simple calculations in SymPy quickly create unwieldy results like the three should_be values below.
Comparisons to the correct values give False (although math.isclose gives True).
from sympy import sqrt
phi = (1 + sqrt(5)) / 2
should_be_phi = -(1/2 + sqrt(5)/2)**2 + (1/2 + sqrt(5)/2)**3
should_be_half = -sqrt(5)/8 + 1/8 + (1/2 + sqrt(5)/2)**2/4
should_be_one = -sqrt(5)/4 + 1/4 + (1/2 + sqrt(5)/2)**2/2
print(should_be_phi == phi, should_be_half == 1/2, should_be_one == 1)
These are the same formulas formatted by Wolfram Alpha:
phi:
half:
one:
should_be_phi was created as phi**3 - phi**2 btw.
Currently I always copy these monstrosities to Wolfram Alpha to get decent formulas and to remove duplicates.
Do you also get False for each comparison? I use Python 3.6.8 and SymPy 1.4.
Is there a way do do symbolic calculations in Python that actually works?
SymPy seems to be unable to do the things it is supposedly made for.

I presume that what you want is for those expressions to be simplified so just use the simplify function:
In [6]: from sympy import *
In [7]: phi = (1 + sqrt(5)) / 2
In [8]: should_be_phi = -(S(1)/2 + sqrt(5)/2)**2 + (S(1)/2 + sqrt(5)/2)**3
In [9]: should_be_phi
Out[9]:
2 3
⎛1 √5⎞ ⎛1 √5⎞
- ⎜─ + ──⎟ + ⎜─ + ──⎟
⎝2 2 ⎠ ⎝2 2 ⎠
In [10]: simplify(should_be_phi)
Out[10]:
1 √5
─ + ──
2 2
Note that you should use S(1)/2 rather than 1/2 which gives a float.
If you want to compare expressions then the obvious way would be to use == but that is for "structural equality" in SymPy. What that means is that expr1 == expr2 will give True only when the expressions are in the exact same form. If you want to test for mathematical equality then you should use Eq(lhs, rhs) or simplify(lhs-rhs):
In [11]: should_be_phi == phi # Expressions are not in the same form
Out[11]: False
In [12]: Eq(should_be_phi, phi)
Out[12]: True
In [13]: simplify(should_be_phi - phi)
Out[13]: 0
Is there a way do do symbolic calculations in Python that actually works?
SymPy seems to be unable to do the things it is supposedly made for.
Unlike Wolfram Alpha, SymPy is not designed to be usable or understandable to someone who has not read any of the documentation. Your questions above would be answered if you had read the first few pages of the SymPy tutorial:
https://docs.sympy.org/latest/tutorial/index.html#tutorial

solving linear system 6x6 matrix using sympy

i am trying to solve a matrix that has 6x6 matrices as it's entries(elements)
i tried multiplying the inverse of gen to the solution matrix, but i don't trust the correctness of the answer am getting.
from sympy import Eq, solve_linear_system, Matrix,count_ops,Mul,horner
import sympy as sp
a, b, c, d, e,f = sp.symbols('a b c d e f')
ad = Matrix(([43.4,26.5,115,-40.5,52.4,0.921],
[3.78,62.9,127,-67.6,110,4.80],
[41.25,75.0,213,-88.9, 131, 5.88],
[-10.6,-68.4,-120,64.6,-132,-8.49],
[6.5,74.3,121,-72.8,179,29.7],
[1.2,30.7,49.7,-28.7,91,29.9]))
fb= Matrix(([1,0,0,0,0,0],
[0,1,0,0,0,0],
[0,0,1,0,0,0],
[0,0,0,1,0,0],
[0,0,0,0,1,0],
[0,0,0,0,0,1]))
ab = Matrix(([-0.0057],
[0.0006],
[-0.0037],
[0.0009],
[0.0025],
[0.0042]))
az = sp.symbols('az')
bz = sp.symbols('bz')
fz = sp.symbols('fz')
gen = Matrix(([az, fz, 0, 0, 0, 0,bz],
[fz,az,fz,0,0,0,bz],
[0,fz,az,fz,0,0,bz],
[0,0,fz,az,fz,0,bz],
[0,0,0,fz,az,fz,bz],
[0,0,0,0,fz,az,bz]))
answer = solve_linear_system(gen,a,b,c,d,e,f)
first_solution = answer[a]
df = count_ops(first_solution)
print(df,first_solution)
disolved = zip(first_solution.simplify().as_numer_denom(),(1,-1))
dft = Mul(*[horner(b)**e for b,e in disolved])
dff = count_ops(dft)
print(dff,dft)
_1st_solution = dft.subs({az:ad,fz:fb,bz:ab},simultaneous = True).doit()
print(_1st_solution)
when i ran my code it raised sympy.matrices.common.ShapeError

You have to be careful when using horner with expressions containing commutative symbols that are actually noncommutative (in your case because they represent matrices). Your dft expression is
(az**2*bz - bz*fz**2)/(az*(az*(az + fz) - 2*fz**2) - fz**3)
but should maybe be
(az**2 - fz**2)*(az*(az*(az + fz) - 2*fz**2) - fz**3)**(-1)*bz
You would have received a correct expression if you had created the symbols as noncommutative (as shown below).
But you can't use horner with non-commutative symbols, so I just rearranged the expression by hand; you will have to check to see that the ordering is right. As an alternative to doing the factoring by hand you might also try using factor_nc to help you -- but it won't handle horner like expression factoring:
>>> ax, bz, fz = symbols('az bz fz, commutative=False)
>>> (az**2*bz - fz**2*bz)
az**2*bz - fz**2*bz
>>> factor_nc(_)
(az**2 - fz**2)*bz

Derivative of summations

I am using sympy from time to time, but am not very good at it. At the moment I am stuck with defining a list of indexed variables, i.e. n1 to nmax and performing a summation on it. Then I want to be able to take the derivative:
So far I tried the following:
numSpecies = 10
n = IndexedBase('n')
i = symbols("i",cls=Idx)
nges = summation(n[i],[i,1,numSpecies])
However, if i try to take the derivative with respect to one variable, this fails:
diff(nges,n[5])
I also tried to avoid working with IndexedBase.
numSpecies = 10
n = symbols('n0:%d'%numSpecies)
k = symbols('k',integer=True)
ntot = summation(n[k],[k,0,numSpecies])
However, here already the summation fails because mixing python tuples and the sympy summation.
How I can perform indexedbase derivatives or some kind of workaround?

With SymPy's development version, your example works.
To install SymPy's development version, just pull it down with git:
git clone git://github.com/sympy/sympy.git
cd sympy
Then run python from that path or set the PYTHONPATH to include that directory before Python's default installation.
Your example on the development version:
In [3]: numSpecies = 10
In [4]: n = IndexedBase('n')
In [5]: i = symbols("i",cls=Idx)
In [6]: nges = summation(n[i],[i,1,numSpecies])
In [7]: nges
Out[7]: n[10] + n[1] + n[2] + n[3] + n[4] + n[5] + n[6] + n[7] + n[8] + n[9]
In [8]: diff(nges,n[5])
Out[8]: 1
You could also use the contracted form of summation:
In [9]: nges_uneval = Sum(n[i], [i,1,numSpecies])
In [10]: nges_uneval
Out[10]:
10
___
╲
╲ n[i]
╱
╱
‾‾‾
i = 1
In [11]: diff(nges_uneval, n[5])
Out[11]:
10
___
╲
╲ δ
╱ 5,i
╱
‾‾‾
i = 1
In [12]: diff(nges_uneval, n[5]).doit()
Out[12]: 1
Also notice that in the next SymPy version you will be able to derive symbols with symbolic indices:
In [13]: j = symbols("j")
In [13]: diff(n[i], n[j])
Out[13]:
δ
j,i
Where you get the Kronecker delta.
If you don't feel like installing the SymPy development version, just wait for the next full version (probably coming out this autumn), it will support derivatives of IndexedBase.

I don't know why the IndexedBase approach does not work (I would be interested to know also). You can, however, do the following:
import sympy as sp
numSpecies = 10
n = sp.symbols('n0:%d'%numSpecies) # note that n equals the tuple (n0, n1, ..., n9)
ntot = sum(n) # sum elements of n using the standard
# Python function for summing tuple elements
#ntot = sp.Add(*n) # same result using Sympy function
sp.diff(ntot, n[5])

I'm not clear about what you want to do. However, perhaps this will help. Edited in response to two comments received.
from sympy import *
nspecies = 10
[var('n%s'%_) for _ in range(nspecies)]
expr = sympify('+'.join(['n%s'%_ for _ in range(nspecies)]))
expr
print ( diff(expr,n1) )
expr = sympify('n0**n1+n1**n2')
expr
print ( diff(expr,n1) )
Only the first expression responds to the original question. This is the output.
1
n0**n1*log(n0) + n1**n2*n2/n1

Smart rewriting of expressions in sympy

I found this to be tricky to explain, but I'll do my best through an example.
Consider the expression assigned to the variable grad below
from sympy import *
a, x, b = symbols("a x b")
y_pred = a * x
loss = log(1 + exp(- b * y_pred))
grad = diff(loss, x, 1)
grad has the following expression:
-a*b*exp(-a*b*x)/(1 + exp(-a*b*x))
Now I want to manipulate grad in two ways.
1) I want sympy to try rewrite the expression grad such that none of its terms look like
exp(-a*b*x)/(1 + exp(-a*b*x)).
2) I also want it to try to rewrite the expression such that it has at least one term that look like this 1./(1 + exp(a*b*x)).
So at the end, grad becomes:
-a*b/(1 + exp(a*b*x)
Note that 1./(1 + exp(a*b*x)) is equivalent to exp(-a*b*x)/(1 + exp(-a*b*x)) but I don't want to mention that to sympy explicitly :).
I'm not sure if this is feasible at all, but it would be interesting to know whether it's possible to do this to some extent.

cancel does this
In [16]: cancel(grad)
Out[16]:
-a⋅b
──────────
a⋅b⋅x
ℯ + 1
This works because it sees the expression as -a*b*(1/A)/(1 + 1/A), where A = exp(a*b*x), and cancel rewrites rational functions as canceled p/q (see the section on cancel in the SymPy tutorial for more information).
Note that this only works because it uses A = exp(a*b*x) instead of A = exp(-a*b*x). So for instance, cancel won't do the similar simplification here
In [17]: cancel(-a*b*exp(a*b*x)/(1 + exp(a*b*x)))
Out[17]:
a⋅b⋅x
-a⋅b⋅ℯ
────────────
a⋅b⋅x
ℯ + 1

Are you just looking for simplify?
>>> grad
-a*b*exp(-a*b*x)/(1 + exp(-a*b*x))
>>> simplify(grad)
-a*b/(exp(a*b*x) + 1)