Related
I want to create a plot of (T/Tmax vs R/R0) for different values of Pa in a single plot like below. I have written this code that appends values to a list but all values of (T/Tmax vs R/R0) are appended in single list which does not give a good plot. What can I do to have such a plot? Also how can I make an excel sheet from the data from the loop where column 1 is T/Tmax list and column 2,3,4...are corresponding R/R0 values for different pa?
KLMDAT1 = []
KLMDAT2 = []
for j in range(z):
pa[j] = 120000-10000*j
i = 0
R = R0
q = 0
T = 0
while (T<Tmax):
k1 = KLM_RKM(i*dT,R,q,pa[j])
k2 = KLM_RKM((i+0.5)*dT,R,q+0.5*dT*k1,pa[j])
k3 = KLM_RKM((i+0.5)*dT,R,q+0.5*dT*k2,pa[j])
k4 = KLM_RKM((i+1)*dT,R,q+dT*k3,pa[j])
q = q +1/6.0*dT*(k1+2*k2+2*k3+k4)
R = R+dT*q
if(R>0):
KLMDAT1.append(T / Tmax)
KLMDAT2.append(R / R0)
if(R>Rmax):
Rmax = R
if (abs(q)>c or R < 0):
break
T=T+dT
i = i+1
wb.save('KLM.xlsx')
np.savetxt('KLM.csv',[KLMDAT1, KLMDAT2])
plt.plot(KLMDAT1, KLMDAT2)
plt.show()
You are plotting it wrong. Your first variable needs to be T/Tmax. So initialize an empty T list, append T values to it, divide it by Tmax, and then plot twice: first KLMDAT1 and then KLMDAT2. Following pseudocode explains it
KLMDAT1 = []
KLMDAT2 = []
T_list = [] # <--- Initialize T list here
for j in range(z):
...
while (T<Tmax):
...
T=T+dT
T_list.append(T) # <--- Append T here
i = i+1
# ... rest of the code
plt.plot(np.array(T_list)/Tmax, KLMDAT1) # <--- Changed here
plt.plot(np.array(T_list)/Tmax, KLMDAT2) # <--- Changed here
plt.show()
I use Sympy solve() function to solve a large number of equations. All variables in the equations are defined as symbols. Variables can start with the letter P or F. I use solve() to express one specific P variable (the one that I observe) with only F variables, so I use solve() to substitute all other P variables with F variables. The sum of the coefficients before the F variables is ideally 1 or almost 1 (e.g.: 0.99).
This produces good results till a certain point where the number of equations becomes pretty big and also their length. There the Sympy solve() function starts to give me wrong results. The sum of the coefficients becomes negative (e.g. -7,...). It looks like that the solve() function gets problems with substituting any carrying over all variables and their coefficients.
Is there a way to correct this problem?
Dictionary of equations under link: https://drive.google.com/open?id=1VBQucrDU-o1diCd6i4rR3MlRh95qycmK
import json
from sympy import Symbol, Add, Eq, solve
# Get data
# data from link above
with open("C:\\\\Test\\dict.json") as f:
equations = json.load(f)
comp =[]
expressions = []
for p, equation_components in equations.items():
p = Symbol(p)
comp.append(p)
expression = []
for name, multiplier in equation_components.items():
if type(multiplier) == float or type(multiplier) == int:
expression.append(Symbol(name) * multiplier)
else:
expression.append(Symbol(name) * Symbol(multiplier))
expressions.append(Eq(p, Add(*expression)))
# Solution for variable P137807
print("Solving...")
# Works for slice :364 !!!!!
solutions = solve(expressions[:364], comp[:364], simplify=False, rational=False)
# Gives wrong results for slice :366 and above !!!!!
# solutions = solve(expressions[:366], comp[:366], simplify=False, rational=False)
vm_symbol = Symbol("P137807")
solution_1 = solutions[vm_symbol]
print("\n")
print("Solution_1:")
print(solution_1)
print("\n")
#Sum of coefficients
list_sum = []
for i in solution_1.args:
if str(i.args[1]) != "ANaN":
list_sum.append(i.args[0])
coeff_sum = sum(list_sum)
print("Sum:")
print(coeff_sum)
...
I just wanted to mark the problem as solved and provide reference to the solution. Please look at numerical instability when solving n=385 linear equations with Float coefficients #17136.
The solution that worked for me was to use the following solver and not the Sympy solve() function:
def ssolve(eqs, syms):
"""return the solution of linear system of equations
with symbolic coefficients and a unique solution.
Examples
========
>>> eqs=[x-1,x+2*y-z-2,x+z+w-6,2*y+z+x-2]
>>> v=[x,y,z,w]
>>> ssolve(eqs, v)
{x: 1, z: 0, w: 5, y: 1/2}
"""
from sympy.solvers.solveset import linear_coeffs
v = list(syms)
N = len(v)
# convert equations to coefficient dictionaries
print('checking linearity')
d = []
v0 = v + [0]
for e in [i.rewrite(Add) for i in eqs]:
co = linear_coeffs(e, *v)
di = dict([(i, c) for i, c in zip(v0, co) if c or not i])
d.append(di)
print('forward solving')
sol = {}
impl = {}
done = False
while not done:
# check for those that are done
more = set([i for i, di in enumerate(d) if len(di) == 2])
did = 0
while more:
di = d[more.pop()]
c = di.pop(0)
x = list(di)[0]
a = di.pop(x)
K = sol[x] = -c/a
v.remove(x)
changed = True
did += 1
# update everyone else
for j, dj in enumerate(d):
if x not in dj:
continue
dj[0] += dj.pop(x)*K
if len(dj) == 2:
more.add(j)
if did: print('found',did,'definitions')
# solve implicitly for the next variable
dcan = [i for i in d if len(i) > 2]
if not dcan:
done = True
else:
# take shortest first
di = next(ordered(dcan, lambda i: len(i)))
done = False
x = next(ordered(i for i in di if i))
c = di.pop(x)
for k in di:
di[k] /= -c
impl[x] = di.copy()
di.clear()
v.remove(x)
# update everyone else
for j, dj in enumerate(d):
if x not in dj:
continue
done = False
c = dj.pop(x)
for k in impl[x]:
dj[k] = dj.get(k, 0) + impl[x][k]*c
have = set(sol)
sol[0] = 1
while N - len(have):
print(N - len(have), 'to backsub')
for k in impl:
if impl[k] and not set(impl[k]) - have - {0}:
sol[k] = sum(impl[k][vi]*sol[vi] for vi in impl[k])
impl[k].clear()
have.add(k)
sol.pop(0)
return sol
I working on a problem and my goal is to replace variables in the file and the name of the files.
The issue is that I have to change a couple of variables at the same time for all combinations (Generally 24 combinations).
I know how to create of all combinations of strings, but I want to put lists inside and iterate over them.
a = [ 'distance', 'T1', 'T2', 'gamma' ]
new_list = list(itertools.permutations(a, 2))
I created the function to pass my values:
def replace_variables(distance ='0', T1 ='0', T2 = '0', gamma = '0'):
template_new = template.replace('*distance*', distance).replace('*T1*', T1).replace('*T2*', T2).replace('*gamma*', gamma)
input_file = input_name.replace('one','T1'+T1).replace('two','T2'+T2).replace('phi','PHI'+Phi).replace('distance','R'+distance)
return template_new, input_file
when I call that function I can pass only names of variables.
for i in new_list:
elem1 = i[0]
elem2 = i[1]
template_new, input_file =replace_variables(elem1, elem2)
print input_file
Though I need to use lists:
distance = ['-3','+3']
T1 = ['-3', '+3']
T2 = ['-3', '+3']
gamma = ['-3', '+3']
And for each pair of variables change values in a file and a name of file such as:
original file: name_file_R_T1_T2_gamma.txt
will be replaced by:
name_file_3_3_0_0.txt, name_file_3_-3_0_0.txt, name_file_-3_3_0_0.txt,
name_file_3_3_0_0.txt, name_file_3_0_3_0.txt, name_file_3_0_-3_0.txt,
and so forth.
The original template looks like:
template = """
R = 3.0 *distance* cm
THETA1 = 60. *T1* degree
THETA2 = 2.0 *T2* degree
GAMMA = 0 *gamma* degree
"""
and I want to obtain:
template = """
R = 3.0 +3 cm
THETA1 = 60. +3 degree
THETA2 = 2.0 +0 degree
GAMMA = 0 +0 degree
"""
and so forth
I think I almost tackled the above problem:
#!/usr/bin/env python
import itertools
import copy
def replace_variables(i, distance ='0', T1 ='0', T2 = '0', gamma = '0' ):
k_ = copy.deepcopy(i)
k_[0][0] = '-2'
k_[1][0] = '2'
template_new = template.replace('*distance*', distance).replace('*T1*', T1).replace('*T2*', T2).replace('*gamma*', gamma)
input_file = input_name.replace('one','T1'+T1).replace('two','T2'+T2).replace('gamma','gamma'+gamma).replace('distance','R'+distance)
f = open(template_new, 'w')
f.write(template_new)
f.close()
input_name = 'name_file_distance_T1_T2_gamma.txt'
template = """
R = 3.0 *distance* cm
THETA1 = 60. *T1* degree
THETA2 = 2.0 *T2* degree
GAMMA = 0 *gamma* degree
"""
a = [['distance','+2','-2'], ['T1','+2','-2'], ['T2','+2','-2'], ['gamma','+2','-2']]
new_list = list(itertools.permutations(a, 2))
for i in new_list:
replace_variables(i, x, y)
Though I faced 2 problems:
1) My code does not change values of variables (apart from default ones) in the replace_variables function and I'm getting:
name_file_Rdistance_T1T1_T20_gamma0.txt, and so on
I think because of default arguments passed to the function.
2) My function does not create a separated files.
I have a class that was taking in lists of 1's and 0's and performing GF(2) finite field arithmetic operations. It used to work until I tried to make it take the input in polynomial format. As for how the finite arithmetic will be done after fixing the regex issue, I was thinking about overloading the operators.
The actual code in parsePolyToListInput(input) works when outside the class. The problem seems to be in the regex, which errors that it will only take in a string (this makes sense), but does not seem to initialize with self.expr as a parameter (that's a problem). The #staticmethod just before the initialization was an attempt to salvage the unbound error as it the polynomial was passed in, but this is apparently completely wrong. Just to save you time if you decide to look at any of the arithmetic operations, modular inverse does not work (seems to be due to the formatting issue after every iteration of that while loop for division in the function and what the return type is):
import re
class gf2poly:
#binary arithemtic on polynomials
##staticmethod
def __init__(self,expr):
self.expr = expr
#self.expr = [int(i) for i in expr]
self.expr = gf2poly.parsePolyToListInput(self.expr)
def convert(self): #to clarify the input if necessary
convertToString = str(self.expr)
print "expression is %s"%(convertToString)
def id(self): #returns modulus 2 (1,0,0,1,1,....) for input lists
return [int(self.expr[i])%2 for i in range(len(self.expr))]
def listToInt(self): #converts list to integer for later use
result = gf2poly.id(self)
return int(''.join(map(str,result)))
def prepBinary(a,b): #converts to base 2 and orders min and max for use
a = gf2poly.listToInt(a); b = gf2poly.listToInt(b)
bina = int(str(a),2); binb = int(str(b),2)
a = min(bina,binb); b = max(bina,binb);
return a,b
#staticmethod
def outFormat(raw):
raw = str(raw[::-1]); g = [] #reverse binary string for enumeration
[g.append(i) for i,c in enumerate(raw) if c == '1']
processed = "x**"+' + x**'.join(map(str, g[::-1]))
if len(g) == 0: return 0 #return 0 if list empty
return processed #returns result in gf(2) polynomial form
def parsePolyToListInput(poly):
c = [int(i.group(0)) for i in re.finditer(r'\d+', poly)] #re.finditer returns an iterator
#m = max(c)
return [1 if x in c else 0 for x in xrange(max(c), -1, -1)]
#return d
def add(self,other): #accepts 2 lists as parameters
a = gf2poly.listToInt(self); b = gf2poly.listToInt(other)
bina = int(str(a),2); binb = int(str(b),2)
m = bina^binb; z = "{0:b}".format(m)
return z #returns binary string
def subtract(self,other): #basically same as add() but built differently
result = [self.expr[i] ^ other.expr[i] for i in range(len(max(self.expr,other.expr)))]
return int(''.join(map(str,result)))
def multiply(a,b): #a,b are lists like (1,0,1,0,0,1,....)
a,b = gf2poly.prepBinary(a,b)
g = []; bitsa = "{0:b}".format(a)
[g.append((b<<i)*int(bit)) for i,bit in enumerate(bitsa)]
m = reduce(lambda x,y: x^y,g); z = "{0:b}".format(m)
return z #returns product of 2 polynomials in gf2
def divide(a,b): #a,b are lists like (1,0,1,0,0,1,....)
a,b = gf2poly.prepBinary(a,b)
bitsa = "{0:b}".format(a); bitsb = "{0:b}".format(b)
difflen = len(str(bitsb)) - len(str(bitsa))
c = a<<difflen; q=0
while difflen >= 0 and b != 0: #a is divisor, b is dividend, b/a
q+=1<<difflen; b = b^c # b/a because of sorting in prep
lendif = abs(len(str(bin(b))) - len(str(bin(c))))
c = c>>lendif; difflen -= lendif
r = "{0:b}".format(b); q = "{0:b}".format(q)
return r,q #returns r remainder and q quotient in gf2 division
def remainder(a,b): #separate function for clarity when calling
r = gf2poly.divide(a,b)[0]; r = int(str(r),2)
return "{0:b}".format(r)
def quotient(a,b): #separate function for clarity when calling
q = gf2poly.divide(a,b)[1]; q = int(str(q),2)
return "{0:b}".format(q)
def extendedEuclideanGF2(a,b): # extended euclidean. a,b are GF(2) polynomials in list form
inita,initb=a,b; x,prevx=0,1; y,prevy = 1,0
while sum(b) != 0:
q = gf2poly.quotient(a,b);
q = list(q); q = [int(x) for x in q]
#q = list(q);
#q = tuple([int(i) for i in q])
q = gf2poly(q)
a,b = b,gf2poly.remainder(a,b);
#a = map(list, a);
#b = [list(x) for x in a];
#a = [int(x) for x in a]; b = [int(x) for x in b];
b = list(b); b = [int(x) for x in b]
#b = list(b);
#b = tuple([int(i) for i in b])
b = gf2poly(b)
#x,prevx = (prevx-q*x, x);
#y,prevy=(prevy-q*y, y)
print "types ",type(q),type(a),type(b)
#q=a//b; a,b = b,a%b; x,prevx = (prevx-q*x, x); y,prevy=(prevy-q*y, y)
#print("%d * %d + %d * %d = %d" % (inita,prevx,initb,prevy,a))
return a,prevx,prevy # returns gcd of (a,b), and factors s and t
def modular_inverse(a,mod): # where a,mod are GF(2) polynomials in list form
gcd,s,t = gf2poly.extendedEuclideanGF2(a,mod); mi = gf2poly.remainder(s,mod)
#gcd,s,t = ext_euc_alg_i(a,mod); mi = s%mod
if gcd !=1: return False
#print ("%d * %d mod %d = 1"%(a,mi,mod))
return mi # returns modular inverse of a,mod
I usually test it with this input:
a = x**14 + x**1 + x**0
p1 = gf2poly(a)
b = x**6 + x**2 + x**1
p2 = gf2poly(b)
The first thing you might notice about my code is that it's not very good. There are 2 reasons for that:
1) I wrote it so that the 1st version could do work in the finite field GF(2), and output in polynomial format. Then the next versions were supposed to be able to take polynomial inputs, and also perform the crucial 'modular inverse' function which is not working as planned (this means it's actually not working at all).
2) I'm teaching myself Python (I'm actually teaching myself programming overall), so any constructive criticism from pro Python programmers is welcome as I'm trying to break myself of beginner habits as quickly as possible.
EDIT:
Maybe some more of the code I've been testing with will help clarify what works and what doesn't:
t1 = [1,1,1]; t2 = [1,0,1]; t3 = [1,1]; t4 = [1, 0, 1, 1, 1, 1, 1]
t5 = [1,1,1,1]; t6 = [1,1,0,1]; t7 = [1,0,1,1,0]
f1 = gf2poly(t1); f2 = gf2poly(t2); f3 = gf2poly(t3); f4 = gf2poly(t4)
f5 = gf2poly(t5);f6 = gf2poly(t6);f7 = gf2poly(t7)
##print "subtract: ",a.subtract(b)
##print "add: ",a.add(b)
##print "multiply: ",gf2poly.multiply(f1,f3)
##print "multiply: ",gf2poly.multiply(f1,f2)
##print "multiply: ",gf2poly.multiply(f3,f4)
##print "degree a: ",a.degree()
##print "degree c: ",c.degree()
##print "divide: ",gf2poly.divide(f1,b)
##print "divide: ",gf2poly.divide(f4,a)
##print "divide: ",gf2poly.divide(f4,f2)
##print "divide: ",gf2poly.divide(f2,a)
##print "***********************************"
##print "quotient: ",gf2poly.quotient(f2,f5)
##print "remainder: ",gf2poly.remainder(f2,f5)
##testq = gf2poly.quotient(f4,f2)
##testr = gf2poly.remainder(f4,f2)
##print "quotient: ",testq,type(testq)
##print "remainder: ",testr,type(testr)
##print "***********************************"
##print "outFormat testp: ",gf2poly.outFormat(testq)
##print "outFormat testr: ",gf2poly.outFormat(testr)
##print "***********************************"
#print "gf2poly.modular_inverse(): ",gf2poly.modular_inverse(f2,f3)
print "p1 ",p1 #,type(f2),type(f3)
#print "parsePolyToListInput ",gf2poly.parsePolyToListInput(a)
Part of your problem is that you haven't declared self as an argument for parsePolyToListInput. When you call a method, the instance you call it on is implicitly bound as the first argument. Naming the first argument self is a convention, not a strict requirement - the instance is being bound to poly, which you then try to run a regexp over.
It looks me like there's some confusion in your design here about what's behavior of individual instances of the class and what's class-level or module-level behavior. In Python, it's perfectly acceptable to leave something that doesn't take an instance of a class as a parameter defined as a module-level function rather than shoehorning it in awkwardly. parsePolyToListInput might be one such function.
Your add implementation, similarly, has a comment saying it "accepts 2 lists as parameters". In fact, it's going to get a gf2poly instance as its first argument - this is probably right if you're planning to do operator overloading, but it means the second argument should also be a gf2poly instance as well.
EDIT:
Yeah, your example code shows a breakdown between class behavior and instance behavior. Either your multiply call should look something like this:
print "multiply: ",f1.multiply(f3)
Or multiply shouldn't be a method at all:
gfpoly.py:
def multiply(f1, f2):
a,b = prepBinary(a,b)
g = []; bitsa = "{0:b}".format(a)
[g.append((b<<i)*int(bit)) for i,bit in enumerate(bitsa)]
m = reduce(lambda x,y: x^y,g); z = "{0:b}".format(m)
return z #returns product of 2 polynomials in gf2
That latter approach is, for instance, how the standard math library does things.
The advantage of defining a multiplication method is that you could name it appropriately (http://docs.python.org/2/reference/datamodel.html#special-method-names) and use it with the * operator:
print "multiply: ",f1 *f3
I am writing a function to select randomly elements stored in a dictionary:
import random
from liblas import file as lasfile
from collections import defaultdict
def point_random_selection(list,k):
try:
sample_point = random.sample(list,k)
except ValueError:
sample_point = list
return(sample_point)
def world2Pixel_Id(x,y,X_Min,Y_Max,xDist,yDist):
col = int((x - X_Min)/xDist)
row = int((Y_Max - y)/yDist)
return("{0}_{1}".format(col,row))
def point_GridGroups(inFile,X_Min,Y_Max,xDist,yDist):
Groups = defaultdict(list)
for p in lasfile.File(inFile,None,'r'):
id = world2Pixel_Id(p.x,p.y,X_Min,Y_Max,xDist,yDist)
Groups[id].append(p)
return(Groups)
where k is the number of element to select. Groups is the dictionary
file_out = lasfile.File("outPut",mode='w',header= h)
for m in Groups.iteritems():
# select k point for each dictionary key
point_selected = point_random_selection(m[1],k)
for l in xrange(len(point_selected)):
# save the data
file_out.write(point_selected[l])
file_out.close()
My problem is that this approach is extremely slow (for file of ~800 Mb around 4 days)
You could try and update your samples as you read the coordinates. This at least saves you from having to store everything in memory before running your sample. This is not guaranteed to make things faster.
The following is based off of BlkKnght's excellent answer to build a random sample from file input without retaining all the lines. This just expanded it to keep multiple samples instead.
import random
from liblas import file as lasfile
from collections import defaultdict
def world2Pixel_Id(x, y, X_Min, Y_Max, xDist, yDist):
col = int((x - X_Min) / xDist)
row = int((Y_Max - y) / yDist)
return (col, row)
def random_grouped_samples(infile, n, X_Min, Y_Max, xDist, yDist):
"""Select up to n points *per group* from infile"""
groupcounts = defaultdict(int)
samples = defaultdict(list)
for p in lasfile.File(inFile, None, 'r'):
id = world2Pixel_Id(p.x, p.y, X_Min, Y_Max, xDist, yDist)
i = groupcounts[id]
r = random.randint(0, i)
if r < n:
if i < n:
samples[id].insert(r, p) # add first n items in random order
else:
samples[id][r] = p # at a decreasing rate, replace random items
groupcounts[id] += 1
return samples
The above function takes inFile and your boundary coordinates, as well as the sample size n, and returns grouped samples that have at most n items in each group, picked uniformly.
Because all you use the id for is as a group key, I reduced it to only calculating the col, row tuple, there is no need to make it a string.
You can write these out to a file with:
file_out = lasfile.File("outPut",mode='w',header= h)
for group in samples.itervalues():
for p in group:
file_out.write(p)
file_out.close()