I am given a bunch of expressions in prefix notation in an ANSI text file. I would like to produce another ANSI text file containing the step-by-step evaluation of these expressions. For example:
- + ^ x 2 ^ y 2 1
should be turned into
t1 = x^2
t2 = y^2
t3 = t1 + t2
t4 = t3 - 1
t4 is the result
I also have to identify common subexpressions. For example given
expression_1: z = ^ x 2
expression_2: - + z ^ y 2 1
expression_3: - z y
I have to generate an output saying that x appears in expressions 1, 2 and 3 (through z).
I have to identify dependecies: expression_1 depends only on x, expression_2 depends on x and y, etc.
The original problem is more difficult than the examples above and I have no control over the input format, it is in prefix notation in a much more complicated way than the above ones.
I already have a working implementation in C++ however it is a lot of pain doing such things in C++.
What programming language is best suited for these type problems?
Could you recommend a tutorial / website / book where I could start?
What keywords should I look for?
UPDATE: Based on the answers, the above examples are somewhat unfortunate, I have unary, binary and n-ary operators in the input. (If you are wondering, exp is an unary operator, sum over a range is an n-ary operator.)
To give you an idea how this would look like in Python, here is some example code:
operators = "+-*/^"
def parse(it, count=1):
token = next(it)
if token in operators:
op1, count = parse(it, count)
op2, count = parse(it, count)
tmp = "t%s" % count
print tmp, "=", op1, token, op2
return tmp, count + 1
return token, count
s = "- + ^ x 2 ^ y 2 1"
a = s.split()
res, dummy = parse(iter(a))
print res, "is the result"
The output is the same as your example output.
This example aside, I think any of the high-level languages you listed are almost equally suited for the task.
The sympy python package does symbolic algebra, including common subexpression elimination and generating evaluation steps for a set of expressions.
See: http://docs.sympy.org/dev/modules/rewriting.html (Look at the cse method at the bottom of the page).
The Python example is elegantly short, but I suspect that you won't actually get enough control over your expressions that way. You're much better off actually building an expression tree, even though it takes more work, and then querying the tree. Here's an example in Scala (suitable for cutting and pasting into the REPL):
object OpParser {
private def estr(oe: Option[Expr]) = oe.map(_.toString).getOrElse("_")
case class Expr(text: String, left: Option[Expr] = None, right: Option[Expr] = None) {
import Expr._
def varsUsed: Set[String] = text match {
case Variable(v) => Set(v)
case Op(o) =>
left.map(_.varsUsed).getOrElse(Set()) ++ right.map(_.varsUsed).getOrElse(Set())
case _ => Set()
}
def printTemp(n: Int = 0, depth: Int = 0): (String,Int) = text match {
case Op(o) =>
val (tl,nl) = left.map(_.printTemp(n,depth+1)).getOrElse(("_",n))
val (tr,nr) = right.map(_.printTemp(nl,depth+1)).getOrElse(("_",n))
val t = "t"+(nr+1)
println(t + " = " + tl + " " + text + " " + tr)
if (depth==0) println(t + " is the result")
(t, nr+1)
case _ => (text, n)
}
override def toString: String = {
if (left.isDefined || right.isDefined) {
"(" + estr(left) + " " + text + " " + estr(right) + ")"
}
else text
}
}
object Expr {
val Digit = "([0-9]+)"r
val Variable = "([a-z])"r
val Op = """([+\-*/^])"""r
def parse(s: String) = {
val bits = s.split(" ")
val parsed = (
if (bits.length > 2 && Variable.unapplySeq(bits(0)).isDefined && bits(1)=="=") {
parseParts(bits,2)
}
else parseParts(bits)
)
parsed.flatMap(p => if (p._2<bits.length) None else Some(p._1))
}
def parseParts(as: Array[String], from: Int = 0): Option[(Expr,Int)] = {
if (from >= as.length) None
else as(from) match {
case Digit(n) => Some(Expr(n), from+1)
case Variable(v) => Some(Expr(v), from+1)
case Op(o) =>
parseParts(as, from+1).flatMap(lhs =>
parseParts(as, lhs._2).map(rhs => (Expr(o,Some(lhs._1),Some(rhs._1)), rhs._2))
)
case _ => None
}
}
}
}
This may be a little much to digest all at once, but then again, this does rather a lot.
Firstly, it's completely bulletproof (note the heavy use of Option where a result might fail). If you throw garbage at it, it will just return None. (With a bit more work, you could make it complain about the problem in an informative way--basically the case Op(o) which then does parseParts nested twice could instead store the results and print out an informative error message if the op didn't get two arguments. Likewise, parse could complain about trailing values instead of just throwing back None.)
Secondly, when you're done with it, you have a complete expression tree. Note that printTemp prints out the temporary variables you wanted, and varsUsed lists the variables used in a particular expression, which you can use to expand to a full list once you parse multiple lines. (You might need to fiddle with the regexp a little if your variables can be more than just a to z.) Note also that the expression tree prints itself out in normal infix notation. Let's look at some examples:
scala> OpParser.Expr.parse("4")
res0: Option[OpParser.Expr] = Some(4)
scala> OpParser.Expr.parse("+ + + + + 1 2 3 4 5 6")
res1: Option[OpParser.Expr] = Some((((((1 + 2) + 3) + 4) + 5) + 6))
scala> OpParser.Expr.parse("- + ^ x 2 ^ y 2 1")
res2: Option[OpParser.Expr] = Some((((x ^ 2) + (y ^ 2)) - 1))
scala> OpParser.Expr.parse("+ + 4 4 4 4") // Too many 4s!
res3: Option[OpParser.Expr] = None
scala> OpParser.Expr.parse("Q#$S!M$#!*)000") // Garbage!
res4: Option[OpParser.Expr] = None
scala> OpParser.Expr.parse("z =") // Assigned nothing?!
res5: Option[OpParser.Expr] = None
scala> res2.foreach(_.printTemp())
t1 = x ^ 2
t2 = y ^ 2
t3 = t1 + t2
t4 = t3 - 1
t4 is the result
scala> res2.map(_.varsUsed)
res10: Option[Set[String]] = Some(Set(x, y))
Now, you could do this in Python also without too much additional work, and in a number of the other languages besides. I prefer to use Scala, but you may prefer otherwise. Regardless, I do recommend creating the full expression tree if you want to retain maximum flexibility for handling tricky cases.
Prefix notation is really simple to do with plain recursive parsers. For instance:
object Parser {
val Subexprs = collection.mutable.Map[String, String]()
val Dependencies = collection.mutable.Map[String, Set[String]]().withDefaultValue(Set.empty)
val TwoArgsOp = "([-+*/^])".r // - at the beginning, ^ at the end
val Ident = "(\\p{Alpha}\\w*)".r
val Literal = "(\\d+)".r
var counter = 1
def getIdent = {
val ident = "t" + counter
counter += 1
ident
}
def makeOp(op: String) = {
val op1 = expr
val op2 = expr
val ident = getIdent
val subexpr = op1 + " " + op + " " + op2
Subexprs(ident) = subexpr
Dependencies(ident) = Dependencies(op1) ++ Dependencies(op2) + op1 + op2
ident
}
def expr: String = nextToken match {
case TwoArgsOp(op) => makeOp(op)
case Ident(id) => id
case Literal(lit) => lit
case x => error("Unknown token "+x)
}
def nextToken = tokens.next
var tokens: Iterator[String] = _
def parse(input: String) = {
tokens = input.trim split "\\s+" toIterator;
counter = 1
expr
if (tokens.hasNext)
error("Input not fully parsed: "+tokens.mkString(" "))
(Subexprs, Dependencies)
}
}
This will generate output like this:
scala> val (subexpressions, dependencies) = Parser.parse("- + ^ x 2 ^ y 2 1")
subexpressions: scala.collection.mutable.Map[String,String] = Map(t3 -> t1 + t2, t4 -> t3 - 1, t1 -> x ^ 2, t2 -> y ^ 2)
dependencies: scala.collection.mutable.Map[String,Set[String]] = Map(t3 -> Set(x, y, t2, 2, t1), t4 -> Set(x, y, t3, t2, 1, 2, t1), t1 -> Set(x, 2), t
2 -> Set(y, 2))
scala> subexpressions.toSeq.sorted foreach println
(t1,x ^ 2)
(t2,y ^ 2)
(t3,t1 + t2)
(t4,t3 - 1)
scala> dependencies.toSeq.sortBy(_._1) foreach println
(t1,Set(x, 2))
(t2,Set(y, 2))
(t3,Set(x, y, t2, 2, t1))
(t4,Set(x, y, t3, t2, 1, 2, t1))
This can be easily expanded. For instance, to handle multiple expression statements you can use this:
object Parser {
val Subexprs = collection.mutable.Map[String, String]()
val Dependencies = collection.mutable.Map[String, Set[String]]().withDefaultValue(Set.empty)
val TwoArgsOp = "([-+*/^])".r // - at the beginning, ^ at the end
val Ident = "(\\p{Alpha}\\w*)".r
val Literal = "(\\d+)".r
var counter = 1
def getIdent = {
val ident = "t" + counter
counter += 1
ident
}
def makeOp(op: String) = {
val op1 = expr
val op2 = expr
val ident = getIdent
val subexpr = op1 + " " + op + " " + op2
Subexprs(ident) = subexpr
Dependencies(ident) = Dependencies(op1) ++ Dependencies(op2) + op1 + op2
ident
}
def expr: String = nextToken match {
case TwoArgsOp(op) => makeOp(op)
case Ident(id) => id
case Literal(lit) => lit
case x => error("Unknown token "+x)
}
def assignment: Unit = {
val ident = nextToken
nextToken match {
case "=" =>
val tmpIdent = expr
Dependencies(ident) = Dependencies(tmpIdent)
Subexprs(ident) = Subexprs(tmpIdent)
Dependencies.remove(tmpIdent)
Subexprs.remove(tmpIdent)
case x => error("Expected assignment, got "+x)
}
}
def stmts: Unit = while(tokens.hasNext) tokens.head match {
case TwoArgsOp(_) => expr
case Ident(_) => assignment
case x => error("Unknown statement starting with "+x)
}
def nextToken = tokens.next
var tokens: BufferedIterator[String] = _
def parse(input: String) = {
tokens = (input.trim split "\\s+" toIterator).buffered
counter = 1
stmts
if (tokens.hasNext)
error("Input not fully parsed: "+tokens.mkString(" "))
(Subexprs, Dependencies)
}
}
Yielding:
scala> val (subexpressions, dependencies) = Parser.parse("""
| z = ^ x 2
| - + z ^ y 2 1
| - z y
| """)
subexpressions: scala.collection.mutable.Map[String,String] = Map(t3 -> z + t2, t5 -> z - y, t4 -> t3 - 1, z -> x ^ 2, t2 -> y ^ 2)
dependencies: scala.collection.mutable.Map[String,Set[String]] = Map(t3 -> Set(x, y, t2, 2, z), t5 -> Set(x, 2, z, y), t4 -> Set(x, y, t3, t2, 1, 2, z
), z -> Set(x, 2), t2 -> Set(y, 2))
scala> subexpressions.toSeq.sorted foreach println
(t2,y ^ 2)
(t3,z + t2)
(t4,t3 - 1)
(t5,z - y)
(z,x ^ 2)
scala> dependencies.toSeq.sortBy(_._1) foreach println
(t2,Set(y, 2))
(t3,Set(x, y, t2, 2, z))
(t4,Set(x, y, t3, t2, 1, 2, z))
(t5,Set(x, 2, z, y))
(z,Set(x, 2))
Ok, since recursive parsers are not your thing, here's an alternative with parse combinators:
object PrefixParser extends JavaTokenParsers {
import scala.collection.mutable
// Maps generated through parsing
val Subexprs = mutable.Map[String, String]()
val Dependencies = mutable.Map[String, Set[String]]().withDefaultValue(Set.empty)
// Initialize, read, parse & evaluate string
def read(input: String) = {
counter = 1
Subexprs.clear
Dependencies.clear
parseAll(stmts, input)
}
// Grammar
def stmts = stmt+
def stmt = assignment | expr
def assignment = (ident <~ "=") ~ expr ^^ assignOp
def expr: P = subexpr | identifier | number
def subexpr: P = twoArgs | nArgs
def twoArgs: P = operator ~ expr ~ expr ^^ twoArgsOp
def nArgs: P = "sum" ~ ("\\d+".r >> args) ^^ nArgsOp
def args(n: String): Ps = repN(n.toInt, expr)
def operator = "[-+*/^]".r
def identifier = ident ^^ (id => Result(id, Set(id)))
def number = wholeNumber ^^ (Result(_, Set.empty))
// Evaluation helper class and types
case class Result(ident: String, dependencies: Set[String])
type P = Parser[Result]
type Ps = Parser[List[Result]]
// Evaluation methods
def assignOp: (String ~ Result) => Result = {
case ident ~ result =>
val value = assign(ident,
Subexprs(result.ident),
result.dependencies - result.ident)
Subexprs.remove(result.ident)
Dependencies.remove(result.ident)
value
}
def assign(ident: String,
value: String,
dependencies: Set[String]): Result = {
Subexprs(ident) = value
Dependencies(ident) = dependencies
Result(ident, dependencies)
}
def twoArgsOp: (String ~ Result ~ Result) => Result = {
case op ~ op1 ~ op2 => makeOp(op, op1, op2)
}
def makeOp(op: String,
op1: Result,
op2: Result): Result = {
val ident = getIdent
assign(ident,
"%s %s %s" format (op1.ident, op, op2.ident),
op1.dependencies ++ op2.dependencies + ident)
}
def nArgsOp: (String ~ List[Result]) => Result = {
case op ~ ops => makeNOp(op, ops)
}
def makeNOp(op: String, ops: List[Result]): Result = {
val ident = getIdent
assign(ident,
"%s(%s)" format (op, ops map (_.ident) mkString ", "),
ops.foldLeft(Set(ident))(_ ++ _.dependencies))
}
var counter = 1
def getIdent = {
val ident = "t" + counter
counter += 1
ident
}
// Debugging helper methods
def printAssignments = Subexprs.toSeq.sorted foreach println
def printDependencies = Dependencies.toSeq.sortBy(_._1) map {
case (id, dependencies) => (id, dependencies - id)
} foreach println
}
This is the kind of results you get:
scala> PrefixParser.read("""
| z = ^ x 2
| - + z ^ y 2 1
| - z y
| """)
res77: PrefixParser.ParseResult[List[PrefixParser.Result]] = [5.1] parsed: List(Result(z,Set(x)), Result(t4,Set(t4, y, t3, t2, z)), Result(t5,Set(z, y
, t5)))
scala> PrefixParser.printAssignments
(t2,y ^ 2)
(t3,z + t2)
(t4,t3 - 1)
(t5,z - y)
(z,x ^ 2)
scala> PrefixParser.printDependencies
(t2,Set(y))
(t3,Set(z, y, t2))
(t4,Set(y, t3, t2, z))
(t5,Set(z, y))
(z,Set(x))
n-Ary operator
scala> PrefixParser.read("""
| x = sum 3 + 1 2 * 3 4 5
| * x x
| """)
res93: PrefixParser.ParseResult[List[PrefixParser.Result]] = [4.1] parsed: List(Result(x,Set(t1, t2)), Result(t4,Set(x, t4)))
scala> PrefixParser.printAssignments
(t1,1 + 2)
(t2,3 * 4)
(t4,x * x)
(x,sum(t1, t2, 5))
scala> PrefixParser.printDependencies
(t1,Set())
(t2,Set())
(t4,Set(x))
(x,Set(t1, t2))
It turns out that this sort of parsing is of interest to me also, so I've done a bit more work on it.
There seems to be a sentiment that things like simplification of expressions is hard. I'm not so sure. Let's take a look at a fairly complete solution. (The printing out of tn expressions is not useful for me, and you've got several Scala examples already, so I'll skip that.)
First, we need to extract the various parts of the language. I'll pick regular expressions, though parser combinators could be used also:
object OpParser {
val Natural = "([0-9]+)"r
val Number = """((?:-)?[0-9]+(?:\.[0-9]+)?(?:[eE](?:-)?[0-9]+)?)"""r
val Variable = "([a-z])"r
val Unary = "(exp|sin|cos|tan|sqrt)"r
val Binary = "([-+*/^])"r
val Nary = "(sum|prod|list)"r
Pretty straightforward. We define the various things that might appear. (I've decided that user-defined variables can only be a single lowercase letter, and that numbers can be floating-point since you have the exp function.) The r at the end means this is a regular expression, and it will give us the stuff in parentheses.
Now we need to represent our tree. There are a number of ways to do this, but I'll choose an abstract base class with specific expressions as case classes, since this makes pattern matching easy. Furthermore, we might want nice printing, so we'll override toString. Mostly, though, we'll use recursive functions to do the heavy lifting.
abstract class Expr {
def text: String
def args: List[Expr]
override def toString = args match {
case l :: r :: Nil => "(" + l + " " + text + " " + r + ")"
case Nil => text
case _ => args.mkString(text+"(", ",", ")")
}
}
case class Num(text: String, args: List[Expr]) extends Expr {
val quantity = text.toDouble
}
case class Var(text: String, args: List[Expr]) extends Expr {
override def toString = args match {
case arg :: Nil => "(" + text + " <- " + arg + ")"
case _ => text
}
}
case class Una(text: String, args: List[Expr]) extends Expr
case class Bin(text: String, args: List[Expr]) extends Expr
case class Nar(text: String, args: List[Expr]) extends Expr {
override def toString = text match {
case "list" =>
(for ((a,i) <- args.zipWithIndex) yield {
"%3d: %s".format(i+1,a.toString)
}).mkString("List[\n","\n","\n]")
case _ => super.toString
}
}
Mostly this is pretty dull--each case class overrides the base class, and the text and args automatically fill in for the def. Note that I've decided that a list is a possible n-ary function, and that it will be printed out with line numbers. (The reason is that if you have multiple lines of input, it's sometimes more convenient to work with them all together as one expression; this lets them be one function.)
Once our data structures are defined, we need to parse the expressions. It's convenient to represent the stuff to parse as a list of tokens; as we parse, we'll return both an expression and the remaining tokens that we haven't parsed--this is a particularly useful structure for recursive parsing. Of course, we might fail to parse anything, so it had better be wrapped in an Option also.
def parse(tokens: List[String]): Option[(Expr,List[String])] = tokens match {
case Variable(x) :: "=" :: rest =>
for ((expr,remains) <- parse(rest)) yield (Var(x,List(expr)), remains)
case Variable(x) :: rest => Some(Var(x,Nil), rest)
case Number(n) :: rest => Some(Num(n,Nil), rest)
case Unary(u) :: rest =>
for ((expr,remains) <- parse(rest)) yield (Una(u,List(expr)), remains)
case Binary(b) :: rest =>
for ((lexp,lrem) <- parse(rest); (rexp,rrem) <- parse(lrem)) yield
(Bin(b,List(lexp,rexp)), rrem)
case Nary(a) :: Natural(b) :: rest =>
val buffer = new collection.mutable.ArrayBuffer[Expr]
def parseN(tok: List[String], n: Int = b.toInt): List[String] = {
if (n <= 0) tok
else {
for ((expr,remains) <- parse(tok)) yield { buffer += expr; parseN(remains, n-1) }
}.getOrElse(tok)
}
val remains = parseN(rest)
if (buffer.length == b.toInt) Some( Nar(a,buffer.toList), remains )
else None
case _ => None
}
Note that we use pattern matching and recursion to do most of the heavy lifting--we pick off part of the list, figure out how many arguments we need, and pass those along recursively. The N-ary operation is a little less friendly, but we create a little recursive function that will parse N things at a time for us, storing the results in a buffer.
Of course, this is a little unfriendly to use, so we add some wrapper functions that let us interface with it nicely:
def parse(s: String): Option[Expr] = parse(s.split(" ").toList).flatMap(x => {
if (x._2.isEmpty) Some(x._1) else None
})
def parseLines(ls: List[String]): Option[Expr] = {
val attempt = ls.map(parse).flatten
if (attempt.length<ls.length) None
else if (attempt.length==1) attempt.headOption
else Some(Nar("list",attempt))
}
Okay, now, what about simplification? One thing we might want to do is numeric simplification, where we precompute the expressions and replace the original expression with the reduced version thereof. That sounds like some sort of a recursive operation--find numbers, and combine them. First we get some helper functions to do calculations on numbers:
def calc(n: Num, f: Double => Double): Num = Num(f(n.quantity).toString, Nil)
def calc(n: Num, m: Num, f: (Double,Double) => Double): Num =
Num(f(n.quantity,m.quantity).toString, Nil)
def calc(ln: List[Num], f: (Double,Double) => Double): Num =
Num(ln.map(_.quantity).reduceLeft(f).toString, Nil)
and then we do the simplification:
def numericSimplify(expr: Expr): Expr = expr match {
case Una(t,List(e)) => numericSimplify(e) match {
case n # Num(_,_) => t match {
case "exp" => calc(n, math.exp _)
case "sin" => calc(n, math.sin _)
case "cos" => calc(n, math.cos _)
case "tan" => calc(n, math.tan _)
case "sqrt" => calc(n, math.sqrt _)
}
case a => Una(t,List(a))
}
case Bin(t,List(l,r)) => (numericSimplify(l), numericSimplify(r)) match {
case (n # Num(_,_), m # Num(_,_)) => t match {
case "+" => calc(n, m, _ + _)
case "-" => calc(n, m, _ - _)
case "*" => calc(n, m, _ * _)
case "/" => calc(n, m, _ / _)
case "^" => calc(n, m, math.pow)
}
case (a,b) => Bin(t,List(a,b))
}
case Nar("list",list) => Nar("list",list.map(numericSimplify))
case Nar(t,list) =>
val simple = list.map(numericSimplify)
val nums = simple.collect { case n # Num(_,_) => n }
if (simple.length == 0) t match {
case "sum" => Num("0",Nil)
case "prod" => Num("1",Nil)
}
else if (nums.length == simple.length) t match {
case "sum" => calc(nums, _ + _)
case "prod" => calc(nums, _ * _)
}
else Nar(t, simple)
case Var(t,List(e)) => Var(t, List(numericSimplify(e)))
case _ => expr
}
Notice again the heavy use of pattern matching to find when we're in a good case, and to dispatch the appropriate calculation.
Now, surely algebraic substitution is much more difficult! Actually, all you need to do is notice that an expression has already been used, and assign a variable. Since the syntax I've defined above allows in-place variable substitution, we can actually just modify our expression tree to include more variable assignments. So we do (edited to only insert variables if the user hasn't):
def algebraicSimplify(expr: Expr): Expr = {
val all, dup, used = new collection.mutable.HashSet[Expr]
val made = new collection.mutable.HashMap[Expr,Int]
val user = new collection.mutable.HashMap[Expr,Expr]
def findExpr(e: Expr) {
e match {
case Var(t,List(v)) =>
user += v -> e
if (all contains e) dup += e else all += e
case Var(_,_) | Num(_,_) => // Do nothing in these cases
case _ => if (all contains e) dup += e else all += e
}
e.args.foreach(findExpr)
}
findExpr(expr)
def replaceDup(e: Expr): Expr = {
if (made contains e) Var("x"+made(e),Nil)
else if (used contains e) Var(user(e).text,Nil)
else if (dup contains e) {
val fixed = replaceDupChildren(e)
made += e -> made.size
Var("x"+made(e),List(fixed))
}
else replaceDupChildren(e)
}
def replaceDupChildren(e: Expr): Expr = e match {
case Una(t,List(u)) => Una(t,List(replaceDup(u)))
case Bin(t,List(l,r)) => Bin(t,List(replaceDup(l),replaceDup(r)))
case Nar(t,list) => Nar(t,list.map(replaceDup))
case Var(t,List(v)) =>
used += v
Var(t,List(if (made contains v) replaceDup(v) else replaceDupChildren(v)))
case _ => e
}
replaceDup(expr)
}
That's it--a fully functional algebraic replacement routine. Note that it builds up sets of expressions that it's seen, keeping special track of which ones are duplicates. Thanks to the magic of case classes, all the equalities are defined for us, so it just works. Then we can replace any duplicates as we recurse through to find them. Note that the replace routine is split in half, and that it matches on an unreplaced version of the tree, but uses a replaced version.
Okay, now let's add a few tests:
def main(args: Array[String]) {
val test1 = "- + ^ x 2 ^ y 2 1"
val test2 = "+ + +" // Bad!
val test3 = "exp sin cos sum 5" // Bad!
val test4 = "+ * 2 3 ^ 3 2"
val test5 = List(test1, test4, "^ y 2").mkString("list 3 "," ","")
val test6 = "+ + x y + + * + x y + 4 5 * + x y + 4 y + + x y + 4 y"
def performTest(test: String) = {
println("Start with: " + test)
val p = OpParser.parse(test)
if (p.isEmpty) println(" Parsing failed")
else {
println("Parsed: " + p.get)
val q = OpParser.numericSimplify(p.get)
println("Numeric: " + q)
val r = OpParser.algebraicSimplify(q)
println("Algebraic: " + r)
}
println
}
List(test1,test2,test3,test4,test5,test6).foreach(performTest)
}
}
How does it do?
$ scalac OpParser.scala; scala OpParser
Start with: - + ^ x 2 ^ y 2 1
Parsed: (((x ^ 2) + (y ^ 2)) - 1)
Numeric: (((x ^ 2) + (y ^ 2)) - 1)
Algebraic: (((x ^ 2) + (y ^ 2)) - 1)
Start with: + + +
Parsing failed
Start with: exp sin cos sum 5
Parsing failed
Start with: + * 2 3 ^ 3 2
Parsed: ((2 * 3) + (3 ^ 2))
Numeric: 15.0
Algebraic: 15.0
Start with: list 3 - + ^ x 2 ^ y 2 1 + * 2 3 ^ 3 2 ^ y 2
Parsed: List[
1: (((x ^ 2) + (y ^ 2)) - 1)
2: ((2 * 3) + (3 ^ 2))
3: (y ^ 2)
]
Numeric: List[
1: (((x ^ 2) + (y ^ 2)) - 1)
2: 15.0
3: (y ^ 2)
]
Algebraic: List[
1: (((x ^ 2) + (x0 <- (y ^ 2))) - 1)
2: 15.0
3: x0
]
Start with: + + x y + + * + x y + 4 5 * + x y + 4 y + + x y + 4 y
Parsed: ((x + y) + ((((x + y) * (4 + 5)) + ((x + y) * (4 + y))) + ((x + y) + (4 + y))))
Numeric: ((x + y) + ((((x + y) * 9.0) + ((x + y) * (4 + y))) + ((x + y) + (4 + y))))
Algebraic: ((x0 <- (x + y)) + (((x0 * 9.0) + (x0 * (x1 <- (4 + y)))) + (x0 + x1)))
So I don't know if that's useful for you, but it turns out to be useful for me. And this is the sort of thing that I would be very hesitant to tackle in C++ because various things that were supposed to be easy ended up being painful instead.
Edit: Here's an example of using this structure to print temporary assignments, just to demonstrate that this structure is perfectly okay for doing such things.
Code:
def useTempVars(expr: Expr): Expr = {
var n = 0
def temp = { n += 1; "t"+n }
def replaceTemp(e: Expr, exempt: Boolean = false): Expr = {
def varify(x: Expr) = if (exempt) x else Var(temp,List(x))
e match {
case Var(t,List(e)) => Var(t,List(replaceTemp(e, exempt = true)))
case Una(t,List(u)) => varify( Una(t, List(replaceTemp(u,false))) )
case Bin(t,lr) => varify( Bin(t, lr.map(replaceTemp(_,false))) )
case Nar(t,ls) => varify( Nar(t, ls.map(replaceTemp(_,false))) )
case _ => e
}
}
replaceTemp(expr)
}
def varCut(expr: Expr): Expr = expr match {
case Var(t,_) => Var(t,Nil)
case Una(t,List(u)) => Una(t,List(varCut(u)))
case Bin(t,lr) => Bin(t, lr.map(varCut))
case Nar(t,ls) => Nar(t, ls.map(varCut))
case _ => expr
}
def getAssignments(expr: Expr): List[Expr] = {
val children = expr.args.flatMap(getAssignments)
expr match {
case Var(t,List(e)) => children :+ expr
case _ => children
}
}
def listAssignments(expr: Expr): List[String] = {
getAssignments(expr).collect(e => e match {
case Var(t,List(v)) => t + " = " + varCut(v)
}) :+ (expr.text + " is the answer")
}
Selected results (from listAssignments(useTempVars(r)).foreach(printf(" %s\n",_))):
Start with: - + ^ x 2 ^ y 2 1
Assignments:
t1 = (x ^ 2)
t2 = (y ^ 2)
t3 = (t1 + t2)
t4 = (t3 - 1)
t4 is the answer
Start with: + + x y + + * + x y + 4 5 * + x y + 4 y + + x y + 4 y
Algebraic: ((x0 <- (x + y)) + (((x0 * 9.0) + (x0 * (x1 <- (4 + y)))) + (x0 + x1)))
Assignments:
x0 = (x + y)
t1 = (x0 * 9.0)
x1 = (4 + y)
t2 = (x0 * x1)
t3 = (t1 + t2)
t4 = (x0 + x1)
t5 = (t3 + t4)
t6 = (x0 + t5)
t6 is the answer
Second edit: finding dependencies is also not too bad.
Code:
def directDepends(expr: Expr): Set[Expr] = expr match {
case Var(t,_) => Set(expr)
case _ => expr.args.flatMap(directDepends).toSet
}
def indirectDepends(expr: Expr) = {
val depend = getAssignments(expr).map(e =>
e -> e.args.flatMap(directDepends).toSet
).toMap
val tagged = for ((k,v) <- depend) yield (k.text -> v.map(_.text))
def percolate(tags: Map[String,Set[String]]): Option[Map[String,Set[String]]] = {
val expand = for ((k,v) <- tags) yield (
k -> (v union v.flatMap(x => tags.get(x).getOrElse(Set())))
)
if (tags.exists(kv => expand(kv._1) contains kv._1)) None // Cyclic dependency!
else if (tags == expand) Some(tags)
else percolate(expand)
}
percolate(tagged)
}
def listDependents(expr: Expr): List[(String,String)] = {
def sayNothing(s: String) = if (s=="") "nothing" else s
val e = expr match {
case Var(_,_) => expr
case _ => Var("result",List(expr))
}
indirectDepends(e).map(_.toList.map(x =>
(x._1, sayNothing(x._2.toList.sorted.mkString(" ")))
)).getOrElse(List((e.text,"cyclic")))
}
And if we add new test cases val test7 = "list 3 z = ^ x 2 - + z ^ y 2 1 w = - z y" and val test8 = "list 2 x = y y = x" and show the answers with for ((v,d) <- listDependents(r)) println(" "+v+" requires "+d) we get (selected results):
Start with: - + ^ x 2 ^ y 2 1
Dependencies:
result requires x y
Start with: list 3 z = ^ x 2 - + z ^ y 2 1 w = - z y
Parsed: List[
1: (z <- (x ^ 2))
2: ((z + (y ^ 2)) - 1)
3: (w <- (z - y))
]
Dependencies:
z requires x
w requires x y z
result requires w x y z
Start with: list 2 x = y y = x
Parsed: List[
1: (x <- y)
2: (y <- x)
]
Dependencies:
result requires cyclic
Start with: + + x y + + * + x y + 4 5 * + x y + 4 y + + x y + 4 y
Algebraic: ((x0 <- (x + y)) + (((x0 * 9.0) + (x0 * (x1 <- (4 + y)))) + (x0 + x1)))
Dependencies:
x0 requires x y
x1 requires y
result requires x x0 x1 y
So I think that on top of this sort of structure, all of your individual requirements are met by blocks of one or two dozen lines of Scala code.
Edit: here's expression evaluation, if you're given a mapping from vars to values:
def numericEvaluate(expr: Expr, initialValues: Map[String,Double]) = {
val chain = new collection.mutable.ArrayBuffer[(String,Double)]
val evaluated = new collection.mutable.HashMap[String,Double]
def note(xv: (String,Double)) { chain += xv; evaluated += xv }
evaluated ++= initialValues
def substitute(expr: Expr): Expr = expr match {
case Var(t,List(n # Num(v,_))) => { note(t -> v.toDouble); n }
case Var(t,_) if (evaluated contains t) => Num(evaluated(t).toString,Nil)
case Var(t,ls) => Var(t,ls.map(substitute))
case Una(t,List(u)) => Una(t,List(substitute(u)))
case Bin(t,ls) => Bin(t,ls.map(substitute))
case Nar(t,ls) => Nar(t,ls.map(substitute))
case _ => expr
}
def recurse(e: Expr): Expr = {
val sub = numericSimplify(substitute(e))
if (sub == e) e else recurse(sub)
}
(recurse(expr), chain.toList)
}
and it's used like so in the testing routine:
val (num,ops) = numericEvaluate(r,Map("x"->3,"y"->1.5))
println("Evaluated:")
for ((v,n) <- ops) println(" "+v+" = "+n)
println(" result = " + num)
giving results like these (with input of x = 3 and y = 1.5):
Start with: list 3 - + ^ x 2 ^ y 2 1 + * 2 3 ^ 3 2 ^ y 2
Algebraic: List[
1: (((x ^ 2) + (x0 <- (y ^ 2))) - 1)
2: 15.0
3: x0
]
Evaluated:
x0 = 2.25
result = List[
1: 10.25
2: 15.0
3: 2.25
]
Start with: list 3 z = ^ x 2 - + z ^ y 2 1 w = - z y
Algebraic: List[
1: (z <- (x ^ 2))
2: ((z + (y ^ 2)) - 1)
3: (w <- (z - y))
]
Evaluated:
z = 9.0
w = 7.5
result = List[
1: 9.0
2: 10.25
3: 7.5
]
The other challenge--picking out the vars that haven't already been used--is just set subtraction off of the dependencies result list. diff is the name of the set subtraction method.
The problem consists of two subproblems: parsing and symbolic manipulation. It seems to me the answer boils down to two possible solutions.
One is to implement everything from scratch: "I do recommend creating the full expression tree if you want to retain maximum flexibility for handling tricky cases." - proposed by Rex. As Sven points out: "any of the high-level languages you listed are almost equally suited for the task," however "Python (or any of the high-level languages you listed) won't take away the complexity of the problem."
I have received very nice solutions in Scala (many thanks for Rex and Daniel), a nice little example in Python (from Sven). However, I am still interested in Lisp, Haskell or Erlang solutions.
The other solution is to use some existing library/software for the task, with all the implied pros and cons. Candidates are Maxima (Common Lisp), SymPy (Python, proposed by payne) and GiNaC (C++).
Related
I am studying algorithms in Python and solving a question that is:
Let x(k) be a recursively defined string with base case x(1) = "123"
and x(k) is "1" + x(k-1) + "2" + x(k-1) + "3". Given three positive
integers k,s, and t, find the substring x(k)[s:t].
For example, if k = 2, s = 1 and t = 5,x(2) = 112321233 and x(2)[1:5]
= 1232.
I have solved it using a simple recursive function:
def generate_string(k):
if k == 1:
return "123"
part = generate_string(k -1)
return ("1" + part + "2" + part + "3")
print(generate_string(k)[s,t])
Although my first approach gives correct answer, the problem is that it takes too long to build string x when k is greater than 20. The program need to be finished within 16 seconds while k is below 50. I have tried to use memoization but it does not help as I am not allowed to cache each test case. I thus think that I must avoid using recursive function to speed up the program. Is there any approaches I should consider?
We can see that the string represented by x(k) grows exponentially in length with increasing k:
len(x(1)) == 3
len(x(k)) == len(x(k-1)) * 2 + 3
So:
len(x(k)) == 3 * (2**k - 1)
For k equal to 100, this amounts to a length of more than 1030. That's more characters than there are atoms in a human body!
Since the parameters s and t will take (in comparison) a tiny, tiny slice of that, you should not need to produce the whole string. You can still use recursion though, but keep passing an s and t range to each call. Then when you see that this slice will actually be outside of the string you would generate, then you can just exit without recursing deeper, saving a lot of time and (string) space.
Here is how you could do it:
def getslice(k, s, t):
def recur(xsize, s, t):
if xsize == 0 or s >= xsize or t <= 0:
return ""
smaller = (xsize - 3) // 2
return ( ("1" if s <= 0 else "")
+ recur(smaller, s-1, t-1)
+ ("2" if s <= smaller+1 < t else "")
+ recur(smaller, s-smaller-2, t-smaller-2)
+ ("3" if t >= xsize else "") )
return recur(3 * (2**k - 1), s, t)
This doesn't use any caching of x(k) results... In my tests this was fast enough.
Based on #FMc's answer, here's some python3 code that calculates x(k, s, t):
from functools import lru_cache
from typing import *
def f_len(k) -> int:
return 3 * ((2 ** k) - 1)
#lru_cache(None)
def f(k) -> str:
if k == 1:
return "123"
return "1" + f(k - 1) + "2" + f(k - 1) + "3"
def substring_(k, s, t, output) -> None:
# Empty substring.
if s >= t or k == 0:
return
# (An optimization):
# If all the characters need to be included, just calculate the string and cache it.
if s == 0 and t == f_len(k):
output.append(f(k))
return
if s == 0:
output.append("1")
sub_len = f_len(k - 1)
substring_(k - 1, max(0, s - 1), min(sub_len, t - 1), output)
if s <= 1 + sub_len < t:
output.append("2")
substring_(k - 1, max(0, s - sub_len - 2), min(sub_len, t - sub_len - 2), output)
if s <= 2 * (1 + sub_len) < t:
output.append("3")
def substring(k, s, t) -> str:
output: List[str] = []
substring_(k, s, t, output)
return "".join(output)
def test(k, s, t) -> bool:
actual = substring(k, s, t)
expected = f(k)[s:t]
return actual == expected
assert test(1, 0, 3)
assert test(2, 2, 6)
assert test(2, 1, 5)
assert test(2, 0, f_len(2))
assert test(3, 0, f_len(3))
assert test(8, 44, 89)
assert test(10, 1001, 2022)
assert test(14, 12345, 45678)
assert test(17, 12345, 112345)
# print(substring(30, 10000, 10100))
print("Tests passed")
This is an interesting problem. I'm not sure whether I'll have time to write the code, but here's an outline of how you can solve it. Note: see the better answer from trincot.
As discussed in the comments, you cannot generate the actual string: you will quickly run out of memory as k grows. But you can easily compute the length of that string.
First some notation:
f(k) : The generated string.
n(k) : The length of f(k).
nk1 : n(k-1), which is used several times in table below.
For discussion purposes, we can divide the string into the following regions. The start/end values use standard Python slice numbering:
Region | Start | End | Len | Subtring | Ex: k = 2
-------------------------------------------------------------------
A | 0 | 1 | 1 | 1 | 0:1 1
B | 1 | 1 + nk1 | nk1 | f(k-1) | 1:4 123
C | 1 + nk1 | 2 + nk1 | 1 | 2 | 4:5 2
D | 2 + nk1 | 2 + nk1 + nk1 | nk1 | f(k-1) | 5:8 123
E | 2 + nk1 + nk1 | 3 + nk1 + nk1 | 1 | 3 | 8:9 3
Given k, s, and t we need to figure out which region of the string is relevant. Take a small example:
k=2, s=6, and t=8.
The substring defined by 6:8 does not require the full f(k). We only need
region D, so we can turn our attention to f(k-1).
To make the shift from k=2 to k=1, we need to adjust s and t: specifically,
we need to subtract the total length of regions A + B + C. For k=2, that
length is 5 (1 + nk1 + 1).
Now we are dealing with: k=1, s=1, and t=3.
Repeat as needed.
Whenever k gets small enough, we stop this nonsense and actually generate the string so we can grab the needed substring directly.
It's possible that some values of s and t could cross region boundaries. In that case, divide the problem into two subparts (one for each region needed). But the general idea is the same.
Here's a commented iterative version in JavaScript that's very easy to convert to Python.
In addition to being what you asked for, that is non-recursive, it allows us to solve things like f(10000, 10000, 10050), which seem to exceed Python default recursion depth.
// Generates the full string
function g(k){
if (k == 1)
return "123";
prev = g(k - 1);
return "1" + prev + "2" + prev + "3";
}
function size(k){
return 3 * ((1 << k) - 1);
}
// Given a depth and index,
// we'd like (1) a string to
// output, (2) the possible next
// part of the same depth to
// push to the stack, and (3)
// possibly the current section
// mapped deeper to also push to
// the stack. (2) and (3) can be
// in a single list.
function getParams(depth, i){
const psize = size(depth - 1);
if (i == 0){
return ["1", [[depth, 1 + psize], [depth - 1, 0]]];
} else if (i < 1 + psize){
return ["", [[depth, 1 + psize], [depth - 1, i - 1]]];
} else if (i == 1 + psize){
return ["2", [[depth, 2 + 2 * psize], [depth - 1, 0]]];
} else if (i < 2 + 2 * psize){
return ["", [[depth, 2 + 2 * psize], [depth - 1, i - 2 - psize]]];
} else {
return ["3", []];
}
}
function f(k, s, t){
let len = t - s;
let str = "";
let stack = [[k, s]];
while (str.length < len){
const [depth, i] = stack.pop();
if (depth == 1){
const toTake = Math.min(3 - i, len - str.length);
str = str + "123".substr(i, toTake);
} else {
const [s, rest] = getParams(depth, i);
str = str + s;
stack.push(...rest);
}
}
return str;
}
function test(k, s, t){
const l = g(k).substring(s, t);
const r = f(k, s, t);
console.log(g(k).length);
//console.log(g(k))
console.log(l);
console.log(r);
console.log(l == r);
}
test(1, 0, 3);
test(2, 2, 6);
test(2, 1, 5);
test(4, 44, 45);
test(5, 30, 40);
test(7, 100, 150);
I am still teaching some R mainly to myself (and to my students).
Here's an implementation of the Collatz sequence in R:
f <- function(n)
{
# construct the entire Collatz path starting from n
if (n==1) return(1)
if (n %% 2 == 0) return(c(n, f(n/2)))
return(c(n, f(3*n + 1)))
}
Calling f(13) I get
13, 40, 20, 10, 5, 16, 8, 4, 2, 1
However note that a vector is growing dynamically in size here. Such moves tend to be a recipe for inefficient code. Is there a more efficient version?
In Python I would use
def collatz(n):
assert isinstance(n, int)
assert n >= 1
def __colla(n):
while n > 1:
yield n
if n % 2 == 0:
n = int(n / 2)
else:
n = int(3 * n + 1)
yield 1
return list([x for x in __colla(n)])
I found a way to write into vectors without specifying their dimension a priori. Therefore a solution could be
collatz <-function(n)
{
stopifnot(n >= 1)
# define a vector without specifying the length
x = c()
i = 1
while (n > 1)
{
x[i] = n
i = i + 1
n = ifelse(n %% 2, 3*n + 1, n/2)
}
x[i] = 1
# now "cut" the vector
dim(x) = c(i)
return(x)
}
I was curious to see how a C++ implementation through Rcpp would compare to your two base R approaches. Here are my results.
First let's define a function collatz_Rcpp that returns the Hailstone sequence for a given integer n. The (non-recursive) implementation was adapted from Rosetta Code.
library(Rcpp)
cppFunction("
std::vector<int> collatz_Rcpp(int i) {
std::vector<int> v;
while(true) {
v.push_back(i);
if (i == 1) break;
i = (i % 2) ? (3 * i + 1) : (i / 2);
}
return v;
}
")
We now run a microbenchmark analysis using both your base R and the Rcpp implementation. We calculate the Hailstone sequences for the first 10000 integers
# base R implementation
collatz_R <- function(n) {
# construct the entire Collatz path starting from n
if (n==1) return(1)
if (n %% 2 == 0) return(c(n, collatz(n/2)))
return(c(n, collatz(3*n + 1)))
}
# "updated" base R implementation
collatz_R_updated <-function(n) {
stopifnot(n >= 1)
# define a vector without specifying the length
x = c()
i = 1
while (n > 1) {
x[i] = n
i = i + 1
n = ifelse(n %% 2, 3*n + 1, n/2)
}
x[i] = 1
# now "cut" the vector
dim(x) = c(i)
return(x)
}
library(microbenchmark)
n <- 10000
res <- microbenchmark(
baseR = sapply(1:n, collatz_R),
baseR_updated = sapply(1:n, collatz_R_updated),
Rcpp = sapply(1:n, collatz_Rcpp))
res
# expr min lq mean median uq max
# baseR 65.68623 73.56471 81.42989 77.46592 83.87024 193.2609
#baseR_updated 3861.99336 3997.45091 4240.30315 4122.88577 4348.97153 5463.7787
# Rcpp 36.52132 46.06178 51.61129 49.27667 53.10080 168.9824
library(ggplot2)
autoplot(res)
The (non-recursive) Rcpp implementation seems to be around 30% faster than the original (recursive) base R implementation. The "updated" (non-recursive) base R implementation is significantly slower than the original (recursive) base R approach (the microbenchmark takes around 10 minutes to finish on my MacBook Air due to baseR_updated).
I've done the exercise and it works. But I want to know if there a smarter way to do it. Thanks.
# Consider dividing a string into two halves.
# If the length is even, the front and back halves are the same length.
# If the length is odd, we'll say that the extra char goes in the front half.
# e.g. 'abcde', the front half is 'abc', the back half 'de'.
# Given 2 strings, a and b, return a string of the form
# a-front + b-front + a-back + b-back
My code:
def front_back(a, b):
if len(a) % 2 == 0:
a_front = a[0:len(a) / 2]
else:
a_front = a[0:(len(a) / 2) + 1]
if len(a) % 2 == 0:
a_back = a[len(a) / 2:]
else:
a_back = a[(len(a) / 2) + 1:]
if len(b) % 2 == 0:
b_front = b[0:len(b) / 2]
else:
b_front = b[0:(len(b) / 2) + 1]
if len(b) % 2 == 0:
b_back = b[len(b) / 2:]
else:
b_back = b[(len(b) / 2) + 1:]
return a_front + b_front + a_back + b_back
You probably don't need all the if tests. Using substrings (as you use the slices in python), the following example in Java works.
public static String SplitMixCombine(String a, String b) {
return a.substring(0, (a.length()+1)/2) +
b.substring(0, (b.length()+1)/2) +
a.substring((a.length()+1)/2, a.length()) +
b.substring((b.length()+1)/2, b.length());
}
Just got this to work in python:
>>> def mix_combine(a, b):
... return a[0:int((len(a)+1)/2)] + b[0:int((len(b)+1)/2)] + a[int((len(a)+1)/2):] +b[int((len(b)+1)/2):]
...
>>> a = "abcd"
>>> b = "wxyz"
>>> mix_combine(a,b)
'abwxcdyz'
>>> a = "abcde"
>>> b = "vwxyz"
>>> mix_combine(a,b)
'abcvwxdeyz'
>>>
This is a lot cleaner code.
def front_back(a, b):
print a, b
a_indx = int((len(a)+1)/2)
b_indx = int((len(b)+1)/2)
print a[:a_indx] + b[:b_indx] + a[a_indx:] +b[b_indx:]
print "\n"
front_back("ab", "cd")
front_back("abc", "de")
front_back("ab", "cde")
front_back("abc", "def")
Hope this helps !
I wrote another one-liner for this:
def front_back(a, b):
return a[:len(a) / 2 + len(a) % 2] + b[:len(b) / 2 + len(b) % 2] + a[-(len(a) / 2):] + b[-(len(b) / 2):]
An interesting thing to note is that in Python 5 / 2 yields 2, but -5 / 2 yields -3, you might expect to get -2. That's why I added the parenthesis. See Negative integer division surprising result
def front_back(a, b):
return front_string(a)+front_string(b)+back_string(a)+back_string(b)
def front_string(s):
return s[:int((len(s)+1)/2)]
def back_string(s):
return s[int((len(s)+1)/2):]
# int((len(s)+1)/2) returns the correct index for both odd and even length strings
I'm a beginner in Python, teaching myself off of Google Code University. I had this problem as an exercise, and was able to solve it using the solution shown below:
# F. front_back
# Consider dividing a string into two halves.
# If the length is even, the front and back halves are the same length.
# If the length is odd, we'll say that the extra char goes in the front half.
# e.g. 'abcde', the front half is 'abc', the back half 'de'.
# Given 2 strings, a and b, return a string of the form
# a-front + b-front + a-back + b-back
def front_back(a, b):
if len(a) % 2 == 0:
ad = len(a) / 2
if len(b) % 2 == 0:
bd = len(b) / 2
else:
bd = (len(b) / 2) + 1
else:
ad = (len(a) / 2) + 1
if len(b) % 2 == 0:
bd = len(b) / 2
else:
bd = (len(b) / 2) + 1
return a[:ad] + b[:bd] + a[ad:] + b[bd:]
This produces the correct output and solves the problem. However, I am duplicating the logic of whether to split a string evenly or add the odd number to the first half, and this seems redundant. There has to be a more efficient way of doing this. The same exact check and logic is being applied to a and b. Anyone?
def front_back(a, b):
ad = (len(a) + 1) // 2
bd = (len(b) + 1) // 2
return a[:ad] + b[:bd] + a[ad:] + b[bd:]
Using // for division makes this code work in both Python 2.x and 3.x.
Well, put it in a separate function.
def front_back(string):
offset = len(string) / 2
if len(string) % 2 != 0:
offset += 1
return string[:offset], string[offset:]
def solution(a, b):
front_a, back_a = front_back(a)
front_b, back_b = front_back(b)
return front_a + back_a + front_b + back_b
Since you're adding 1 to the length if it's odd, and 'odd' means that len(a)%2 == 1...
def front_back2(a, b):
ad = (len(a) + len(a)%2) / 2
bd = (len(b) + len(b)%2) / 2
return a[:ad]+b[:bd]+a[ad:]+b[bd:]
Of course, you could even condense it to one line just for kicks (although, it's significantly less readable):
def front_back2(a, b):
return a[:(len(a)+len(a)%2)/2]+b[:(len(b)+len(b)%2)/2]+a[(len(a)+len(a)%2)/2:]+b[(len(b)+len(b)%2)/2:]
You can get the maximum index by using ceil
In [1]: l = [1,2,3]
In [2]: import math
In [4]: math.ceil(len(l)/2.0)
Out[4]: 2.0
In [5]: l.append(4)
In [6]: math.ceil(len(l)/2.0)
Out[6]: 2.0
In [7]: l.append(5)
In [8]: math.ceil(len(l)/2.0)
Out[8]: 3.0
In [9]: l[0:3]
Out[9]: [1, 2, 3]
In [10]: l[3:]
Out[10]: [4, 5]
Mhh trying to understand #Sven answer I got this:
len( s ) + 1 / 2
Will always give you the correct index.
So if we put that in a function:
def d( s ):
return ( len(s) + 1 ) / 2
We can use it in the solution:
def front_back( a, b ):
return a[:d(a)] + b[:d(b)] + a[d(a):] + b[d(b):]
Ok, I got it now.
I'm not quite sure what's the difference between / and // though
from math import ceil
def front_back(a, b):
divide = lambda s: int(ceil(len(s) / 2.0)) # or lambda s: (len(s) + 1) // 2
a_divide, b_divide = divide(a), divide(b)
return a[:a_divide] + b[:b_divide] + a[a_divide:] + b[b_divide:]
Here's mine:
def front_back( a, b ) :
return of(a)[0] + of(b)[0] + of(a)[1] + of(b)[1]
def of( s ):
index = len( s ) / 2 + ( 1 if len( s ) % 2 == 1 else 0 )
return ( s[ : index ] , s[ index : ] )
print front_back('abcde','hola')
Prints:
abchodela
Anybody knows proper python implementation of TEA (Tiny Encryption Algorithm)? I tried the one I've found here: http://sysadminco.com/code/python-tea/ - but it does not seem to work properly.
It returns different results than other implementations in C or Java. I guess it's caused by completely different data types in python (or no data types in fact).
Here's the code and an example:
def encipher(v, k):
y=v[0];z=v[1];sum=0;delta=0x9E3779B9;n=32
w=[0,0]
while(n>0):
y += (z << 4 ^ z >> 5) + z ^ sum + k[sum & 3]
y &= 4294967295L # maxsize of 32-bit integer
sum += delta
z += (y << 4 ^ y >> 5) + y ^ sum + k[sum>>11 & 3]
z &= 4294967295L
n -= 1
w[0]=y; w[1]=z
return w
def decipher(v, k):
y=v[0]
z=v[1]
sum=0xC6EF3720
delta=0x9E3779B9
n=32
w=[0,0]
# sum = delta<<5, in general sum = delta * n
while(n>0):
z -= (y << 4 ^ y >> 5) + y ^ sum + k[sum>>11 & 3]
z &= 4294967295L
sum -= delta
y -= (z << 4 ^ z >> 5) + z ^ sum + k[sum&3]
y &= 4294967295L
n -= 1
w[0]=y; w[1]=z
return w
Python example:
>>> import tea
>>> key = [0xbe168aa1, 0x16c498a3, 0x5e87b018, 0x56de7805]
>>> v = [0xe15034c8, 0x260fd6d5]
>>> res = tea.encipher(v, key)
>>> "%X %X" % (res[0], res[1])
**'70D16811 F935148F'**
C example:
#include <unistd.h>
#include <stdio.h>
void encipher(unsigned long *const v,unsigned long *const w,
const unsigned long *const k)
{
register unsigned long y=v[0],z=v[1],sum=0,delta=0x9E3779B9,
a=k[0],b=k[1],c=k[2],d=k[3],n=32;
while(n-->0)
{
sum += delta;
y += (z << 4)+a ^ z+sum ^ (z >> 5)+b;
z += (y << 4)+c ^ y+sum ^ (y >> 5)+d;
}
w[0]=y; w[1]=z;
}
int main()
{
unsigned long v[] = {0xe15034c8, 0x260fd6d5};
unsigned long key[] = {0xbe168aa1, 0x16c498a3, 0x5e87b018, 0x56de7805};
unsigned long res[2];
encipher(v, res, key);
printf("%X %X\n", res[0], res[1]);
return 0;
}
$ ./tea
**D6942D68 6F87870D**
Please note, that both examples were run with the same input data (v and key), but results were different. I'm pretty sure C implementation is correct - it comes from a site referenced by wikipedia (I couldn't post a link to it because I don't have enough reputation points yet - some antispam thing)
I fixed it. Here is working TEA implementation in python:
#!/usr/bin/env python
#-*- coding: utf-8 -*-
import sys
from ctypes import *
def encipher(v, k):
y = c_uint32(v[0])
z = c_uint32(v[1])
sum = c_uint32(0)
delta = 0x9e3779b9
n = 32
w = [0,0]
while(n>0):
sum.value += delta
y.value += ( z.value << 4 ) + k[0] ^ z.value + sum.value ^ ( z.value >> 5 ) + k[1]
z.value += ( y.value << 4 ) + k[2] ^ y.value + sum.value ^ ( y.value >> 5 ) + k[3]
n -= 1
w[0] = y.value
w[1] = z.value
return w
def decipher(v, k):
y = c_uint32(v[0])
z = c_uint32(v[1])
sum = c_uint32(0xc6ef3720)
delta = 0x9e3779b9
n = 32
w = [0,0]
while(n>0):
z.value -= ( y.value << 4 ) + k[2] ^ y.value + sum.value ^ ( y.value >> 5 ) + k[3]
y.value -= ( z.value << 4 ) + k[0] ^ z.value + sum.value ^ ( z.value >> 5 ) + k[1]
sum.value -= delta
n -= 1
w[0] = y.value
w[1] = z.value
return w
if __name__ == "__main__":
key = [1,2,3,4]
v = [1385482522,639876499]
enc = encipher(v,key)
print enc
print decipher(enc,key)
And a small sample:
>>> v
[1385482522, 639876499]
>>> tea.decipher(tea.encipher(v,key),key)
[1385482522L, 639876499L]
Since TEA is a block cipher and your v is a very small block, I'd guess there may be block padding differences, or as Wikipedia notes:
http://en.wikipedia.org/wiki/Tiny_Encryption_Algorithm:
Note that the reference implementation
is bound to a specific microprocessor
architecture meaning that byte order
considerations are important when
cyphertext is shared and processed on
different systems. The original paper
does not specify any details about
microprocessor architecture and so
anyone implementing a system using TEA
would need to make those
specifications for themselves.
I didn't inspect either implementation in detail. Your &= statements feel suspicious, too.
Tea is broken, do not use it.
XXTEA which is secure does not define endianess and stuff and you should should reinvent whell when you can use AES.
There is no point in using unsecure cryptography.
I strongy advice you to apply AES, it can be implemented in 8bit microcontolers whit just few kB of code
EDIT
Did you checked this code?
http://sysadminco.com/code/python-tea/