We Keep Coding

Convert a string equation to an integer answer

If I have a string
equation = "1+2+3+4"
How do I change it into an int and equate the answer? I was thinking something like this but it would give an error.
answer = (int)equation
print(answer)
The equation could contain +-*/

If you are prepared to accept the risks, namely that you may be letting hostile users run whatever they like on your machine, you could use eval():
>>> equation = "1+2+3+4"
>>> eval(equation)
10
If your code will only ever accept input that you control, then this is the quickest and simplest solution. If you need to allow general input from users other than you, then you'd need to look for a more restrictive expression evaluator.
Update
Thanks to #roippi for pulling me up and pointing out that the code above executes in the environment of the calling code. So the expression would be evaluated with local and global variables available. Suppress that like so:
eval(equation, {'__builtins__': None})
This doesn't make the use of eval() secure. It just gives it a clean, empty environment in which to operate. A hostile user could still hose your system.

If we wanted to expose a calculator to the world, I'd be very interested to see if a cracker could work around this:
import string
def less_dangerous_eval(equation):
if not set(equation).intersection(string.ascii_letters + '{}[]_;\n'):
return eval(equation)
else:
print("illegal character")
return None
less_dangerous_eval('1*2/3^4+(5-6)')
returns:
3
I know that this can be broken by giving it bad syntax (fixable with a try/except block) or an operation that would take up all of the memory in the system (try/except catching this might be more iffy), but I am not currently aware of any way for someone to gain control.

This works if you only have digits and plusses:
answer = sum(float(i) for i in equation.split('+'))
or if you know they will only be integers
answer = sum(int(i) for i in equation.split('+'))
If you want to be able to evaluate more than that, I suggest you do your homework:
Look up the string module (which has string.digits)
the math module, which has your operations
create logic to perform the operations in proper order
Good luck!

I suggest you use eval and check the input:
def evaluate(s):
import string
for i in s:
if i not in "+-*/ " + string.digits:
return False # input is not valid
return eval(s)
As already mentioned, eval is unsafe, and very difficult to make safe. I remember seeing a very good blog post explaining this, does anyone know what it was? However ast.literal_eval is safe, it doesn't allow __import__ etc. I would strongly recomment using ast.literal_eval instead of eval whenever possible. Unfortunately, in this case it isn't possible. However, in a different case, e.g. one where you only need to support addition and multiplication, you could (and should) use literal_eval instead.
On the other hand, if this is homework with the intention for you to learn about parsing, then I would suggest doing this a different way. I know that if I was a teacher, I would respond to an answer using eval with "Very clever, but this won't help you pass a test on abstract syntax trees." (which are incidentally one thing that you should look at if you want to implement this "properly").

PEP8, locals() and interpolation

Here is some code:
foo = "Bears"
"Lions, Tigers and %(foo)s" % locals()
My PEP8 linter (SublimeLinter) complains about this, because foo is "unreferenced". My question is whether PEP8 should count this type of string interpolation as "referenced", or if there is a good reason to consider this "bad style".

Well, it isn't referenced. The part that's questionable style is using locals() to access variables instead of just accessing them by name. See this previous question for why that's a dubious idea. It's not a terrible thing, but it's not good style for a program that you want to maintain in the long term.
Edit: It's true that when you use a literal format string, it seems more explicit. But part of the point of the previous post is that in a larger program, you will probably wind up not using a literal format string. If it's a small program and you don't care, go ahead and use it. But warning about things that are likely to cause maintainability problems later is also part of what style guides and linters are for.
Also, locals isn't a canonical representation of names that are explicitly referenced in the literal. It's a canonical representation of all names in the local namespace. You can still do it if you like, but it's basically a loose/sloppy alternative to explicitly using the names you're using, which is again, exactly the sort of thing linters are supposed to warn you about.

Even if you reject BrenBarn's argument that foo isn't referenced, if you accept the argument that passing locals() in string formatting should be flagged, it may not be worth writing to code to consider foo referenced.
First, in every case where that extra code would help, the construct is not acceptable anyway, and the user is going to have to ignore a lint warning anyway. Yes, there is some harm in giving the user two lint warnings to ignore when there's only actually one problem, especially if one of the warnings is somewhat misleading. But is it enough harm to justify writing very complicated code and introduce new bugs into the linter?
You also have to consider that for this to actually work, the linter has to recognize not just % formatting, but also {} formatting, and every other kind of string formatting, HTML templating, etc. that the user could be using. In practice, this means handling various very common forms, and providing some kind of hook for the user to describe anything else.
And, on top of that, even if you don't think it should work with arbitrarily-generated format strings, it surely has to at least work with l10n. How is that going to work? If the format string is generated by something like gettext, the linter has no way of knowing whether foo is referenced, unless it can check all of the translations and see that at least one of them references foo—which means it has to understand (or have hooks to be taught) every string translation mechanism, and have access to the translation database.
So, I would suggest that, even if you consider the warning spurious in this case, you leave it there anyway. At most, add something which qualifies the warning:
foo is possibly unreferenced, but in a function that uses locals()

The following wouldn't make SublimeLinter happy either, It looks up each variable name referenced in the string and substitutes the corresponding value from the namespace mapping, which defaults to the caller's locals. As such it show the inherent limitation a utility like SublimeLinter has when trying to determine if something has been referenced in Python). My advice is just ignore SublimeLinter or add code to fake it out, like foo = foo. I've had to do something like the latter to get rid of C compiler warnings about things which were both legal and intended.
import re
import sys
SUB_RE = re.compile(r"%\((.*?)\)s")
def local_vars_subst(s, namespace=None):
if namespace is None:
namespace = sys._getframe(1).f_locals
def repl(matchobj):
var = matchobj.group(1).strip()
try:
retval = namespace[var]
except KeyError:
retval = "<undefined>"
return retval
return SUB_RE.sub(repl, s)
foo = "Bears"
print local_vars_subst("Lions, Tigers and %(foo)s")

Best practice for string substitution with gettext using Python

Looking for best practice advice on what string substitution technique to use when using gettext(). Or do all techniques apply equally?
I can think of at least 3 string techniques:
1) Classic "%" based formatting:
"My name is %(name)s" % locals()
2) .format() based formatting:
"My name is {name}".format( locals() )
3) string.Template.safe_substitute()
import string
template = string.Template( "My name is ${name}" )
template.safe_substitute( locals() )
The advantage of the string.Template technique is that a translated string with with an incorrectly spelled variable reference can still yield a usable string value while the other techniques unconditionally raise an exception. The downside of the string.Template technique appears to be the inability for one to customize how a variable is formatted (padding, justification, width, etc).

Actually I would prefer to get an exception during my tests, to fix the error as soon as possible -- "errors should not pass silently". So I consider that approach (2) is the best one in modern Python (which supports the readable and flexible format), and approach (1) a probably inevitable fall-back if you're stuck supporting older Python releases (where % was "the" way to do flexible formatting -- finding, or coding, some backport of format would be the main alternative).

Don't have incorrectly spelled variable names - that's what testing is for.
On the face of it, displaying something seems better than throwing an exception, but sooner or later one of those mistranslations is bound to cause offence to some user.
.format() is the way to go these days.
Reconsider passing locals() straight into the translation. It will save you worrying about what happens if someone works out a way to do something sneaky with self in one of the translations.

How safe is expression evaluation using eval?

I am building a website where I have a need that user should be able to evaluate some expression based from the value in DB tables, instead of using tools like pyparsing etc, I am thinking of using python itself, and have come up with a solution which is sufficient for my purpose. I am basically using eval to evaluate the expression and passing globals dict with empty __builtins__ so that nothing can be accessed and a locals dict for values from DB, if user will need some functions I can pass those too e.g.
import datetime
def today():
return datetime.datetime.now()
expression = """ first_name.lower() == "anurag" and today().year == 2010 """
print eval(expression, {'__builtins__':{}}, {'first_name':'Anurag', 'today':today})
So my question is how safe it would be , I have three criteria
Can user access current state of my program or table etc someshow?
Can user have access to os level calls?
Can user halt my system by looping or using much memory e.g. by doing range(10*8), in some cases he can e.g 100**1000 etc so 3 is not so much of a problem. i may check such op with tokenize and anyway I will be using GAE so it is not not much of concern.
Edit: IMO this is not the duplicate of Q:661084 because where it ends this one starts, I want to know even with __builtins__ blocked, can user do bad things?

It's completely unsafe to use eval, even with built-ins emptied and blocked -- the attacker can start with a literal, get its __class__, etc, etc, up to object, its __subclasses__, and so forth... basically, Python introspection is just too strong to stand up to a skilled, determined attacker.
ast.literal_eval is safe, if you can live by its limitations...

Certainly it's possible to consume all available memory or create an infinite loop even without the builtins. There are many ways to do it such as 'a'*999999*999999 or to make an infinite loop:
>>> print eval('[[x.append(a) for a in x] for x in [[0]]]',
... {'__builtins__':{}}, {'first_name':'Anurag', 'today':today})
As for 1) and 2), I'm not sure but it looks risky. Here is one thing that I tried that I thought would work, but it seems that someone else already considered that line of attack and blocked it:
>>> import datetime
>>> def today():
>>> return datetime.datetime.now()
>>>
>>> print eval('today.func_globals', {'__builtins__':{}}, {'first_name':'Anurag', 'today':today})
RuntimeError: restricted attribute
I was half expecting to get this instead:
{'__builtins__': <module '__builtin__' (built-in)>, ...
So I think it's probably a bad idea. You only need one tiny hole and you give access to your entire system. Have you considered other methods that don't use eval? What is wrong with them?

It is possible to get create and invoke any class defined in the program, which includes ones that can exit the Python interpreter. In addition, you can create and execute arbitrary strings of bytecode, which can segfault the interpreter. See Eval really is dangerous for all the details.

What are the important language features (idioms) of Python to learn early on [duplicate]

This question already has answers here:
The Zen of Python [closed]
(22 answers)
Python: Am I missing something? [closed]
(16 answers)
Closed 8 years ago.
I would be interested in knowing what the StackOverflow community thinks are the important language features (idioms) of Python. Features that would define a programmer as Pythonic.
Python (pythonic) idiom - "code expression" that is natural or characteristic to the language Python.
Plus, Which idioms should all Python programmers learn early on?
Thanks in advance
Related:
Code Like a Pythonista: Idiomatic Python
Python: Am I missing something?

Python is a language that can be described as:
"rules you can fit in the
palm of your hand with a huge bag of
hooks".
Nearly everything in python follows the same simple standards. Everything is accessible, changeable, and tweakable. There are very few language level elements.
Take for example, the len(data) builtin function. len(data) works by simply checking for a data.__len__() method, and then calls it and returns the value. That way, len() can work on any object that implements a __len__() method.
Start by learning about the types and basic syntax:
Dynamic Strongly Typed Languages
bool, int, float, string, list, tuple, dict, set
statements, indenting, "everything is an object"
basic function definitions
Then move on to learning about how python works:
imports and modules (really simple)
the python path (sys.path)
the dir() function
__builtins__
Once you have an understanding of how to fit pieces together, go back and cover some of the more advanced language features:
iterators
overrides like __len__ (there are tons of these)
list comprehensions and generators
classes and objects (again, really simple once you know a couple rules)
python inheritance rules
And once you have a comfort level with these items (with a focus on what makes them pythonic), look at more specific items:
Threading in python (note the Global Interpreter Lock)
context managers
database access
file IO
sockets
etc...
And never forget The Zen of Python (by Tim Peters)
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

This page covers all the major python idioms: http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html

An important idiom in Python is docstrings.
Every object has a __doc__ attribute that can be used to get help on that object. You can set the __doc__ attribute on modules, classes, methods, and functions like this:
# this is m.py
""" module docstring """
class c:
"""class docstring"""
def m(self):
"""method docstring"""
pass
def f(a):
"""function f docstring"""
return
Now, when you type help(m), help(m.f) etc. it will print the docstring as a help message.
Because it's just part of normal object introspection this can be used by documention generating systems like epydoc or used for testing purposes by unittest.
It can also be put to more unconventional (i.e. non-idiomatic) uses such as grammars in Dparser.
Where it gets even more interesting to me is that, even though doc is a read-only attribute on most objects, you can use them anywhere like this:
x = 5
""" pseudo docstring for x """
and documentation tools like epydoc can pick them up and format them properly (as opposed to a normal comment which stays inside the code formatting.

Decorators get my vote. Where else can you write something like:
def trace(num_args=0):
def wrapper(func):
def new_f(*a,**k):
print_args = ''
if num_args > 0:
print_args = str.join(',', [str(x) for x in a[0:num_args]])
print('entering %s(%s)' %(f.__name__,print_args))
rc = f(*a,**k)
if rc is not None:
print('exiting %s(%s)=%s' %(f.__name__,str(rc)))
else:
print('exiting %s(%s)' %(f.__name__))
return rc
return new_f
return wrapper
#trace(1)
def factorial(n):
if n < 2:
return 1
return n * factorial(n-1)
factorial(5)
and get output like:
entering factorial(5)
entering factorial(4)
entering factorial(3)
entering factorial(2)
entering factorial(1)
entering factorial(0)
exiting factorial(0)=1
exiting factorial(1)=1
exiting factorial(2)=2
exiting factorial(3)=6
exiting factorial(4)=24
exiting factorial(5)=120

Everything connected to list usage.
Comprehensions, generators, etc.

Personally, I really like Python syntax defining code blocks by using indentation, and not by the words "BEGIN" and "END" (as in Microsoft's Basic and Visual Basic - I don't like these) or by using left- and right-braces (as in C, C++, Java, Perl - I like these).
This really surprised me because, although indentation has always been very important to me, I didn't make to much "noise" about it - I lived with it, and it is considered a skill to be able to read other peoples, "spaghetti" code. Furthermore, I never heard another programmer suggest making indentation a part of a language. Until Python! I only wish I had realized this idea first.
To me, it is as if Python's syntax forces you to write good, readable code.
Okay, I'll get off my soap-box. ;-)

From a more advanced viewpoint, understanding how dictionaries are used internally by Python. Classes, functions, modules, references are all just properties on a dictionary. Once this is understood it's easy to understand how to monkey patch and use the powerful __gettattr__, __setattr__, and __call__ methods.

Here's one that can help. What's the difference between:
[ foo(x) for x in range(0, 5) ][0]
and
( foo(x) for x in range(0, 5) ).next()
answer:
in the second example, foo is called only once. This may be important if foo has a side effect, or if the iterable being used to construct the list is large.

Two things that struck me as especially Pythonic were dynamic typing and the various flavors of lists used in Python, particularly tuples.
Python's list obsession could be said to be LISP-y, but it's got its own unique flavor. A line like:
return HandEvaluator.StraightFlush, (PokerCard.longFaces[index + 4],
PokerCard.longSuits[flushSuit]), []
or even
return False, False, False
just looks like Python and nothing else. (Technically, you'd see the latter in Lua as well, but Lua is pretty Pythonic in general.)

Using string substitutions:
name = "Joe"
age = 12
print "My name is %s, I am %s" % (name, age)
When I'm not programming in python, that simple use is what I miss most.

Another thing you cannot start early enough is probably testing. Here especially doctests are a great way of testing your code by explaining it at the same time.
doctests are simple text file containing an interactive interpreter session plus text like this:
Let's instantiate our class::
>>> a=Something(text="yes")
>>> a.text
yes
Now call this method and check the results::
>>> a.canify()
>>> a.text
yes, I can
If e.g. a.text returns something different the test will fail.
doctests can be inside docstrings or standalone textfiles and are executed by using the doctests module. Of course the more known unit tests are also available.

I think that tutorials online and books only talk about doing things, not doing things in the best way. Along with the python syntax i think that speed in some cases is important.
Python provides a way to benchmark functions, actually two!!
One way is to use the profile module, like so:
import profile
def foo(x, y, z):
return x**y % z # Just an example.
profile.run('foo(5, 6, 3)')
Another way to do this is to use the timeit module, like this:
import timeit
def foo(x, y, z):
return x**y % z # Can also be 'pow(x, y, z)' which is way faster.
timeit.timeit('foo(5, 6, 3)', 'from __main__ import *', number = 100)
# timeit.timeit(testcode, setupcode, number = number_of_iterations)