What concept is involved here? Example in Python and R. - python

I am trying to find the right language to describe the following concept. Maybe someone can help me out.
This is a general question about programming but I'll use Python and R for examples.
In Python, we can put something in a dictionary like this
myData = {}
myData["myField"] = 14
In R, for example, using the data.table package, we could write something like
data = data.table(x = c(1, 2, 3))
data[,myField: = x^2]
These do different things but compare the second line of each of them. In Python, the "myField" is a string. In the R data.table example, there is no string. The R example is kinda nice because it saves you typing but then it gives you trouble if want to write a program where myField is a variable. In Python that is trivial because you can just do
myData[myVariable] = 14
with myVariable being defined as another string. In R, you can do this too but have to use a different syntax which means you have to know two completely different syntactical ways of programming it.
My question: What is this called? I know it has something to do with scoping rules, (perhaps meta programming?), but can't figure out the right language for it. Anyone?

I believe this programming language "feature" is known as "bare strings".
PHP also has (deprecated) support for this: Why is $foo[bar] wrong?
It's widely considered to be a pretty terrible idea because of the risk of overlap with variable or constant names, and should definitely be avoided in production code. However, JavaScript has an interesting twist on the idea that avoids those issues:
var obj = { key: "value" };
You can define the keys on inline objects without using quotation marks. This works because you can't do it the other way - the key is parsed as a string, never a variable name. I've found this to be a pretty useful tradeoff in programs where you only use predefined keys in dictionaries. The Python version, for reference, requires the quotation marks, but allows you to use a variable, as you demonstrated:
obj = { "key": "value", key_var: "value" }

I'm not sure if this is what you're asking, but, in your Python example, the "myField" in myData["myField"] is a string literal -- i.e., you're hard-coding in the key value. Similarly, if you defined a Python function
def myFun(a):
# magic happens
and you invoked the function as myFun("goober"), you'd be passing in a string literal ("goober"), as opposed to defining a variable b = "goober" then passing in the variable as myFun(b).

Your question seems related to the definition/use of (unevaluated) symbols in Lisp (and related languages, and also Mathematica), c.f. the answers here

Related

why there is space for "?" in the code. How to remove it? [duplicate]

I would like to put an int into a string. This is what I am doing at the moment:
num = 40
plot.savefig('hanning40.pdf') #problem line
I have to run the program for several different numbers, so I'd like to do a loop. But inserting the variable like this doesn't work:
plot.savefig('hanning', num, '.pdf')
How do I insert a variable into a Python string?
See also
If you tried using + to concatenate a number with a string (or between strings, etc.) and got an error message, see How can I concatenate str and int objects?.
If you are trying to assemble a URL with variable data, do not use ordinary string formatting, because it is error-prone and more difficult than necessary. Specialized tools are available. See Add params to given URL in Python.
If you are trying to assemble a SQL query, do not use ordinary string formatting, because it is a major security risk. This is the cause of "SQL injection" which costs real companies huge amounts of money every year. See for example Python: best practice and securest way to connect to MySQL and execute queries for proper techniques.
If you just want to print (output) the string, you can prepare it this way first, or if you don't need the string for anything else, print each piece of the output individually using a single call to print. See How can I print multiple things (fixed text and/or variable values) on the same line, all at once? for details on both approaches.
Using f-strings:
plot.savefig(f'hanning{num}.pdf')
This was added in 3.6 and is the new preferred way.
Using str.format():
plot.savefig('hanning{0}.pdf'.format(num))
String concatenation:
plot.savefig('hanning' + str(num) + '.pdf')
Conversion Specifier:
plot.savefig('hanning%s.pdf' % num)
Using local variable names (neat trick):
plot.savefig('hanning%(num)s.pdf' % locals())
Using string.Template:
plot.savefig(string.Template('hanning${num}.pdf').substitute(locals()))
See also:
Fancier Output Formatting - The Python Tutorial
Python 3's f-Strings: An Improved String Formatting Syntax (Guide) - RealPython
With the introduction of formatted string literals ("f-strings" for short) in Python 3.6, it is now possible to write this with a briefer syntax:
>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'
With the example given in the question, it would look like this
plot.savefig(f'hanning{num}.pdf')
plot.savefig('hanning(%d).pdf' % num)
The % operator, when following a string, allows you to insert values into that string via format codes (the %d in this case). For more details, see the Python documentation:
printf-style String Formatting
You can use + as the normal string concatenation function as well as str().
"hello " + str(10) + " world" == "hello 10 world"
In general, you can create strings using:
stringExample = "someString " + str(someNumber)
print(stringExample)
plot.savefig(stringExample)
If you would want to put multiple values into the string you could make use of format
nums = [1,2,3]
plot.savefig('hanning{0}{1}{2}.pdf'.format(*nums))
Would result in the string hanning123.pdf. This can be done with any array.
Special cases
Depending on why variable data is being used with strings, the general-purpose approaches may not be appropriate.
If you need to prepare an SQL query
Do not use any of the usual techniques for assembling a string. Instead, use your SQL library's functionality for parameterized queries.
A query is code, so it should not be thought about like normal text. Using the library will make sure that any inserted text is properly escaped. If any part of the query could possibly come from outside the program in any way, that is an opportunity for a malevolent user to perform SQL injection. This is widely considered one of the important computer security problems, costing real companies huge amounts of money every year and causing problems for countless customers. Even if you think you know the data is "safe", there is no real upside to using any other approach.
The syntax will depend on the library you are using and is outside the scope of this answer.
If you need to prepare a URL query string
See Add params to given URL in Python. Do not do it yourself; there is no practical reason to make your life harder.
Writing to a file
While it's possible to prepare a string ahead of time, it may be simpler and more memory efficient to just write each piece of data with a separate .write call. Of course, non-strings will still need to be converted to string before writing, which may complicate the code. There is not a one-size-fits-all answer here, but choosing badly will generally not matter very much.
If you are simply calling print
The built-in print function accepts a variable number of arguments, and can take in any object and stringify it using str. Before trying string formatting, consider whether simply passing multiple arguments will do what you want. (You can also use the sep keyword argument to control spacing between the arguments.)
# display a filename, as an example
print('hanning', num, '.pdf', sep='')
Of course, there may be other reasons why it is useful for the program to assemble a string; so by all means do so where appropriate.
It's important to note that print is a special case. The only functions that work this way are ones that are explicitly written to work this way. For ordinary functions and methods, like input, or the savefig method of Matplotlib plots, we need to prepare a string ourselves.
Concatenation
Python supports using + between two strings, but not between strings and other types. To work around this, we need to convert other values to string explicitly: 'hanning' + str(num) + '.pdf'.
Template-based approaches
Most ways to solve the problem involve having some kind of "template" string that includes "placeholders" that show where information should be added, and then using some function or method to add the missing information.
f-strings
This is the recommended approach when possible. It looks like f'hanning{num}.pdf'. The names of variables to insert appear directly in the string. It is important to note that there is not actually such a thing as an "f-string"; it's not a separate type. Instead, Python will translate the code ahead of time:
>>> def example(num):
... return f'hanning{num}.pdf'
...
>>> import dis
>>> dis.dis(example)
2 0 LOAD_CONST 1 ('hanning')
2 LOAD_FAST 0 (num)
4 FORMAT_VALUE 0
6 LOAD_CONST 2 ('.pdf')
8 BUILD_STRING 3
10 RETURN_VALUE
Because it's a special syntax, it can access opcodes that aren't used in other approaches.
str.format
This is the recommended approach when f-strings aren't possible - mainly, because the template string needs to be prepared ahead of time and filled in later. It looks like 'hanning{}.pdf'.format(num), or 'hanning{num}.pdf'.format(num=num)'. Here, format is a method built in to strings, which can accept arguments either by position or keyword.
Particularly for str.format, it's useful to know that the built-in locals, globals and vars functions return dictionaries that map variable names to the contents of those variables. Thus, rather than something like '{a}{b}{c}'.format(a=a, b=b, c=c), we can use something like '{a}{b}{c}'.format(**locals()), unpacking the locals() dict.
str.format_map
This is a rare variation on .format. It looks like 'hanning{num}.pdf'.format_map({'num': num}). Rather than accepting keyword arguments, it accepts a single argument which is a mapping.
That probably doesn't sound very useful - after all, rather than 'hanning{num}.pdf'.format_map(my_dict), we could just as easily write 'hanning{num}.pdf'.format(**my_dict). However, this is useful for mappings that determine values on the fly, rather than ordinary dicts. In these cases, unpacking with ** might not work, because the set of keys might not be determined ahead of time; and trying to unpack keys based on the template is unwieldy (imagine: 'hanning{num}.pdf'.format(num=my_mapping[num]), with a separate argument for each placeholder).
string.Formatter
The string standard library module contains a rarely used Formatter class. Using it looks like string.Formatter().format('hanning{num}.pdf', num=num). The template string uses the same syntax again. This is obviously clunkier than just calling .format on the string; the motivation is to allow users to subclass Formatter to define a different syntax for the template string.
All of the above approaches use a common "formatting language" (although string.Formatter allows changing it); there are many other things that can be put inside the {}. Explaining how it works is beyond the scope of this answer; please consult the documentation. Do keep in mind that literal { and } characters need to be escaped by doubling them up. The syntax is presumably inspired by C#.
The % operator
This is a legacy way to solve the problem, inspired by C and C++. It has been discouraged for a long time, but is still supported. It looks like 'hanning%s.pdf' % num, for simple cases. As you'd expect, literal '%' symbols in the template need to be doubled up to escape them.
It has some issues:
It seems like the conversion specifier (the letter after the %) should match the type of whatever is being interpolated, but that's not actually the case. Instead, the value is converted to the specified type, and then to string from there. This isn't normally necessary; converting directly to string works most of the time, and converting to other types first doesn't help most of the rest of the time. So 's' is almost always used (unless you want the repr of the value, using 'r'). Despite that, the conversion specifier is a mandatory part of the syntax.
Tuples are handled specially: passing a tuple on the right-hand side is the way to provide multiple arguments. This is an ugly special case that's necessary because we aren't using function-call syntax. As a result, if you actually want to format a tuple into a single placeholder, it must be wrapped in a 1-tuple.
Other sequence types are not handled specially, and the different behaviour can be a gotcha.
string.Template
The string standard library module contains a rarely used Template class. Instances provide substitute and safe_substitute methods that work similarly to the built-in .format (safe_substitute will leave placeholders intact rather than raising an exception when the arguments don't match). This should also be considered a legacy approach to the problem.
It looks like string.Template('hanning$num.pdf').substitute(num=num), and is inspired by traditional Perl syntax. It's obviously clunkier than the .format approach, since a separate class has to be used before the method is available. Braces ({}) can be used optionally around the name of the variable, to avoid ambiguity. Similarly to the other methods, literal '$' in the template needs to be doubled up for escaping.
I had a need for an extended version of this: instead of embedding a single number in a string, I needed to generate a series of file names of the form 'file1.pdf', 'file2.pdf' etc. This is how it worked:
['file' + str(i) + '.pdf' for i in range(1,4)]
You can make dict and substitute variables in your string.
var = {"name": "Abdul Jalil", "age": 22}
temp_string = "My name is %(name)s. I am %(age)s years old." % var

How to use single quotes in Python keywords/variable name?

I want to use single quote in a variable name and keyword. I know how to do this for a variable value using single and double quotes("5'_Exon_Annotation") or using black slash to escape the single quote. But not sure how to do this with variable name/ key word. Below are examples of how I want to assign values to column names and a variable that needs single quote.
5'co-ordinate = "something"
gene_df = gene_df.assign(gene_position = gene_pos,
5'_co-ordinate = bkpt1_list[1],
3'_co-ordinate = bkpt2_list[1])
First of all is it possible in Python? I couldn't find a post related to this.
While I wouldn't recommend it at all, it is possible to use all kinds of characters for attributes of classes using setattr(), like so:
class Test:
def __init__(self):
setattr(self, 'this variable has ""', 1)
test = Test()
print(getattr(test, 'this variable has ""'))
# test.this variable has "" does NOT work
But I really wouldn't recommend it. The intestesting thing about this is, that setting an attribute this way makes it kind of hidden, because you can only access it using getattr().
If you use globals() or locals() you can also set variables this way:
locals()['this variable has ""'] = 1 # or globals()
print(locals()['this variable has ""'])
# this variable has "" = 2 does NOT work
That makes it really cumbersome to access the variable though, and again, I really wouldn't recommend it. It's a neat trick (debateable I guess) if you really want to hide something though.
Here are the available characters for Python identifiers. You might find something which is close to an apostrophe but not the common one.
Btw. I'd not suggest to deviate from the ASCII table. If the language naturally support it and the editors offer helpers to write those, it's fine (like Julia) but for Python it can be quite confusing for others.
https://docs.python.org/3/reference/lexical_analysis.html#identifiers
Here is an example of a valid Python code using some kind of a quote in the function name (copy-paste it, but the single quote is for example not displayed in Safari on macOS, however it's there as you can see in the screenshot of my terminal):
In [1]: def f֜(x): return 2*x
In [2]: f֜(3)
Out[2]: 6

how to insert variable into a re.compile in python [duplicate]

I would like to put an int into a string. This is what I am doing at the moment:
num = 40
plot.savefig('hanning40.pdf') #problem line
I have to run the program for several different numbers, so I'd like to do a loop. But inserting the variable like this doesn't work:
plot.savefig('hanning', num, '.pdf')
How do I insert a variable into a Python string?
See also
If you tried using + to concatenate a number with a string (or between strings, etc.) and got an error message, see How can I concatenate str and int objects?.
If you are trying to assemble a URL with variable data, do not use ordinary string formatting, because it is error-prone and more difficult than necessary. Specialized tools are available. See Add params to given URL in Python.
If you are trying to assemble a SQL query, do not use ordinary string formatting, because it is a major security risk. This is the cause of "SQL injection" which costs real companies huge amounts of money every year. See for example Python: best practice and securest way to connect to MySQL and execute queries for proper techniques.
If you just want to print (output) the string, you can prepare it this way first, or if you don't need the string for anything else, print each piece of the output individually using a single call to print. See How can I print multiple things (fixed text and/or variable values) on the same line, all at once? for details on both approaches.
Using f-strings:
plot.savefig(f'hanning{num}.pdf')
This was added in 3.6 and is the new preferred way.
Using str.format():
plot.savefig('hanning{0}.pdf'.format(num))
String concatenation:
plot.savefig('hanning' + str(num) + '.pdf')
Conversion Specifier:
plot.savefig('hanning%s.pdf' % num)
Using local variable names (neat trick):
plot.savefig('hanning%(num)s.pdf' % locals())
Using string.Template:
plot.savefig(string.Template('hanning${num}.pdf').substitute(locals()))
See also:
Fancier Output Formatting - The Python Tutorial
Python 3's f-Strings: An Improved String Formatting Syntax (Guide) - RealPython
With the introduction of formatted string literals ("f-strings" for short) in Python 3.6, it is now possible to write this with a briefer syntax:
>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'
With the example given in the question, it would look like this
plot.savefig(f'hanning{num}.pdf')
plot.savefig('hanning(%d).pdf' % num)
The % operator, when following a string, allows you to insert values into that string via format codes (the %d in this case). For more details, see the Python documentation:
printf-style String Formatting
You can use + as the normal string concatenation function as well as str().
"hello " + str(10) + " world" == "hello 10 world"
In general, you can create strings using:
stringExample = "someString " + str(someNumber)
print(stringExample)
plot.savefig(stringExample)
If you would want to put multiple values into the string you could make use of format
nums = [1,2,3]
plot.savefig('hanning{0}{1}{2}.pdf'.format(*nums))
Would result in the string hanning123.pdf. This can be done with any array.
Special cases
Depending on why variable data is being used with strings, the general-purpose approaches may not be appropriate.
If you need to prepare an SQL query
Do not use any of the usual techniques for assembling a string. Instead, use your SQL library's functionality for parameterized queries.
A query is code, so it should not be thought about like normal text. Using the library will make sure that any inserted text is properly escaped. If any part of the query could possibly come from outside the program in any way, that is an opportunity for a malevolent user to perform SQL injection. This is widely considered one of the important computer security problems, costing real companies huge amounts of money every year and causing problems for countless customers. Even if you think you know the data is "safe", there is no real upside to using any other approach.
The syntax will depend on the library you are using and is outside the scope of this answer.
If you need to prepare a URL query string
See Add params to given URL in Python. Do not do it yourself; there is no practical reason to make your life harder.
Writing to a file
While it's possible to prepare a string ahead of time, it may be simpler and more memory efficient to just write each piece of data with a separate .write call. Of course, non-strings will still need to be converted to string before writing, which may complicate the code. There is not a one-size-fits-all answer here, but choosing badly will generally not matter very much.
If you are simply calling print
The built-in print function accepts a variable number of arguments, and can take in any object and stringify it using str. Before trying string formatting, consider whether simply passing multiple arguments will do what you want. (You can also use the sep keyword argument to control spacing between the arguments.)
# display a filename, as an example
print('hanning', num, '.pdf', sep='')
Of course, there may be other reasons why it is useful for the program to assemble a string; so by all means do so where appropriate.
It's important to note that print is a special case. The only functions that work this way are ones that are explicitly written to work this way. For ordinary functions and methods, like input, or the savefig method of Matplotlib plots, we need to prepare a string ourselves.
Concatenation
Python supports using + between two strings, but not between strings and other types. To work around this, we need to convert other values to string explicitly: 'hanning' + str(num) + '.pdf'.
Template-based approaches
Most ways to solve the problem involve having some kind of "template" string that includes "placeholders" that show where information should be added, and then using some function or method to add the missing information.
f-strings
This is the recommended approach when possible. It looks like f'hanning{num}.pdf'. The names of variables to insert appear directly in the string. It is important to note that there is not actually such a thing as an "f-string"; it's not a separate type. Instead, Python will translate the code ahead of time:
>>> def example(num):
... return f'hanning{num}.pdf'
...
>>> import dis
>>> dis.dis(example)
2 0 LOAD_CONST 1 ('hanning')
2 LOAD_FAST 0 (num)
4 FORMAT_VALUE 0
6 LOAD_CONST 2 ('.pdf')
8 BUILD_STRING 3
10 RETURN_VALUE
Because it's a special syntax, it can access opcodes that aren't used in other approaches.
str.format
This is the recommended approach when f-strings aren't possible - mainly, because the template string needs to be prepared ahead of time and filled in later. It looks like 'hanning{}.pdf'.format(num), or 'hanning{num}.pdf'.format(num=num)'. Here, format is a method built in to strings, which can accept arguments either by position or keyword.
Particularly for str.format, it's useful to know that the built-in locals, globals and vars functions return dictionaries that map variable names to the contents of those variables. Thus, rather than something like '{a}{b}{c}'.format(a=a, b=b, c=c), we can use something like '{a}{b}{c}'.format(**locals()), unpacking the locals() dict.
str.format_map
This is a rare variation on .format. It looks like 'hanning{num}.pdf'.format_map({'num': num}). Rather than accepting keyword arguments, it accepts a single argument which is a mapping.
That probably doesn't sound very useful - after all, rather than 'hanning{num}.pdf'.format_map(my_dict), we could just as easily write 'hanning{num}.pdf'.format(**my_dict). However, this is useful for mappings that determine values on the fly, rather than ordinary dicts. In these cases, unpacking with ** might not work, because the set of keys might not be determined ahead of time; and trying to unpack keys based on the template is unwieldy (imagine: 'hanning{num}.pdf'.format(num=my_mapping[num]), with a separate argument for each placeholder).
string.Formatter
The string standard library module contains a rarely used Formatter class. Using it looks like string.Formatter().format('hanning{num}.pdf', num=num). The template string uses the same syntax again. This is obviously clunkier than just calling .format on the string; the motivation is to allow users to subclass Formatter to define a different syntax for the template string.
All of the above approaches use a common "formatting language" (although string.Formatter allows changing it); there are many other things that can be put inside the {}. Explaining how it works is beyond the scope of this answer; please consult the documentation. Do keep in mind that literal { and } characters need to be escaped by doubling them up. The syntax is presumably inspired by C#.
The % operator
This is a legacy way to solve the problem, inspired by C and C++. It has been discouraged for a long time, but is still supported. It looks like 'hanning%s.pdf' % num, for simple cases. As you'd expect, literal '%' symbols in the template need to be doubled up to escape them.
It has some issues:
It seems like the conversion specifier (the letter after the %) should match the type of whatever is being interpolated, but that's not actually the case. Instead, the value is converted to the specified type, and then to string from there. This isn't normally necessary; converting directly to string works most of the time, and converting to other types first doesn't help most of the rest of the time. So 's' is almost always used (unless you want the repr of the value, using 'r'). Despite that, the conversion specifier is a mandatory part of the syntax.
Tuples are handled specially: passing a tuple on the right-hand side is the way to provide multiple arguments. This is an ugly special case that's necessary because we aren't using function-call syntax. As a result, if you actually want to format a tuple into a single placeholder, it must be wrapped in a 1-tuple.
Other sequence types are not handled specially, and the different behaviour can be a gotcha.
string.Template
The string standard library module contains a rarely used Template class. Instances provide substitute and safe_substitute methods that work similarly to the built-in .format (safe_substitute will leave placeholders intact rather than raising an exception when the arguments don't match). This should also be considered a legacy approach to the problem.
It looks like string.Template('hanning$num.pdf').substitute(num=num), and is inspired by traditional Perl syntax. It's obviously clunkier than the .format approach, since a separate class has to be used before the method is available. Braces ({}) can be used optionally around the name of the variable, to avoid ambiguity. Similarly to the other methods, literal '$' in the template needs to be doubled up for escaping.
I had a need for an extended version of this: instead of embedding a single number in a string, I needed to generate a series of file names of the form 'file1.pdf', 'file2.pdf' etc. This is how it worked:
['file' + str(i) + '.pdf' for i in range(1,4)]
You can make dict and substitute variables in your string.
var = {"name": "Abdul Jalil", "age": 22}
temp_string = "My name is %(name)s. I am %(age)s years old." % var

How to use gettext with python >3.6 f-strings

Previously you would use gettext as following:
_('Hey {},').format(username)
but what about new Python's f-string?
f'Hey {username}'
'Hey {},' is contained in your translation dictionary as is.
If you use f'Hey {username},', that creates another string, which won't be translated.
In that case, the format method remains the only one useable, but you could approach the f-string features by using named parameters
_('Hey {username},').format(username=username)
or if you have a dictionary containing your data, this cool trick where format picks the required information in the input dictionary:
d = {"username":"John", "city":"New York", "unused":"doesn't matter"}
_('Hey {username} from {city},').format(**d)
My solution is to make a function f() which performs the f-string interpolation after gettext has been called.
from copy import copy
from inspect import currentframe
def f(s):
frame = currentframe().f_back
kwargs = copy(frame.f_globals)
kwargs.update(frame.f_locals)
return eval(s.format(**kwargs))
Now you just wrap _(...) in f() and don’t preface the string with an f:
f(_('Hey, {username}'))
Note of caution
I’m usually against the use of eval as it could make the function potentially unsafe, but I personally think it should be justified here, so long as you’re aware of what’s being formatted. That said use at your own risk.
Remember
This isn’t a perfect solution, this is just my solution. As per PEP 498 states each formatting method “have their advantages, but in addition have disadvantages” including this.
For example if you need to change the expression inside the string then it will no longer match, therefore not be translated unless you also update your .po file as well. Also if you’re not the one translating them and you use an expression that’s hard to decipher what the outcome will be then that can cause miscommunication or other issues in translation.

Dash in dictionary key and eval

Before I get beat to death by the 'eval is evil' crowd, it's a necessary evil in this case and I can't change it. Eval has it's uses, and in a tightly controlled environment it's very powerful.
However, I have an issue with no obvious solution and I'm hoping for outside the box thinking.
>>> mydict = {"a-b": "woohoo"}
>>> mydict["a-b"]
'woohoo'
>>> eval('mydict["a-b"]')
'woohoo'
>>> eval('a-b', mydict)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1, in <module>
NameError: name 'a' is not defined
Unfortuately, the last case is the one I'm forced to use and obviously it doesn't work. Any ideas how to eval an expression into my 'globals' or 'locals' objects and not have it interpret the - as a minus operator? Some of the keys in my 'globals' object do have dashes in the key names, and this is data I cannot control.
To the comments below.
the data is coming to me from an external source. I cannot dictate or control the format of the data at all.
the 'clause' I'm evaluating is coming to me from user configuration that is stored.
This is part of a larger system, where users can push JSON data in via an api, we handle the data internally as a dictionary, then we apply certain rules to the data. The rules are provided as configuration by the administrators from a web interface.
Ultimately, I need to allow the user to give me a (possibly complex) python one liner and evaluate it against a dictionary. Isn't that exactly what eval is for? If there is a better way given I cannot dictating the format of the data and must allow the user to give me a string with an evaluation? Eval is amazing because it lets the user do quite a few cool things, like use .get() and len(), but obviously it has downside as well like the aforementioned inability to distinguish or escape out the -.
Thanks!
You're trying to make the string "a-b" a symbol during your evaluation. Traditionally, this doesn't work because "-" (hyphen) isn't word. Only the [A-Za-z0-9_] (word) characters can be used in symbol names. Changing hyphen to underscore works fine:
>>> mydict = {"a_b": "woohoo"}
>>> eval('a_b', mydict)
'woohoo'
>>>
However, in Python3, many Unicode characters can be used in a symbol and some might be adequate substitutions for ASCII hyphen:
>>> mydict = {"aᐨb": "woohoo"}
>>> eval('aᐨb', mydict)
'woohoo'
>>>
Here I used a Canadian syllabics final short horizontal stroke (though clearly an abuse of this code's intended purpose.) See the posting What Unicode symbols are accepted in Python3 variable names? for more about this approach.
I need to allow the user to give me a (possibly complex) python one
liner and evaluate it against a dictionary.
If that's the case, shouldn't a and b be part of that dictionary, which solves the problem:
>>> mydict = {"a": 34, "b": 13}
>>> eval('a-b', mydict)
21
Instead of using mydict as the global variable dict for the evaluated expression, give the user access to it as a dict:
eval(user_expression, {'data': mydict})
Then the user accesses it with expressions like
data['a-b']
instead of trying to use a-b as a variable name and needing to somehow break the Python parser. This is particularly nice if you might have a JSON array or other JSON type instead of a JSON object, since a Python list can't be used as the global variable environment for eval.
If you want to make the syntax a bit nicer, you can give the user Javascript-like dotted attribute access:
class ItemsAsAttributesDict(dict):
def __getattr__(self, name):
return self[name]
# when loading the JSON
dict = json.loads(json_string, object_hook=ItemsAsAttributesDict)
Then just like in Javascript, dict entries like data['a'] can be accessed as data.a, but entries like data['a-b'] still require bracket notation.
If you're set on using mydict as the global variables dict, the user will have to use globals() to access keys that aren't valid variable names:
globals()['a-b']
Be aware that using eval opens up nasty attack vectors. People are going to think these queries are safe, and they'll evaluate queries from untrusted sources, and then someone will ask for the value of
__import__('os').system('arbitrary_evil_command')
and everyone will hate you.
Also, using eval ties your program to Python syntax. You'll have a hell of a time porting it to any other language, especially since users will be depending on things like list comprehensions and other Python features you might not have expected them to use. You might even have a hard time transitioning between Python versions, or supporting different Python versions.

Categories