We Keep Coding

why there is space for "?" in the code. How to remove it? [duplicate]

I would like to put an int into a string. This is what I am doing at the moment:
num = 40
plot.savefig('hanning40.pdf') #problem line
I have to run the program for several different numbers, so I'd like to do a loop. But inserting the variable like this doesn't work:
plot.savefig('hanning', num, '.pdf')
How do I insert a variable into a Python string?
See also
If you tried using + to concatenate a number with a string (or between strings, etc.) and got an error message, see How can I concatenate str and int objects?.
If you are trying to assemble a URL with variable data, do not use ordinary string formatting, because it is error-prone and more difficult than necessary. Specialized tools are available. See Add params to given URL in Python.
If you are trying to assemble a SQL query, do not use ordinary string formatting, because it is a major security risk. This is the cause of "SQL injection" which costs real companies huge amounts of money every year. See for example Python: best practice and securest way to connect to MySQL and execute queries for proper techniques.
If you just want to print (output) the string, you can prepare it this way first, or if you don't need the string for anything else, print each piece of the output individually using a single call to print. See How can I print multiple things (fixed text and/or variable values) on the same line, all at once? for details on both approaches.

Using f-strings:
plot.savefig(f'hanning{num}.pdf')
This was added in 3.6 and is the new preferred way.
Using str.format():
plot.savefig('hanning{0}.pdf'.format(num))
String concatenation:
plot.savefig('hanning' + str(num) + '.pdf')
Conversion Specifier:
plot.savefig('hanning%s.pdf' % num)
Using local variable names (neat trick):
plot.savefig('hanning%(num)s.pdf' % locals())
Using string.Template:
plot.savefig(string.Template('hanning${num}.pdf').substitute(locals()))
See also:
Fancier Output Formatting - The Python Tutorial
Python 3's f-Strings: An Improved String Formatting Syntax (Guide) - RealPython

With the introduction of formatted string literals ("f-strings" for short) in Python 3.6, it is now possible to write this with a briefer syntax:
>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'
With the example given in the question, it would look like this
plot.savefig(f'hanning{num}.pdf')

plot.savefig('hanning(%d).pdf' % num)
The % operator, when following a string, allows you to insert values into that string via format codes (the %d in this case). For more details, see the Python documentation:
printf-style String Formatting

You can use + as the normal string concatenation function as well as str().
"hello " + str(10) + " world" == "hello 10 world"

In general, you can create strings using:
stringExample = "someString " + str(someNumber)
print(stringExample)
plot.savefig(stringExample)

If you would want to put multiple values into the string you could make use of format
nums = [1,2,3]
plot.savefig('hanning{0}{1}{2}.pdf'.format(*nums))
Would result in the string hanning123.pdf. This can be done with any array.

Special cases
Depending on why variable data is being used with strings, the general-purpose approaches may not be appropriate.
If you need to prepare an SQL query
Do not use any of the usual techniques for assembling a string. Instead, use your SQL library's functionality for parameterized queries.
A query is code, so it should not be thought about like normal text. Using the library will make sure that any inserted text is properly escaped. If any part of the query could possibly come from outside the program in any way, that is an opportunity for a malevolent user to perform SQL injection. This is widely considered one of the important computer security problems, costing real companies huge amounts of money every year and causing problems for countless customers. Even if you think you know the data is "safe", there is no real upside to using any other approach.
The syntax will depend on the library you are using and is outside the scope of this answer.
If you need to prepare a URL query string
See Add params to given URL in Python. Do not do it yourself; there is no practical reason to make your life harder.
Writing to a file
While it's possible to prepare a string ahead of time, it may be simpler and more memory efficient to just write each piece of data with a separate .write call. Of course, non-strings will still need to be converted to string before writing, which may complicate the code. There is not a one-size-fits-all answer here, but choosing badly will generally not matter very much.
If you are simply calling print
The built-in print function accepts a variable number of arguments, and can take in any object and stringify it using str. Before trying string formatting, consider whether simply passing multiple arguments will do what you want. (You can also use the sep keyword argument to control spacing between the arguments.)
# display a filename, as an example
print('hanning', num, '.pdf', sep='')
Of course, there may be other reasons why it is useful for the program to assemble a string; so by all means do so where appropriate.
It's important to note that print is a special case. The only functions that work this way are ones that are explicitly written to work this way. For ordinary functions and methods, like input, or the savefig method of Matplotlib plots, we need to prepare a string ourselves.
Concatenation
Python supports using + between two strings, but not between strings and other types. To work around this, we need to convert other values to string explicitly: 'hanning' + str(num) + '.pdf'.
Template-based approaches
Most ways to solve the problem involve having some kind of "template" string that includes "placeholders" that show where information should be added, and then using some function or method to add the missing information.
f-strings
This is the recommended approach when possible. It looks like f'hanning{num}.pdf'. The names of variables to insert appear directly in the string. It is important to note that there is not actually such a thing as an "f-string"; it's not a separate type. Instead, Python will translate the code ahead of time:
>>> def example(num):
... return f'hanning{num}.pdf'
...
>>> import dis
>>> dis.dis(example)
2 0 LOAD_CONST 1 ('hanning')
2 LOAD_FAST 0 (num)
4 FORMAT_VALUE 0
6 LOAD_CONST 2 ('.pdf')
8 BUILD_STRING 3
10 RETURN_VALUE
Because it's a special syntax, it can access opcodes that aren't used in other approaches.
str.format
This is the recommended approach when f-strings aren't possible - mainly, because the template string needs to be prepared ahead of time and filled in later. It looks like 'hanning{}.pdf'.format(num), or 'hanning{num}.pdf'.format(num=num)'. Here, format is a method built in to strings, which can accept arguments either by position or keyword.
Particularly for str.format, it's useful to know that the built-in locals, globals and vars functions return dictionaries that map variable names to the contents of those variables. Thus, rather than something like '{a}{b}{c}'.format(a=a, b=b, c=c), we can use something like '{a}{b}{c}'.format(**locals()), unpacking the locals() dict.
str.format_map
This is a rare variation on .format. It looks like 'hanning{num}.pdf'.format_map({'num': num}). Rather than accepting keyword arguments, it accepts a single argument which is a mapping.
That probably doesn't sound very useful - after all, rather than 'hanning{num}.pdf'.format_map(my_dict), we could just as easily write 'hanning{num}.pdf'.format(**my_dict). However, this is useful for mappings that determine values on the fly, rather than ordinary dicts. In these cases, unpacking with ** might not work, because the set of keys might not be determined ahead of time; and trying to unpack keys based on the template is unwieldy (imagine: 'hanning{num}.pdf'.format(num=my_mapping[num]), with a separate argument for each placeholder).
string.Formatter
The string standard library module contains a rarely used Formatter class. Using it looks like string.Formatter().format('hanning{num}.pdf', num=num). The template string uses the same syntax again. This is obviously clunkier than just calling .format on the string; the motivation is to allow users to subclass Formatter to define a different syntax for the template string.
All of the above approaches use a common "formatting language" (although string.Formatter allows changing it); there are many other things that can be put inside the {}. Explaining how it works is beyond the scope of this answer; please consult the documentation. Do keep in mind that literal { and } characters need to be escaped by doubling them up. The syntax is presumably inspired by C#.
The % operator
This is a legacy way to solve the problem, inspired by C and C++. It has been discouraged for a long time, but is still supported. It looks like 'hanning%s.pdf' % num, for simple cases. As you'd expect, literal '%' symbols in the template need to be doubled up to escape them.
It has some issues:
It seems like the conversion specifier (the letter after the %) should match the type of whatever is being interpolated, but that's not actually the case. Instead, the value is converted to the specified type, and then to string from there. This isn't normally necessary; converting directly to string works most of the time, and converting to other types first doesn't help most of the rest of the time. So 's' is almost always used (unless you want the repr of the value, using 'r'). Despite that, the conversion specifier is a mandatory part of the syntax.
Tuples are handled specially: passing a tuple on the right-hand side is the way to provide multiple arguments. This is an ugly special case that's necessary because we aren't using function-call syntax. As a result, if you actually want to format a tuple into a single placeholder, it must be wrapped in a 1-tuple.
Other sequence types are not handled specially, and the different behaviour can be a gotcha.
string.Template
The string standard library module contains a rarely used Template class. Instances provide substitute and safe_substitute methods that work similarly to the built-in .format (safe_substitute will leave placeholders intact rather than raising an exception when the arguments don't match). This should also be considered a legacy approach to the problem.
It looks like string.Template('hanning$num.pdf').substitute(num=num), and is inspired by traditional Perl syntax. It's obviously clunkier than the .format approach, since a separate class has to be used before the method is available. Braces ({}) can be used optionally around the name of the variable, to avoid ambiguity. Similarly to the other methods, literal '$' in the template needs to be doubled up for escaping.

I had a need for an extended version of this: instead of embedding a single number in a string, I needed to generate a series of file names of the form 'file1.pdf', 'file2.pdf' etc. This is how it worked:
['file' + str(i) + '.pdf' for i in range(1,4)]

You can make dict and substitute variables in your string.
var = {"name": "Abdul Jalil", "age": 22}
temp_string = "My name is %(name)s. I am %(age)s years old." % var

how to insert variable into a re.compile in python [duplicate]

I would like to put an int into a string. This is what I am doing at the moment:
num = 40
plot.savefig('hanning40.pdf') #problem line
I have to run the program for several different numbers, so I'd like to do a loop. But inserting the variable like this doesn't work:
plot.savefig('hanning', num, '.pdf')
How do I insert a variable into a Python string?
See also
If you tried using + to concatenate a number with a string (or between strings, etc.) and got an error message, see How can I concatenate str and int objects?.
If you are trying to assemble a URL with variable data, do not use ordinary string formatting, because it is error-prone and more difficult than necessary. Specialized tools are available. See Add params to given URL in Python.
If you are trying to assemble a SQL query, do not use ordinary string formatting, because it is a major security risk. This is the cause of "SQL injection" which costs real companies huge amounts of money every year. See for example Python: best practice and securest way to connect to MySQL and execute queries for proper techniques.
If you just want to print (output) the string, you can prepare it this way first, or if you don't need the string for anything else, print each piece of the output individually using a single call to print. See How can I print multiple things (fixed text and/or variable values) on the same line, all at once? for details on both approaches.

Using f-strings:
plot.savefig(f'hanning{num}.pdf')
This was added in 3.6 and is the new preferred way.
Using str.format():
plot.savefig('hanning{0}.pdf'.format(num))
String concatenation:
plot.savefig('hanning' + str(num) + '.pdf')
Conversion Specifier:
plot.savefig('hanning%s.pdf' % num)
Using local variable names (neat trick):
plot.savefig('hanning%(num)s.pdf' % locals())
Using string.Template:
plot.savefig(string.Template('hanning${num}.pdf').substitute(locals()))
See also:
Fancier Output Formatting - The Python Tutorial
Python 3's f-Strings: An Improved String Formatting Syntax (Guide) - RealPython

With the introduction of formatted string literals ("f-strings" for short) in Python 3.6, it is now possible to write this with a briefer syntax:
>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'
With the example given in the question, it would look like this
plot.savefig(f'hanning{num}.pdf')

plot.savefig('hanning(%d).pdf' % num)
The % operator, when following a string, allows you to insert values into that string via format codes (the %d in this case). For more details, see the Python documentation:
printf-style String Formatting

You can use + as the normal string concatenation function as well as str().
"hello " + str(10) + " world" == "hello 10 world"

In general, you can create strings using:
stringExample = "someString " + str(someNumber)
print(stringExample)
plot.savefig(stringExample)

If you would want to put multiple values into the string you could make use of format
nums = [1,2,3]
plot.savefig('hanning{0}{1}{2}.pdf'.format(*nums))
Would result in the string hanning123.pdf. This can be done with any array.

Special cases
Depending on why variable data is being used with strings, the general-purpose approaches may not be appropriate.
If you need to prepare an SQL query
Do not use any of the usual techniques for assembling a string. Instead, use your SQL library's functionality for parameterized queries.
A query is code, so it should not be thought about like normal text. Using the library will make sure that any inserted text is properly escaped. If any part of the query could possibly come from outside the program in any way, that is an opportunity for a malevolent user to perform SQL injection. This is widely considered one of the important computer security problems, costing real companies huge amounts of money every year and causing problems for countless customers. Even if you think you know the data is "safe", there is no real upside to using any other approach.
The syntax will depend on the library you are using and is outside the scope of this answer.
If you need to prepare a URL query string
See Add params to given URL in Python. Do not do it yourself; there is no practical reason to make your life harder.
Writing to a file
While it's possible to prepare a string ahead of time, it may be simpler and more memory efficient to just write each piece of data with a separate .write call. Of course, non-strings will still need to be converted to string before writing, which may complicate the code. There is not a one-size-fits-all answer here, but choosing badly will generally not matter very much.
If you are simply calling print
The built-in print function accepts a variable number of arguments, and can take in any object and stringify it using str. Before trying string formatting, consider whether simply passing multiple arguments will do what you want. (You can also use the sep keyword argument to control spacing between the arguments.)
# display a filename, as an example
print('hanning', num, '.pdf', sep='')
Of course, there may be other reasons why it is useful for the program to assemble a string; so by all means do so where appropriate.
It's important to note that print is a special case. The only functions that work this way are ones that are explicitly written to work this way. For ordinary functions and methods, like input, or the savefig method of Matplotlib plots, we need to prepare a string ourselves.
Concatenation
Python supports using + between two strings, but not between strings and other types. To work around this, we need to convert other values to string explicitly: 'hanning' + str(num) + '.pdf'.
Template-based approaches
Most ways to solve the problem involve having some kind of "template" string that includes "placeholders" that show where information should be added, and then using some function or method to add the missing information.
f-strings
This is the recommended approach when possible. It looks like f'hanning{num}.pdf'. The names of variables to insert appear directly in the string. It is important to note that there is not actually such a thing as an "f-string"; it's not a separate type. Instead, Python will translate the code ahead of time:
>>> def example(num):
... return f'hanning{num}.pdf'
...
>>> import dis
>>> dis.dis(example)
2 0 LOAD_CONST 1 ('hanning')
2 LOAD_FAST 0 (num)
4 FORMAT_VALUE 0
6 LOAD_CONST 2 ('.pdf')
8 BUILD_STRING 3
10 RETURN_VALUE
Because it's a special syntax, it can access opcodes that aren't used in other approaches.
str.format
This is the recommended approach when f-strings aren't possible - mainly, because the template string needs to be prepared ahead of time and filled in later. It looks like 'hanning{}.pdf'.format(num), or 'hanning{num}.pdf'.format(num=num)'. Here, format is a method built in to strings, which can accept arguments either by position or keyword.
Particularly for str.format, it's useful to know that the built-in locals, globals and vars functions return dictionaries that map variable names to the contents of those variables. Thus, rather than something like '{a}{b}{c}'.format(a=a, b=b, c=c), we can use something like '{a}{b}{c}'.format(**locals()), unpacking the locals() dict.
str.format_map
This is a rare variation on .format. It looks like 'hanning{num}.pdf'.format_map({'num': num}). Rather than accepting keyword arguments, it accepts a single argument which is a mapping.
That probably doesn't sound very useful - after all, rather than 'hanning{num}.pdf'.format_map(my_dict), we could just as easily write 'hanning{num}.pdf'.format(**my_dict). However, this is useful for mappings that determine values on the fly, rather than ordinary dicts. In these cases, unpacking with ** might not work, because the set of keys might not be determined ahead of time; and trying to unpack keys based on the template is unwieldy (imagine: 'hanning{num}.pdf'.format(num=my_mapping[num]), with a separate argument for each placeholder).
string.Formatter
The string standard library module contains a rarely used Formatter class. Using it looks like string.Formatter().format('hanning{num}.pdf', num=num). The template string uses the same syntax again. This is obviously clunkier than just calling .format on the string; the motivation is to allow users to subclass Formatter to define a different syntax for the template string.
All of the above approaches use a common "formatting language" (although string.Formatter allows changing it); there are many other things that can be put inside the {}. Explaining how it works is beyond the scope of this answer; please consult the documentation. Do keep in mind that literal { and } characters need to be escaped by doubling them up. The syntax is presumably inspired by C#.
The % operator
This is a legacy way to solve the problem, inspired by C and C++. It has been discouraged for a long time, but is still supported. It looks like 'hanning%s.pdf' % num, for simple cases. As you'd expect, literal '%' symbols in the template need to be doubled up to escape them.
It has some issues:
It seems like the conversion specifier (the letter after the %) should match the type of whatever is being interpolated, but that's not actually the case. Instead, the value is converted to the specified type, and then to string from there. This isn't normally necessary; converting directly to string works most of the time, and converting to other types first doesn't help most of the rest of the time. So 's' is almost always used (unless you want the repr of the value, using 'r'). Despite that, the conversion specifier is a mandatory part of the syntax.
Tuples are handled specially: passing a tuple on the right-hand side is the way to provide multiple arguments. This is an ugly special case that's necessary because we aren't using function-call syntax. As a result, if you actually want to format a tuple into a single placeholder, it must be wrapped in a 1-tuple.
Other sequence types are not handled specially, and the different behaviour can be a gotcha.
string.Template
The string standard library module contains a rarely used Template class. Instances provide substitute and safe_substitute methods that work similarly to the built-in .format (safe_substitute will leave placeholders intact rather than raising an exception when the arguments don't match). This should also be considered a legacy approach to the problem.
It looks like string.Template('hanning$num.pdf').substitute(num=num), and is inspired by traditional Perl syntax. It's obviously clunkier than the .format approach, since a separate class has to be used before the method is available. Braces ({}) can be used optionally around the name of the variable, to avoid ambiguity. Similarly to the other methods, literal '$' in the template needs to be doubled up for escaping.

I had a need for an extended version of this: instead of embedding a single number in a string, I needed to generate a series of file names of the form 'file1.pdf', 'file2.pdf' etc. This is how it worked:
['file' + str(i) + '.pdf' for i in range(1,4)]

You can make dict and substitute variables in your string.
var = {"name": "Abdul Jalil", "age": 22}
temp_string = "My name is %(name)s. I am %(age)s years old." % var

Why do two or more string arguments without commas separating them results in concatenation when given in argument list [duplicate]

I can create a multi-line string using this syntax:
string = str("Some chars "
"Some more chars")
This will produce the following string:
"Some chars Some more chars"
Is Python joining these two separate strings or is the editor/compiler treating them as a single string?
P.s: I just want to understand the internals. I know there are other ways to declare or create multi-line strings.

Read the reference manual, it's in there.
Specifically:
Multiple adjacent string or bytes literals (delimited by whitespace), possibly using different quoting conventions, are allowed, and their meaning is the same as their concatenation. Thus, "hello" 'world' is equivalent to "helloworld". This feature can be used to reduce the number of backslashes needed, to split long strings conveniently across long lines, or even to add comments to parts of strings,
(emphasis mine)
This is why:
string = str("Some chars "
"Some more chars")
is exactly the same as: str("Some chars Some more chars").
This action is performed wherever a string literal might appear, list initiliazations, function calls (as is the case with str above) et-cetera.
The only caveat is when a string literal is not contained between one of the grouping delimiters (), {} or [] but, instead, spreads between two separate physical lines. In that case we can alternatively use the backslash character to join these lines and get the same result:
string = "Some chars " \
"Some more chars"
Of course, concatenation of strings on the same physical line does not require the backslash. (string = "Hello " "World" is just fine)
Is Python joining these two separate strings or is the editor/compiler treating them as a single string?
Python is, now when exactly does Python do this is where things get interesting.
From what I could gather (take this with a pinch of salt, I'm not a parsing expert), this happens when Python transforms the parse tree (LL(1) Parser) for a given expression to it's corresponding AST (Abstract Syntax Tree).
You can get a view of the parsed tree via the parser module:
import parser
expr = """
str("Hello "
"World")
"""
pexpr = parser.expr(expr)
parser.st2list(pexpr)
This dumps a pretty big and confusing list that represents concrete syntax tree parsed from the expression in expr:
-- rest snipped for brevity --
[322,
[323,
[3, '"hello"'],
[3, '"world"']]]]]]]]]]]]]]]]]],
-- rest snipped for brevity --
The numbers correspond to either symbols or tokens in the parse tree and the mappings from symbol to grammar rule and token to constant are in Lib/symbol.py and Lib/token.py respectively.
As you can see in the snipped version I added, you have two different entries corresponding to the two different str literals in the expression parsed.
Next, we can view the output of the AST tree produced by the previous expression via the ast module provided in the Standard Library:
p = ast.parse(expr)
ast.dump(p)
# this prints out the following:
"Module(body = [Expr(value = Call(func = Name(id = 'str', ctx = Load()), args = [Str(s = 'hello world')], keywords = []))])"
The output is more user friendly in this case; you can see that the args for the function call is the single concatenated string Hello World.
In addition, I also stumbled upon a cool module that generates a visualization of the tree for ast nodes. Using it, the output of the expression expr is visualized like this:
Image cropped to show only the relevant part for the expression.
As you can see, in the terminal leaf node we have a single str object, the joined string for "Hello " and "World", i.e "Hello World".
If you are feeling brave enough, dig into the source, the source code for transforming expressions into a parse tree is located at Parser/pgen.c while the code transforming the parse tree into an Abstract Syntax Tree is in Python/ast.c.
This information is for Python 3.5 and I'm pretty sure that unless you're using some really old version (< 2.5) the functionality and locations should be similar.
Additionally, if you are interested in the whole compilation step python follows, a good gentle intro is provided by one of the core contributors, Brett Cannon, in the video From Source to Code: How CPython's Compiler Works.

Python string literal concatenation

I can create a multi-line string using this syntax:
string = str("Some chars "
"Some more chars")
This will produce the following string:
"Some chars Some more chars"
Is Python joining these two separate strings or is the editor/compiler treating them as a single string?
P.s: I just want to understand the internals. I know there are other ways to declare or create multi-line strings.

Read the reference manual, it's in there.
Specifically:
Multiple adjacent string or bytes literals (delimited by whitespace), possibly using different quoting conventions, are allowed, and their meaning is the same as their concatenation. Thus, "hello" 'world' is equivalent to "helloworld". This feature can be used to reduce the number of backslashes needed, to split long strings conveniently across long lines, or even to add comments to parts of strings,
(emphasis mine)
This is why:
string = str("Some chars "
"Some more chars")
is exactly the same as: str("Some chars Some more chars").
This action is performed wherever a string literal might appear, list initiliazations, function calls (as is the case with str above) et-cetera.
The only caveat is when a string literal is not contained between one of the grouping delimiters (), {} or [] but, instead, spreads between two separate physical lines. In that case we can alternatively use the backslash character to join these lines and get the same result:
string = "Some chars " \
"Some more chars"
Of course, concatenation of strings on the same physical line does not require the backslash. (string = "Hello " "World" is just fine)
Is Python joining these two separate strings or is the editor/compiler treating them as a single string?
Python is, now when exactly does Python do this is where things get interesting.
From what I could gather (take this with a pinch of salt, I'm not a parsing expert), this happens when Python transforms the parse tree (LL(1) Parser) for a given expression to it's corresponding AST (Abstract Syntax Tree).
You can get a view of the parsed tree via the parser module:
import parser
expr = """
str("Hello "
"World")
"""
pexpr = parser.expr(expr)
parser.st2list(pexpr)
This dumps a pretty big and confusing list that represents concrete syntax tree parsed from the expression in expr:
-- rest snipped for brevity --
[322,
[323,
[3, '"hello"'],
[3, '"world"']]]]]]]]]]]]]]]]]],
-- rest snipped for brevity --
The numbers correspond to either symbols or tokens in the parse tree and the mappings from symbol to grammar rule and token to constant are in Lib/symbol.py and Lib/token.py respectively.
As you can see in the snipped version I added, you have two different entries corresponding to the two different str literals in the expression parsed.
Next, we can view the output of the AST tree produced by the previous expression via the ast module provided in the Standard Library:
p = ast.parse(expr)
ast.dump(p)
# this prints out the following:
"Module(body = [Expr(value = Call(func = Name(id = 'str', ctx = Load()), args = [Str(s = 'hello world')], keywords = []))])"
The output is more user friendly in this case; you can see that the args for the function call is the single concatenated string Hello World.
In addition, I also stumbled upon a cool module that generates a visualization of the tree for ast nodes. Using it, the output of the expression expr is visualized like this:
Image cropped to show only the relevant part for the expression.
As you can see, in the terminal leaf node we have a single str object, the joined string for "Hello " and "World", i.e "Hello World".
If you are feeling brave enough, dig into the source, the source code for transforming expressions into a parse tree is located at Parser/pgen.c while the code transforming the parse tree into an Abstract Syntax Tree is in Python/ast.c.
This information is for Python 3.5 and I'm pretty sure that unless you're using some really old version (< 2.5) the functionality and locations should be similar.
Additionally, if you are interested in the whole compilation step python follows, a good gentle intro is provided by one of the core contributors, Brett Cannon, in the video From Source to Code: How CPython's Compiler Works.

Decoding RFC 2231 headers

Trying to address this issue, I'm trying to wrap my head around the various functions in the Python standard library aimed at supporting RFC 2231. The main aim of that RFC appears to be three-fold: allowing non-ASCII encoding in header parameters, noting the language of a given value, and allowing header parameters to span multiple lines. The email.util library provides several functions to deal with various aspects of this. As far as I can tell, they work as follows:
decode_rfc2231 only splits the value of such a parameter into its parts, like this:
>>> email.utils.decode_rfc2231("utf-8''T%C3%A4st.txt")
['utf-8', '', 'T%C3%A4st.txt']
decode_params takes care of detecting RFC2231-encoded parameters. It collects parts which belong together, and also decodes the url-encoded string to a byte sequence. This byte sequence, however, is then encoded as latin1. And all values are enclosed in quotation marks. Furthermore, there is some special handling for the first argument, which still has to be a tuple of two elements, but those two get passed to the result without modification.
>>> email.utils.decode_params([
... (1,2),
... ("foo","bar"),
... ("name*","utf-8''T%C3%A4st.txt"),
... ("baz*0","two"),("baz*1","-part")])
[(1, 2), ('foo', '"bar"'), ('baz', '"two-part"'), ('name', ('utf-8', '', '"Täst.txt"'))]
collapse_rfc2231_value can be used to convert this triple of encoding, language and byte sequence into a proper unicode string. What has me confused, though, is the fact that if the input was such a triple, then the quotes will be carried over to the output. If, on the other hand, the input was a single quoted string, then these quotes will be removed.
>>> [(k, email.utils.collapse_rfc2231_value(v)) for k, v in
... email.utils.decode_params([
... (1,2),
... ("foo","bar"),
... ("name*","utf-8''T%C3%A4st.txt"),
... ("baz*0","two"),("baz*1","-part")])[1:]]
[('foo', 'bar'), ('baz', 'two-part'), ('name', '"Täst.txt"')]
So it seems that in order to use all this machinery, I'd have to add yet another step to unquote the third element of any tuple I'd encounter. Is this true, or am I missing some point here? I had to figure out a lot of the above with help from the source code, since the docs are a bit vague on the details. I cannot imagine what could be the point behind this selective unquoting. Is there a point to it?
What is the best reference on how to use these functions?
The best I found so far is the email.message.Message implementation. There, the process seems to be roughly the one outlined above, but every field gets unquoted via _unquotevalue after the decode_params, and only get_filename and get_boundary collapse their values, all others return a tuple instead. I hope there is something more useful.

Currently the functions from email.utils are rarely used besides within email.message. Most users seem to prefer using email.message.Message directly. There's even a somewhat old issue report on adding unit tests (that would certainly be usable as examples) to Python, even if I'm not sure on how it relates to email.util.
A short example I found is this blogpost which, however, doesn't contain more than once sentence and a few SLOCs of information about RFC2231 parsing. The author notes, however, that many MTAs use RFC2047 instead. Depending on your usecase, that might also be an issue.
Judging from the few examples I could find I assume your way of parsing using email.util is the only way to go, even if the long list comprehension is somewhat ugly.
Because of the lack of examples in some respect it could be wise to write a new RFC2231 parser (if you really need a better, maybe faster or more beautiful codebase). A new implementation could be based on existing implementations like the Dovecot RFC2231 parser for compatibility reasons (you could even use the Dovecot unit test. As the C code seems quite complex to me and since I can't find any python implementation besides email.util and Python2 backports of email.util the task of porting to Python won't be easy (note that Dovecot is LGPL-licensed, which might be an issue in your project)
I think the email.util RFC2231 API has not been designed for easy standalone usage but more as a pile of utility methods for use in email.message.Message.

Old question, but I could not find a complete answer that works on this. So this is what I ended up doing (on Python 2.7):
def decode_rfc2231_header(header):
"""Decode a RFC 2231 header"""
# Remove any quotes
header = email.utils.unquote(header)
encoding, language, value = email.utils.decode_rfc2231(header)
value = urllib.unquote(value)
return email.utils.collapse_rfc2231_value((encoding, language, value))
For example:
>>> name = u'èéêëēėęûüùúūàáâäæãåāāîïíīįì test ôöòóœøōõssśšłžźżçćčñń'
>>> encoded_header = email.utils.encode_rfc2231(name.encode("utf8"), 'utf8', 'en')
>>> print encoded_header
utf8'en'%C3%A8%C3%A9%C3%AA%C3%AB%C4%93%C4%97%C4%99%C3%BB%C3%BC%C3%B9%C3%BA%C5%AB%C3%A0%C3%A1%C3%A2%C3%A4%C3%A6%C3%A3%C3%A5%C4%81%C4%81%C3%AE%C3%AF%C3%AD%C4%AB%C4%AF%C3%AC%20test%20%C3%B4%C3%B6%C3%B2%C3%B3%C5%93%C3%B8%C5%8D%C3%B5ss%C5%9B%C5%A1%C5%82%C5%BE%C5%BA%C5%BC%C3%A7%C4%87%C4%8D%C3%B1%C5%84
>>> print decode_rfc2231_header(encoded_header)
èéêëēėęûüùúūàáâäæãåāāîïíīįì test ôöòóœøōõssśšłžźżçćčñń