We Keep Coding

TypeError: a bytes-like object is required, not 'str' when writing to a file in Python3 [duplicate]

I've very recently migrated to Python 3.5.
This code was working properly in Python 2.7:
with open(fname, 'rb') as f:
lines = [x.strip() for x in f.readlines()]
for line in lines:
tmp = line.strip().lower()
if 'some-pattern' in tmp: continue
# ... code
But in 3.5, on the if 'some-pattern' in tmp: continue line, I get an error which says:
TypeError: a bytes-like object is required, not 'str'
I was unable to fix the problem using .decode() on either side of the in, nor could I fix it using
if tmp.find('some-pattern') != -1: continue
What is wrong, and how do I fix it?

You opened the file in binary mode:
with open(fname, 'rb') as f:
This means that all data read from the file is returned as bytes objects, not str. You cannot then use a string in a containment test:
if 'some-pattern' in tmp: continue
You'd have to use a bytes object to test against tmp instead:
if b'some-pattern' in tmp: continue
or open the file as a textfile instead by replacing the 'rb' mode with 'r'.

You can encode your string by using .encode()
Example:
'Hello World'.encode()
As the error describes, in order to write a string to a file you need to encode it to a byte-like object first, and encode() is encoding it to a byte-string.

Like it has been already mentioned, you are reading the file in binary mode and then creating a list of bytes. In your following for loop you are comparing string to bytes and that is where the code is failing.
Decoding the bytes while adding to the list should work. The changed code should look as follows:
with open(fname, 'rb') as f:
lines = [x.decode('utf8').strip() for x in f.readlines()]
The bytes type was introduced in Python 3 and that is why your code worked in Python 2. In Python 2 there was no data type for bytes:
>>> s=bytes('hello')
>>> type(s)
<type 'str'>

You have to change from wb to w:
def __init__(self):
self.myCsv = csv.writer(open('Item.csv', 'wb'))
self.myCsv.writerow(['title', 'link'])
to
def __init__(self):
self.myCsv = csv.writer(open('Item.csv', 'w'))
self.myCsv.writerow(['title', 'link'])
After changing this, the error disappears, but you can't write to the file (in my case). So after all, I don't have an answer?
Source: How to remove ^M
Changing to 'rb' brings me the other error: io.UnsupportedOperation: write

Use the encode() function along with the hardcoded string value given in a single quote.
Example:
file.write(answers[i] + '\n'.encode())
Or
line.split(' +++$+++ '.encode())

For this small example, adding the only b before
'GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\n\n' solved my problem:
import socket
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('www.py4inf.com', 80))
mysock.send(b'GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\n\n')
while True:
data = mysock.recv(512)
if (len(data) < 1):
break
print (data);
mysock.close()
What does the 'b' character do in front of a string literal?

You opened the file in binary mode:
The following code will throw
a TypeError: a bytes-like object is required, not 'str'.
for line in lines:
print(type(line))# <class 'bytes'>
if 'substring' in line:
print('success')
The following code will work - you have to use the decode() function:
for line in lines:
line = line.decode()
print(type(line))# <class 'str'>
if 'substring' in line:
print('success')

Try opening your file as text:
with open(fname, 'rt') as f:
lines = [x.strip() for x in f.readlines()]
Additionally, here is a link for Python 3.x on the official page:
io — Core tools for working with streams.
And this is the open function: open
If you are really trying to handle it as a binary then consider encoding your string.

I got this error when I was trying to convert a char (or string) to bytes, the code was something like this with Python 2.7:
# -*- coding: utf-8 -*-
print(bytes('ò'))
This is the way of Python 2.7 when dealing with Unicode characters.
This won't work with Python 3.6, since bytes require an extra argument for encoding, but this can be little tricky, since different encoding may output different result:
print(bytes('ò', 'iso_8859_1')) # prints: b'\xf2'
print(bytes('ò', 'utf-8')) # prints: b'\xc3\xb2'
In my case I had to use iso_8859_1 when encoding bytes in order to solve the issue.

Summary
Python 2.x encouraged many bad habits WRT text handling. In particular, its type named str does not actually represent text per the Unicode standard (that type is unicode), and the default "string literal" in fact produces a sequence of raw bytes - with some convenience functions for treating it like a string, if you can get away with assuming a "code page" style encoding.
In 3.x, "string literals" now produce actual strings, and built-in functionality no longer does any implicit conversions between the two types. Thus, the same code now has a TypeError, because the literal and the variable are of incompatible types. To fix the problem, one of the values must be either replaced or converted, so that the types match.
The Python documentation has an extremely detailed guide to working with Unicode properly.
In the example in the question, the input file is processed as if it contains text. Therefore, the file should have been opened in a text mode in the first place. The only good reason the file would have been opened in binary mode even in 2.x is to avoid universal newline translation; in 3.x, this is done by specifying the newline keyword parameter when opening a file in text mode.
To read a file as text properly requires knowing a text encoding, which is specified in the code by (string) name. The encoding iso-8859-1 is a safe fallback; it interprets each byte separately, as representing one of the first 256 Unicode code points, in order (so it will never raise an exception due to invalid data). utf-8 is much more common as of the time of writing, but it does not accept arbitrary data. (However, in many cases, for English text, the distinction will not matter; both of those encodings, and many more, are supersets of ASCII.)
Thus:
with open(fname, 'r', newline='\n', encoding='iso-8859-1') as f:
lines = [x.strip() for x in f.readlines()]
# proceed as before
# If the results are wrong, take additional steps to ascertain the correct encoding
How the error is created when migrating from 2.x to 3.x
In 2.x, 'some-pattern' creates a str, i.e. a sequence of bytes that the programmer is then likely to pretend is text. The str type is the same as the bytes type, and different from the unicode type that properly represents text. Many methods are offered to treat this data as if it were text, but it is not a proper representation of text. The meaning of each value as a text character (the encoding) is assumed. (In order to enable the illusion of raw data as "text", there would sometimes be implicit conversions between the str and unicode types. However, this results in confusing errors of its own - such as getting UnicodeDecodeError from an attempt to encode, or vice-versa).
In 3.x, 'some-pattern' creates what is also called a str; but now str means the Unicode-using, properly-text-representing string type. (unicode is no longer used as a type name, and only bytes refers to the sequence-of-bytes type.) Some changes were made to bytes to dissociate it from the text-with-assumed-encoding interpretation (in particular, indexing into a bytes object now results in an int, rather than a 1-element bytes), but many strange legacy methods persist (including ones rarely used even with actual strings any more, like zfill).
Why this causes a problem
The data, tmp, is a bytes instance. It came from a binary source: in this case, a file opened with a 'b' file mode. In other cases, it could come from a raw network socket, a web request made with urllib or similar, or some other API call.
This means that it cannot do anything meaningful in combination with a string. The elements of a string are Unicode code points (i.e., abstractions that represent, for the most part, text characters, in a universal form that represents all world languages and many other symbols). The elements of a bytes are, well, bytes. (Specifically in 3.x, they are interpreted as unsigned integers ranging from 0 to 255 inclusive.)
When the code was migrated, the literal 'some-pattern' went from describing a bytes, to describing text. Thus, the code went from making a legal comparison (byte-sequence to byte-sequence), to making an illegal one (string to byte-sequence).
Fixing the problem
In order to operate on a string and a byte-sequence - whether it's checking for equality with ==, lexicographic comparison with <, substring search with in, concatenation with +, or anything else - either the string must be converted to a byte-sequence, or vice-versa. In general, only one of these will be the correct, sensible answer, and it will depend on the context.
Fixing the source
Sometimes, one of the values can be seen to be "wrong" in the first place. For example, if reading the file was intended to result in text, then it should have been opened in a text mode. In 3.x, the file encoding can simply be passed as an encoding keyword argument to open, and conversion to Unicode is handled seamlessly without having to feed a binary file to an explicit translation step (thus, universal newline handling still takes place seamlessly).
In the case of the original example, that could look like:
with open(fname, 'r') as f:
lines = [x.strip() for x in f.readlines()]
This example assumes a platform-dependent default encoding for the file. This will normally work for files that were created in straightforward ways, on the same computer. In the general case, however, the encoding of the data must be known in order to work with it properly.
If the encoding is known to be, for example, UTF-8, that is trivially specified:
with open(fname, 'r', encoding='utf-8') as f:
lines = [x.strip() for x in f.readlines()]
Similarly, a string literal that should have been a bytes literal is simply missing a prefix: to make the bytes sequence representing integer values [101, 120, 97, 109, 112, 108, 101] (i.e., the ASCII values of the letters example), write the bytes literal b'example', rather than the string literal `'example'). Similarly the other way around.
In the case of the original example, that would look like:
if b'some-pattern' in tmp:
There is a safeguard built in to this: the bytes literal syntax only allows ASCII characters, so something like b'ëxãmþlê' will be caught as a SyntaxError, regardless of the encoding of the source file (since it is not clear which byte values are meant; in the old implied-encoding schemes, the ASCII range was well established, but everything else was up in the air.) Of course, bytes literals with elements representing values 128..255 can still be written by using \x escaping for those values: for example, b'\xebx\xe3m\xfel\xea' will produce a byte-sequence corresponding to the text ëxãmþlê in Latin-1 (ISO 8859-1) encoding.
Converting, when appropriate
Conversion between byte-sequences and text is only possible when an encoding has been determined. It has always been so; we just used to assume an encoding locally, and then mostly ignore that we had done so. (Programmers in places like East Asia have been more aware of the problem historically, because they commonly need to work with scripts that have more than 256 distinct symbols, and thus their text requires multi-byte encodings.)
In 3.x, because there is no pressure to be able to treat byte-sequences implicitly as text with an assumed encoding, there are therefore no implicit conversion steps behind the scenes. This means that understanding the API is straightforward: Bytes are raw data; therefore, they are used to encode text, which is an abstraction. Therefore, the .encode() method is provided by str (which represents text), in order to encode text into raw data. Similarly, the .decode() method is provided by bytes (which represents a byte-sequence), in order to decode raw data into text.
Applying these to the example code, again supposing UTF-8 encoding is appropriate, gives:
if 'some-pattern'.encode('utf-8') in tmp:
and
if 'some-pattern' in tmp.decode('utf-8'):

How can you use python to append lower-level ASCII characters to a (notepad) document?

I need to reach a specific byte count and want to fill the rest of document with null bytes such as lower-level ascii characters. I've heard there's a way using python but I'm unsure how to. Any suggestions?

You can use chr() for this, for example:
t = "BLA"
t += chr(33)
print(t)
will print BLA!. 33 is the decimal value for "!". Use any other value you like, loop over your string and write it to a new file (use open, read, and write, of course to accomplish this on a file base).
Edit: Use
file_ptr = open("filename.txt", "r")
t = file_ptr.read()
for example to grab your file's contents.

Save file/TypeError: a bytes-like object is required, not 'str' [duplicate]

I've very recently migrated to Python 3.5.
This code was working properly in Python 2.7:
with open(fname, 'rb') as f:
lines = [x.strip() for x in f.readlines()]
for line in lines:
tmp = line.strip().lower()
if 'some-pattern' in tmp: continue
# ... code
But in 3.5, on the if 'some-pattern' in tmp: continue line, I get an error which says:
TypeError: a bytes-like object is required, not 'str'
I was unable to fix the problem using .decode() on either side of the in, nor could I fix it using
if tmp.find('some-pattern') != -1: continue
What is wrong, and how do I fix it?

You opened the file in binary mode:
with open(fname, 'rb') as f:
This means that all data read from the file is returned as bytes objects, not str. You cannot then use a string in a containment test:
if 'some-pattern' in tmp: continue
You'd have to use a bytes object to test against tmp instead:
if b'some-pattern' in tmp: continue
or open the file as a textfile instead by replacing the 'rb' mode with 'r'.

You can encode your string by using .encode()
Example:
'Hello World'.encode()
As the error describes, in order to write a string to a file you need to encode it to a byte-like object first, and encode() is encoding it to a byte-string.

Like it has been already mentioned, you are reading the file in binary mode and then creating a list of bytes. In your following for loop you are comparing string to bytes and that is where the code is failing.
Decoding the bytes while adding to the list should work. The changed code should look as follows:
with open(fname, 'rb') as f:
lines = [x.decode('utf8').strip() for x in f.readlines()]
The bytes type was introduced in Python 3 and that is why your code worked in Python 2. In Python 2 there was no data type for bytes:
>>> s=bytes('hello')
>>> type(s)
<type 'str'>

You have to change from wb to w:
def __init__(self):
self.myCsv = csv.writer(open('Item.csv', 'wb'))
self.myCsv.writerow(['title', 'link'])
to
def __init__(self):
self.myCsv = csv.writer(open('Item.csv', 'w'))
self.myCsv.writerow(['title', 'link'])
After changing this, the error disappears, but you can't write to the file (in my case). So after all, I don't have an answer?
Source: How to remove ^M
Changing to 'rb' brings me the other error: io.UnsupportedOperation: write

Use the encode() function along with the hardcoded string value given in a single quote.
Example:
file.write(answers[i] + '\n'.encode())
Or
line.split(' +++$+++ '.encode())

For this small example, adding the only b before
'GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\n\n' solved my problem:
import socket
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('www.py4inf.com', 80))
mysock.send(b'GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\n\n')
while True:
data = mysock.recv(512)
if (len(data) < 1):
break
print (data);
mysock.close()
What does the 'b' character do in front of a string literal?

You opened the file in binary mode:
The following code will throw
a TypeError: a bytes-like object is required, not 'str'.
for line in lines:
print(type(line))# <class 'bytes'>
if 'substring' in line:
print('success')
The following code will work - you have to use the decode() function:
for line in lines:
line = line.decode()
print(type(line))# <class 'str'>
if 'substring' in line:
print('success')

Try opening your file as text:
with open(fname, 'rt') as f:
lines = [x.strip() for x in f.readlines()]
Additionally, here is a link for Python 3.x on the official page:
io — Core tools for working with streams.
And this is the open function: open
If you are really trying to handle it as a binary then consider encoding your string.

I got this error when I was trying to convert a char (or string) to bytes, the code was something like this with Python 2.7:
# -*- coding: utf-8 -*-
print(bytes('ò'))
This is the way of Python 2.7 when dealing with Unicode characters.
This won't work with Python 3.6, since bytes require an extra argument for encoding, but this can be little tricky, since different encoding may output different result:
print(bytes('ò', 'iso_8859_1')) # prints: b'\xf2'
print(bytes('ò', 'utf-8')) # prints: b'\xc3\xb2'
In my case I had to use iso_8859_1 when encoding bytes in order to solve the issue.

Summary
Python 2.x encouraged many bad habits WRT text handling. In particular, its type named str does not actually represent text per the Unicode standard (that type is unicode), and the default "string literal" in fact produces a sequence of raw bytes - with some convenience functions for treating it like a string, if you can get away with assuming a "code page" style encoding.
In 3.x, "string literals" now produce actual strings, and built-in functionality no longer does any implicit conversions between the two types. Thus, the same code now has a TypeError, because the literal and the variable are of incompatible types. To fix the problem, one of the values must be either replaced or converted, so that the types match.
The Python documentation has an extremely detailed guide to working with Unicode properly.
In the example in the question, the input file is processed as if it contains text. Therefore, the file should have been opened in a text mode in the first place. The only good reason the file would have been opened in binary mode even in 2.x is to avoid universal newline translation; in 3.x, this is done by specifying the newline keyword parameter when opening a file in text mode.
To read a file as text properly requires knowing a text encoding, which is specified in the code by (string) name. The encoding iso-8859-1 is a safe fallback; it interprets each byte separately, as representing one of the first 256 Unicode code points, in order (so it will never raise an exception due to invalid data). utf-8 is much more common as of the time of writing, but it does not accept arbitrary data. (However, in many cases, for English text, the distinction will not matter; both of those encodings, and many more, are supersets of ASCII.)
Thus:
with open(fname, 'r', newline='\n', encoding='iso-8859-1') as f:
lines = [x.strip() for x in f.readlines()]
# proceed as before
# If the results are wrong, take additional steps to ascertain the correct encoding
How the error is created when migrating from 2.x to 3.x
In 2.x, 'some-pattern' creates a str, i.e. a sequence of bytes that the programmer is then likely to pretend is text. The str type is the same as the bytes type, and different from the unicode type that properly represents text. Many methods are offered to treat this data as if it were text, but it is not a proper representation of text. The meaning of each value as a text character (the encoding) is assumed. (In order to enable the illusion of raw data as "text", there would sometimes be implicit conversions between the str and unicode types. However, this results in confusing errors of its own - such as getting UnicodeDecodeError from an attempt to encode, or vice-versa).
In 3.x, 'some-pattern' creates what is also called a str; but now str means the Unicode-using, properly-text-representing string type. (unicode is no longer used as a type name, and only bytes refers to the sequence-of-bytes type.) Some changes were made to bytes to dissociate it from the text-with-assumed-encoding interpretation (in particular, indexing into a bytes object now results in an int, rather than a 1-element bytes), but many strange legacy methods persist (including ones rarely used even with actual strings any more, like zfill).
Why this causes a problem
The data, tmp, is a bytes instance. It came from a binary source: in this case, a file opened with a 'b' file mode. In other cases, it could come from a raw network socket, a web request made with urllib or similar, or some other API call.
This means that it cannot do anything meaningful in combination with a string. The elements of a string are Unicode code points (i.e., abstractions that represent, for the most part, text characters, in a universal form that represents all world languages and many other symbols). The elements of a bytes are, well, bytes. (Specifically in 3.x, they are interpreted as unsigned integers ranging from 0 to 255 inclusive.)
When the code was migrated, the literal 'some-pattern' went from describing a bytes, to describing text. Thus, the code went from making a legal comparison (byte-sequence to byte-sequence), to making an illegal one (string to byte-sequence).
Fixing the problem
In order to operate on a string and a byte-sequence - whether it's checking for equality with ==, lexicographic comparison with <, substring search with in, concatenation with +, or anything else - either the string must be converted to a byte-sequence, or vice-versa. In general, only one of these will be the correct, sensible answer, and it will depend on the context.
Fixing the source
Sometimes, one of the values can be seen to be "wrong" in the first place. For example, if reading the file was intended to result in text, then it should have been opened in a text mode. In 3.x, the file encoding can simply be passed as an encoding keyword argument to open, and conversion to Unicode is handled seamlessly without having to feed a binary file to an explicit translation step (thus, universal newline handling still takes place seamlessly).
In the case of the original example, that could look like:
with open(fname, 'r') as f:
lines = [x.strip() for x in f.readlines()]
This example assumes a platform-dependent default encoding for the file. This will normally work for files that were created in straightforward ways, on the same computer. In the general case, however, the encoding of the data must be known in order to work with it properly.
If the encoding is known to be, for example, UTF-8, that is trivially specified:
with open(fname, 'r', encoding='utf-8') as f:
lines = [x.strip() for x in f.readlines()]
Similarly, a string literal that should have been a bytes literal is simply missing a prefix: to make the bytes sequence representing integer values [101, 120, 97, 109, 112, 108, 101] (i.e., the ASCII values of the letters example), write the bytes literal b'example', rather than the string literal `'example'). Similarly the other way around.
In the case of the original example, that would look like:
if b'some-pattern' in tmp:
There is a safeguard built in to this: the bytes literal syntax only allows ASCII characters, so something like b'ëxãmþlê' will be caught as a SyntaxError, regardless of the encoding of the source file (since it is not clear which byte values are meant; in the old implied-encoding schemes, the ASCII range was well established, but everything else was up in the air.) Of course, bytes literals with elements representing values 128..255 can still be written by using \x escaping for those values: for example, b'\xebx\xe3m\xfel\xea' will produce a byte-sequence corresponding to the text ëxãmþlê in Latin-1 (ISO 8859-1) encoding.
Converting, when appropriate
Conversion between byte-sequences and text is only possible when an encoding has been determined. It has always been so; we just used to assume an encoding locally, and then mostly ignore that we had done so. (Programmers in places like East Asia have been more aware of the problem historically, because they commonly need to work with scripts that have more than 256 distinct symbols, and thus their text requires multi-byte encodings.)
In 3.x, because there is no pressure to be able to treat byte-sequences implicitly as text with an assumed encoding, there are therefore no implicit conversion steps behind the scenes. This means that understanding the API is straightforward: Bytes are raw data; therefore, they are used to encode text, which is an abstraction. Therefore, the .encode() method is provided by str (which represents text), in order to encode text into raw data. Similarly, the .decode() method is provided by bytes (which represents a byte-sequence), in order to decode raw data into text.
Applying these to the example code, again supposing UTF-8 encoding is appropriate, gives:
if 'some-pattern'.encode('utf-8') in tmp:
and
if 'some-pattern' in tmp.decode('utf-8'):

Messing up my unicode output - but where and how?

I am doing a word count on some text files, storing the results in a dictionary. My problem is that after outputting to file, the words are not displayed right even if they were in the original text. (I use TextWrangler to look at them).
For instance, dashes show up as dashes in the original but as \u2014 in the output; in the output, very word is prefixed by a u as well.
Problem
I do not know where, when and how in my script this happens.
I am reading the file with codecs.open() and outputting them with codecs.open() and as json.dump(). They both go wrong in the same way. In between, all is do is
tokenizing
regular expressions
collect in dictionary
And I don't know where I mess things up; I have de-activated tokenizing and most other functions to no effect. All this is happening in Python 2.
Following previous advice, I tried to keep everything within the script in Unicode.
Here is what I do (non-relevant code omitted):
#read in file, iterating over a list of "fileno"s
with codecs.open(os.path.join(dir,unicode(fileno)+".txt"), "r", "utf-8") as inputfili:
inputtext=inputfili.read()
#process the text: tokenize, lowercase, remove punctuation and conjugation
content=regular expression to extract text w/out metadata
contentsplit=nltk.tokenize.word_tokenize(content)
text=[i.lower() for i in contentsplit if not re.match(r"\d+", i)]
text= [re.sub(r"('s|s|s's|ed)\b", "", i) for i in text if i not in string.punctuation]
#build the dictionary of word counts
for word in text:
dicti[word].append(word)
#collect counts for each word, make dictionary of unique words
dicti_nos={unicode(k):len(v) for k,v in dicti.items()}
hapaxdicti= {k:v for k,v in perioddicti_nos.items() if v == 1}
#sort the dictionary
sorteddict=sorted(dictionary.items(), key=lambda x: x[1], reverse=True)
#output the results as .txt and json-file
with codecs.open(file_name, "w", "utf-8") as outputi:
outputi.write("\n".join([unicode(i) for i in sorteddict]))
with open(file_name+".json", "w") as jsonoutputi:
json.dump(dictionary, jsonoutputi, encoding="utf-8")
EDIT: Solution
Looks like my main issue was writing the file in the wrong way. If I change my code to what's reproduced below, things work out. Looks like joining a list of (string, number) tuples messed the string part up; if I join the tuples first, things work.
For the json output, I had to change to codecs.open() and set ensure_ascii to False. Apparently just setting the encoding to utf-8 does not do the trick like I thought.
with codecs.open(file_name, "w", "utf-8") as outputi:
outputi.write("\n".join([":".join([i[0],unicode(i[1])]) for i in sorteddict]))
with codecs.open(file_name+".json", "w", "utf-8") as jsonoutputi:
json.dump(dictionary, jsonoutputi, ensure_ascii=False)
Thanks for your help!

As your example is partially pseudocode there's no way to run a real test and give you something that runs and has been tested, but from reading what you have provided I think you may misunderstand the way Unicode works in Python 2.
The unicode type (such as is produced via the unicode() or unichr() functions) is meant to be an internal representation of a Unicode string that can be used for string manipulation and comparison purposes. It has no associated encoding. The unicode() function will take a buffer as its first argument and an encoding as its second argument and interpret that buffer using that encoding to produce an internally usable Unicode string that is from that point forward unencumbered by encodings.
That Unicode string isn't meant to be written out to a file; all file formats assume some encoding, and you're supposed to provide one again before writing that Unicode string out to a file. Everyplace you have a construct like unicode(fileno) or unicode(k) or unicode(i) is suspect both because you're relying on a default encoding (which probably isn't what you want) and because you're going on to expose most of these values directly to the file system.
After you're done working with these Unicode strings you can use the built-in method encode() on them with your desired encoding as an argument to pack them into strings of ordinary bytes set as required by your encoding.
So looking back at your example above, your inputtext variable is an ordinary string containing data encoded per the UTF-8 encoding. This isn't Unicode. You could convert it to a Unicode string with an operation like inputuni = unicode(inputtext, 'utf-8') and operate on it like that if you chose, but for what you're doing you may not even find it necessary. If you did convert it to Unicode though you'd have to perform the equivalent of a inputuni.encode('UTF-8') on any Unicode string that you were planning on writing out to your file.

Python doesn't interpret UTF8 correctly

I know similar questions have been asked a million times, but despite reading through many of them I can't find a solution that applies to my situation.
I have a django application, in which I've created a management script. This script reads some text files, and outputs them to the terminal (it will do more useful stuff with the contents later, but I'm still testing it out) and the characters come out with escape sequences like \xc3\xa5 instead of the intended å. Since that escape sequence means Ã¥, which is a common misinterpretation of å because of encoding problems, I suspect there are at least two places where this is going wrong. However, I can't figure out where - I've checked all the possible culprits I can think of:
The terminal encoding is UTF-8; echo $LANG gives en_US.UTF-8
The text files are encoded in UTF-8; file * in the directory where they reside results in all entries being listed as "UTF-8 Unicode text" except one, which does not contain any non-ASCII characters and is listed as "ASCII text". Running iconv -f ascii -t utf8 thefile.txt > utf8.txt on that file yields another file with ASCII text encoding.
The Python scripts are all UTF-8 (or, in several cases, ASCII with no non-ASCII characters). I tried inserting a comment in my management script with some special characters to force it to save as UTF-8, but it did not change the behavior. The above observations on the text files apply on all Python script files as well.
The Python script that handles the text files has # -*- encoding: utf-8 -*- at the top; the only line preceding that is #!/usr/bin/python3, but I've tried both changing to .../python for Python 2.7 or removing it entirely to leave it up to Django, without results.
According to the documentation, "Django natively supports Unicode data", so I "can safely pass around Unicode strings" anywhere in the application.
I really can't think of anywhere else to look for a non-UTF-8 link in the chain. Where could I possibly have missed a setting to change to UTF-8?
For completeness: I'm reading from the files with lines = file.readlines() and printing with the standard print() function. No manual encoding or decoding happens at either end.
UPDATE:
In response to quiestions in comments:
print(sys.getdefaultencoding(), sys.stdout.encoding, f.encoding) yields ('ascii', 'UTF-8', None) for all files.
I started compiling an SSCCE, and quickly found that the problem is only there if I try to print the value in a tuple. In other words, print(lines[0].strip()) works fine, but print(lines[0].strip(), lines[1].strip()) does not. Adding .decode('utf-8') yields a tuple where both strings are marked with a prepending u and \xe5 (the correct escape sequence for å) instead of the odd characters before - but I can't figure out how to print them as regular strings, with no escape characters. I've tested another call to .decode('utf-8') as well as wrapping in str() but both fail with UnicodeEncodeError complaining that \xe5 can't be encoded in ascii. Since a single string works correctly, I don't know what else to test.
SSCCE:
# -*- coding: utf-8 -*-
import os, sys
for root,dirs,files in os.walk('txt-songs'):
for filename in files:
with open(os.path.join(root,filename)) as f:
print(sys.getdefaultencoding(), sys.stdout.encoding, f.encoding)
lines = f.readlines()
print(lines[0].strip()) # works
print(lines[0].strip(), lines[1].strip()) # does not work

The big problem here is that you're mixing up Python 2 and Python 3. In particular, you've written Python 3 code, and you're trying to run it in Python 2.7. But there are a few other problems along the way. So, let me try to explain everything that's going wrong.
I started compiling an SSCCE, and quickly found that the problem is only there if I try to print the value in a tuple. In other words, print(lines[0].strip()) works fine, but print(lines[0].strip(), lines[1].strip()) does not.
The first problem here is that the str of a tuple (or any other collection) includes the repr, not the str, of its elements. The simple way to solve this problem is to not print collections. In this case, there is really no reason to print a tuple at all; the only reason you have one is that you've built it for printing. Just do something like this:
print '({}, {})'.format(lines[0].strip(), lines[1].strip())
In cases where you already have a collection in a variable, and you want to print out the str of each element, you have to do that explicitly. You can print the repr of the str of each with this:
print tuple(map(str, my_tuple))
… or print the str of each directly with this:
print '({})'.format(', '.join(map(str, my_tuple)))
Notice that I'm using Python 2 syntax above. That's because if you actually used Python 3, there would be no tuple in the first place, and there would also be no need to call str.
You've got a Unicode string. In Python 3, unicode and str are the same type. But in Python 2, it's bytes and str that are the same type, and unicode is a different one. So, in 2.x, you don't have a str yet, which is why you need to call str.
And Python 2 is also why print(lines[0].strip(), lines[1].strip()) prints a tuple. In Python 3, that's a call to the print function with two strings as arguments, so it will print out two strings separated by a space. In Python 2, it's a print statement with one argument, which is a tuple.
If you want to write code that works the same in both 2.x and 3.x, you either need to avoid ever printing more than one argument, or use a wrapper like six.print_, or do a from __future__ import print_function, or be very careful to do ugly things like adding in extra parentheses to make sure your tuples are tuples in both versions.
So, in 3.x, you've got str objects and you just print them out. In 2.x, you've got unicode objects, and you're printing out their repr. You can change that to print out their str, or to avoid printing a tuple in the first place… but that still won't help anything.
Why? Well, printing anything, in either version, just calls str on it and then passes it to sys.stdio.write. But in 3.x, str means unicode, and sys.stdio is a TextIOWrapper; in 2.x, str means bytes, and sys.stdio is a binary file.
So, the pseudocode for what ultimately happens is:
sys.stdio.wrapped_binary_file.write(s.encode(sys.stdio.encoding, sys.stdio.errors))
sys.stdio.write(s.encode(sys.getdefaultencoding()))
And, as you saw, those will do different things, because:
print(sys.getdefaultencoding(), sys.stdout.encoding, f.encoding) yields ('ascii', 'UTF-8', None)
You can simulate Python 3 here by using a io.TextIOWrapper or codecs.StreamWriter and then using print >>f, … or f.write(…) instead of print, or you can explicitly encode all your unicode objects like this:
print '({})'.format(', '.join(element.encode('utf-8') for element in my_tuple)))
But really, the best way to deal with all of these problems is to run your existing Python 3 code in a Python 3 interpreter instead of a Python 2 interpreter.
If you want or need to use Python 2.7, that's fine, but you have to write Python 2 code. If you want to write Python 3 code, that's great, but you have to run Python 3.3. If you really want to write code that works properly in both, you can, but it's extra work, and takes a lot more knowledge.
For further details, see What's New In Python 3.0 (the "Print Is A Function" and "Text Vs. Data Instead Of Unicode Vs. 8-bit" sections), although that's written from the point of view of explaining 3.x to 2.x users, which is backward from what you need. The 3.x and 2.x versions of the Unicode HOWTO may also help.

For completeness: I'm reading from the files with lines = file.readlines() and printing with the standard print() function. No manual encoding or decoding happens at either end.
In Python 3.x, the standard print function just writes Unicode to sys.stdout. Since that's a io.TextIOWrapper, its write method is equivalent to this:
self.wrapped_binary_file.write(s.encode(self.encoding, self.errors))
So one likely problem is that sys.stdout.encoding does not match your terminal's actual encoding.
And of course another is that your shell's encoding does not match your terminal window's encoding.
For example, on OS X, I create a myscript.py like this:
print('\u00e5')
Then I fire up Terminal.app, create a session profile with encoding "Western (ISO Latin 1)", create a tab with that session profile, and do this:
$ export LANG=en_US.UTF-8
$ python3 myscript.py
… and I get exactly the behavior you're seeing.

It seems from your comment that you are using python-2 and not python-3.
If you are using python-3, it's worth reading the unicode howto guide on reading/writing to understand what python is doing.
The basic flow if encoding is:
DECODE from encoding to unicode -> Processing -> Encode from unicode to encoding
In python3 the bytes are decoded to strings and strings are encoded to bytes.
The bytes to string decoding is handled for you with open().
[..] the built-in open() function can return a file-like object that
assumes the file’s contents are in a specified encoding and accepts
Unicode parameters for methods such as read() and write(). This works
through open()‘s encoding and errors parameters [..]
So to read in unicode from a utf-8 encoded file you should be doing this:
# python-3
with open('utf8.txt', mode='r', encoding='utf-8') as f:
lines = f.readlines() # returns unicode
If you want similar functionality using python-2, you can use codecs.open():
# python-2
import codecs
with codecs.open('utf8.txt', mode='r', encoding='utf-8') as f:
lines = f.readlines() # returns unicode