We Keep Coding

Subprocess stdout decode string wont work with greek letters

Im using the following subprocess call c = subprocess.Popen("wmic nic where 'netconnectionid like '%οπικ%' and PhysicalAdapter=True' get netconnectionid",stdout=subprocess.PIPE). What i want to do is pass the netconnectionid (aka network adapter name) so i can pass it into a variable and use it here
ip = self.ip_list.currentText()
subprocess.Popen('netsh interface ipv4 set address name="' + sk + '"
static 192.168.131.' + ip + ' 255.255.255.0 192.168.131.1')
Im doing this because adapter names differ in different pcs. The problem is here :
os = c.stdout.read()
sk = os.splitlines()[2].strip().decode('cp437')
The result is 'Τοπική σύνδεση'.Im trying to convert the bytes to greek letters.I have tried utf-8 and it doesnt work. I tried printing the sk variable with .decode('cp437') and its the only decoding option that does right but after that,the netsh command wont work anyway.I tried renaming the adapter with english letters and decoding with utf-8 and it works so well.This is what i basically want to convert :
b'\xe7?\xe3??? \xe5??\xeb\xee\xe5?'
Any ideas on how to make this work?

Try using encoding utf-8.
greek_string = 'Τοπική σύνδεση'
encoded_string = greek_string.encode('utf-8')
decoded_string = encoded_string.decode()
print('Original: {}\n\nEncoded: {}\nDecoded: {}'.format(greek_string,
encoded_string,
decoded_string))
Output:
Original: Τοπική σύνδεση
Encoded: b'\xce\xa4\xce\xbf\xcf\x80\xce\xb9\xce\xba\xce\xae \xcf\x83\xcf\x8d\xce\xbd\xce\xb4\xce\xb5\xcf\x83\xce\xb7'
Decoded: Τοπική σύνδεση

Python imaplib: Display non-ASCII characters correctly

I am using Python 3.5 and imaplib to fetch an e-mail from GMail and print its body. The body contains non-ASCII characters.
These are 'encoded' in a strange way and I cannot find out how to fix this.
import email
import imaplib
c = imaplib.IMAP4_SSL('imap.gmail.com')
c.login('example#gmail.com', 'password')
c.select('Inbox')
_, data = c.fetch(b'12345', '(RFC822)')
mail = data[0][1]
message = email.message_from_bytes(mail)
payload = message.get_payload()
body = mail[0].as_string()
print(body)
Gives
>> ... Mit freundlichen Gr=C3=BC=C3=9Fen ...
instead of the desired
>> ... Mit freundlichen Grüßen ...
It looks to me like this is not an issue of encoding but one of conversion. But how do I tell Python to convert the characters correctly? Is there a more convenient library?

The text is encoded with quoted-printable encoding, which is a way to encode non-ascii characters in ascii text. You can decode it using python's quopri module.
>>> import quopri
>>> bs = b'Gr=C3=BC=C3=9Fen'
>>> # Decode quoted-printable to raw bytes.
>>> utf8 = quopri.decodestring(bs)
>>> # Decode bytes to text.
>>> s = utf8.decode('utf-8')
>>> print(s)
Grüßen
You may find that quoted-printable is the value of the email's content-transfer-encoding header.

How to parse German Umlaute and other special characters from emails

I am trying to parse an email using python's email-module with its Parser() provided by the email.utils-submodule.
However, there are some special characters which I was not able to parse / convert correctly.
Here is the script I got so far:
import sys
import email
from email.parser import Parser
full_msg = Parser().parse(sys.stdin)
msg = full_msg # this ugly line is part of former debugging
sender = msg['from']
to = msg['to']
subject = msg['subject']
body = msg.get_payload()
date = msg['Date']
fname = '{}.txt'.format(date)
with open(fname, 'w') as f:
f.write('{:10}{}\n'.format('Von:', sender))
f.write('{:10}{}\n'.format('An:', to))
f.write('{:10}{}\n'.format('Betreff:', subject))
f.write('{}\n'.format(body))
Since I am parsing both international as well as German mails I have to deal with the so called 'Umlaute' (ä, ü, ö) and some other characters like ß and the ellipsis (...).
So for example a body like
Würde Dürfte Könnte
get's
W=C3=BCrde D=C3=BCrfte K=C3=B6nnte=
and a subject of
Das dürfte jetzt klappen
becomes
=?utf-8?Q?Das_d=C3=BCrfte_jetzt_klappen?=
Is there a way to deal with those encoding/decoding issues? What am I missing?
UPDATE 1:
The system's language resp. encoding was set to en_US.UTF-8. I changed that to de_DE.UTF-8 by reconfiguring the available locales. However, this did not change the output at all. locale gives:
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=
UPDATE 2:
I found out that this type of string formatting is called Quoted-printable. There is a Python module called quopri to handle this format, but I was unable to get satisfying results. However, I switched to JavaScript using MailParser which works like a charm.

Parse multipart/form-data file in UTF-8

I am parsing a multipart/form input with Python's cgi module:
body_file = StringIO.StringIO(self.request.body)
pdict = {'boundary': 'xYzZY'}
httpbody = cgi.parse_multipart(body_file, pdict)
text = self.trim(httpbody['text'])
and I want to print some elements of httpbody that are the UTF-8 encoded.
I tried text.decode('utf-8') and unicode(text, encoding='utf-8'), but nothing seems to work. Am I missing something here?

Try the following:
text = self.trim(httpbody['text'])
text.encode('utf-8')
I'm assuming the text variable is in string, if not sure str(). Otherwise, you'll get another error thrown at you.

Unable to display Japanese (UTF-8) characters in email body with webbrowser

I am reading text from two different .txt files and concatenating them together. Then add that to a body of the email through by using webbrowser.
One text file is English characters (ascii) and the other Japanese (UTF-8). The text will display fine if I write it to a text file. But if I use webbrowser to insert the text into an email body the Japanese text displays as question marks.
I have tried running the script on multiple machines that have different mail clients as their defaults. Initially I thought maybe that was the issue, but that does not appear to be. Thunderbird and Mail (MacOSX) display question marks.
Hello. Today is 2014-05-09
????????????????2014-05-09????
I have looked at similar issues around on SO but they have not solved the issue.
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in
position 20: ordinal not in
range(128)
Japanese in python function
Printing out Japanese (Chinese) characters
python utf-8 japanese
Is there a way to have the Japanese (UTF-8) display in the body of an email created with webbrowser in python? I could use the email functionality but the requirement is the script needs to open the default mail client and insert all the information.
The code and text files I am using are below. I have simplified it to focus on the issue.
email-template.txt
Hello. Today is {{date}}
email-template-jp.txt
こんにちは。今日は {{date}} です。
Python Script
#
# -*- coding: utf-8 -*-
#
import sys
import re
import os
import glob
import webbrowser
import codecs,sys
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
# vars
date_range = sys.argv[1:][0]
email_template_en = "email-template.txt"
email_template_jp = "email-template-jp.txt"
email_to_send = "email-to-send.txt" # finished email is saved here
# Default values for the composed email that will be opened
mail_list = "test#test.com"
cc_list = "test1#test.com, test2#test.com"
subject = "Email Subject"
# Open email templates and insert the date from the parameters sent in
try:
f_en = open(email_template_en, "r")
f_jp = codecs.open(email_template_jp, "r", "UTF-8")
try:
email_content_en = f_en.read()
email_content_jp = f_jp.read()
email_en = re.sub(r'{{date}}', date_range, email_content_en)
email_jp = re.sub(r'{{date}}', date_range, email_content_jp).encode("UTF-8")
# this throws an error
# UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 26: ordinal not in range(128)
# email_en_jp = (email_en + email_jp).encode("UTF-8")
email_en_jp = (email_en + email_jp)
finally:
f_en.close()
f_jp.close()
pass
except Exception, e:
raise e
# Open the default mail client and fill in all the information
try:
f = open(email_to_send, "w")
try:
f.write(email_en_jp)
# Does not send Japanese text to the mail client. But will write to the .txt file fine. Unsure why.
webbrowser.open("mailto:%s?subject=%s&cc=%s&body=%s" %(mail_list, subject, cc_list, email_en_jp), new=1) # open mail client with prefilled info
finally:
f.close()
pass
except Exception, e:
raise e
edit: Forgot to add I am using Python 2.7.1

EDIT 2: Found a workable solution after all.
Replace your webbrowser call with this.
import subprocess
[... other code ...]
arg = "mailto:%s?subject=%s&cc=%s&body=%s" % (mail_list, subject, cc_list, email_en_jp)
subprocess.call(["open", arg])
This will open your default email client on MacOS. For other OSes please replace "open" in the subprocess line with the proper executable.
EDIT: I looked into it a bit more and Mark's comment above made me read the RFC (2368) for mailto URL scheme.
The special hname "body" indicates that the associated hvalue is the
body of the message. The "body" hname should contain the content for
the first text/plain body part of the message. The mailto URL is
primarily intended for generation of short text messages that are
actually the content of automatic processing (such as "subscribe"
messages for mailing lists), not general MIME bodies.
And a bit further down:
8-bit characters in mailto URLs are forbidden. MIME encoded words (as
defined in [RFC2047]) are permitted in header values, but not for any
part of a "body" hname."
So it looks like this is not possible as per RFC, although that makes me question why the JavaScript solution in the JSFiddle provided by naota works at all.
I leave my previous answer as is below, although it does not work.
I have run into same issues with Python 2.7.x quite a couple of times now and every time a different solution somehow worked.
So here are several suggestions that may or may not work, as I haven't tested them.
a) Force unicode strings:
webbrowser.open(u"mailto:%s?subject=%s&cc=%s&body=%s" % (mail_list, subject, cc_list, email_en_jp), new=1)
Notice the small u right after the opening ( and before the ".
b) Force the regex to use unicode:
email_jp = re.sub(ur'{{date}}', date_range, email_content_jp).encode("UTF-8")
# or maybe
email_jp = re.sub(ur'{{date}}', date_range, email_content_jp)
c) Another idea regarding the regex, try compiling it first with the re.UNICODE flag, before applying it.
pattern = re.compile(ur'{{date}}', re.UNICODE)
d) Not directly related, but I noticed you write the combined text via the normal open method. Try using the codecs.open here as well.
f = codecs.open(email_to_send, "w", "UTF-8")
Hope this helps.