We Keep Coding

parse and decode mail text in aiosmtpd, perform string substitution, and reinject

I began with smtpd in order to process mailqueue, parse inbound emails and send them back to recipients (using smtpdlib.sendmail).
I switched to aiosmtpd since i needed multithread processing (while smtpd is single-threaded, and besides that looks like discontinued).
By the way I'm puzzled by aiosmtpd management of mail envelope contents, that seems much more granular than before, so good if you need really fine tuning, but somewhat oversized if you just want to process body without modifying the rest.
To make an example, smtpd process_message method just needed data_decode=True parameter to process and decode mail body without touching anything, while aiosmtpd HANDLE_data method seems unable to automagically decode mail envelope and often gives exceptions with embedded images, attachments, and so on...
EDIT added code examples, smtpd first: following code will instantiate smtp server waiting for mail on port 10025 and delivering to 10027 via smtplib (both localhost). It is safe to work on data variable (basically perform string substitutions, my goal) for all kind of mail (text/html based, with embedded images, attachments...)
class PROXY_SMTP(smtpd.SMTPServer):
def process_message(self, peer, mailfrom, rcpttos, data, decode_data=True):
server = smtplib.SMTP('localhost', 10027)
server.sendmail(mailfrom, rcpttos, data)
server.quit()
server = PROXY_SMTP(('127.0.0.1', 10025), None)
asyncore.loop()
Previous code works well but in a single thread fashion (= 1 mail at once), so i switched to aiosmtpd to have concurrent mail processing. Same example with aiosmtpd would be roughly:
class MyHandler:
async def handle_DATA(self, server, session, envelope):
peer = session.peer
mailfrom = envelope.mail_from
rcpttos = envelope.rcpt_tos
data = envelope.content.decode()
server = smtplib.SMTP('localhost', 10027)
server.sendmail(mailfrom, rcpttos, data)
server.quit()
my_handler = MyHandler()
async def main(loop):
my_controller = Controller(my_handler, hostname='127.0.0.1', port=10025)
my_controller.start()
loop = asyncio.get_event_loop()
loop.create_task(main(loop=loop))
try:
loop.run_forever()
This code works well for text emails, but will give exceptions when decoding envelope.content with any complex mail (mime content, attachments...)
How could I parse and decode mail text in aiosmtpd, perform string substitution as I did with smtpd, and reinject via smtplib?

You are calling decode() on something whose encoding you can't possibly know or predict in advance. Modifying the raw RFC5322 message is extremely problematic anyway, because you can't easily look inside quoted-printable or base64 body parts if you want to modify the contents. Also watch out for RFC2047 encapsulation in human-visible headers, file names in RFC2231 (or some dastardly non-compliant perversion - many clients don't get this even almost right) etc. See below for an example.
Instead, if I am guessing correctly what you want, I would parse it into an email object, then take it from there.
from email import message_from_bytes
from email.policy import default
class MyHandler:
async def handle_DATA(self, server, session, envelope):
peer = session.peer
mailfrom = envelope.mail_from
rcpttos = envelope.rcpt_tos
message = message_from_bytes(envelope.content, policy=default)
# ... do things with the message,
# maybe look into the .walk() method to traverse the MIME structure
server = smtplib.SMTP('localhost', 10027)
server.send_message(message, mailfrom, rcpttos)
server.quit()
return '250 OK'
The policy argument selects the modern email.message.EmailMessage class which replaces the legacy email.message.Message class from Python 3.2 and earlier. (A lot of online examples still promote the legacy API; the new one is more logical and versatile, so you want to target that if you can.)
This also adds the missing return statement which each handler should provide as per the documentation.
Here's an example message which contains the string "Hello" in two places. Because the content-transfer-encoding obscures the content, you need to analyze the message (such as by parsing it into an email object) to be able to properly manipulate it.
From: me <me#example.org>
To: you <recipient#example.net>
Subject: MIME encapsulation demo
Mime-Version: 1.0
Content-type: multipart/alternative; boundary="covfefe"
--covfefe
Content-type: text/plain; charset="utf-8"
Content-transfer-encoding: quoted-printable
You had me at "H=
ello."
--covfefe
Content-type: text/html; charset="utf-8"
Content-transfer-encoding: base64
PGh0bWw+PGhlYWQ+PHRpdGxlPkhlbGxvLCBpcyBpdCBtZSB5b3UncmUgbG9va2luZyBmb3I/PC
90aXRsZT48L2hlYWQ+PGJvZHk+PHA+VGhlIGNvdiBpbiB0aGUgZmUgZmU8L3A+PC9ib2R5Pjwv
aHRtbD4K
--covfefe--

The OP incorrectly added this text to the question; I'm moving it here as a (half) answer.
--- SOLVED ---
This is what i gotten so far, minor adjustments are still needed (mainly for mime content separate handling and "rebuilding") but this solves my main problem: receive mail on separated threads, make room for text processing, sleep for fixed amount of time before final delivery. Thanks to tripleee answers and comments I found correct way.
import asyncio
from aiosmtpd.controller import Controller
import smtplib
from email import message_from_bytes
from email.policy import default
class MyHandler:
async def handle_DATA(self, server, session, envelope):
peer = session.peer
mailfrom = envelope.mail_from
rcpttos = envelope.rcpt_tos
message = message_from_bytes(envelope.content, policy=default)
#HERE MAYBE WOULD BE SAFER TO WALK CONTENTS AND PARSE/MODIFY ONLY MAIL BODY, BUT NO SIDE EFFECTS UNTIL NOW WITH MIME, ATTACHMENTS...
messagetostring = message.as_string() ### smtplib.sendmail WANTED BYTES or STRING, NOT email OBJECT.
### HERE HAPPENS TEXT PROCESSING, STRING SUBSTITUTIONS...
### THIS WAS MY CORE NEED, ASYNCWAIT ON EACH THREAD
await asyncio.sleep(15)
server = smtplib.SMTP('localhost', 10027)
server.send_message(mailfrom, rcpttos, messagetostring) ### NEEDED TO INVERT ARGS ORDER
server.quit()
return '250 OK' ### ADDED RETURN
my_handler = MyHandler()
async def main(loop):
my_controller = Controller(my_handler, hostname='127.0.0.1', port=10025)
my_controller.start()
loop = asyncio.get_event_loop()
loop.create_task(main(loop=loop))
try:
loop.run_forever()

Importing e-mails from one gmail mailbox to another

I have 2 Gmail accounts, and I use one ("gmail1") to forward all e-mail to the other one ("gmail2"). This works fine, but I discovered recently (after years!) that in fact, Gmail does not forward all my e-mail, but only the e-mail it considers not to be spam. Therefore, checking my spam folder in gmail2 for missing e-mails, I have often blamed senders, when really the e-mail had gotten lost on gmail1. I want to solve this programatically, by regularly checking for spam e-mails in gmail1 and importing them into gmail2 (then I'm fine with gmail2 categorizing as spam or not, as long as the mail makes its way there).
I'm only a very amateur programmer, and so far thanks to some samples from the gmail API docs, I've managed to log in to my accounts, and retrieve some messages corresponding to a query, including for example recent spam e-mails. However, I'm struggling with the "import" concept and how to use it. There are no Python examples for this, and I couldn't solve the problems I'm facing.
Where I am right now:
- if I "get" a message from gmail1 and attempt an import_ call on gmail2, using the message I just retrieved, I get an error because threadId is not allowed
- if I do the same and then "del message['threadId']" then the error becomes error 400 : 'raw' RFC822 payload message string or uploading message via /upload/* URL required. I've seen that there are some situations where upload is required, but I am completely lost as to what I should do to make this work.
Here's what I have so far (sorry for the very hacky style):
# skipping imports
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly','https://www.googleapis.com/auth/gmail.modify']
def getMessage(service, user_id, msg_id):
"""from gmail API examples"""
try:
message = service.users().messages().get(userId=user_id, id=msg_id).execute()
return message
except errors.HttpError as error:
print ('An error occurred:' , error)
def listMessagesMatchingQuery(service, user_id, query=''):
"""from gmail API examples"""
# skipping some code
return messages
def login(accountId):
"""from gmail API examples, adapted to handle 2 accounts"""
# skipping some code
return build('gmail', 'v1', credentials=creds)
def importMessage(service, user_id, msg):
"""my daring attempt at using import, without any Python sample to use as a basis"""
try:
message = service.users().messages().import_(userId=user_id, body=msg).execute()
return message
except errors.HttpError as error:
print ('An error occurred:' , error)
if __name__ == '__main__':
service_gmail = login('gmail2')
service_dnt = login('gmail1')
messages = listMessagesMatchingQuery(service_dnt,"me","in:spam is:unread after:" + str(int((datetime.now() - timedelta(hours=12)).timestamp())))
# this gets me some recent unread spam messages
m=getMessage(service_dnt,"me",messages[0]['id'])
# now I have a full message - I'm just investigating for now so the first message is enough
del m['threadId']
# if I don't do that, the error I get is that threadId is not allowed here, so I remove it
imported = importMessage(service_gmail,"me",m)
# this now gives me error 400 : 'raw' RFC822 payload message string or uploading message via /upload/* URL required
I'd like to find the way to make this work, so that the e-mail appears in gmail2 as if it had been received by gmail2 directly (though I would like to keep the To: address, as I use a catch-all on gmail1 and want to know which e-mail address the e-mail was directed to). But right now, I get only errors about having to use upload; I'm not sure if that's what I really should be doing, and if it is , I have no idea how.
Thanks a lot in advance for any help!

In the end I managed. I had missed the fact that there are 2 ways to "get" messages, a simple one and a "raw" one. With the "raw" way to access the message, I can use the import_ function easily:
message = service_dnt.users().messages().get(userId="me", id=messageId,format='raw').execute()
imported = importMessage(service_gmail,"me",{'raw': message['raw']})
whereby the importMessage function is unchanged vs. the original code I posted.

How do i handle streaming messages with Python gRPC

I'm following this Route_Guide sample.
The sample in question fires off and reads messages without replying to a specific message. The latter is what i'm trying to achieve.
Here's what i have so far:
import grpc
...
channel = grpc.insecure_channel(conn_str)
try:
grpc.channel_ready_future(channel).result(timeout=5)
except grpc.FutureTimeoutError:
sys.exit('Error connecting to server')
else:
stub = MyService_pb2_grpc.MyServiceStub(channel)
print('Connected to gRPC server.')
this_is_just_read_maybe(stub)
def this_is_just_read_maybe(stub):
responses = stub.MyEventStream(stream())
for response in responses:
print(f'Received message: {response}')
if response.something:
# okay, now what? how do i send a message here?
def stream():
yield my_start_stream_msg
# this is fine, i receive this server-side
# but i can't check for incoming messages here
I don't seem to have a read() or write() on the stub, everything seems to be implemented with iterators.
How do i send a message from this_is_just_read_maybe(stub)?
Is that even the right approach?
My Proto is a bidirectional stream:
service MyService {
rpc MyEventStream (stream StreamingMessage) returns (stream StreamingMessage) {}
}

What you're trying to do is perfectly possible and will probably involve writing your own request iterator object that can be given responses as they arrive rather than using a simple generator as your request iterator. Perhaps something like
class MySmarterRequestIterator(object):
def __init__(self):
self._lock = threading.Lock()
self._responses_so_far = []
def __iter__(self):
return self
def _next(self):
# some logic that depends upon what responses have been seen
# before returning the next request message
return <your message value>
def __next__(self): # Python 3
return self._next()
def next(self): # Python 2
return self._next()
def add_response(self, response):
with self._lock:
self._responses.append(response)
that you then use like
my_smarter_request_iterator = MySmarterRequestIterator()
responses = stub.MyEventStream(my_smarter_request_iterator)
for response in responses:
my_smarter_request_iterator.add_response(response)
. There will probably be locking and blocking in your _next implementation to handle the situation of gRPC Python asking your object for the next request that it wants to send and your responding (in effect) "wait, hold on, I don't know what request I want to send until after I've seen how the next response turned out".

Instead of writing a custom iterator, you can also use a blocking queue to implement send and receive like behaviour for client stub:
import queue
...
send_queue = queue.SimpleQueue() # or Queue if using Python before 3.7
my_event_stream = stub.MyEventStream(iter(send_queue.get, None))
# send
send_queue.push(StreamingMessage())
# receive
response = next(my_event_stream) # type: StreamingMessage
This makes use of the sentinel form of iter, which converts a regular function into an iterator that stops when it reaches a sentinel value (in this case None).

Telnet send command and then read response

This shouldn't be that complicated, but it seems that both the Ruby and Python Telnet libs have awkward APIs. Can anyone show me how to write a command to a Telnet host and then read the response into a string for some processing?
In my case "SEND" with a newline retrieves some temperature data on a device.
With Python I tried:
tn.write(b"SEND" + b"\r")
str = tn.read_eager()
which returns nothing.
In Ruby I tried:
tn.puts("SEND")
which should return something as well, the only thing I've gotten to work is:
tn.cmd("SEND") { |c| print c }
which you can't do much with c.
Am I missing something here? I was expecting something like the Socket library in Ruby with some code like:
s = TCPSocket.new 'localhost', 2000
while line = s.gets # Read lines from socket
puts line # and print them
end

I found out that if you don't supply a block to the cmd method, it will give you back the response (assuming the telnet is not asking you for anything else). You can send the commands all at once (but get all of the responses bundled together) or do multiple calls, but you would have to do nested block callbacks (I was not able to do it otherwise).
require 'net/telnet'
class Client
# Fetch weather forecast for NYC.
#
# #return [String]
def response
fetch_all_in_one_response
# fetch_multiple_responses
ensure
disconnect
end
private
# Do all the commands at once and return everything on one go.
#
# #return [String]
def fetch_all_in_one_response
client.cmd("\nNYC\nX\n")
end
# Do multiple calls to retrieve the final forecast.
#
# #return [String]
def fetch_multiple_responses
client.cmd("\r") do
client.cmd("NYC\r") do
client.cmd("X\r") do |forecast|
return forecast
end
end
end
end
# Connect to remote server.
#
# #return [Net::Telnet]
def client
#client ||= Net::Telnet.new(
'Host' => 'rainmaker.wunderground.com',
'Timeout' => false,
'Output_log' => File.open('output.log', 'w')
)
end
# Close connection to the remote server.
def disconnect
client.close
end
end
forecast = Client.new.response
puts forecast

how to sign request tokens?

I am currently trying to write a script to send off a request token, I have the header, and the claimset, but I don't understand the signature! OAuth requires my private key to be encrypted with SHA256withRSA (also known as RSASSA-PKCS1-V1_5-SIGN with the SHA-256 hash function), but the closest I could find was RSAES-PKCS1-v1_5 (has RSA, and the SHA-256 hash). I followed the example, and tweaked it, so I could get it set, but heres my dillema:
signature = ""
h = SHA.new (signature)
key = RSA.importKey(open('C:\Users\Documents\Library\KEY\My Project 905320c6324f.json').read())
cipher = PKCS1_v1_5.new(key)
ciphertext = cipher.encrypt(message+h.digest())
print(ciphertext)
I'm a bit lost, the JSON file I was given has both public key, and private, do I copy and paste the private key into the signature variable (it gave me a invalid syntax)? Or do I past the directory again? I am so lost, and way over my head haha. I am currently running Python 3.4, with pyCrypto for the signature.

Based on what you've said below about wanting to write a command system using gmail, I wrote a simple script to do this using IMAP. I think this is probably simpler than trying to use Google APIs for a single user, unless you were wanting to do that simply for the exercise.
import imaplib, logging
from time import sleep
USERNAME = 'YOUR_USERNAME_HERE' # For gmail, this is your full email address.
PASSWORD = 'YOUR_PASSWORD_HERE'
CHECK_DELAY = 60 # In seconds
LOGGING_FORMAT = '%(asctime)s %(message)s'
logging.basicConfig(filename='imapTest.log', format=LOGGING_FORMAT, level=logging.INFO)
logging.info("Connecting to IMAP server...")
imap = imaplib.IMAP4_SSL('imap.gmail.com')
imap.login(USERNAME, PASSWORD)
logging.info("Connected to IMAP server.")
def get_command_messages():
logging.info("Checking for new commands.")
imap.check()
# Search the inbox (server-side) for messages containing the subject 'COMMAND' and which are from you.
# Substitute USERNAME below for the sending email address if it differs.
typ, data = imap.search(None, '(FROM "%s" SUBJECT "COMMAND")' %(USERNAME))
return data[0]
def delete_messages(message_nums):
logging.info("Deleting old commands.")
for message in message_nums.split():
imap.store(message, '+FLAGS', '\\DELETED')
imap.expunge()
# Select the inbox
imap.select()
# Delete any messages left over that match commands, so we are starting 'clean'.
# This probably isn't the nicest way to do this, but saves checking the DATE header.
message_nums = get_command_messages()
delete_messages(message_nums)
try:
while True:
sleep(CHECK_DELAY)
# Get the message body and sent time. Use BODY.PEEK instead of BODY if you don't want to mark the message as read, but we're deleting it anyway below.
message_nums = get_command_messages()
if message_nums:
# search returns space-separated message IDs, but we need them comma-separated for fetch.
typ, messages = imap.fetch(message_nums.replace(' ', ','), '(BODY[TEXT])')
logging.info("Found %d commands" %(len(messages[0])))
for message in messages[0]:
# You now have the message body in the message variable.
# From here, you can check against it to perform commands, e.g:
if 'shutdown' in message:
print("I got a shutdown command!")
# Do stuff
delete_messages(message_nums)
finally:
try:
imap.close()
except:
pass
imap.logout()
If you're set on using the Gmail API, though, Google strongly encourage you to use their existing Python library rather than attempt to do full authentication etc. yourself as you appear to be. With that, it should - more or less - be a case of replacing the imap calls above with the relevant Gmail API ones.