We Keep Coding

aiosmtpd weirdness with custom controller and SMTP factory

To start, I'm working with aiosmtpd, and am trying to write a class that wraps around it to start up the SMTP server programmatically with StartTLS. Now, up until very recently, this code worked as expected with any handler you might pass into it, such as a basic Message handler that I wrote to adjust parameters of the message, etc. and passing that in as part of the message headers.
import asyncio
import aiosmtpd
import aiosmtpd.controller
import aiosmtpd.handlers
import aiosmtpd.smtp
import email
import regex
import logging
import ssl
EMPTYBYTES = b''
COMMASPACE = ', '
CRLF = b'\r\n'
NLCRE = regex.compile(br'\r\n|\r|\n')
class StartTLSServer(aiosmtpd.controller.Controller):
def __init__(self, handler, ssl_cert_file, ssl_key_file, loop=None, hostname=None,
port=8025, *, ready_timeout=1.0, enable_SMTPUTF8=True, decode_data=False,
require_starttls=True, smtp_ident=None, data_size_limit=10485760,
smtp_timeout=300):
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain(ssl_cert_file, ssl_key_file)
self.tls_context = context
self.require_starttls = require_starttls
self.enable_SMTPUTF8 = enable_SMTPUTF8
self.decode_data = decode_data
self.smtp_ident = smtp_ident
self.data_size_limit = data_size_limit
self.smtp_timeout = smtp_timeout
super().__init__(handler, loop=loop, hostname=hostname, port=port,
ready_timeout=ready_timeout, enable_SMTPUTF8=enable_SMTPUTF8)
def factory(self):
return aiosmtpd.smtp.SMTP(self.handler, data_size_limit=self.data_size_limit,
enable_SMTPUTF8=self.enable_SMTPUTF8,
decode_data=self.decode_data,
require_starttls=self.require_starttls,
hostname=self.smtp_ident,
ident=self.smtp_ident,
tls_context=self.tls_context,
timeout=self.smtp_timeout)
class MessageHandler(aiosmtpd.handlers.Message):
def __init__(self, message_class=None, *, loop=None):
super().__init__(message_class)
self.loop = loop or asyncio.get_event_loop()
async def handle_DATA(self, server, session, envelope):
message = self.prepare_message(session, envelope)
await self.handle_message(message)
return '250 OK'
def prepare_message(self, session, envelope):
# If the server was created with decode_data True, then data will be a
# str, otherwise it will be bytes.
data = envelope.content
if isinstance(data, bytes):
message = email.message_from_bytes(data, self.message_class)
else:
assert isinstance(data, str), (
'Expected str or bytes, got {}'.format(type(data)))
message = email.message_from_string(data, self.message_class)
message['X-Peer'] = str(session.peer)
message['X-Envelope-MailFrom'] = envelope.mail_from
message['X-Envelope-RcptTo'] = COMMASPACE.join(envelope.rcpt_tos)
return message # This is handed off to handle_message directly.
async def handle_message(self, message):
print(message.as_string())
return
This resides in custom_handlers.py which is then subsequently called in testing via the Python console as follows:
>>> from custom_handlers import StartTLSServer, MessageHandler
>>> server = StartTLSServer(MessageHandler, ssl_cert_file="valid cert path", ssl_key_file="valid key path", hostname="0.0.0.0", port=25, require_starttls=True, smtp_ident="StartTLSSMTPServer01")
>>> server.start()
When I want to stop the test server, I'll simply do a server.stop() however during processing of any message, we get hard-stopped by this evil error:
Traceback (most recent call last):
File "/home/sysadmin/.local/lib/python3.8/site-packages/aiosmtpd/smtp.py", line 728, in _handle_client
await method(arg)
File "/home/sysadmin/.local/lib/python3.8/site-packages/aiosmtpd/smtp.py", line 1438, in smtp_DATA
status = await self._call_handler_hook('DATA')
File "/home/sysadmin/.local/lib/python3.8/site-packages/aiosmtpd/smtp.py", line 465, in _call_handler_hook
status = await hook(self, self.session, self.envelope, *args)
TypeError: handle_DATA() missing 1 required positional argument: 'envelope'
Now, I can replicate this with ANY handler passed into the SMTP factory.
However, I can't replicate this with a plain aiosmtpd with a Debugging handler, like defined in the docs:
aiosmtpd -c aiosmtpd.handlers.Debugging stdout -l 0.0.0.0:8025
... which works fine. Passing the Debugging handler into the StartTLSServer causes the same error as the custom MessageHandler class, even with the Debugging handler.
Am I missing something obvious here about my class that's exploding here in a way that is different to the programmatic usage as expected by aiosmtpd?

You are missing () in order to instantiate an object of your MessageHandler class:
>>> server = StartTLSServer(MessageHandler(), ...)
When you simply pass MessageHandler without the (), aiosmtpd will try to invoke the regular function MessageHandler.handle_DATA(...) (as opposed to the bound method function MessageHandler().handle_DATA(...)).
This regular function takes four arguments: an instance of MessageHandler as its first argument followed by the usual server, session and envelope arguments. This explains why the error message complains about a missing positional argument.
PS, do note that your handle_DATA implementation is superfluous since it is identical to the implementation in the base class aiosmtpd.handlers.Message - so you can just delete it, and it should still work just fine.

Python msfrpc works with python2, throws authentication error with python 3

I'm writing a web app using python flask and python3, and would like to use the metasploit API. When writing code with python2, everything works as it should (as the lib was written for python2). However, when attempting to use it in python 3, i'm given this error:
File "/usr/local/lib/python3.6/dist-packages/msfrpc.py", line 64, in login
raise self.MsfAuthError("MsfRPC: Authentication failed")
msfrpc.MsfAuthError: 'MsfRPC: Authentication failed'
The only difference in the msfrpc.py file between the python2 and python 3 versions is that the py2 version includes "httplib" and uses "httplib.HTTPSConnection" to connect to the msgrpc service, and the py3 version includes "http.client" and uses "http.client.HTTPConnection" to connect to the service.
Anyone know why this error is happening?
Here is the source code for msfrpc.py:
import msgpack
import http.client
class Msfrpc:
class MsfError(Exception):
def __init__(self,msg):
self.msg = msg
def __str__(self):
return repr(self.msg)
class MsfAuthError(MsfError):
def __init__(self,msg):
self.msg = msg
def __init__(self,opts=[]):
self.host = opts.get('host') or "127.0.0.1"
self.port = opts.get('port') or 55552
self.uri = opts.get('uri') or "/api/"
self.ssl = opts.get('ssl') or False
self.authenticated = False
self.token = False
self.headers = {"Content-type" : "binary/message-pack" }
if self.ssl:
self.client = http.client.HTTPConnection(self.host,self.port)
else:
self.client = http.client.HTTPConnection(self.host,self.port)
def encode(self,data):
return msgpack.packb(data)
def decode(self,data):
return msgpack.unpackb(data)
def call(self,meth,opts = []):
if meth != "auth.login":
if not self.authenticated:
raise self.MsfAuthError("MsfRPC: Not Authenticated")
if meth != "auth.login":
opts.insert(0,self.token)
opts.insert(0,meth)
params = self.encode(opts)
self.client.request("POST",self.uri,params,self.headers)
resp = self.client.getresponse()
return self.decode(resp.read())
def login(self,user,password):
ret = self.call('auth.login',[user,password])
if ret.get('result') == 'success':
self.authenticated = True
self.token = ret.get('token')
return True
else:
raise self.MsfAuthError("MsfRPC: Authentication failed")

I'm not sure if I should bring this question back from the dead, but I had the same question a bit ago and I figured out the problem/solution so I'll post it here:
Python msfrpc works with python2, throws authentication error with python 3
The problem was that msfrpc was originally written in Python 2, and therefore the prefix of 'b' in the msgpack RPC responses from the Metasploit host was ignored but it is required in Python 3 to indicate that the literal should become a bytes literal instead of a Unicode string type.
For some more info, here's a link to a post that helped me figure it out:
https://timothybramlett.com/Strings_Bytes_and_Unicode_in_Python_2_and_3.html
The corrected code I posted in the link above works fine (you just have to put a "b" in front of the strings returned from the server to tell Python that the 0's and 1's in the responses are bits/bytes and not Unicode code points), but really a better solution is to use the msfrpc module written by Dan McInerney which in addition to "indicating bytes", instead of 'http.client' uses the 'requests' package.
https://github.com/DanMcInerney/msfrpc

How do i handle streaming messages with Python gRPC

I'm following this Route_Guide sample.
The sample in question fires off and reads messages without replying to a specific message. The latter is what i'm trying to achieve.
Here's what i have so far:
import grpc
...
channel = grpc.insecure_channel(conn_str)
try:
grpc.channel_ready_future(channel).result(timeout=5)
except grpc.FutureTimeoutError:
sys.exit('Error connecting to server')
else:
stub = MyService_pb2_grpc.MyServiceStub(channel)
print('Connected to gRPC server.')
this_is_just_read_maybe(stub)
def this_is_just_read_maybe(stub):
responses = stub.MyEventStream(stream())
for response in responses:
print(f'Received message: {response}')
if response.something:
# okay, now what? how do i send a message here?
def stream():
yield my_start_stream_msg
# this is fine, i receive this server-side
# but i can't check for incoming messages here
I don't seem to have a read() or write() on the stub, everything seems to be implemented with iterators.
How do i send a message from this_is_just_read_maybe(stub)?
Is that even the right approach?
My Proto is a bidirectional stream:
service MyService {
rpc MyEventStream (stream StreamingMessage) returns (stream StreamingMessage) {}
}

What you're trying to do is perfectly possible and will probably involve writing your own request iterator object that can be given responses as they arrive rather than using a simple generator as your request iterator. Perhaps something like
class MySmarterRequestIterator(object):
def __init__(self):
self._lock = threading.Lock()
self._responses_so_far = []
def __iter__(self):
return self
def _next(self):
# some logic that depends upon what responses have been seen
# before returning the next request message
return <your message value>
def __next__(self): # Python 3
return self._next()
def next(self): # Python 2
return self._next()
def add_response(self, response):
with self._lock:
self._responses.append(response)
that you then use like
my_smarter_request_iterator = MySmarterRequestIterator()
responses = stub.MyEventStream(my_smarter_request_iterator)
for response in responses:
my_smarter_request_iterator.add_response(response)
. There will probably be locking and blocking in your _next implementation to handle the situation of gRPC Python asking your object for the next request that it wants to send and your responding (in effect) "wait, hold on, I don't know what request I want to send until after I've seen how the next response turned out".

Instead of writing a custom iterator, you can also use a blocking queue to implement send and receive like behaviour for client stub:
import queue
...
send_queue = queue.SimpleQueue() # or Queue if using Python before 3.7
my_event_stream = stub.MyEventStream(iter(send_queue.get, None))
# send
send_queue.push(StreamingMessage())
# receive
response = next(my_event_stream) # type: StreamingMessage
This makes use of the sentinel form of iter, which converts a regular function into an iterator that stops when it reaches a sentinel value (in this case None).

Python twisted irc: Wait for a whois reply inside privmsg method

I'm trying to make an IRC bot using the twisted.words.protocols.irc module.
The bot will parse messages from a channel and parse them for command strings.
Everything works fine except when I need the bot to identify a nick by sending a whois command. The whois reply will not be handled until the privmsg method (the method from which I'm doing the parsing) returns.
example:
from twisted.words.protocols import irc
class MyBot(irc.IRClient):
..........
def privmsg(self, user, channel, msg):
"""This method is called when the client recieves a message"""
if msg.startswith(':whois '):
nick = msg.split()[1]
self.whois(nick)
print(self.whoislist)
def irc_RPL_WHOISCHANNELS(self, prefix, params):
"""This method is called when the client recieves a reply for whois"""
self.whoislist[prefix] = params
Is there a way to somehow make the bot wait for a reply after self.whois(nick)?
Perhaps use a thread (I don't have any experience with those).

Deferred is a core concept in Twisted, you must be familiar with it to use Twisted.
Basically, your whois checking function should return a Deferred that will be fired when you receive whois-reply.

I managed to fix this by running all handler methods as threads, and then setting a field, following
kirelagin's suggestion, before running a whois query, and modifying the method that recieves the data
to change the field when it recieves a reply. Its not the most elegant solution but it works.
Modified code:
class MyBot(irc.IRClient):
..........
def privmsg(self, user, channel, msg):
"""This method is called when the client recieves a message"""
if msg.startswith(':whois '):
nick = msg.split()[1]
self.whois_status = 'REQUEST'
self.whois(nick)
while not self.whois_status == 'ACK':
sleep(1)
print(self.whoislist)
def irc_RPL_WHOISCHANNELS(self, prefix, params):
"""This method is called when the client recieves a reply for whois"""
self.whoislist[prefix] = params
def handleCommand(self, command, prefix, params):
"""Determine the function to call for the given command and call
it with the given arguments.
"""
method = getattr(self, "irc_%s" % command, None)
try:
# all handler methods are now threaded.
if method is not None:
thread.start_new_thread(method, (prefix, params))
else:
thread.start_new_thread(self.irc_unknown, (prefix, command, params))
except:
irc.log.deferr()
def irc_RPL_WHOISCHANNELS(self, prefix, params):
"""docstring for irc_RPL_WHOISCHANNELS"""
self.whoislist[prefix] = params
def irc_RPL_ENDOFWHOIS(self, prefix, params):
self.whois_status = 'ACK'

Fetching Email in Twisted using IMAP

I am trying to fetch emails on a gmail account using twisted, and to say the least it has been a pain, looking at email is their a clear explanation and structure (it seems hacked together at best). I am trying to grab attachments yet the attachment isn't any where in sight.
I am using the example IMAP Client from twisted and modified it, I am using fetchAll('1:') to get the email and then getting the first email but I can't find the email attachment that is on that email (I checked it is there in google). Also what is with the 1: and I can't seem to find any thing that actually explains email (as though no one understands it)
So Stackoverflow, what am I missing?
Code
#!/usr/bin/env python
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.
"""
Simple IMAP4 client which displays the subjects of all messages in a
particular mailbox.
"""
import sys
from twisted.internet import protocol
from twisted.internet import ssl
from twisted.internet import defer
from twisted.internet import stdio
from twisted.mail import imap4
from twisted.protocols import basic
from twisted.python import util
from twisted.python import log
class TrivialPrompter(basic.LineReceiver):
from os import linesep as delimiter
promptDeferred = None
def prompt(self, msg):
assert self.promptDeferred is None
self.display(msg)
self.promptDeferred = defer.Deferred()
return self.promptDeferred
def display(self, msg):
self.transport.write(msg)
def lineReceived(self, line):
if self.promptDeferred is None:
return
d, self.promptDeferred = self.promptDeferred, None
d.callback(line)
class SimpleIMAP4Client(imap4.IMAP4Client):
"""
A client with callbacks for greeting messages from an IMAP server.
"""
greetDeferred = None
def serverGreeting(self, caps):
self.serverCapabilities = caps
if self.greetDeferred is not None:
d, self.greetDeferred = self.greetDeferred, None
d.callback(self)
class SimpleIMAP4ClientFactory(protocol.ClientFactory):
usedUp = False
protocol = SimpleIMAP4Client
def __init__(self, username, onConn):
self.ctx = ssl.ClientContextFactory()
self.username = username
self.onConn = onConn
def buildProtocol(self, addr):
"""
Initiate the protocol instance. Since we are building a simple IMAP
client, we don't bother checking what capabilities the server has. We
just add all the authenticators twisted.mail has. Note: Gmail no
longer uses any of the methods below, it's been using XOAUTH since
2010.
"""
assert not self.usedUp
self.usedUp = True
p = self.protocol(self.ctx)
p.factory = self
p.greetDeferred = self.onConn
p.registerAuthenticator(imap4.PLAINAuthenticator(self.username))
p.registerAuthenticator(imap4.LOGINAuthenticator(self.username))
p.registerAuthenticator(
imap4.CramMD5ClientAuthenticator(self.username))
return p
def clientConnectionFailed(self, connector, reason):
d, self.onConn = self.onConn, None
d.errback(reason)
def cbServerGreeting(proto, username, password):
"""
Initial callback - invoked after the server sends us its greet message.
"""
# Hook up stdio
tp = TrivialPrompter()
stdio.StandardIO(tp)
# And make it easily accessible
proto.prompt = tp.prompt
proto.display = tp.display
# Try to authenticate securely
return proto.authenticate(password
).addCallback(cbAuthentication, proto
).addErrback(ebAuthentication, proto, username, password
)
def ebConnection(reason):
"""
Fallback error-handler. If anything goes wrong, log it and quit.
"""
log.startLogging(sys.stdout)
log.err(reason)
return reason
def cbAuthentication(result, proto):
"""
Callback after authentication has succeeded.
Lists a bunch of mailboxes.
"""
return proto.list("", "*"
).addCallback(cbMailboxList, proto
)
def ebAuthentication(failure, proto, username, password):
"""
Errback invoked when authentication fails.
If it failed because no SASL mechanisms match, offer the user the choice
of logging in insecurely.
If you are trying to connect to your Gmail account, you will be here!
"""
failure.trap(imap4.NoSupportedAuthentication)
return proto.prompt(
"No secure authentication available. Login insecurely? (y/N) "
).addCallback(cbInsecureLogin, proto, username, password
)
def cbInsecureLogin(result, proto, username, password):
"""
Callback for "insecure-login" prompt.
"""
if result.lower() == "y":
# If they said yes, do it.
return proto.login(username, password
).addCallback(cbAuthentication, proto
)
return defer.fail(Exception("Login failed for security reasons."))
def cbMailboxList(result, proto):
"""
Callback invoked when a list of mailboxes has been retrieved.
"""
result = [e[2] for e in result]
s = '\n'.join(['%d. %s' % (n + 1, m) for (n, m) in zip(range(len(result)), result)])
if not s:
return defer.fail(Exception("No mailboxes exist on server!"))
return proto.prompt(s + "\nWhich mailbox? [1] "
).addCallback(cbPickMailbox, proto, result
)
def cbPickMailbox(result, proto, mboxes):
"""
When the user selects a mailbox, "examine" it.
"""
mbox = mboxes[int(result or '1') - 1]
return proto.examine(mbox
).addCallback(cbExamineMbox, proto
)
def cbExamineMbox(result, proto):
"""
Callback invoked when examine command completes.
Retrieve the subject header of every message in the mailbox.
"""
# FETCH ALL HEADERS? WHERE IS A ONE FOR AN ATTACHMENT
return proto.fetchAll('1:*').addCallback(cbFetch, proto)
def cbFetch(result, proto):
"""
Finally, display headers.
"""
if result:
keys = result.keys()
keys.sort()
k = keys[-1]
proto.display('%s %s' % (k, result[k]))
else:
print "Hey, an empty mailbox!"
return proto.logout()
def cbClose(result):
"""
Close the connection when we finish everything.
"""
from twisted.internet import reactor
reactor.stop()
def main():
hostname = raw_input('IMAP4 Server Hostname: ')
port = raw_input('IMAP4 Server Port (the default is 143, 993 uses SSL): ')
username = raw_input('IMAP4 Username: ')
password = util.getPassword('IMAP4 Password: ')
onConn = defer.Deferred(
).addCallback(cbServerGreeting, username, password
).addErrback(ebConnection
).addBoth(cbClose)
factory = SimpleIMAP4ClientFactory(username, onConn)
from twisted.internet import reactor
if port == '993':
reactor.connectSSL(hostname, int(port), factory, ssl.ClientContextFactory())
else:
if not port:
port = 143
reactor.connectTCP(hostname, int(port), factory)
reactor.run()
if __name__ == '__main__':
main()

First off, imap4 is a (perhaps needlessly) complex protocol for mail handling, and twisted's client implementation is (necessarily) complex to fully support that protocol. To get much of anywhere, you should consider spending some time reading the standard that explains the protocol: rfc3501 and the relevent parts of twisted's api.
That said, it looks like you're using IMAP4Client.fetchAll(), which, paradoxically, fetches "envelope" data, the headers and metadata about the message. the comparable call that returns the body of the email, too, is actually fetchFull().

Thanks to the pointers to RFC3501 and RFC822 by SingleNegationElimination and Jean-Paul, I think I have a better understanding of how to achieve this.
The solution I found is to the API fetchSpecific source code of imap4. You can specify which "part" of the message (i.e. a mail in your inbox) to retrieve. For attachment, the content of the attached file(s) is embedded in the TEXT part and encoded with base64. In my particular example, there is a pdf attachment in the mail, and
proto.fetchSpecific(imap4.MessageSet(247), uid=True, headerType='TEXT').addCallback(cbViewAttachment, proto)
def cbViewAttachment(result, proto):
for k, value in result.items():
print(value[0][4][:400])
...
gives something like
------_CANON_2007111239350128_
Content-Type: Application/pdf;
name="0128_20200711123935_001.pdf"
Content-Disposition: attachment;
filename="0128_20200711123935_001.pdf"
Content-Transfer-Encoding: base64
JVBERi0xLjYKJeLjz9MNCjEgMCBvYmoKPDwgCi9DcmVhdGlvbkRhdGUgKEQ6MjAyMDA3MTEyMDM5
MThaMDAnMDAnKQovQ3JlYXRvciAoXDM3NlwzNzdcMDAwQ1wwMDBhXDAwMG5cMDAwb1wwMDBuXDAw
MCBcMDAwTVwwMDBGXDAwMDJcMDAwNFw
Some quick explanation. The above code does fetch the TEXT part of the message with UID = 247. Upon success of the call to fetchSpecific, cbDownloadAttachment is called, where the first argument is a dictionary of TEXT parts - one entry for each of the specified message. In this case there is only one message being specified, so the result will just be a dictionary with one single entry corresponding to the UID = 247 message.
The actual attachment content is embedded in the [0][4] of the value of the entry. It looks like above. It is almost obvious that the JVB... part is the base64-encoded content. I tried to decode the first small portion of it with base64 and it looks like
%PDF-1.6\n%\xe2\xe3
which means that this is actually the beginning of a PDF file.
You can then write the content to a local file so as to "download" the attachment. In the above example it is basically
with open('your/file/name', 'wb') as f:
f.write(base64.urlsafe_b64decode(value[0][4][213:]))
Note that 213 is hard-coded as the start of the content. In the real-world case you should definitely parse the TEXT a bit to find it out.
Hope this helps someone in the similar situation as me.