We Keep Coding

How to compress email (without any attachment) using Gmail API send() method?

I have tried manipulating headers, tried adding/replacing encoding to 'gzip' but adding new headers for to accept 'gzip' format or replacing current headers only increases the overall size of the email rather than reducing it.
My code:
def create_message(sender, to, subject, message_text):
"""Create a message for an email.
Args:
sender: Email address of the sender.
to: Email address of the receiver.
subject: The subject of the email message.
message_text: The text of the email message.
Returns:
An object containing a base64url encoded email object.
"""
message = MIMEText(message_text)
message['to'] = to
message['from'] = sender
message['subject'] = subject
# message.add_header('Accept-Encoding', 'gzip')
# message.add_header('User-Agent', 'gzip')
message.add_header('Content-Encoding', 'gzip')
# message.replace_header('Content-Transfer-Encoding', 'gzip')
to_be_sent = {'raw': base64.urlsafe_b64encode(message.as_string().encode()).decode()}
return to_be_sent
def send_message(service, user_id, message):
"""Send an email message.
Args:
service: Authorized Gmail API service instance.
user_id: User's email address. The special value "me"
can be used to indicate the authenticated user.
message: Message to be sent.
Returns:
Sent Message.
"""
try:
message = (service.users().messages().send(userId=user_id, body=message).execute())
# print('Message Id: %s' % message['id'])
return message
except Exception as error:
print('An error occurred: %s' % error)
To read the message:
# Call the Gmail API - read
message = service.users().messages().get(userId="me", id=MESSAGE_ID, format="full").execute()
The Gmail API performance tip states that:
An easy and convenient way to reduce the bandwidth needed for each request is to enable gzip compression.In order to receive a gzip-encoded response you must do two things: Set an Accept-Encoding header, and modify your user agent to contain the string gzip.
Accept-Encoding: gzip
User-Agent: my program (gzip)
Still not working for me. Only keeps the size either constant or increases it.

It won't be possible with the method you are asking for. As #VPfB mentioned, the documentation says response.
Although I thought of alternative and it is to compress the content/body if your sole purpose is to diminish the mail size. You can also decompress the content afterwards if needed be.
I did the test earlier. See the following results below:
Test conditions:
Tested in Google Apps Script using Gmail API (python is unavailable in my device)
Long string content test length: 8802
Short string content test length: 452
Test cases are generated through here
Compression used is LZ string compression
Output (sizeEstimate):
Long uncompressed: 18928
Long compression: 18421
Short uncompressed: 1627
Short compression: 2081
Conclusion:
If you are planning to use this on fairly short content emails, don't bother, it will not save you any storage and will only enlarge the size.
If you are using it on long content emails, it saves a very little storage but it's up to you to decide if that is worth saving that much but having the need to decompress it every time you want to read the email.
Reference includes both compress and decompress functions.
See Python implemenation by eduardtomasek if you want to test it on your end.

parse and decode mail text in aiosmtpd, perform string substitution, and reinject

I began with smtpd in order to process mailqueue, parse inbound emails and send them back to recipients (using smtpdlib.sendmail).
I switched to aiosmtpd since i needed multithread processing (while smtpd is single-threaded, and besides that looks like discontinued).
By the way I'm puzzled by aiosmtpd management of mail envelope contents, that seems much more granular than before, so good if you need really fine tuning, but somewhat oversized if you just want to process body without modifying the rest.
To make an example, smtpd process_message method just needed data_decode=True parameter to process and decode mail body without touching anything, while aiosmtpd HANDLE_data method seems unable to automagically decode mail envelope and often gives exceptions with embedded images, attachments, and so on...
EDIT added code examples, smtpd first: following code will instantiate smtp server waiting for mail on port 10025 and delivering to 10027 via smtplib (both localhost). It is safe to work on data variable (basically perform string substitutions, my goal) for all kind of mail (text/html based, with embedded images, attachments...)
class PROXY_SMTP(smtpd.SMTPServer):
def process_message(self, peer, mailfrom, rcpttos, data, decode_data=True):
server = smtplib.SMTP('localhost', 10027)
server.sendmail(mailfrom, rcpttos, data)
server.quit()
server = PROXY_SMTP(('127.0.0.1', 10025), None)
asyncore.loop()
Previous code works well but in a single thread fashion (= 1 mail at once), so i switched to aiosmtpd to have concurrent mail processing. Same example with aiosmtpd would be roughly:
class MyHandler:
async def handle_DATA(self, server, session, envelope):
peer = session.peer
mailfrom = envelope.mail_from
rcpttos = envelope.rcpt_tos
data = envelope.content.decode()
server = smtplib.SMTP('localhost', 10027)
server.sendmail(mailfrom, rcpttos, data)
server.quit()
my_handler = MyHandler()
async def main(loop):
my_controller = Controller(my_handler, hostname='127.0.0.1', port=10025)
my_controller.start()
loop = asyncio.get_event_loop()
loop.create_task(main(loop=loop))
try:
loop.run_forever()
This code works well for text emails, but will give exceptions when decoding envelope.content with any complex mail (mime content, attachments...)
How could I parse and decode mail text in aiosmtpd, perform string substitution as I did with smtpd, and reinject via smtplib?

You are calling decode() on something whose encoding you can't possibly know or predict in advance. Modifying the raw RFC5322 message is extremely problematic anyway, because you can't easily look inside quoted-printable or base64 body parts if you want to modify the contents. Also watch out for RFC2047 encapsulation in human-visible headers, file names in RFC2231 (or some dastardly non-compliant perversion - many clients don't get this even almost right) etc. See below for an example.
Instead, if I am guessing correctly what you want, I would parse it into an email object, then take it from there.
from email import message_from_bytes
from email.policy import default
class MyHandler:
async def handle_DATA(self, server, session, envelope):
peer = session.peer
mailfrom = envelope.mail_from
rcpttos = envelope.rcpt_tos
message = message_from_bytes(envelope.content, policy=default)
# ... do things with the message,
# maybe look into the .walk() method to traverse the MIME structure
server = smtplib.SMTP('localhost', 10027)
server.send_message(message, mailfrom, rcpttos)
server.quit()
return '250 OK'
The policy argument selects the modern email.message.EmailMessage class which replaces the legacy email.message.Message class from Python 3.2 and earlier. (A lot of online examples still promote the legacy API; the new one is more logical and versatile, so you want to target that if you can.)
This also adds the missing return statement which each handler should provide as per the documentation.
Here's an example message which contains the string "Hello" in two places. Because the content-transfer-encoding obscures the content, you need to analyze the message (such as by parsing it into an email object) to be able to properly manipulate it.
From: me <me#example.org>
To: you <recipient#example.net>
Subject: MIME encapsulation demo
Mime-Version: 1.0
Content-type: multipart/alternative; boundary="covfefe"
--covfefe
Content-type: text/plain; charset="utf-8"
Content-transfer-encoding: quoted-printable
You had me at "H=
ello."
--covfefe
Content-type: text/html; charset="utf-8"
Content-transfer-encoding: base64
PGh0bWw+PGhlYWQ+PHRpdGxlPkhlbGxvLCBpcyBpdCBtZSB5b3UncmUgbG9va2luZyBmb3I/PC
90aXRsZT48L2hlYWQ+PGJvZHk+PHA+VGhlIGNvdiBpbiB0aGUgZmUgZmU8L3A+PC9ib2R5Pjwv
aHRtbD4K
--covfefe--

The OP incorrectly added this text to the question; I'm moving it here as a (half) answer.
--- SOLVED ---
This is what i gotten so far, minor adjustments are still needed (mainly for mime content separate handling and "rebuilding") but this solves my main problem: receive mail on separated threads, make room for text processing, sleep for fixed amount of time before final delivery. Thanks to tripleee answers and comments I found correct way.
import asyncio
from aiosmtpd.controller import Controller
import smtplib
from email import message_from_bytes
from email.policy import default
class MyHandler:
async def handle_DATA(self, server, session, envelope):
peer = session.peer
mailfrom = envelope.mail_from
rcpttos = envelope.rcpt_tos
message = message_from_bytes(envelope.content, policy=default)
#HERE MAYBE WOULD BE SAFER TO WALK CONTENTS AND PARSE/MODIFY ONLY MAIL BODY, BUT NO SIDE EFFECTS UNTIL NOW WITH MIME, ATTACHMENTS...
messagetostring = message.as_string() ### smtplib.sendmail WANTED BYTES or STRING, NOT email OBJECT.
### HERE HAPPENS TEXT PROCESSING, STRING SUBSTITUTIONS...
### THIS WAS MY CORE NEED, ASYNCWAIT ON EACH THREAD
await asyncio.sleep(15)
server = smtplib.SMTP('localhost', 10027)
server.send_message(mailfrom, rcpttos, messagetostring) ### NEEDED TO INVERT ARGS ORDER
server.quit()
return '250 OK' ### ADDED RETURN
my_handler = MyHandler()
async def main(loop):
my_controller = Controller(my_handler, hostname='127.0.0.1', port=10025)
my_controller.start()
loop = asyncio.get_event_loop()
loop.create_task(main(loop=loop))
try:
loop.run_forever()

Keeping track of multiple clients on grpc server

I am trying to create a grpc python server that can keep track of all clients connected.
I am referencing a talk/demo that Ray Tsang did where he kept a collection of StreamObservers and just iterated through them to send to all the clients. Here is a video of that for reference.
Now my question is how do you get a StreamObserver in python? I only see self, request and context as being available to me in the definition.
This is my first python project so there might be something obvious I am missing here.
Here is my proto, its basically the sample proto
syntax = "proto3";
package hellostreamingworld;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
// Sends another greeting
rpc SayHelloAgain (HelloRequest) returns (stream HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}

If I have understand you need to create a class ClientInterceptor that extends grpc.UnaryUnaryClientInterceptor(https://grpc.io/grpc/python/grpc.html?highlight=unaryunaryclientinterceptor#grpc.UnaryUnaryClientInterceptor)
and then assign it with the intercept_channel method in this way
self.channel = grpc.insecure_channel(address, options)
self.channel = grpc.intercept_channel(
self.channel,
ClientInterceptor()
)
You can use the intercept_unary_unary method for receive infos about the various clients.
If you want to have infos server side extends the ServerInterceptor(https://grpc.io/grpc/python/grpc.html?highlight=serverinterceptor#grpc.ServerInterceptor) and assign it on the server init
self.server = grpc.server(futures.ThreadPoolExecutor(max_workers=1000),
options=(('grpc.max_send_message_length', 1000 * 1024 * 1024,),
('grpc.max_receive_message_length', 1000 * 1024 * 1024,),
),
interceptors=(MyServerInterceptor())
)
And the use the intercept_service method for receive infos.

What you probably need is a bidirectional stream service. That is both the client and server are continuously sending data over. In this case gRPC + python will keep track of your client!
Check out this example code and corresponding explanation.

Get account summary details from interactive brokers using python

I have some questions based on automated trading via IB using python.
I can access to TWS, but when I am request for account summary I can't put them into constant variables to use it, I only received them as an printing output.
Here my code:
from ib.ext.Contract import Contract
from ib.ext.Order import Order
from ib.opt import Connection, message
def error_handler(msg):
"""Handles the capturing of error messages"""
print "Server Error: %s" % msg
def reply_handler(msg):
"""Handles of server replies"""
print "Server Response: %s, %s" % (msg.typeName, msg)
if __name__ == "__main__":
tws_conn = Connection.create(port=4096, clientId=150)
tws_conn.connect()
tws_conn.registerAll(reply_handler)
tws_conn.reqAccountSummary(119, "All", "TotalCashValue")
time.sleep(4)
The output on the screen(cmd):
Server Response: accountSummary, <accountSummary reqId=119,
account=DU860294, tag=TotalCashValue, value=980232.77, currency=USD>
Server Response: accountSummary, <accountSummary reqId=119,
account=DUC00074, tag=TotalCashValue, value=610528.18, currency=USD>
Server Response: accountSummaryEnd, <accountSummaryEnd reqId=119>
My needed is to put all these informations into variables to use it in my program.
Thanks in advance.

You can obtain this information via updateAccountValue event too. Subscribe for this event using tws_conn.reqAccountUpdates() and you'll receive account related informations in
updateAccountValue(string key, string value, string currency, string accountName)
Then you can filter the messages like key=="TotalCashValue" and convert the value string into a double variable.

How to send multicast message (multiple users) using xmpp and Python xmpppy (XEP-0033: Extended Stanza Addressing)

I am using xmpppy http://xmpppy.sourceforge.net/ to send Jabber notifications, it is working well for single destinations using the following code:
# pip install https://github.com/rochacbruno/xmpppy/tarball/master
import xmpp
JABBER_SETTINGS = {"USERNAME": None, "PASSWORD": None, "DOMAIN": None, "RESOURCE": None}
def get_jabber_client():
client = xmpp.Client(JABBER_SETTINGS.get('DOMAIN'))
client.connect(server=(JABBER_SETTINGS.get('DOMAIN'), '5222'))
client.auth(
JABBER_SETTINGS.get('USERNAME'),
JABBER_SETTINGS.get('PASSWORD'),
JABBER_SETTINGS.get('RESOURCE')
)
client.sendInitPresence()
return client
def send_message(to, message):
client = get_jabber_client()
xmpp_message = xmpp.Message(to, message)
client.send(xmpp_message)
client.disconnect()
send_message("single.destination#domain.com", "Hello World!")
But now I need to send the message to multiple destinations, for now I am doing.
for users in list_of_users:
send_message(user, "Hello World!")
Which works fine, but every time I call it starts the process of authentication and takes a lot of time.
I've tried to create a single client and use the same client to send the message.
def send_message(to, message):
if isinstance(to, basestring):
to = [to]
assert isinstance(to, (list, tuple))
client = get_jabber_client()
for destination in to:
xmpp_message = xmpp.Message(destination, message)
client.send(xmpp_message)
client.disconnect()
send_message(['user1...', 'user2...'], "Hello World!")
The code above works, but only the first user in the list gets the message well formatted, the other users receives the message in pure XML.
I saw this code (in .net), mentioning XEP-0033: Extended Stanza Addressing http://forum.ag-software.net/thread/1482-Send-Message-To-all-users-in-contact-list
var addresses = new Addresses();
addresses.AddAddress(new Address
{
Type = Type.to,
Jid = "hildjj#jabber.org/Work",
Description = "Joe Hildebrand"
});
addresses.AddAddress(new Address
{
Type = Type.cc,
Jid = "jer#jabber.org/Home",
Description = "Jeremie Miller"
});
var msg = new Matrix.Xmpp.Client.Message();
msg.Add(addresses);
msg.To = "multicast.jabber.org";
msg.Body = "Hello, world!";
builds the following Xml:
<message to='multicast.jabber.org'>
<addresses xmlns='http://jabber.org/protocol/address'>
<address type='to' jid='hildjj#jabber.org/Work' desc='Joe Hildebrand'/>
<address type='cc' jid='jer#jabber.org/Home' desc='Jeremie Miller'/>
</addresses>
<body>Hello, world!</body>
</message>
But I did not found the way to do the same in Python using xmpppy, any idea on how to build the multicast stanza and send the message to multiple users using Python?
Thanks.

You could probably get this done without XEP-0033. In this loop, you overwrite the value of the variable message the first time, and subsequent messages will get garbled contents:
for destination in to:
message = xmpp.Message(destination, message)
client.send(message)
Try this instead:
for destination in to:
xmpp_message = xmpp.Message(destination, message)
client.send(xmpp_message)