I have written an email parsing mechanism in python.
It finds a new email and passes the data correctly. I am 99.999% certain that my code is functioning correctly, so there should be no issue there. The problem is that occasionally, the Gmail inbox will get flooded with messages that are considered "unseen". At this point, there is nothing that my code can do.
It fails with:
imaplib.error: FETCH command error: BAD ['Could not parse command']
This is distressing, and I would love to have either
a way to check whether the unseen messages have overflown to this state, or
a way to manually (via imaplib) mark all messages as read, including a way to detect this particular error.
Any thoughts on how to accomplish this?
Here is my code:
#!/usr/bin/env python
import imaplib, re, sys, time, OSC, threading, os
iparg = 'localhost'
oportarg = 9000
iportarg = 9002
usern = 'myusrname#gmail.com'
gpass = 'mypass'
kill_program = False
server = imaplib.IMAP4_SSL('imap.googlemail.com', 993)
oclient = OSC.OSCClient()
email_interval = 2.0
def login():
server.login(usern, gpass)
oclient.connect((iparg, oportarg))
def logout_handle(addr, tags, stuff, source):
print 'received kill call'
global kill_program
kill_program = True
def filter_signature(s): #so annoying; wish i didn't have to do this
try:
a_sig = re.sub(r'Sent|--Sent', '', s)
b_sig = re.sub(r'using SMS-to-email. Reply to this email to text the sender back and', '', a_sig)
c_sig = re.sub(r'save on SMS fees.', '', b_sig)
d_sig = re.sub(r'https://www.google.com/voice', '', c_sig)
no_lines = re.sub(r'\n|=|\r?', '', d_sig) #add weird characters to this as needed
except:
nolines = s
return no_lines
def parse_email(interval):
while True:
server.select('INBOX')
status, ids = server.search(None, 'UnSeen')
print 'status is: ', status
if not ids or ids[0] is '':
print 'no new messages'
else:
try:
print 'found a message; attempting to parse...'
latest_id = ids[0]
status, msg_data = server.fetch(latest_id, '(UID BODY[TEXT])')
raw_data = msg_data[0][1]
raw_filter = raw_data
print 'message result: ', raw_filter
time.sleep(interval)
#execute main block
while not kill_program:
login()
parse_email(email_interval)
st.kill()
sys.exit()
Based upon the error, I would very carefully check the parameters that you're passing to fetch. Gmail is telling you that it could not parse the command that you sent to it.
Also, you can do a STORE +FLAGS \SEEN to mark the messages as read.
Related
So I have the ability to open outlook and insert all the needed values to create a draft of an email, however, when I reopen the window and attempt to send another email it gives met this error.
self.mailItem = olApp.CreateItem(0)
File "<COMObject Outlook.Application>", line 2, in CreateItem
pywintypes.com_error: (-2147023174, 'The RPC server is unavailable.', None, None)
Is it the way I call it or is it the way this pywin32 works, how would I go about resolving this issue to the point where i can send as much emails, using the same code? Replacing self.mailItem.Display() with self.mailItem.Send(), allows me to send the multiple I desire however I want the user to edit any composed email before sending it. thus this not being a option.
my email class
import win32com.client as win32
from PyQt5.QtWidgets import QMessageBox
olApp = win32.Dispatch('Outlook.Application')
olNS = olApp.GetNameSpace('MAPI')
class emailComposition():
def __init__(self):
self.mailItem = olApp.CreateItem(0)
self.mailItem.BodyFormat = 1
self.mailItem.Sensitivity = 2
attachment = self.mailItem.Attachments.Add("C:\\Users\\----\Desktop\\Krypt\\Images\\Logos\\Logo1.png")
attachment.PropertyAccessor.SetProperty("http://schemas.microsoft.com/mapi/proptag/0x3712001F", "MyId1")
def setSubject(self, subj):
self.mailItem.Subject = subj
def setSender(self, senderEmail):
self.sender = senderEmail
def setRecipient(self, recipientEmail, recipientName):
self.mailItem.To = recipientEmail
self.rName = recipientName
def setText(self, ItemList):
items = "<br>• xyz <br>• xyz"
# still need to do something with the ItemList, removing it has no effect on the error outcome!
self.mailItem.HTMLBody = "<p>Good Day " + self.rName + " <br>Could you please send me a quote on the following: " + items + " </p> <p>Thanks</p><img src=""cid:MyId1"" width=""100"" height=""100"" >"
def displayEmail(self):
try:
self.mailItem._oleobj_.Invoke(*(64209, 0, 8, 0, olNS.Accounts.Item(self.sender)))
except:
msg = QMessageBox()
msg.setIcon(QMessageBox.Warning)
msg.setText("Unregistered email")
msg.setInformativeText("""The email provided isn't a registered email within outlook.\nA default email will be used.""")
msg.setWindowTitle("Krypt")
msg.exec_()
self.mailItem.Display()
Code i use to call this class, all of it is in a button press connect
def createEmail(self):
input = self.ui.tblQuoteItems.item(0, 0)
if input is not None:
if self.ui.edtSupplier.text() != "" and self.ui.edtSupplierRep.text() != "":
x = emailComposition()
x.setRecipient(self.ui.edtSupplierEmail.text(), self.ui.edtSupplierRep.text())
x.setSubject("Official Quote")
x.setSender(self.ui.edtUserEmail.text())
self.read() #Loads self.itemList with needed items
x.setText(self.itemList)
x.displayEmail()
Try to get a Namespace and any standard folder reference after creating a new Outlook Application instance in the code:
import win32com.client as win32
olApp = win32.Dispatch('Outlook.Application')
olNS = olApp.GetNameSpace('MAPI')
objFolder = objNS.GetDefaultFolder(olFolderInbox)
Read more about that in the Automating Outlook from a Visual Basic Applications article.
Also you may find the Application Shutdown Changes in Outlook article helpful.
I am trying to write a program that will scan my email inbox and take further action if the first word in the email is "Password". However, my specific condition if message_str[:9].lower() == 'password': always defaults to false, even though the string being compared is clearly 'password' as I have tested by making print statements. Why is this so?
code:
import imapclient, pyzmail, time, sys
while True:
imapObj = imapclient.IMAPClient('imap.gmail.com', ssl=True)
imapObj.login('xxxxxxxx#gmail.com', sys.argv[1])
imapObj.select_folder('INBOX', readonly=False)
UIDs = imapObj.search(['SINCE', '01-Mar-2021', 'FROM', 'xxxxxxxx#gmail.com'])
rawMessages = imapObj.fetch(UIDs, ['BODY[]'])
for i in UIDs:
message = pyzmail.PyzMessage.factory(rawMessages[i][b'BODY[]'])
if message.text_part:
message_str = message.text_part.get_payload().decode(message.text_part.charset)
a = (repr(message_str))
if message_str[:9].lower() == 'password':
continue
'password' has 8 not 9 letters, so you need to use message_str[:8].lower()
Below is my code and I am hoping someone can help me with the cleaning up the code and making it more effiencient. Basically, the code should iterate through all the volumes in my AWS account and then list all untagged volumes and then send out an email. However, it times out when running it as a lambda function in AWS but if i run it locally, it will take over 30 mins to complete (however it does complete). Im sure its iterating through things it doesnt need.
Also if I print the ec2_instances list, I can see duplicate values, so I want to only have unique values so that its not repeating the script for each ec2 instance.
import logging
import boto3
from smtplib import SMTP, SMTPException
from email.mime.text import MIMEText
logger = logging.getLogger()
logger.setLevel(logging.INFO)
session = boto3.Session(profile_name="prod")
client = session.client('ec2')
untagged_volumes = []
detached_volumes = []
ec2_instances = []
response = client.describe_volumes()
for volume in response['Volumes']:
if 'Tags' in str(volume):
continue
else:
if 'available' in str(volume):
detached_volumes.append(volume['VolumeId'])
else:
untagged_volumes.append(volume['VolumeId'])
untagged_volumes.append(volume['Attachments'][0]['InstanceId'])
ec2_instances.append(volume['Attachments'][0]['InstanceId'])
unique_instances = list(set(ec2_instances))
# Create the msg body.
msg_body_list = []
for instance in unique_instances:
desc_instance = client.describe_instances()
# append to the msg_body_list the lines that we would like to show on the email
msg_body_list.append("VolumeID: {}".format(desc_instance['Reservations'][0]['Instances'][0]['BlockDeviceMappings'][0]['Ebs']['VolumeId']))
msg_body_list.append("Attached Instance: {}".format(desc_instance['Reservations'][0]['Instances'][0]['InstanceId']))
# if there are tags, we will append it as singles lines as far we have tags
if 'Tags' in desc_instance['Reservations'][0]['Instances'][0]:
msg_body_list.append("Tags:")
for tag in desc_instance['Reservations'][0]['Instances'][0]['Tags']:
msg_body_list.append(" Key: {} | Value: {}".format(tag['Key'], tag['Value']))
# in case we don't have tags, just append no tags.
else:
msg_body_list.append("Tags: no tags")
msg_body_list.append("--------------------")
# send email
mail_from = "xxx#xxx.com"
mail_to = 'xxx#xxx.com'
msg = MIMEText("\n".join(msg_body_list))
msg["Subject"] = "EBS Tagged Instance Report for"
msg["From"] = mail_from
msg["To"] = mail_to
try:
server = SMTP('xxx.xxx.xxx.xxx', 'xx')
server.sendmail(mail_from, mail_to.split(','), msg.as_string())
server.quit()
print('Email sent')
except SMTPException:
print('ERROR! Unable to send mail')
Lambda functions have a time limit of 15 minutes. That is the reason for the timeout - if you need to run scripts for longer, look up AWS Fargate.
I'm using poplib to connect to a gmail account and I want it to reply to emails that are sent to it.
However sometimes my regular expression's statement grabs random emails that aren't the one I wanted to reply to.
For example:
5952f967.8543630a.1c70d.283f#mx.google.com
Code below:
def checkEmail():
mail = poplib.POP3_SSL('pop.gmail.com')
mail.user(USER)
mail.pass_(PASS)
numEmails = len(mail.list()[1])
count = 0
if(numEmails>=1):
for i in range(numEmails):
for msg in mail.retr(i+1)[1]:
sender = re.findall(r'[\w\.-]+#[\w\.-]+',msg)
for s in sender:
if s and count == 0:
count += 1
replySender(s)
print s
message = email.message_from_string(msg)
count = 0
mail.quit()
I have this python script for inserting raw email to db. Do not ask me why I am inserting raw mail to database.
import sys
from DB import *
import email
full_msg = sys.stdin.readlines()
j = ''.join(full_msg)
msg = email.message_from_string(j)
sql = '''INSERT INTO Email(Id, Message) VALUES (NULL, %s)'''
db = DB()
db.query(sql, (msg, ))
It would be great if I can get uid of that message, so if I for example delete message from db I can also delete message my uid on imap server.
I do not want to login to imap server and then delete message by uid because I do not know user password since it is encrypted.
I was thinking to get for example msg['Message-Id'] and then to grep files in user maildir for that Message-Id and delete actual file but that sound totally wrong to me.
I know in python you have something like UIDNEXT in imaplib but that is under assumption I am logged in which I'm not.
UPDATE:
With this I can fetch next uid but I have to login. How to get UIDNEXT without login?
By the way I use postfix/dovecot with mysql.
import getpass, sys
from imapclient import IMAPClient
try:
hostname, username = sys.argv[1:]
except ValueError:
print 'usage %s hostname username' % sys.argv[0]
c = IMAPClient(hostname, ssl=True)
try:
c.login(username, getpass.getpass())
except c.Error, e:
print "Could not login in:", e
sys.exit(1)
else:
select_dict = c.select_folder('INBOX', readonly=True)
for k, v in select_dict.items():
if k == 'UIDNEXT':
print '%s: %r' % (k,v)
c.logout()
NEW UPDATE
Sample of dovecot-uidlist
16762 W105493 S104093 :1417408077.2609_1.zumance
16763 S18340 W18608 :1417429204.3464_1.zumance
Code for geeting last line of dovecot-uidlist uid:
l = open("dovecot-uidlist").readlines()
print l[-1].split(" ")[0]
This is completed script for mail pipe:
import sys
import email
import re
from DB import *
full_msg = sys.stdin.readlines()
j = ''.join(full_msg)
msg = email.message_from_string(j)
match = re.search(r'[\w\.-]+#[\w\.-]+', msg['to'])
address = match.group(0)
address = address.split("#")
with open("/var/vmail/"+address[1]+"/"+address[0]+"/dovecot-uidlist", 'r') as f:
first_line = f.readline()
nextuid = first_line.split(" ")
nextuid = re.findall(r'\d+', nextuid[2])
sql = '''INSERT INTO Email(Id, Message, Uid, Domain, Mbox) VALUES (NULL, %s, %s, %s, %s)'''
db = DB()
db.query(sql, (msg, nextuid[0], address[1], address[0], ))
Blog post with files at https://pregmatch.org/read/python-procmail
Dovecot maintains the mapping between UID and filename in the file dovecot-uidlist. The file contains first a header line and then one line per message.
The header line looks like this:
1 1173189136 20221
The first digit is the version, the second the IMAP UIDVALIDITY, and the last is the next UID that will be used.
After that, each message has its own line looking like this:
20220 1035478339.27041_118.example.org:2,S
The first word is the UID, the next is the filename.
There's more information at the dovecot wiki.
Since parsing dovecot-uidlist will not work because list will not be updated until you check email with your email client, I decide to go with other solution. That solution is dovecot pre-auth mechanism. In my python procmail pipe script I decided to do somethig like this:
import subprocess
p = subprocess.Popen( "/usr/libexec/dovecot/imap -u "+user, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
p.stdin.write("b select INBOX\n")
p.stdin.write("e logout\n")
(stdoutdata, stderrdata) = p.communicate()
print stdoutdata
print stderrdata
stdoutdata gives me my output that look like this:
* PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in as nikola#pregmatch.org
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk NotJunk $Forwarded)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk NotJunk $Forwarded \*)] Flags permitted.
* 5574 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1412448500] UIDs valid
* OK [UIDNEXT 16875] Predicted next UID
* OK [HIGHESTMODSEQ 3051] Highest
b OK [READ-WRITE] Select completed (0.009 secs).
* BYE Logging out
e OK Logout completed.
Now all i have to do is parse this part of that output:
* OK [UIDVALIDITY 1412448500] UIDs valid
* OK [UIDNEXT 16875] Predicted next UID
This pre-auth solved my problem. I will post complete solution with parsing part later today (and on my blog).
SOLUTION:
import subprocess
pSub = subprocess.Popen( "/usr/libexec/dovecot/imap -u "+username+"#"+domain_parsed, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
pSub.stdin.write("b select INBOX\n")
pSub.stdin.write("e logout\n")
(stdoutdata, stderrdata) = pSub.communicate()
dovecotStream = open("/var/www/domain.com/scripts/server/dovecot","w")
dovecotStream.write(stdoutdata)
dovecotStream.close()
nextuidNo = []
with open("/var/www/domain.com/scripts/server/dovecot") as dovecotFile:
dovecotFilelines = dovecotFile.read()
for dovecotFileline in dovecotFilelines.split('\n'):
matchCheck = re.findall(r'\[UIDNEXT.+\]', dovecotFileline)
if len(matchCheck):
nextuidNo = re.findall(r'\d+', matchCheck[0])
print nextuidNo #this is list
Complete blog post at: https://pregmatch.org/read/dovecot-preauth-python