As of May this year, Google has discontinued the option to allow "less secure apps" to access Gmail accounts using "simply" username and password information. This has meant that a Python code I had been operating to extract emails (see below some of the modules / elements used) now leads to the following error:
imaplib.IMAP4.error: b'[AUTHENTICATIONFAILED] Invalid credentials (Failure)'
Is there still a reliable way to extract email information from Gmail using a Python module? Any advice greatly appreciated
Selected code from current programme
import smtplib
import time
import imaplib
import email
import traceback
import re
from bs4 import BeautifulSoup
import datetime
import pandas as pd
import requests
import env
ORG_EMAIL = "#gmail.com"
FROM_EMAIL = "xxxx" + ORG_EMAIL
FROM_PWD = "xxx"
SMTP_SERVER = "imap.gmail.com"
SMTP_PORT = 993
emails_regex_2= r'[\w\.-]+#[\w\.-]+(?:\.[\w]+)+'
time_format = '%d %b %Y'
mail = imaplib.IMAP4_SSL(SMTP_SERVER)
mail.login(FROM_EMAIL,FROM_PWD)
mail.select('inbox')
data = mail.search(None, 'ALL')
I had the same problem. I am also to lazy to rewrite my utility scripts. And in addition, I prefer not to, since I meant them to be scripts to check emails account, not specifically gmail account (I know google would want to abolish standardization of protocol, but there are still other email providers around).
The way that worked with me (so far. Google is now saying that this method is also "less secure", which is usually what they say 2 years before deactivating it), is to create app password.
For that you need to enable 2-steps validation (I know, it seems to be a step in the wrong direction for what you are trying to do, but it is not). And then to go to your google account, section "Security" and create a app password. That password is meant to be used by an application only. You can create many of them, one for each application that use your email.
Then, you can use it in your script as you were using your user password before.
The idea, I surmise (since it can be weird that Google deems insecure a protocol with one password but not with another, even tho you can basically do anything with it) is that this password is supposed to be specific to an app. So your could easily revoke it if needed. Plus, it is generated, you can't choose it. And, well, you can basically do anything with it... except google specific operations. For example, a hacker who gets your app password could not change your google password, or close your account with it.
But, well, still, it is surprisingly simple (I would have expected at least that, when creating the app password, you could fine-restrict what can be done with that password)
Related
I am trying to use imaplib in order to fetch my mails
import imaplib
mail= imaplib.IMAP4_SSL("imap.gmail.com")
mail.login("mymailaddress","mypassword")
mail.select("inbox")
but unfortunately, I am getting the following error
File , line 3, in <module>
mail.login("mymailaddress","mypassword")
raise self.error(dat[-1])
imaplib.error: b'[ALERT] Application-specific password required: https://support.google.com/accounts/answer/185833 (Failure)'
I have checked all other suggestion given on similar question but none are working (similar problem). I even enabled IMAP on my account. I also allow less secure app. What might be the reason for this error?
As stated in the error message you are getting You need to go to Sign in with App Passwords
The issue you are having is that you are using the users password. You should be using an apps password
Side note: Update as of May 30, 2022
To help keep your account secure, starting May 30, 2022, Google will no longer support the use of third-party apps or devices which ask you to sign in to your Google Account using only your username and password.
This means anyone connecting to the smtp / imap server and using the users password will get an error. Less secure apps will no work anymore.
You need to switch to using xoauth2 or trying an apps password. At this time it is not clear if apps password will work or not.
At this time it is not clear if apps password will work or not.
I was able to set up apps password today following an article in the google help center
I am new to Python and have recently tried out two approaches to automating the sending out of an email on Outlook 365, one with greater success than the other. I'd like to ask what the key differences are since they look quite vastly different.
The first method is that essentially outlined in the Automate the Boring Stuff book, using SMTP or IMAP. I tried this, but didn't get it to work perhaps because of authentication issues using an office computer.
The second method, which has worked for me, doesn't involve authentication, and I simply import the win32com client and the following code:
outlook = client.Dispatch('Outlook.Application')
message = outlook.CreateItem(0)
message.Display()
message.To = "redacted"
message.CC = "redacted"
message.Subject = "Hello"
I'd like to ask what are the main ways in which the two methods differ. It seems that the second might rely on Outlook being open and me being logged on, but would the first also work if my computer were put on sleep?
Why go through the first approach which involves authentication when I'm already logged on to Windows and have access to Outlook without needing to enter my user id and password?
I think this is a question that might be useful to others new to Python and email automation, as they may also encounter the two approaches in their search for solutions.
tldr; if you need to send mail from different mailboxes, use smtplib. If you are automating stuff that you can do manually using outlook, use win32com.client.
SMTP
Referencing the official python docs for SMTP, the methods described there simply allow you to send a mail. (yes, that's it. You cannot even look at the inbox. You'll need imaplib or poplib.)
To me the advantage of smtp is that you can send emails from another person's mailbox if you have his/her credentials. If you were to use win32com.client, you will need to sign out of your own outlook, sign in to that specific person's outlook, then run the code. And for me, I faced the issue where I had to wait for his/her inbox to finish loading before anything gets sent. This is not feasible if you only need to send mail (and not interested in any other functionality such as read or delete mail) from many mailboxes.
[Update] I've recently used smtplib and email (a python builtin package) in a personal project. As it is a personal project, I didn't want to use my office email, hence I decided to use smtplib instead. While there is a need to setup an initial connection, it is very straightforward. Since there is no way to save the email as a draft before sending, the logical workaround is to send it to your own email addresses (or any other 'safe' emails) to test if it works as intended.
import smtplib
from email.message import EmailMessage
msg = EmailMessage()
msg['From'] = 'YOUR_EMAIL#GMAIL.COM'
msg['Subject'] = 'Some subject here'
msg['To'] = ', '.join(['adam#gmail.com','bob#gmail.com','candice#gmail.com'])
msg.set_content('Some text here')
with smtplib.SMTP_SSL('smtp.gmail.com', 465) as smtp:
smtp.login('YOUR_EMAIL#GMAIL.COM', 'PASSWORD123')
smtp.send_message(msg)
print('Email sent!')
win32com.client (focusing on the Outlook application)
You should use this library if you want to automate what you can do on outlook mailboxes that you have access to. Syntax tends to be simpler and it allows you to do stuff that is not possible using only smtplib.
Here are 2 examples to illustrate my point.
Example 1: Automate sending of a calendar invite.
If you were to do it using SMTP, you'll require more code and another library email, specifically .MIMEMultipart, .MIMEBase, .MIMEText, .Utils. The syntax looks intimidating, to say the least. Just take a look at the ical variable of the following stackoverflow answer:
ical = "BEGIN:VCALENDAR"+CRLF+"PRODID:pyICSParser"+CRLF+"VERSION:2.0"+CRLF+"CALSCALE:GREGORIAN"+CRLF
ical+="METHOD:REQUEST"+CRLF+"BEGIN:VEVENT"+CRLF+"DTSTART:"+dtstart+CRLF+"DTEND:"+dtend+CRLF+"DTSTAMP:"+dtstamp+CRLF+organizer+CRLF
ical+= "UID:FIXMEUID"+dtstamp+CRLF
ical+= attendee+"CREATED:"+dtstamp+CRLF+description+"LAST-MODIFIED:"+dtstamp+CRLF+"LOCATION:"+CRLF+"SEQUENCE:0"+CRLF+"STATUS:CONFIRMED"+CRLF
ical+= "SUMMARY:test "+ddtstart.strftime("%Y%m%d # %H:%M")+CRLF+"TRANSP:OPAQUE"+CRLF+"END:VEVENT"+CRLF+"END:VCALENDAR"+CRLF
win32com.client is so much easier (phew~). There are code examples all over the internet (here and here), and here's a simple example:
ol = w32.Dispatch('Outlook.Application')
appt = ol.CreateItem(1)
appt.Start = '2021-02-06 15:00'
appt.Save()
Example 2: Saving an email as a draft
I often end up automating work for colleagues and when you are sending emails as a batch, it is highly recommended to test by saving the created mail items as a draft. win32com.client allows you to save the mail item as a draft (i.e. .Save()) but smtplib doesn't allow you to do that (reiterate, it only allows you to send a mail.)
[Disclaimer] I done some automation work on outlook and I've always used win32com.client and I have only recently started to use smtplib for a personal project. This question intrigued me and I decided that it is time to at least read a bit more about smtplib.
I am trying to connect to an email account with the imaplib python module but it doesn't work. With #outlook it works fine for me, but with #mydomain it doesn't work.
It's not a credential issue because I tried a POP connection and it worked, but not with imap.
I have searched in many places and it happens to many but I do not see a solution. Someone who can help me?
There's a few things to look at.
First, make sure IMAP access is enabled in the settings of whatever service you're using.
(Eg, Gmail: Settings > Forwarding and POP/IMAP > Enable IMAP). This is often not enough to fix the issue though, you need to set up an app password.
(Eg, Google: https://myaccount.google.com/security > Enable 2-step verification if it is off > App passwords).
An app password lets you sign into a service using a generated password in lieu of your main account password. Use this app password in your credentials while logging in with imaplib, and it should work. I'd suggest considering using a Microsoft app password too when using imaplib for Outlook, as it's more secure. You aren't storing your actual MS password in plain-text anywhere.
Hope that helps.
I've created a simple Python program using Spotipy that shows some recommended tracks based on the tracks downloaded in the user device. But I'm having some trouble on making the program user-friendly.
First of all, is there any problem by sharing my Client ID and my Client Secret with the user by, for example, uploading my code in GitHub? Can I use Redirect URI as being http://localhost/ or should I create a website for my program for securing purposes? In Username field, it should be the username of the account to be analyzed or it can be anything, like "Brian Rogers"?
In the authentication part, it shows the user in Python console the following message:
User authentication requires interaction with your
web browser. Once you enter your credentials and
give authorization, you will be redirected to
a url. Paste that url you were directed to to
complete the authorization.
Opening https://... in your browser
Enter the URL you were redirected to:
My question is: since I'm managing to use Tkinter, how can I redirect the input from the Tkinter input box to the Python console?
Finally, how long does the authentication token take to expire? And if so, how to renew it (if possible, so that only the user enters when they run the program for the first time)?
Thanks in advance for the patient!
I'll address all your questions one by one.
is there any problem by sharing my Client ID and my Client Secret with the user by, for example, uploading my code in GitHub?
One should always avoid putting personal credentials in the source. If someone misuses your credentials, you'll be the one who will be held responsible because they are YOUR credentials. In any case, the only havoc I can imagine one could cause is to spam requests to Spotify's API which I believe Spotify's API already has protections and will drop further requests if it detects request spam. I've had and seen some projects put their Spotify and YouTube API credentials by creating special accounts for generating API credentials for their projects, in their source code and push to GitHub to make the tool easier to setup for use.
Can I use Redirect URI as being http://localhost/ or should I create a website for my program for securing purposes? In Username field, it should be the username of the account to be analyzed or it can be anything, like "Brian Rogers"?
As you're only searching for relevant tracks on Spotify, I believe you probably don't need to access the personal information of the Spotify user whose credentials you are using. If so, you can avoid both passing the username and verifying the redirect URI, by using oauth2.SpotifyClientCredentials to authorize yourself:
import spotipy
import spotipy.oauth2 as oauth2
credentials = oauth2.SpotifyClientCredentials(
client_id=client_id,
client_secret=client_secret)
token = credentials.get_access_token()
# This won't prompt for verification of Redirect URI
sp = spotipy.Spotify(auth=token)
My question is: since I'm managing to use Tkinter, how can I redirect the input from the Tkinter input box to the Python console?
You won't need to, if you use oauth2.SpotifyClientCredentials as mentioned above.
Finally, how long does the authentication token take to expire? And if so, how to renew it (if possible, so that only the user enters when they run the program for the first time)?
As of writing this, the token stays valid for exactly one hour. You can confirm by checking the value of credentials.token_info["expires_in"] which displays the time in seconds.
Also, spotipy raises spotipy.client.SpotifyException when a dependent method has been called but the token has already expired. So, you could catch this exception and overwrite your previous spotipy.client.Spotify instance with a new one. At the minimal you would do something similar to this:
import spotipy
import spotipy.oauth2 as oauth2
def authenticate_calls():
credentials = oauth2.SpotifyClientCredentials(
client_id=client_id,
client_secret=client_secret,
)
token = credentials.get_access_token()
sp = spotipy.Spotify(auth=token)
return sp
sp = authenticate_calls()
try:
do_something_that_needs_authentication(sp)
except spotipy.client.SpotifyException:
sp = authenticate_calls()
do_something_that_needs_authentication(sp)
You could also create a decorator function which would refresh the token if expired and decorate your functions with it!
I have a postfix server listening and receiving all emails received at mywebsite.com Now I want to show these postfix emails in a customized interface and that too for each user
To be clear, all the users of mywebsite.com will be given mail addresses like someguy#mywebsite.com who receives email on my production machine but he sees them in his own console built into his dashboard at mywebsite.com.
So to make the user see the mail he received, I need to create an email replica of the postfix mail so that mywebsite(which runs on django-python) will be reflecting them readily. How do I achieve this. To be precise this is my question, how do I convert a postfix mail to a python mail object(so that my system/website)understands it?
Just to be clear I have written psuedo code to achieve what I want:
email_as_python_object = postfix_email_convertor(postfix_email)
attachments_list = email_as_python_object.attachments
body = email_as_python_object.body # be it html or whatever
And by the way I have tried default email module which comes with python but thats not handy for all the cases. And even I need to deal with mail attachments manually(which I hate). I just need a simple way to deal with cases like these(I was wondering how postfix understands a email received. ie.. how it automatically figures out different headers,attachments etc..). Please help me.
You want to have postfix deliver to a local mailbox, and then use a webmail system for people to access that stored mail.
Don't get hung up on postfix - it just a transfer agent - it takes messages from one place, and puts them somewhere else, it doesn't store messages.
So postfix will take the messages over SMTP, and put them in local mail files.
Then IMAP or some webmail system will display those messages to your users.
If you want the mail integrated in your webapp, then you should probably run an IMAP server, and use python IMAP libraries to get the messages.
First of all, Postfix mail routing rules can be very complex and your presumably preferred solution involves a lot of trickery in the wrong places. You do not want to accidentally show some user anothers mails, do you? Second, although Postfix can do almost anything, it shouldn't as it only is a MDA (mail delivery agent).
Your solution is best solved by using a POP3 or IMAP server (Cyrus IMAPd, Courier, etc). IMAP servers can have "superuser accounts" who can read mails of all users. Your web application can then connect to the users mailbox and retreive the headers and bodys.
If you only want to show the subject-line you can fetch those with a special IMAP command and very low overhead. The Python IMAP library has not the easiest to understand API though. I'll give it a shot (not checked!) with an example taken from the standard library:
import imaplib
sess = imaplib.IMAP4()
sess.login('superuser', 'password')
# Honor the mailbox syntax of your server!
sess.select('INBOX/Luke') # Or something similar.
typ, data = sess.search(None, 'ALL') # All Messages.
subjectlines = []
for num in data[0].split():
typ, msgdata = sess.fetch(num, '(RFC822.SIZE BODY[HEADER.FIELDS (SUBJECT)])')
subject = msgdata[0][1].lstrip('Subject: ').strip()
subjectlines.append(subject)
This logs into the IMAP server, selects the users mailbox, fetches all the message-ids then fetches (hopefully) only the subjectlines and appends the resulting data onto the subjectlines list.
To fetch other parts of the mail vary the line with sess.fetch. For the specific syntax of fetch have a look at RFC 2060 (Section 6.4.5).
Good luck!
I'm not sure that I understand the question.
If you want your remote web application to be able to view users' mailbox, you could install a pop or imap server and use a mail client (you should be able to find one off the shelf) to read the emails. Alternatively, you could write something to interrogate the pop/imap server using the relevant libraries that come with Python itself.
If you want to replicate the mail to another machine, you could use procmail and set up actions to do this. Postfix can be set up to invoke procmail in this wayy.