We Keep Coding

GMail Oauth2 authentication for yagmail stops working after a few days

I've started using yagmail a while ago for sending mails via GMail using OAuth2.
The code that I use is straightforward:
def send_with_yagmail(self):
yag = yagmail.SMTP(self.our_email, oauth2_file="/<path_to>/credentials.json")
yag.send(self.to_email, self.message_subject, contents=self.message_body)
All was well for a few days, and then suddenly Gmail stopped sending mails with the following as part of the stack trace:
...
return get_oauth_string(user, oauth2_info)
File "/usr/local/lib/p│ython3.9/site-packages/yagmail/oauth2.py", line 96, in get_oauth_string
access_token, expires_in = refresh_authorization(**oauth2_info)
File "/usr/local/lib/python3.9/site-packages/yagmail/oauth2.py", line 91, in refresh_authorization
response = call_refresh_token(google_client_id, google_client_secret, google_refresh_token)
File "/usr/local/lib/python3.9/site-packages/yagmail/oauth2.py", line 71, in call_refresh_token
response = urlopen(request_url, encoded_params).read().decode('UTF-8')
File "/usr/local/lib/python3.9/urllib/request.py", line 214, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.9/urllib/request.py", line 523, in open
response = meth(req, response)
File "/usr/local/lib/python3.9/urllib/request.py", line 632, in http_response
response = self.parent.error(
File "/usr/local/lib/python3.9/urllib/request.py", line 561, in error
return self._call_chain(*args)
File "/usr/local/lib/python3.9/urllib/request.py", line 494, in _call_chain
result = func(*args)│Ju
File "/usr/local/lib/python3.9/urllib/request.py", line 641, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
I've solved it by regenerating the credentials.json locally on my machine and uploading to the server thinking that it's a one-off event. It worked for a few days and now it stopped working again. So I'm wondering: what should I do so I don't have to regenerate the credentials every few days? My credentials.json looks like below:
{
"email_address": "my.team#gmail.com",
"google_client_id": "74<blabla>0e1feov.apps.googleusercontent.com",
"google_client_secret": "GOCSP<blabla<kOz",
"google_refresh_token": "1//09<blabla>-2_dxZH8"
}
Isn't the point of the refresh token to be used by the library such that I don't have to regenerate this thing by hand?
Can anybody recommend something that can be done? Thanks!

I ran into this same problem, is your app in 'testing' mode with the Google API? Testing tokens have an expiration of 7 days, which is why you had to re-generate your token. I found this answer, but after I changed my status to "In Production" it now gives me an authentication error. Based on this page if you're using Gmail, a production app needs to be verified by Google, but my credential says it doesn't need to be verified and I don't see an option to request verification. Might need to create a new credential?

downloading file using python format becomes invalid

hey i am trying to download stock data from the nse website of india
so i am using python for this
the link is
import urllib
urllib.urlretrieve("https://www.nseindia.com/content/historical/DERIVATIVES/2016/JAN/fo01JAN2016bhav.csv.zip","fo01JAN2016bhav.csv.zip")
but when i try to open the file that is downloaded it says that the
compressed zipped file is invalid
when i try it normal download from the website by simply pasting the link the file that gets downloaded gets opened
link
https://www.nseindia.com/content/historical/DERIVATIVES/2016/JAN/fo01JAN2016bhav.csv.zip
so if i try using urllib 2
i get this
f=urllib2.urlopen('https://www.nseindia.com/content/historical/DERIVATIVES/2016/JAN/fo01JAN2016bhav.csv.zip')
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
f=urllib2.urlopen('https://www.nseindia.com/content/historical/DERIVATIVES/2016/JAN/fo01JAN2016bhav.csv.zip')
File "C:\Python27\lib\urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 410, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 448, in error
return self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 382, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
how do i fix this ?
it happens for this link only i have tried downloading images from imgur and the code works fine
why is the http 403 error coming when i can normaly access it through my browser?

This link provides an example of what you want to do: https://stackoverflow.com/a/22776/6595777
Found another question regarding downloading zip files. Try this:
url = "http://www.nseindia.com/content/historical/DERIVATIVES/2016/JAN/fo01JAN2016bhav.csv.zip"
download = urllib2.urlopen(url)
with open(os.path.basename(url), "wb") as f:
f.write(download.read())
I don't have commenting permissions yet so I'm posting as an answer.
I can't browse to your link via https, http works though. Have you tried changing your link in your script to http?
It is possible that your script is downloading the error page that I get when trying to use https (ERR_SSL_PROTOCOL_ERROR.) This means that what you download will have the file name you specify (ending in .zip,) but it is actually html. This means it will give you the error that the zip file is invalid

hey i do not know why this is happening in urllib and urllib2 libraries but when i used the requests library
r = requests.get(url)
with open("code3.zip", "wb") as code:
code.write(r.content)
it worked
this might be an indirect solution to my answer

Amazon Simple Product API error when looking up product

from amazon.api import AmazonAPI
AMAZON_ACCESS_KEY = "A******************A"
AMAZON_SECRET_KEY = "7***********************E"
AMAZON_ASSOC_TAG = "j*****-20"
amazon = AmazonAPI(AMAZON_ACCESS_KEY, AMAZON_SECRET_KEY, AMAZON_ASSOC_TAG, region='US')
print(amazon)
#product = amazon.lookup(ItemId='B002RL8FBQ')
When I run the code above it works fine and I get this output from the print function:
<amazon.api.AmazonAPI object at 0x7fb6e59f7b38>
So everything is working fine with my access key, secret key, and associate tag.
However, if I un-comment the last line #product = amazon.lookup(ItemId='B00EOE0WKQ') then I get this error traceback:
Traceback (most recent call last):
File "test.py", line 8, in <module>
product = amazon.lookup(ItemId='B00EOE0WKQ')
File "/home/darren/Python_projects/amazon_wp/myvenv/lib/python3.4/site-packages/amazon/api.py", line 173, in lookup
response = self.api.ItemLookup(ResponseGroup=ResponseGroup, **kwargs)
File "/home/darren/Python_projects/amazon_wp/myvenv/lib/python3.4/site-packages/bottlenose/api.py", line 251, in __call__
{'api_url': api_url, 'cache_url': cache_url})
File "/home/darren/Python_projects/amazon_wp/myvenv/lib/python3.4/site-packages/bottlenose/api.py", line 212, in _call_api
return urllib2.urlopen(api_request, timeout=self.Timeout)
File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 469, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 507, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
I have followed the instruction from the official github for this https://github.com/yoavaviram/python-amazon-simple-product-api and the code I am using you will see I have used from the "Usage" subheading on the github page, thus I am not sure what's going wrong.
For added info, I am using a virtual environment and to show that I have the correct packages installed here is my out put of pip freeze:
(myvenv) darren#my_comp:~/Python_projects/amazon_wp$ pip3 freeze
bottlenose==0.6.3
lxml==3.6.0
python-amazon-simple-product-api==2.1.0
python-dateutil==2.5.3
six==1.10.0
Also, I have tried several different asin numbers of valid products and I get the same error message.
I am using python 3.4 on ubuntu 14.04

I think, problem is with region. Please select valid value from here. Explanation can be that AWS can validate your credentials, but when it comes to "real" call it fails, since 'US' is not valid region...

You might need to authorize your account for API access. This step-by-step should walk you through it.
Edit:
I've installed all of the same versions and am using the same python code with my own keys and it works fine.
The only time I encountered that error was when I didn't specify the region (which you are clearly doing).
One thing I'd try is to add the following code into your script:
import logging
logging.basicConfig(level=logging.DEBUG)
which should display the following request url:
DEBUG:bottlenose.api:Amazon URL:
http://webservices.amazon.co.uk/onca/xml?AWSAccessKeyId=&AssociateTag=&ItemId=B00EOE0WKQ&Operation=ItemLookup&ResponseGroup=Large&S
ervice=AWSECommerceService&Timestamp=&Version=2013-08-01&Signature=
You can visit this in your browser and should see an XML document returned. If it fails it should hopefully give you a better error than what the pythonlib gives you.
For instance, if I visit https://associates-amazon.s3.amazonaws.com/scratchpad/index.html (never got this to work for me) but it provides a list of base URLs for the region.
I created my associate account on the .co.uk region, thus my requests will only be valid for http://webservices.amazon.co.uk, if I instead try to query http://webservices.amazon.com then I will see:
The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method. Consult
the service documentation for details.
If you've got an associate account on amazon.com, try it without specifying region as I believe that's default. Apart from the above, check that your VM has internet connectivity and if nothing else works try creating another access key and using that.

Error 400 with python-amazon-simple-product-api via pythonanywhere

I've been at this for the better part of a day but have been coming up with the same Error 400 for quite some time. Basically, the application's goal is to parse a book's ISBN from the Amazon referral url and use it as the reference key to pull images from Amazon's Product Advertising API. The webpage is written in Python 3.4 and Django 1.8. I spent quite a while researching on here and settled for using python-amazon-simple-product-api since it would make parsing results from Amazon a little easier.
Answers like: How to use Python Amazon Simple Product API to get price of a product
Make it seem pretty simple, but I haven't quite gotten it to lookup a product successfully yet. Here's a console printout of what my method usually does, with the correct ISBN already filled:
>>> from amazon.api import AmazonAPI
>>> access_key='amazon-access-key'
>>> secret ='amazon-secret-key'
>>> assoc ='amazon-associate-account-name'
>>> amazon = AmazonAPI(access_key, secret, assoc)
>>> product = amazon.lookup(ItemId='1632360705')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/tsuko/.virtualenvs/django17/lib/python3.4/site-packages/amazon/api.py", line 161, in lo
okup
response = self.api.ItemLookup(ResponseGroup=ResponseGroup, **kwargs)
File "/home/tsuko/.virtualenvs/django17/lib/python3.4/site-packages/bottlenose/api.py", line 242, i
n __call__
{'api_url': api_url, 'cache_url': cache_url})
File "/home/tsuko/.virtualenvs/django17/lib/python3.4/site-packages/bottlenose/api.py", line 203, i
n _call_api
return urllib2.urlopen(api_request, timeout=self.Timeout)
File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 461, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 571, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 499, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 579, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
Now I guess I'm curious if this is some quirk with PythonAnywhere, or if I've missed a configuration setting in Django? As far as I can tell through AWS and the Amazon Associates page my keys are correct. I'm not too worried about parsing at this point, just getting the object. I've even tried bypassing the API and just using Bottlenose (which the API extends) but I get the same error 400 result.
I'm really new to Django and Amazon's API, any assistance would be appreciated!

You still haven't authorised your account for API access. To do so, you can go to https://affiliate-program.amazon.com/gp/advertising/api/registration/pipeline.html

What is causing urllib2.urlopen() to connect via proxy?

I'm trying to read a URL within our corporate network. Spesifically the server I'm contacting is in one office and the client PC is in another:
print(urlopen(r"http://london.mycompany/mydir/").read())
Whenever I run this function I get:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Python24\lib\urllib2.py", line 130, in urlopen
return _opener.open(url, data)
File "C:\Python24\lib\urllib2.py", line 364, in open
response = meth(req, response)
File "C:\Python24\lib\urllib2.py", line 471, in http_response
response = self.parent.error(
File "C:\Python24\lib\urllib2.py", line 402, in error
return self._call_chain(*args)
File "C:\Python24\lib\urllib2.py", line 337, in _call_chain
result = func(*args)
File "C:\Python24\lib\urllib2.py", line 480, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 407: Proxy Authentication Required
The odd thing is that there's no firewall between these two computers - for some reason url has decided to connect to the web-server via the proxy which we'd normally use to connect to content outside the company, and in this case that's failing because I've not authenticated it.
I'm pretty sure that the fault occurs within the client PC: I did a nslookup and a ping to the server to confirm that there's a connection between the two computers, however when I watch the transaction using TCPView for Windows I can see that the python.exe process is connecting to a completely different server (yes, the proxy!).
So what could be causing this? Note that the os.environ["http_proxy"] variable is NOT set - this variable is often used to make urllib connect via a proxy server. That's not the case here. Could there be something else which might have the same effect?
FYI, Running Python 2.4.4 on Windows XP 32bit in a very locked-down corporate environment.

It reads from the system settings. Use urllib.FancyURLOpener:
opener = urllib.FancyURLopener({})
f = opener.open("http://london.mycompany/mydir/")
f.read()