This is more of a linux/security related question (than Python's urllib2 authentication question)
My setup is:
I am running a Ubuntu Server in my company's corporate network
I notice that when I try to access the internet via a browser (chrome or firefox), I intermittently get redirected to company's security page asking me for my company's credentials.
Ubuntu server's firewall is disabled.
I am not sure why, but when I try running the following script (to fetch data from internet, even google.com), this script intermittently fails due to 401 Authorization Required Error. Once this happens, then I have to open a browser (obviously access via VNC) and then go to any page and I have to enter my credentials. And once I do, then the script runs just fine for a while. And after a while, it fails due 401 error again.
Script
import urllib2
url = 'http://nominatim.openstreetmap.org/search.php?countrycodes=us&state=colorado&street=6900+W+25th+Ave&format=json&addressdetails=1&polygon_geojson=1'
request = urllib2.Request(url)
response = urllib2.urlopen(request).read()
print response
Traceback (most recent call last):
File "/home/amit/workspace/clink/device_polling/mydb/dbmanager.py", line 1554, in _poll_device
self.update__device_geoloc(deviced_alldb, mydbc, hpnac, logobj)
File "/home/amit/workspace/clink/device_polling/mydb/dbmanager.py", line 1387, in update__device_geoloc
geoinfo = get_coordinates_geolocation(state=state, city=city, street=street, countrycodes=countrycodes)
File "/home/amit/workspace/clink/device_polling/utils/utils.py", line 386, in get_coordinates_geolocation
response = urllib2.urlopen(req)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 442, in error
result = self._call_chain(*args)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 629, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(*args)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 401: Authorization Required
I don't understand why this is happening? Does any security guy know what I should look here for? I am sure it has to do with my corporate firewall which periodically requests my ubuntu server for password
Related
I'm trying to make some search on Amazon Product Ads and using the botlenose to help me do this. But, I'm just receive the HTTP Error 400.
Some other important infos:
I'm from Brazil, and my TAG from Amazon too. Is it a problem?
I did check my KEY, Secret and TAG and it's OK. I did look some other questions on StackOverflow, but nothing work for me.
import bottlenose
AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXXX"
AWS_SECRET_ACCESS_KEY="xxxxxxxxxxxxxxxxxxxxxxx"
AWS_ASSOCIATE_TAG="yyyyyyyyyyy"
amazon = bottlenose.Amazon(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ASSOCIATE_TAG, Region='BR')
response = amazon.ItemLookup(ItemId="B007OZNUCE")
enter code here# Of course, I changed the keys for security reasons.
Traceback (most recent call last):
File "", line 6, in
File "/Users/am/Documents/PycharmProjects/08/lib/python3.7/site-packages/bottlenose/api.py", line 274, in call
{'api_url': api_url, 'cache_url': cache_url})
File "/Users/am/Documents/PycharmProjects/08lib/python3.7/site-packages/bottlenose/api.py", line 235, in _call_api
return urllib2.urlopen(api_request, timeout=self.Timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open
response = meth(req, response)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
After that, I tried to generate the signed url by:
http://associates-amazon.s3.amazonaws.com/signed-requests/helper/index.html
and tried do put the both urls (by bottlenose and by Amazon link), and both return the same error> Later I tried by Browser the same URLs and I received the error:
<ItemLookupErrorResponse xmlns="http://ecs.amazonaws.com/doc/2013-08-01/">
<Error>
<Code>AWS.InvalidAccount</Code>
<Message>
Your AccessKey Id is not registered for Product Advertising API. Please use the AccessKey Id obtained after registering at https://affiliate-program.amazon.com/assoc_credentials/home.
</Message>
</Error>
<RequestID>4d761947-8b00-44d4-a9c0-0a9079e8d603</RequestID>
</ItemLookupErrorResponse>
Now, I'm contacting the Amazon Support to fix it.
Thanks for all!
We are trying to integrate nagios (4.1.1) notifications with Hipchat using Hipsaint. We followed the instructions posted here (keep in mind that the instructions mention nagios 3). We filled in the Token and Room_ID. We get the following output:
:~# hipsaint --token=XXXXXXXXXX --room=XXXXXXX --type=host --inputs="$HOSTNAME$|$LONGDATETIME$|$NOTIFICATIONTYPE$|$HOSTADDRESS$|$HOSTSTATE$|$HOSTOUTPUT$" -n
Traceback (most recent call last):
File "/usr/local/bin/hipsaint", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/hipsaint/bin/commands.py", line 78, in main
msg.deliver_payload()
File "/usr/local/lib/python2.7/dist-packages/hipsaint/messages.py", line 68, in deliver_payload_v1
raw_response = urlopen(self.urlv1, message_params)
File "/usr/lib/python2.7/urllib2.py", line 133, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 416, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 529, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 454, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 388, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 537, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized
We also tried the following:
Using a new Token
Updating Python from ver 2.7 to 3
Opening the ports for hipchat on the nagios Security Group
Any idea how we can proceed?
Test the validity of your auth token and make sure the test is not responding with "401 Unauthorized" status.
If you're using your own private HipChat Server, replace api.hipchat.com with your HipChat Server hostname.
Replace YOUR_TOKEN with the auth token you generated. Execute the following from your Nagios server:
curl https://api.hipchat.com/v1/rooms/list?auth_token=YOUR_TOKEN&auth_test=true
Source: https://www.hipchat.com/docs/api/auth
I am currently leveraging Sinch SMS to send SMS notifications based on events for a web application in python using sinchsms. When I send a message I get this exception:
HTTP Error 403: Sandobx SMS only allowed to be sent to verified numbers.
With this stacktrace:
File "/usr/local/lib/python2.7/site-packages/sinchsms.py", line 79, in send_message
return self._request(self.SEND_SMS_URL + to_number, values)
File "/usr/local/lib/python2.7/site-packages/sinchsms.py", line 40, in _request
connection = urllib2.urlopen(request)
File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/usr/local/lib/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python2.7/urllib2.py", line 475, in error
return self._call_chain(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Sandobx SMS only allowed to be sent to verified numbers.
I don't see a way on the sinch sms dashboard to validate the number or register a number
In sandbox mode you can send sms to one number only (an unfortunate side effect of some scammers/fraudsters). You can verify you number under https://www.sinch.com/dashboard/#/account. To send to any number, you need to make a payment and request a production key.
I'm having trouble using Google URL Shortener API in AppEngine production environment.
In the Developers console, I have the URL Shortener API turned on, and oAuth 2 is also turned on. On top of that I have the simple API Access Browser key obtained from the API Access screen.
Here is the problem. When I run the following code, I get "HTTPError: HTTP Error 403: Forbidden" in the Developers Console log. Interestingly, the same code properly returns the short url in the development environment.
def goo_shorten_url(url):
post_url = 'https://www.googleapis.com/urlshortener/v1/url?fields=id'
logging.info('post_url: {}'.format(post_url))
postdata = {'longUrl':url}
headers = {'Content-Type':'application/json'}
req = urllib2.Request(
post_url,
json.dumps(postdata),
headers
)
ret = urllib2.urlopen(req).read()
print ret
return json.loads(ret)['id']
If I include the API key in the post url as follows,
post_url = 'https://www.googleapis.com/urlshortener/v1/url?fields=id&key=MYAPIKEY'
Prod and Dev both return HTTP Error 403.
I suspect one of these three is true, but would like to hear your thoughts.
An API key is required, but I'm not using the right API key.
An API key is not required (which explains why it work with no key in Dev), but my API key is wrong resulting both Prod and Dev fail.
Google doesn't allow applications to programmatically submit a POST request to its Url shortener API.(this doesn't explain why it would work in Dev at all)
Thanks for reading.
Prod
File "/base/data/home/apps/s~myapp/1.377367579804576653/util/test_module.py", line 50, in get
strin = goo_shorten_url(longurl)
File "/base/data/home/apps/s~myapp/1.377367579804576653/util/JOTools.py", line 41, in goo_shorten_url
ret = urllib2.urlopen(req).read()
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(*args)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
Dev with API Key
File "C:_dev\eclipse-work\gae\MyProj\util\test_module.py", line 50, in get
strin = goo_shorten_url(longurl)
File "C:_dev\eclipse-work\gae\MyProj\util\JOTools.py", line 41, in goo_shorten_url
ret = urllib2.urlopen(req).read()
File "C:\PYTHON27\lib\urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "C:\PYTHON27\lib\urllib2.py", line 410, in open
response = meth(req, response)
File "C:\PYTHON27\lib\urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "C:\PYTHON27\lib\urllib2.py", line 448, in error
return self._call_chain(*args)
File "C:\PYTHON27\lib\urllib2.py", line 382, in _call_chain
result = func(*args)
File "C:\PYTHON27\lib\urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
Google has a nice API for this. You can test your requests here. Hope this helps.
I have been tasked with creating a script that logs on to a corporate portal goes to a particular page, downloads the page, compares it to an earlier version and then emails a certain person depending on changes that have been made. The last parts are easy enough but it has been the first step that is giving me the most trouble.
After unsuccessfully using urllib2(I am trying to do this in python) to connect and about 4 or 5 hours of googling I have determined that the reason I can't connect is due to NTLM authentication on the web page. I have tried a bunch of different processes for connecting found on this site and others to no avail. Based on the NTLM example I have done:
import urllib2
from ntlm import HTTPNtlmAuthHandler
user = 'username'
password = "password"
url = "https://portal.whatever.com/"
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, user, password)
# create the NTLM authentication handler
auth_NTLM = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(passman)
# create and install the opener
opener = urllib2.build_opener(auth_NTLM)
urllib2.install_opener(opener)
# create a header
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
header = { 'Connection' : 'Keep-alive', 'User-Agent' : user_agent}
response = urllib2.urlopen(urllib2.Request(url, None, header))
When I run this (with a real username, password and url) I get the following:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ntlm2.py", line 21, in <module>
response = urllib2.urlopen(urllib2.Request(url, None, header))
File "C:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 619, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 619, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 438, in error
return self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized
The thing that is most interesting about this trace to me is that the final line says a 401 error was sent back. From what I have read the 401 error is the first message sent back to the client when NTLM is started. I was under the impression that the purpose of python-ntml was to handle the NTLM process for me. Is that wrong or am I just using it incorrectly? Also I'm not bounded to using python for this, so if there is an easier way to do this in another language let me know (From what I seen a-googling there isn't).
Thanks!
If the site is using NTLM authentication, the headers attribute of the resulting HTTPError should say so:
>>> try:
... handle = urllib2.urlopen(req)
... except IOError, e:
... print e.headers
...
<other headers>
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM