I want to download a dataset (SemEval 2016/2017) as described in
http://alt.qcri.org/semeval2017/task4/index.php?id=data-and-tools
which in turn reminds to
https://github.com/seirasto/twitter_download
the last link provides instructions on how to run the scripts to download the dataset.
I use Ubuntu, and from the command prompt I run
twitter
as described in the readme file. It opens the web browser from which I log in with my twitter username and password. I obtain a PIN code and I enter it into the prompt generated by the script.
Then, I try to download the dataset with the command
python3 download_tweets_api.py --dist=tweeti-a.dist.tsv --output=downloaded.tsv
but I get
fp#fp-X58LE:~/twitter_download-master$ python3 download_tweets_api.py --dist=tweeti-a.dist.tsv --output=downloaded.tsv
Hi there! We're gonna get you all set up to use Semeval sentiment analysis.
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/twitter/api.py", line 341, in _handle_response
handle = urllib_request.urlopen(req, **kwargs)
File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 469, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 507, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Authorization Required
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "download_tweets_api.py", line 21, in <module>
oauth_dance("Semeval sentiment analysis", CONSUMER_KEY, CONSUMER_SECRET, MY_TWITTER_CREDS)
File "/usr/local/lib/python3.4/dist-packages/twitter/oauth_dance.py", line 93, in oauth_dance
twitter.oauth.request_token(oauth_callback="oob"))
File "/usr/local/lib/python3.4/dist-packages/twitter/api.py", line 334, in __call__
return self._handle_response(req, uri, arg_data, _timeout)
File "/usr/local/lib/python3.4/dist-packages/twitter/api.py", line 367, in _handle_response
raise TwitterHTTPError(e, uri, self.format, arg_data)
twitter.api.TwitterHTTPError: Twitter sent status 401 for URL: oauth/request_token using parameters: (oauth_callback=oob&oauth_consumer_key=JEdRRoDsfwzCtupkir4ivQ&oauth_nonce=6666274229238613502&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1592954865&oauth_version=1.0&oauth_signature=T8xiOX4HMOLQcertNeMC1vlnoHM%3D)
details: {"errors":[{"code":32,"message":"Could not authenticate you."}]}
I disinstalled and reinstalled python and twitter, I can't figure out how to proceed at the moment.
Before running the script, following groups.google.com/forum/#!topic/semevaltweet/4e45k1QkHk4 I first copied the training data ids file twitter-2016train-A.txt (downloaded from the data & tools page alt.qcri.org/semeval2017/task4/index.php?id=data-and-tools) in the same directory of the scripts and I renamed it as tweeti-a.dist.tsv.
Related
I get an error when i try to download a video from youtube with pytube.
from pytube import YouTube
yt = YouTube('https://www.youtube.com/watch?v=9bZkp7q19f0')
stream = yt.streams.first()
stream
stream.download()
The error says "Forbidden", but fetching e.g. with curl from the same URL seems to work just fine. What could be wrong here?
Traceback (most recent call last):
File "C:/Users/Mr. jarvis/Desktop/youtube/youtube.py", line 7, in <module>
stream.download()
File "C:\Users\Mr. jarvis\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pytube\streams.py", line 217, in download
bytes_remaining = self.filesize
File "C:\Users\Mr. jarvis\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pytube\streams.py", line 164, in filesize
headers = request.get(self.url, headers=True)
File "C:\Users\Mr. jarvis\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pytube\request.py", line 21, in get
response = urlopen(url)
File "C:\Users\Mr. jarvis\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\Mr. jarvis\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Users\Mr. jarvis\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\Mr. jarvis\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Users\Mr. jarvis\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\Mr. jarvis\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
Use pytube3 instead, it fixes this error (but is Python 3 only): https://github.com/hbmartin/pytube3
Experienced the same problem. I found unfinished downloaded file in the project folder. After deleting the unfinished downloaded file and running stream.download() again, it worked. So maybe worth checking.
pytube version 12.0.0
I'm setting my jenkins environment and, as part of this process, I need to create a slave node.
Bellow follow the script that is crashing:
import jenkins
server = jenkins.Jenkins('http://localhost:9090')
server.create_node('slave')
Follows the output:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line 1156, in create_node
self._build_url(CREATE_NODE, params), b''))
File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line 341, in jenkins_open
response = urlopen(req, timeout=self.timeout).read()
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 400: Bad Request
Use jenkinsapi module,
from jenkinsapi.jenkins import Jenkins
server = Jenkins('http://localhost:9090')
server.create_node('slave')
It appears to be a bug in python-jenkins 0.4.11, which was fixed by commit
302eb5dad6439842e076fead5bd76f20f615e6a8.
I got a similar error, upgrading python-jenkins fixed it.
We are trying to integrate nagios (4.1.1) notifications with Hipchat using Hipsaint. We followed the instructions posted here (keep in mind that the instructions mention nagios 3). We filled in the Token and Room_ID. We get the following output:
:~# hipsaint --token=XXXXXXXXXX --room=XXXXXXX --type=host --inputs="$HOSTNAME$|$LONGDATETIME$|$NOTIFICATIONTYPE$|$HOSTADDRESS$|$HOSTSTATE$|$HOSTOUTPUT$" -n
Traceback (most recent call last):
File "/usr/local/bin/hipsaint", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/hipsaint/bin/commands.py", line 78, in main
msg.deliver_payload()
File "/usr/local/lib/python2.7/dist-packages/hipsaint/messages.py", line 68, in deliver_payload_v1
raw_response = urlopen(self.urlv1, message_params)
File "/usr/lib/python2.7/urllib2.py", line 133, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 416, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 529, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 454, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 388, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 537, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized
We also tried the following:
Using a new Token
Updating Python from ver 2.7 to 3
Opening the ports for hipchat on the nagios Security Group
Any idea how we can proceed?
Test the validity of your auth token and make sure the test is not responding with "401 Unauthorized" status.
If you're using your own private HipChat Server, replace api.hipchat.com with your HipChat Server hostname.
Replace YOUR_TOKEN with the auth token you generated. Execute the following from your Nagios server:
curl https://api.hipchat.com/v1/rooms/list?auth_token=YOUR_TOKEN&auth_test=true
Source: https://www.hipchat.com/docs/api/auth
This is more of a linux/security related question (than Python's urllib2 authentication question)
My setup is:
I am running a Ubuntu Server in my company's corporate network
I notice that when I try to access the internet via a browser (chrome or firefox), I intermittently get redirected to company's security page asking me for my company's credentials.
Ubuntu server's firewall is disabled.
I am not sure why, but when I try running the following script (to fetch data from internet, even google.com), this script intermittently fails due to 401 Authorization Required Error. Once this happens, then I have to open a browser (obviously access via VNC) and then go to any page and I have to enter my credentials. And once I do, then the script runs just fine for a while. And after a while, it fails due 401 error again.
Script
import urllib2
url = 'http://nominatim.openstreetmap.org/search.php?countrycodes=us&state=colorado&street=6900+W+25th+Ave&format=json&addressdetails=1&polygon_geojson=1'
request = urllib2.Request(url)
response = urllib2.urlopen(request).read()
print response
Traceback (most recent call last):
File "/home/amit/workspace/clink/device_polling/mydb/dbmanager.py", line 1554, in _poll_device
self.update__device_geoloc(deviced_alldb, mydbc, hpnac, logobj)
File "/home/amit/workspace/clink/device_polling/mydb/dbmanager.py", line 1387, in update__device_geoloc
geoinfo = get_coordinates_geolocation(state=state, city=city, street=street, countrycodes=countrycodes)
File "/home/amit/workspace/clink/device_polling/utils/utils.py", line 386, in get_coordinates_geolocation
response = urllib2.urlopen(req)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 442, in error
result = self._call_chain(*args)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 629, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(*args)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/miniconda/envs/py_env_clink/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 401: Authorization Required
I don't understand why this is happening? Does any security guy know what I should look here for? I am sure it has to do with my corporate firewall which periodically requests my ubuntu server for password
I try to learn python network programming from book "Foundation of Python Network Programming" Apres,
first we learn about 'virtualenv' and I install virtualenv , try to get 'googlemap package' and try using simple program call 'search1' :
#!/usr/bin/env python
# Foundations of Python Network Programming - Chapter 1 - search1.py
from googlemaps import GoogleMaps
address = '207 N. Defiance St, Archbold, OH'
print GoogleMaps().address_to_latlng(address)
after that I try running it,
the result from the book is
$ python search1.py
(41.5228242, -84.3063479)
the result I get is error liket this
Traceback (most recent call last):
File "search1.py", line 6, in
print GoogleMaps().address_to_latlng(address)
File "/home/administrator/gmapenv/local/lib/python2.7/site-packages/googlemaps.py", line 310, in address_to_latlng
return tuple(self.geocode(address)['Placemark'][0]['Point']['coordinates'][1::-1])
File "/home/administrator/gmapenv/local/lib/python2.7/site-packages/googlemaps.py", line 259, in geocode
url, response = fetch_json(self._GEOCODE_QUERY_URL, params=params)
File "/home/administrator/gmapenv/local/lib/python2.7/site-packages/googlemaps.py", line 50, in fetch_json
response = urllib2.urlopen(request)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 407, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 520, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 445, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 528, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden
What make this error , because it I run down step by step like in the book