We Keep Coding

Dictionary key error when parsing tweets with twitter api

from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream
import pw
import json
access_token = pw.access_token
access_token_secret = pw.access_token_secret
consumer_key = pw.consumer_key
consumer_secret = pw.consumer_secret
class StdOutListener(StreamListener):
def on_data(self, data):
dicto = json.loads(data)
print(dicto['user'])
return True
def on_error(self, status):
print(status)
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, StdOutListener())
stream.filter(track="music")
So I just followed a simple tutorial to process tweets. I'm just trying to parse the tweets, however when I use print(dicto['user']) it prints some and then throws a key error. The weird thing is that it's always after the 48th one. If I simply print the whole dictionary then it happily prints away. Right after the 48th piece of information is says traceback (most recent call last).
I'm a little confused what's going on here.

tweepy bad authentication data

I am trying to access twitter api via tweepy. And I get tweepy.error.TweepError: [{'code': 215, 'message': 'Bad Authentication data.'}] error.
My API access is decribed in twitter_client.py:
import os
import sys
from tweepy import API
from tweepy import OAuthHandler
def get_twitter_auth():
"""Setup twitter authentication
Return: tweepy.OAuthHandler object
"""
try:
consumer_key = os.environ['TWITTER_CONSUMER_KEY']
consumer_secret = os.environ['TWITTER_CONSUMER_SECRET']
access_token = os.environ['TWITTER_ACCESS_TOKEN']
access_secret = os.environ['TWITTER_ACCESS_SECRET']
except KeyError:
sys.stderr.write("TWITTER_* environment variable not set\n")
sys.exit(1)
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
return auth
def get_twitter_client():
"""Setup twitter api client
Return: tweepy.API object
"""
auth = get_twitter_auth()
client = API(auth)
return client
Then I try to get my last 4 tweets:
from tweepy import Cursor
from twitter_client import get_twitter_client
if __name__ == '__main__':
client = get_twitter_client()
for status in Cursor(client.home_timeline()).items(4):
print(status.text)
And get that error. How do I fix it?
I am using python 3.6 and I've installed tweepy via pip whithout specifying a version, so it should be the last version of tweepy.
Upd: I found out that the problem is in environ variables. Somehow twitter api can't get it. However, when I just print(consumer_key, consumer_secret, access_token, access_secret), everything is on it's place

import tweepy
Importing this way improves code readability especially when using it.
eg tweepy.API()
client.home_timeline()
The brackets after home_timeline shouldn't be there.
should be
for status in Cursor(client.home_timeline).items(4):
print(status.text)
.
import tweepy
auth = tweepy.OAuthHandler(<consumer_key>, <consumer_secret>)
auth.set_access_token(<access_token>, <access_secret>)
client = tweepy.API(auth)
for status in tweepy.Cursor(client.user_timeline).items(200):
process_status(status.text)

You may solve this problem by importing load dotenv, which will get .env variables.
import os
import sys
from tweepy import API
from tweepy import OAuthHandler
from dotenv import load_dotenv
def get_twitter_auth():
"""Setup twitter authentication
Return: tweepy.OAuthHandler object
"""
try:
consumer_key = os.getenv('CONSUMER_KEY')
consumer_secret = os.getenv('CONSUMER_SECRET')
access_token = os.getenv('ACCESS_TOKEN')
access_secret = os.getenv('TWITTER_ACCESS_SECRET')
except KeyError:
sys.stderr.write("TWITTER_* environment variable not set\n")
sys.exit(1)
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
return auth
def get_twitter_client():
"""Setup twitter api client
Return: tweepy.API object
"""
auth = get_twitter_auth()
client = API(auth)
return client
from tweepy import Cursor
if __name__ == '__main__':
client = get_twitter_client()
for status in Cursor(client.home_timeline()).items(4):
print(status.text)

Tweepy Attribute Error

I am new to python and tweepy and get an error message I cannot really understand:
import json
from tweepy import Cursor
from twitter_client import get_twitter_client
if __name__ == '__main__':
client = get_twitter_client()
with open('home_timeline.jsonl', 'w') as f:
for page in Cursor(client.home_timeline, count=200).pages(4):
for status in page:
f.write(json.dumps(status._json)+"\n")
Running this code gives the following error-messages:
Traceback (most recent call last):
File "twitter_get_user_timeline.py", line 10, in <module>
for page in Cursor(client.home_timeline, count=200).pages(4):
File "/home/projects/webscraping/testEnv/lib/python3.5/site-packages/tweepy/cursor.py", line 49, in __next__
return self.next()
File "/home/projects/webscraping/testEnv/lib/python3.5/site-packages/tweepy/cursor.py", line 108, in next
data = self.method(max_id=self.max_id, parser=RawParser(), *self.args, **self.kargs)
File "/home/projects/webscraping/testEnv/lib/python3.5/site-packages/tweepy/binder.py", line 239, in _call
return method.execute()
File "/home/projects/webscraping/testEnv/lib/python3.5/site-packages/tweepy/binder.py", line 174, in execute
auth = self.api.auth.apply_auth()
AttributeError: 'function' object has no attribute 'apply_auth'
Since this goes really deep into Tweepy, I cannot really understand where my mistake in the code lies (and the code is from the book: Marco Bonzanini: "Mastering Social Media Mining with Python"). Does anybody have an idea what's going wrong here?
The authentification is done in the twitter_client that is imported. The code there is:
import os
import sys
from tweepy import API
from tweepy import OAuthHandler
def get_twitter_auth():
"""Setup Twitter authentication.
Return: tweepy.OAuthHandler object
"""
try:
consumer_key = os.environ['TWITTER_CONSUMER_KEY']
consumer_secret = os.environ['TWITTER_CONSUMER_SECRET']
access_token = os.environ['TWITTER_ACCESS_TOKEN']
access_secret = os.environ['TWITTER_ACESS_SECRET']
except KeyError:
sys.stderr.write("TWITER_* environment variables not set\n")
sys.exit(1)
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
return auth
def get_twitter_client():
"""Setup Twitter API client
Return: tweepy.API object
"""
auth = get_twitter_auth
client = API(auth)
return client
Thanks a lot for any advice!

In second file, function get_twitter_client, line auth = get_twitter_auth.
You are saving the get_twitter_authfunction in auth and not the returned value.
Fix it with auth = get_twitter_auth()

How to download twitter feed

I am quite the novice when it comes to coding. How do I modify this sample code to download tweets using Python?
def get_tweets(api, input_query):
for tweet in tweepy.Cursor(api.search, q=input_query, lang="en").items():
yield tweet
if __name__ == "__version__":
input_query = sys.argv[1]
access_token = "REPLACE_YOUR_KEY_HERE"
access_token_secret = "REPLACE_YOUR_KEY_HERE"
consumer_key = "REPLACE_YOUR_KEY_HERE"
consumer_secret = "REPLACE_YOUR_KEY_HERE"
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
tweets = get_tweets(api, input_query)
for tweet in tweets:
print(tweet.text)
I entered my keys.
I should be able to download tweets using a command like this print_tweets.py "Enter subject here"
Where do I enter this command? In Python? In the command prompt?
I am getting this error:
NameError Traceback (most recent call
last) in ()
----> 1 print_tweets.py
NameError: name 'print_tweets' is not defined
Please help!

The NameError is saying the python script isn't receiving command line arguments, sys.argv[1] being the "subject". Replace "Enter subject here" with the subject you wish to search.
In this example, springbreak is sys.argv[1]:
C:\> python print_tweets.py springbreak
should return and print out tweet texts containing your "subject".
You may also need to change:
if __name__ == "__version__":
to
if __name__ == "__main__":
as __main__ is the entry-point to the script.
The entire script:
#!/usr/bin/env python
import sys
import tweepy
def get_tweets(api, input_query):
for tweet in tweepy.Cursor(api.search, q=input_query, lang="en").items():
yield tweet
if __name__ == "__main__":
input_query = sys.argv[1]
access_token = "REPLACE_YOUR_KEY_HERE"
access_token_secret = "REPLACE_YOUR_KEY_HERE"
consumer_key = "REPLACE_YOUR_KEY_HERE"
consumer_secret = "REPLACE_YOUR_KEY_HERE"
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
tweets = get_tweets(api, input_query)
for tweet in tweets:
print(tweet.text)

Query Twitter Status by Using Python and Tweepy

I try to query a specified user's tweets with a specified key word included in the tweet text. Here is my code:
# Import Tweepy, sleep, credentials.py
import tweepy
from time import sleep
from credentials import *
# Access and authorize our Twitter credentials from credentials.py
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
SCREEN_NAME = "BachelorABC"
KEYWORD = "TheBachelor"
def twtr2():
raw_tweets = tweepy.Cursor(api.search, q=KEYWORD, lang="en").items(50)
for tweet in raw_tweets:
if tweet['user']['screen_name'] == SCREEN_NAME:
print tweet
twtr2()
I get the error message as below:
Traceback (most recent call last):
File "test2.py", line 19, in <module>
twtr2()
File "test2.py", line 17, in twtr2
if tweet['user']['screen_name'] == SCREEN_NAME:
TypeError: 'Status' object has no attribute '__getitem__'
I googled a lot and thought that maybe I needed to save Twitter's JSON in python first, so I tried the following:
import tweepy, json
from time import sleep
from credentials import *
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
SCREEN_NAME = "BachelorABC"
KEYWORD = "TheBachelor"
raw_tweets = tweepy.Cursor(api.search, q=KEYWORD, lang="en").items(50)
for tweet in raw_tweets:
load_tweet = json.loads(tweet)
if load_tweet['user']['screen_name'] == SCREEN_NAME:
print tweet
However, the result is sad:
Traceback (most recent call last):
File "test2.py", line 35, in <module>
load_tweet = json.loads(tweet)
File "C:\Python27\lib\json\__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
Does anyone know what's wrong with my code? And can you help me to fix it?
Thanks in advance!

I figured out. Here is the solution:
# Import Tweepy, sleep, credentials.py
import tweepy
from time import sleep
from credentials import *
# Access and authorize our Twitter credentials from credentials.py
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
SCREEN_NAME = "BachelorABC"
KEYWORD = "TheBachelor"
for tweet in tweepy.Cursor(api.search, q=KEYWORD, lang="en").items(200):
if tweet.user.screen_name == SCREEN_NAME:
print tweet.text
print tweet.user.screen_name
Please do note that this is not an efficient way to locate the tweets with both specified conditions (screen_name and keyword) satisfied. This is because we query by keyword first, and then query by screen_name. If the keyword is very popular, like what I use here "TheBachelor", with a limited number of tweets (200), we may find none of the 200 tweets are sent by the specified screen_name. I think if we can query by screen_name first, and then by keyword, maybe it will provide a better result. But that's out of the discussion.
I will leave you here.

The issue is with the
load_tweet = json.loads(tweet)
The "tweet" object is not a JSON object. If you want to use JSON objects, follow this stackoverflow post on how to use JSON objects with tweepy.
To achieve what you are trying to do (print each tweet of a feed of 50), I would follow what was stated in the getting started docs:
import tweepy
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
public_tweets = api.home_timeline()
for tweet in public_tweets:
print(tweet.text)