While running this program to retrieve Twitter data using Python 2.7.8 :
#imports
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
#setting up the keys
consumer_key = '…………...'
consumer_secret = '………...'
access_token = '…………...'
access_secret = '……………..'
class TweetListener(StreamListener):
# A listener handles tweets are the received from the stream.
#This is a basic listener that just prints received tweets to standard output
def on_data(self, data):
print (data)
return True
def on_error(self, status):
print (status)
#printing all the tweets to the standard output
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
stream = Stream(auth, TweetListener())
t = u"سوريا"
stream.filter(track=[t])
after running this program for 5 hours i got this Error message:
Traceback (most recent call last):
File "/Users/Mona/Desktop/twitter.py", line 32, in <module>
stream.filter(track=[t])
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 316, in filter
self._start(async)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 237, in _start
self._run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 173, in _run
self._read_loop(resp)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 225, in _read_loop
next_status_obj = resp.read( int(delimited_string) )
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 543, in read
return self._read_chunked(amt)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 612, in _read_chunked
value.append(self._safe_read(chunk_left))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 660, in _safe_read
raise IncompleteRead(''.join(s), amt)
IncompleteRead: IncompleteRead(0 bytes read, 976 more expected)
>>>
Actually i don't know what to do with this problem !!!
You should check to see if you're failing to process tweets quickly enough using the stall_warnings parameter.
stream.filter(track=[t], stall_warnings=True)
These messages are handled by Tweepy (check out implementation here) and will inform you if you're falling behind. Falling behind means that you're unable to process tweets as quickly as the Twitter API is sending them to you. From the Twitter docs:
Setting this parameter to the string true will cause periodic messages to be delivered if the client is in danger of being disconnected. These messages are only sent when the client is falling behind, and will occur at a maximum rate of about once every 5 minutes.
In theory, you should receive a disconnect message from the API in this situation. However, that is not always the case:
The streaming API will attempt to deliver a message indicating why a stream was closed. Note that if the disconnect was due to network issues or a client reading too slowly, it is possible that this message will not be received.
The IncompleteRead could also be due to a temporary network issue and may never happen again. If it happens reproducibly after about 5 hours though, falling behind is a pretty good bet.
I've just had this problem. The other answer is factually correct, in that it's almost certainly:
Your program isn't keeping up with the stream
you get a stall warning if that's the case.
In my case, I was reading the tweets into postgres for later analysis, across a fairly dense geographic area, as well as keywords (London, in fact, and about 100 keywords). It's quite possible that, even though you're just printing it, your local machine is doing a bunch of other things, and system processes get priority, so the tweets will back up until Twitter disconnects you. (This is typically manifests as an apparent memory leak - the program increases in size until it gets killed, or twitter disconnects - whichever is first.)
The thing that made sense here was to push off the processing to a queue. So, I used a redis and django-rq solution - it took about 3 hours to implement on dev and then my production server, including researching, installing, rejigging existing code, being stupid about my installation, testing, and misspelling things as I went.
Install redis on your machine
Start the redis server
Install Django-RQ (or just Install RQ if you're working solely in python)
Now, in your django directory (where appropriate - ymmv for straight python applications) run:
python manage.py rqworker &
You now have a queue! You can add jobs to that like by changing your handler like this:
(At top of file)
import django_rq
Then in your handler section:
def on_data(self, data):
django_rq.enqueue(print, data)
return True
As an aside - if you're interested in stuff emanating from Syria, rather than just mentioning Syria, then you could add to the filter like this:
stream.filter(track=[t], locations=[35.6626, 32.7930, 42.4302, 37.2182]
That's a very rough geobox centred on Syria, but which will pick up bits of Iraq/Turkey around the edges. Since this is an optional extra, it's worth pointing this out:
Bounding boxes do not act as filters for other filter parameters. For
example track=twitter&locations=-122.75,36.8,-121.75,37.8 would match
any tweets containing the term Twitter (even non-geo tweets) OR coming
from the San Francisco area.
From this answer, which helped me, and the twitter docs.
Edit: I see from your subsequent posts that you're still going down the road of using Twitter API, so hopefully you got this sorted anyway, but hopefully this will be useful for someone else! :)
This worked for me.
l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
while True:
try:
stream.filter(track=['python', 'java'], stall_warnings=True)
except (ProtocolError, AttributeError):
continue
A solution is restarting the stream immediately after catching exception.
# imports
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
# setting up the keys
consumer_key = "XXXXX"
consumer_secret = "XXXXX"
access_token = "XXXXXX"
access_secret = "XXXXX"
# printing all the tweets to the standard output
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
class TweetListener(StreamListener):
# A listener handles tweets are the received from the stream.
# This is a basic listener that just prints received tweets to standard output
def on_data(self, data):
print(data)
return True
def on_exception(self, exception):
print('exception', exception)
start_stream()
def on_error(self, status):
print(status)
def start_stream():
stream = Stream(auth, TweetListener())
t = u"سوريا"
stream.filter(track=[t])
start_stream()
For me the back end application to which the URL is pointing is directly returning the string
I changed it to
return Response(response=original_message, status=200, content_type='application/text')
in the start I just returned text like
return original_message
I think this answer works only for my case
Related
I recently started to learn python, i had 0 knowledge on pyhton, and in the last few weeks i've been studying python and the twitter api.
I decided to work on a simple twitter bot, that automatically posts whatever people send on my direct messages, and i maneged to do so.
Here's my code:
import tweepy
import json
import sys
import re
import time
print("acessing api...")
consumer_key = 'consumer_key'
consumer_secret = 'consumer_secret'
key = 'key'
secret = 'key_secret'
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(key, secret)
api = tweepy.API(auth, wait_on_rate_limit = True)
print('api accessed')
print('colecting dms')
direct_message = api.list_direct_messages()
text = direct_message[0].message_create["message_data"]["text"]
def send_tweet():
print('sending tweet...')
api.update_status('/bot: ' + text)
while True:
send_tweet()
time.sleep(30)
The code works, but not 100%, i'm able to extract the text from the Dm and use it on the api.update_status("/bot: " + text) but when it loops i get the following error:
tweepy.error.TweepError: [{'code': 187, 'message': 'Status is a duplicate.'}]
i've looked it up, and tried many things, for example:
while True:
try:
send_tweet()
time.sleep(30)
except tweepy.TweepError as error:
if error.api_code == 187:
print('duplicate message')
else:
print('waiting for new message')
time.sleep(30)
All i want is a way to keep the code going and ignore messages and tweets that were already sent, also wait for new dm's
Another thing that kept happening was that when i tried testing the extraction of the text, i kept getting the same text from the same dm, instead of getting a nother newer one.
Direct Messages stay in the Inbox even after your read or reply. It is best to delete a DM after you process it (in your case tweet it) so you don't need to worry about it next time you fetch the messages.
I have a python script that streams tweets into a csv file. I have provided the runtime parameter of 46800 seconds which is equal to 13 hrs, this dictates for how long the tweets have to be streamed into that csv. It was running fine for the specified duration until yesterday, but yesterday the script ran for 7.5 hrs only and it stopped streaming afterwards. I believe, there was no tweet about the topic i was streaming for a certain duration and hence the disconnect happened. So even if, when people started tweeting about the topic in question at later point, the connection didn't get re-establish and hence the script didn’t stream those tweets to csv file. So, i had to restart the script in another instance and let the script stream the tweets into another csv file. Today also, i ran into similar issue, the stream got disconnected after running for 6 hours and so i had to re-start again.
But i am not sure if that was the case. Below is the script that i used, please advise what could have happened. And if so, then how can i avoid this?
runtime = 46800
class listener(StreamListener):
def on_data(self,data):
data1 = json.loads(data)
time = data1["created_at"]
tweet1 = BeautifulSoup(tweet, "lxml").get_text()
url = "https://twitter.com/{}/status/{}".format(data1["user"]["screen_name"], data1["id_str"])
file = open('MARCH_DATA.csv', 'a')
csv_writer = csv.writer(file)
csv_writer.writerow([time, tweet1, url])
file.close()
auth = OAuthHandler(consumer_key,consumer_secret)
auth.set_access_token(access_token,access_token_secret)
twitterStream = Stream(auth, listener())
twitterStream.filter(track=["MTA"], async = True)
time.sleep(runtime)
twitterStream.disconnect()
Thanks
This worked for my streaming exercise.
# the regular imports, as well as this:
from urllib3.exceptions import ProtocolError
auth = OAuthHandler(consumer_key,consumer_secret)
auth.set_access_token(access_token,access_token_secret)
twitterStream = Stream(auth, listener())
while True:
try:
twitterStream.filter(track=["MTA"], async = True, stall_warnings=True)
except (ProtocolError, AttributeError):
continue
So I am relatively new to Python and don't know a ton about programming in general. I've been trying to just write a stupid meme twitter bot that me and my friends thought of as a joke. But I just wanted to see if I could get it running. And of course I can't. And I'm having trouble understanding how to handle a traceback error based on documentation and forum posts alone.
I am running python 3.7 in Pycharm on Windows, if any of this makes any difference. Will update with details as asked/needed.
Up until this point I've tried:
-Removing other Java interpreter versions.
-Uninstalling and reinstalling tweepy using pip.
-Maybe some other stuff can't quite remember but these being the main things I tried.
Code shown here:
import random
import tweepy
consumer_key = 'Insert Consumer Key'
consumer_secret = 'Insert Consumer Secret'
access_token = 'Access Token Goes Here'
access_token_secret = 'access token secret goes here'
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
chance = random.randint(1, 1001)
def powder(num=chance / 100):
if num == 0.01:
api.update_status("*Special 0.01% chance tweet* I have a better monitor so I should be better at tekken.")
print("*Special 0.01% chance tweet* I have a better monitor so I should be better at tekken.")
elif num <= 27.3:
api.update_status("Hmmm. That's a lot of powder.")
print("Hmmm. That's a lot of powder.")
else:
api.update_status("Yes.")
print("Yes.")
powder()
And the error thrown looks like this:
C:\Users\colem\AppData\Local\Programs\Python\Python37\python.exe
C:/Users/colem/PycharmProjects/Playground/powder.py
Traceback (most recent call last):
File "C:/Users/colem/PycharmProjects/Playground/powder.py", line 29, in <module>
powder()
File "C:/Users/colem/PycharmProjects/Playground/powder.py", line 22, in powder
api.update_status("Hmmm. That's a lot of powder.")
File "C:\Users\colem\AppData\Local\Programs\Python\Python37\lib\site-packages\tweepy\api.py", line 194, in update_status
)(post_data=post_data, *args, **kwargs)
File "C:\Users\colem\AppData\Local\Programs\Python\Python37\lib\site-packages\tweepy\binder.py", line 250, in _call
return method.execute()
File "C:\Users\colem\AppData\Local\Programs\Python\Python37\lib\site-packages\tweepy\binder.py", line 234, in execute
raise TweepError(error_msg, resp, api_code=api_error_code)
tweepy.error.TweepError: [{'code': 32, 'message': 'Could not authenticate you.'}]
Process finished with exit code 1
Sorry if this formatting is terrible, my first Stack Overflow post.
I've tried to make a python script that can delete / undo favorite all my twitter favorites for me. I have seen MATHEW INKSON's post to do the job. I don't need to delete my tweets, just want to clear the favorites. Besides that script is almost two years old and incompatible with latest python. So I've edited a little to run it with python 3.6.0 and my script look like this:
import tweepy
from datetime import datetime, timedelta
test_mode = False
verbose = False
delete_favs = True
days_to_keep = 7
consumer_key = 'my consumer key'
consumer_secret = 'my consumer secret'
access_token = 'my access token'
access_token_secret = 'my access token secret'
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
cutoff_date = datetime.utcnow() - timedelta(days=days_to_keep)
if delete_favs:
print ("Retrieving favorited tweets")
favorites = tweepy.Cursor(api.favorites).items()
unfav_count = 0
kept_count = 0
for tweet in favorites:
if tweet.created_at < cutoff_date:
if verbose:
print ("Unfavoring %d: [%s] %s % (tweet.created_at,tweet.text)")
if not test_mode:
api.destroy_favorite(tweet.id)
unfav_count += 1
else:
kept_count += 1
print ("Unfavored %d tweets, ignored %d" % (unfav_count, kept_count))
But, everytime by running the script on my windows command I am getting this following error:
Traceback (most recent call last):
File "C:\Users\xyz\Desktop\New folder\Unfavorite.py", line 25, in <module>
for tweet in favorites:
File "C:\Users\xyz\AppData\Local\Programs\Python\Python36\lib\site-packages\tweepy-3.6.0-py3.6.egg\tweepy\cursor.py", line 49, in __next__
File "C:\Users\xyz\AppData\Local\Programs\Python\Python36\lib\site-packages\tweepy-3.6.0-py3.6.egg\tweepy\cursor.py", line 197, in next
File "C:\Users\xyz\AppData\Local\Programs\Python\Python36\lib\site-packages\tweepy-3.6.0-py3.6.egg\tweepy\cursor.py", line 108, in next
File "C:\Users\xyz\AppData\Local\Programs\Python\Python36\lib\site-packages\tweepy-3.6.0-py3.6.egg\tweepy\binder.py", line 250, in _call
File "C:\Users\xyz\AppData\Local\Programs\Python\Python36\lib\site-packages\tweepy-3.6.0-py3.6.egg\tweepy\binder.py", line 234, in execute
tweepy.error.TweepError: Twitter error response: status code = 429
I am using Python 3.6, my app's permissions are all correct. Everything is fine with my twitter app. I guess something is wrong with my script.
Please someone help fix my code. I've seen some other script too. Those didn't work out. Suggestions will be appreciated.
Thanks is advance.
As per Twitter response codes, Code 429 is Returned in when a request cannot be served due to the application’s rate limit having been exhausted for the resource. Which means your app has made too many requests and you have to look into API Rate limits.
Okay, so I'm making a bot framework in python, and I'm having issues with creating a twitter command. I'm not sure how much sense this will make to anyone else, I think I did my best to explain this, so let me know if I can provide anymore information or code.
So, my twitter command looks like this
#Command(name="tweets", aliases="tweet")
def tweets(chat, message, args, sender):
if len(args) == 0:
chat.SendMessage("Provide a user")
return
def get_tweets():
consumer_key = JakeBot.conf.get_value("twitter_api-key")
consumer_secret = JakeBot.conf.get_value("twitter_api-secret")
access_token_key = JakeBot.conf.get_value("twitter_access")
access_token_secret = JakeBot.conf.get_value("twitter_access-secret")
api = twitter.Api(consumer_key, consumer_secret, access_token_key, access_token_secret)
public_tweets = api.GetUserTimeline(api.GetUser(args[0]).id)
return public_tweets
print get_tweets()[0].text
Now this, is my standard command format, which works for any command.
I call my command by searching by command name and getting the function and calling it in a separate thread like so:
func = commands[command]
thread.start_new_thread(func, (message.Chat, message.Body, get_args(args), message.Sender))
This works for all my other commands like !echo
For reference, I am using the following lib for twitter: https://code.google.com/p/python-twitter/
Now, the problem is that the application 'hangs' on:
api = twitter.Api(consumer_key, consumer_secret, access_token_key, access_token_secret)
I load my command modules through a file reader from my commands package, which I don't think is my issue here. However, if I just throw this in a quick test module, it works perfectly fine, and prints my latest tweet. Instead, when I call it as stated earlier, it hangs on creating the instance of the twitter API, and I really don't know why, even after hours of debugging this.