I am trying to fetch a subscription list according to the Subscriptions: list documentation. I want to get all my subscribers so I am using mySubscribers=True in the parameter list in a loop after my first request.
while "nextPageToken" in my_dict:
next_page_token = my_dict["nextPageToken"]
my_dict = subscriptions_list_by_channel_id(client,
part='snippet,contentDetails',
mySubscribers=True,
maxResults=50,
pageToken=next_page_token
)
for item in my_dict["items"]:
file.write("{}\n".format(item["snippet"]["channelId"]))
The problem is at page 20 my loop breaks, i.e. I don't recieve a nextPageToken key in the response capping my data to 1000 total subscribers fetched. But I have more than 1000 subs. The documentation states that myRecentSubscribers has a limit at 1000 but that mySubscribers does not.
Can not really find much help with this anywhere. Any light on my situation?
I chose to list channels instead of listing subscriptions, passing the same argument mySubscribers. The documentations says it's deprecated and gives them back in a weird order with duplicates but it does not have a limit.
Related
Here is the code I am using from this link. I have updated the original code as I need the full .json object. But I am having a problem with pagination as I am not getting the full 3200 Tweets.
api = tweepy.API(auth, parser=tweepy.parsers.JSONParser(),wait_on_rate_limit=True)
jsonFile = open(path+filname+'.json', "a+",encoding='utf-8')
page=1
max_pages=3200
result_limit=2
last_tweet_id=False
while page <= max_pages:
if last_tweet_id:
tweet = api.user_timeline(screen_name=user,
count=result_limit,
max_id=last_tweet_id - 1,
tweet_mode = 'extended',
include_retweets=True
)
else:
tweet = api.user_timeline(screen_name=user,
count=result_limit,
tweet_mode = 'extended',
include_retweets=True)
json_str = json.dumps(tweet, ensure_ascii=False, indent=4)
as per author "result_limit and max_pages are multiplied together to get the number of tweets called."
Then shouldn't I get 6400 Tweets by this definition. But the problem is I am getting 2 Tweets 3200 times. I also updated the values to
max_pages=3200
result_limit=5000
You can say it as a super limit so I should at least get 3200 Tweets. But in this case I got 200 Tweets repeated many times (as I terminated the code).
I just want 3200 Tweets per user profile, nothing fancy. Consider that I have 100 users list, so I want that in an efficient way. Currently seems like I am just sending so many requests and wasting time and assets.
Even though I update the code with a smaller value of max_pages, I am still not sure what should be that value, How am I supposed to know that a one-page covers how many Tweets?
Note: "This answer is not useful" as it has an error at .item() so please don't mark it duplicate.
You don't change last_tweet_id after setting it to False, so only the code in the else block is executing. None of the parameters in that method call change while looping, so you're making the same request and receiving the same response back over and over again.
Also, neither page nor max_pages changes within your loop, so this will loop infinitely.
I would recommend looking into using tweepy.Cursor instead, as it handles pagination for you.
I would like to pick up a specific keyword from a discord message as quickly as possible in Python, I'm currently using a request but the issue is that it takes too much time to actualize and grab the new message (it takes 200-500ms to receive messages), I am sure there is a better way of doing it.
def retrieve_messages(channelid):
while True:
##DISCORD
headers = {'authorization':""}
r= requests.get('https://discord.com/api/v9/channels/xxxxxxxxxxxxxxx/messages',headers=headers)
jsonn = json.loads(r.text)
for value in jsonn:
s1=str(value['content'])
s2=(re.findall('code:(0x......................)', s1))
if s2 !=[]:
print(s2)
retrieve_messages('')
According to the reference, the default number of messages returned from the endpoint is 50 (Link: https://discord.com/developers/docs/resources/channel#get-channel-messages). Using the limit-parameter it should be possible to maybe only get 1 or 5, which should limit the amount of time it takes to retrieve the messages and the time it takes to loop through them.
I have been familiarising myself with PRAW for reddit. I am trying to get the top x posts for the week, however I am having trouble changing the limit for the "top" method.
The documentation doesn't seem to mention how to do it, unless I am missing something. I can change the time peroid ok by just passing in the string "week", but the limit has be flummoxed. The image shows that there is a param for limit and it is set to 100.
r = self.getReddit()
sub = r.subreddit('CryptoCurrency')
results = sub.top("week")
for r in results:
print(r.title)
DOCS: subreddit.top()
IMAGE: Inspect listing generator params
From the docs you've linked:
Additional keyword arguments are passed in the initialization of
ListingGenerator.
So we follow that link and see the limit parameter for ListingGenerator:
limit – The number of content entries to fetch. If limit is None, then
fetch as many entries as possible. Most of reddit’s listings contain a
maximum of 1000 items, and are returned 100 at a time. This class will
automatically issue all necessary requests (default: 100).
So using the following should do it for you:
results = sub.top("week", limit=500)
I am using ZenPy to search for a few tickets in ZenDesk:
open_tickets = zenpy_client.search(type='ticket', status=['new', 'open'], subject=subject, group=group_name, created_between=[two_weeks_ago_date, current_date])
The problem is when I have too many results from this Search call (more than 1.000 since it's the new query limit for ZenDesk API). I get the following exception:
<Invalid search: Requested response size was greater than Search Response Limits>
I'm trying to look into ZenPy documentation but couldn't find any parameter that I could use to limit the search call for 10 pages (in this case, 1.000 records since we get 100 tickets per request).
I ended up putting a try-catch in the call but I'm sure that's not the best solution:
from zenpy.lib.exception import APIException
try:
open_tickets = zenpy_client.search(type='ticket', status=['new', 'open'], subject=subject, group=group_name, created_between=[two_weeks_ago_date, current_date])
except APIException as ex:
...
What is the best solution to limit this search?
I also know that I can limit even more the dates but we are creating a lot of tickets in one specific day of the week so there's no way to filter more, I just need to go until the limit.
Reference:
https://developer.zendesk.com/rest_api/docs/support/search
https://develop.zendesk.com/hc/en-us/articles/360022563994--BREAKING-New-Search-API-Result-Limits
Thanks!
The generator returned by search supports Python slices, so the following code will pull results up to 1000 and prevent exceeding the new limit:
ticket_generator = zenpy_client.search(type="ticket")
print(ticket_generator[:1000])
I am working on a project using Tweepy where I need to first grab all followers for a particular Twitter user, and then do the same for all of those followers' followers. I would like to store the latter part in a dictionary where the keys are the first set of followers, and the values are a list of their followers.
Here is my code:
followers_dict = {}
for h in myHandleList:
try:
c = tweepy.Cursor(api.followers_ids, id = h)
for page in c.pages():
followers_dict[h] = page
except tweepy.TweepError:
pass
This code works well for users with under 5000 followers. However, for users with more than 5000 followers, when I run the same code, the code splits their followers into separate lists of no more than 5000 values, and then only adds the second list as values in the dictionary.
For example, one user has 5,400 followers, so when I download their followers, it is formatted as two lists of 5000 and 400. When I use my loop to add their followers to a dictionary, it only adds the second list of 400. I would like to add all 5,400 as values in the dictionary.
I am a noob when it comes to Python, and as someone pointed out in the comments, this is surely an issue with my code - Any suggestions for how to fix this?
Thanks in advance!