We Keep Coding

My tweepy Twitter bot is not working, what am I doing wrong?

It returns several errors and most of them show: <framework not available>
I'm new at coding and I'm just trying making a bot for fun with the Twitter API but had a lot of errors and I don't know what to do.
The main error that appears is this one:
(from collections import namedtuple, Mapping ImportError: cannot
import name 'Mapping' from 'collections')
can someone help me please?
import tweepy
# Authenticate to Twitter
auth = tweepy.OAuthHandler("", "")
auth.set_access_token("", "")
api = tweepy.API(auth)
# Pegar ID do bot
bot_id = int(api.me().id_str)
class MyStreamListener(tweepy.StreamListener):
def on_status(self, tweet):
print("tweet encontrado!")
print(f"TWEET: {tweet.author.scream_name} - {tweet.text}")
if tweet.author.id != bot_id and tweet.in_reply_to_status_id is None:
if not tweet.retweeted:
try:
print("Tentando retuite...")
api.retweet(tweet.id)
print("Retuitado com sucesso!")
except Exception as err:
print(err)
stream_listener = MyStreamListener()
stream = tweepy.Stream(auth=api.auth, Listener=stream_listener)
stream.filter(track=['Squirrel', 'Squirrels', 'squirrel', 'squirrels', 'Esquilo', 'Esquilos'], languages=['en', 'pt'])
The erros that appeers:
Traceback (most recent call last):
File "C:\Users\wgama\PycharmProjects\botesquilo\botesquilo.py", line 1, in
import tweepy
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\tweepy_init_.py", line 12, in
from tweepy.api import API
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\tweepy\api.py", line 15, in
import requests
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\requests_init_.py", line 43, in
import urllib3
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3_init_.py", line 8, in
from .connectionpool import (
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 29, in
from .connection import (
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 39, in
from .util.ssl_ import (
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util_init_.py", line 3, in
from .connection import is_connection_dropped
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\connection.py", line 3, in
from .wait import wait_for_read
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\wait.py", line 1, in
from .selectors import (
File "C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\selectors.py", line 14, in
from collections import namedtuple, Mapping
ImportError: cannot import name 'Mapping' from 'collections' (C:\Users\wgama\AppData\Local\Programs\Python\Python310\lib\collections_init_.py)

As investigated in the comments, the problem comes from the fact that you are using an old version of Tweepy which is incompatible with Python 3.10.
So an update to the lastest version of Tweepy (currently 4.8.0) should solve your problem.
But please note that the API.me() method have been removed since the 4.0.0 version of Tweepy, so you will have to replaced it by the API.verify_credentials() method.
Edit : I just read your code more carefully and I could have found other mistakes: there is not author field in a tweet object but a user field, and I guess you meant screen_name instead of scream_name (!). Finally, after your update to the lastest version of Tweepy, you will also have to replace tweepy.StreamListener by tweepy.Stream.

Why does this python code give "TypeError: the JSON object must be str, not 'bytes'" on one machine but not another

I have a small public repo with the following python code:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import urllib.request, urllib.parse, urllib.error
import json
from urllib.request import urlopen, Request
import os
def get_json_from_url(url):
config = json.loads(open(os.path.dirname(os.path.abspath(__file__))+'/config.json').read())
token = config['token']
request = Request(url)
request.add_header('Authorization', 'token %s' % token)
request.add_header('Accept', "application/vnd.github.inertia-preview+json" )
response = urlopen(request)
return json.loads(response.read())
def process_cards(pri,url, tag=""):
cards= get_json_from_url(url)
for card in cards:
payload=""
if card['note']:
payload="map project:"+ card['note']
else:
payload="Work on: "+ card['content_url']
print("({}) {} {}".format(pri,payload,tag))
def process_project_board(url,tag=""):
board= get_json_from_url(url)
columns_url= board["columns_url"]
columns= get_json_from_url(columns_url)
priorities=['*', 'A', 'B', 'C', 'D','E']
for x in columns:
process_cards(priorities.pop(0),x['cards_url'],tag)
if __name__ == "__main__":
process_project_board("https://api.github.com/projects/1613733","+EQT")
process_project_board("https://api.github.com/projects/1659667","+PersonalProjects")
It works perfectly from my desktop when I run it with
python3 vision.py
but when I clone it on a server, I get the following error:
Traceback (most recent call last):
File "vision.py", line 37, in <module>
process_project_board("https://api.github.com/projects/1613733","+EQT")
File "vision.py", line 29, in process_project_board
board= get_json_from_url(url)
File "vision.py", line 15, in get_json_from_url
return json.loads(response.read())
File "/usr/lib/python3.4/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
What is causing the difference between behaviours? There are no missing library errors- the config.json file (where my Github Token is) is identical, and I've never had a similar problem moving things to the server...

You must be using a Python version lower than 3.6 on your server and JSON doesn't behave like expected. Upgrade your Python version or use a virualenv so you can use the same code on your Desktop and Server

NameError in function to retrieve JSON data

I'm using python 3.6.1 and have the following code which successfully retrieves data in JSON format:
import urllib.request,json,pprint
url = "https://someurl"
response = urllib.request.urlopen(url)
data = json.loads(response.read())
pprint.pprint(data)
I want to wrap this in a function, so i can reuse it. This is what i have tried in a file called getdata.py:
from urllib.request import urlopen
import json
def get_json_data(url):
response = urlopen(url)
return json.loads(response.read())
and this is the error i get after importing the file and attempting to print out the response:
>>> import getdata
>>> print(getdata.get_json_data("https://someurl"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Nick\getdata.py", line 6, in get_json_data
from urllib.request import urlopen
NameError: name 'urllib' is not defined
i also tried this and got the same error:
import urllib.request,json
def get_json_data(url):
response = urllib.request.urlopen(url)
return json.loads(response.read())
What do i need to do to get this to work please?
cheers

Its working now ! I think the problem was the hydrogen addon i have for the Atom editor. I uninstalled it, tried again and it worked. Thanks for looking.

Unit testing bottle py application that uses request body results in KeyError: 'wsgi.input'

When unit testing a bottle py route function:
from bottle import request, run, post
#post("/blah/<boo>")
def blah(boo):
body = request.body.readline()
return "body is %s" % body
blah("booooo!")
The following exception is raised:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in blah
File "bottle.py", line 1197, in body
self._body.seek(0)
File "bottle.py", line 166, in __get__
if key not in storage: storage[key] = self.getter(obj)
File "bottle.py", line 1164, in _body
read_func = self.environ['wsgi.input'].read
KeyError: 'wsgi.input'
The code will work if running as a server via bottle's run function, it's purely when I call it as a normal Python function e.g. in a unit test.
What am I missing? How can I invoke this as a normal python func inside a unit test?

I eventually worked out what the problem is. I needed to "fake" the request environment for bottle to play nicely:
from bottle import request, run, post, tob
from io import BytesIO
body = "abc"
request.environ['CONTENT_LENGTH'] = str(len(tob(body)))
request.environ['wsgi.input'] = BytesIO()
request.environ['wsgi.input'].write(tob(body))
request.environ['wsgi.input'].seek(0)
# Now call your route function and assert

Another issue is that Bottle uses thread locals and reads the BytesIO object you put into request.environ the first time you access the body property on request. Therefore if you run multiple tests with post data, e.g. in a TestCase, when you come to read it in your request callback it will only return the value it was initially given, not your updated value.
The solution is to scrub all the values stored on the request object before each test, so in your setUp(self) you can do something like this:
class MyTestCase(TestCase):
def setUp():
# Flush any cached values
request.bind({})

Check out https://pypi.python.org/pypi/boddle. In your test you could do:
from bottle import request, run, post
from boddle import boddle
#post("/blah/<boo>")
def blah(boo):
body = request.body.readline()
return "body is %s" % body
with boddle(body='woot'):
print blah("booooo!")
Which would print body is woot.
Disclaimer: I authored. (Wrote it for work.)

example urllib3 and threading in python

I am trying to use urllib3 in simple thread to fetch several wiki pages.
The script will
Create 1 connection for every thread (I don't understand why) and Hang forever.
Any tip, advice or simple example of urllib3 and threading
import threadpool
from urllib3 import connection_from_url
HTTP_POOL = connection_from_url(url, timeout=10.0, maxsize=10, block=True)
def fetch(url, fiedls):
kwargs={'retries':6}
return HTTP_POOL.get_url(url, fields, **kwargs)
pool = threadpool.ThreadPool(5)
requests = threadpool.makeRequests(fetch, iterable)
[pool.putRequest(req) for req in requests]
#Lennart's script got this error:
http://en.wikipedia.org/wiki/2010-11_Premier_LeagueTraceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/threadpool.py", line 156, in run
http://en.wikipedia.org/wiki/List_of_MythBusters_episodeshttp://en.wikipedia.org/wiki/List_of_Top_Gear_episodes http://en.wikipedia.org/wiki/List_of_Unicode_characters result = request.callable(*request.args, **request.kwds)
File "crawler.py", line 9, in fetch
print url, conn.get_url(url)
AttributeError: 'HTTPConnectionPool' object has no attribute 'get_url'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/threadpool.py", line 156, in run
result = request.callable(*request.args, **request.kwds)
File "crawler.py", line 9, in fetch
print url, conn.get_url(url)
AttributeError: 'HTTPConnectionPool' object has no attribute 'get_url'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/threadpool.py", line 156, in run
result = request.callable(*request.args, **request.kwds)
File "crawler.py", line 9, in fetch
print url, conn.get_url(url)
AttributeError: 'HTTPConnectionPool' object has no attribute 'get_url'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/threadpool.py", line 156, in run
result = request.callable(*request.args, **request.kwds)
File "crawler.py", line 9, in fetch
print url, conn.get_url(url)
AttributeError: 'HTTPConnectionPool' object has no attribute 'get_url'
After adding import threadpool; import urllib3 and tpool = threadpool.ThreadPool(4) #user318904's code got this error:
Traceback (most recent call last):
File "crawler.py", line 21, in <module>
tpool.map_async(fetch, urls)
AttributeError: ThreadPool instance has no attribute 'map_async'

Here is my take, a more current solution using Python3 and concurrent.futures.ThreadPoolExecutor.
import urllib3
from concurrent.futures import ThreadPoolExecutor
urls = ['http://en.wikipedia.org/wiki/2010-11_Premier_League',
'http://en.wikipedia.org/wiki/List_of_MythBusters_episodes',
'http://en.wikipedia.org/wiki/List_of_Top_Gear_episodes',
'http://en.wikipedia.org/wiki/List_of_Unicode_characters',
]
def download(url, cmanager):
response = cmanager.request('GET', url)
if response and response.status == 200:
print("+++++++++ url: " + url)
print(response.data[:1024])
connection_mgr = urllib3.PoolManager(maxsize=5)
thread_pool = ThreadPoolExecutor(5)
for url in urls:
thread_pool.submit(download, url, connection_mgr)
Some remarks
My code is based on a similar example from the Python Cookbook by Beazley and Jones.
I particularly like the fact that you only need a standard module besides urllib3.
The setup is extremely simple, and if you are only going for side-effects in download (like printing, saving to a file, etc.), there is no additional effort in joining the threads.
If you want something different, ThreadPoolExecutor.submit actually returns whatever download would return, wrapped in a Future.
I found it helpful to align the number of threads in the thread pool with the number of HTTPConnection's in a connection pool (via maxsize). Otherwise you might encounter (harmless) warnings when all threads try to access the same server (as in the example).

Obviously it will create one connection per thread, how should else each thread be able to fetch a page? And you try to use the same connection, made from one url, for all urls. That can hardly be what you intended.
This code worked just fine:
import threadpool
from urllib3 import connection_from_url
def fetch(url):
kwargs={'retries':6}
conn = connection_from_url(url, timeout=10.0, maxsize=10, block=True)
print url, conn.get_url(url)
print "Done!"
pool = threadpool.ThreadPool(4)
urls = ['http://en.wikipedia.org/wiki/2010-11_Premier_League',
'http://en.wikipedia.org/wiki/List_of_MythBusters_episodes',
'http://en.wikipedia.org/wiki/List_of_Top_Gear_episodes',
'http://en.wikipedia.org/wiki/List_of_Unicode_characters',
]
requests = threadpool.makeRequests(fetch, urls)
[pool.putRequest(req) for req in requests]
pool.wait()

Thread programming is hard, so I wrote workerpool to make exactly what you're doing easier.
More specifically, see the Mass Downloader example.
To do the same thing with urllib3, it looks something like this:
import urllib3
import workerpool
pool = urllib3.connection_from_url("foo", maxsize=3)
def download(url):
r = pool.get_url(url)
# TODO: Do something with r.data
print "Downloaded %s" % url
# Initialize a pool, 5 threads in this case
pool = workerpool.WorkerPool(size=5)
# The ``download`` method will be called with a line from the second
# parameter for each job.
pool.map(download, open("urls.txt").readlines())
# Send shutdown jobs to all threads, and wait until all the jobs have been completed
pool.shutdown()
pool.wait()
For more sophisticated code, have a look at workerpool.EquippedWorker (and the tests here for example usage). You can make the pool be the toolbox you pass in.

I use something like this:
#excluding setup for threadpool etc
upool = urllib3.HTTPConnectionPool('en.wikipedia.org', block=True)
urls = ['/wiki/2010-11_Premier_League',
'/wiki/List_of_MythBusters_episodes',
'/wiki/List_of_Top_Gear_episodes',
'/wiki/List_of_Unicode_characters',
]
def fetch(path):
# add error checking
return pool.get_url(path).data
tpool = ThreadPool()
tpool.map_async(fetch, urls)
# either wait on the result object or give map_async a callback function for the results