We Keep Coding

PRAW | submissions | prawcore.exceptions.NotFound: received 404 HTTP response

i try to get this running. In the past the script was running, i cant remember that i have changed something but now its no longer working. I alltimes get a 404 error at the line for submission in submissions.hot():. This script should upvote all posts from a specific redditor (selected by username). Does someone have an idea? I first tought submissions.hot() would be empty but i get back <praw.models.listing.generator.ListingGenerator object at 0x000001A43D81AD40> so that is not the case and there are enough posts on my profile.
import praw
reddit = praw.Reddit(
client_id="XXXXXXX", #personal use script code
client_secret="XXXXXXXX",
password="XXXXXXXXXXXXXX",
user_agent="XXXXXXXXX", #AppName
username="XXXXXXXXXXXX",
)
redditorName = "LofiBeatsMusicLovers"
redditor = reddit.redditor(redditorName)
submissions = redditor.submissions
print(str(submissions))
print(str(submissions.hot()))
for submission in submissions.hot():
submission.upvote()
print("upvoted hot one!!")
Here is the output from the script:
<praw.models.listing.mixins.redditor.SubListing object at 0x000001A43AC37730>
<praw.models.listing.generator.ListingGenerator object at 0x000001A43D81AD40>
Traceback (most recent call last):
File "C:\Users\erdtm\Desktop\YouTubeStreamingSetup\RedditDMBotSWCombination\RedditBot\UpvoteAllPostsOfUser.py", line 20, in <module>
for submission in submissions.hot():
File "C:\Users\erdtm\AppData\Local\Programs\Python\Python310\lib\site-packages\praw\models\listing\generator.py", line 63, in __next__
self._next_batch()
File "C:\Users\erdtm\AppData\Local\Programs\Python\Python310\lib\site-packages\praw\models\listing\generator.py", line 89, in _next_batch
self._listing = self._reddit.get(self.url, params=self.params)
File "C:\Users\erdtm\AppData\Local\Programs\Python\Python310\lib\site-packages\praw\util\deprecate_args.py", line 43, in wrapped
return func(**dict(zip(_old_args, args)), **kwargs)
File "C:\Users\erdtm\AppData\Local\Programs\Python\Python310\lib\site-packages\praw\reddit.py", line 634, in get
return self._objectify_request(method="GET", params=params, path=path)
File "C:\Users\erdtm\AppData\Local\Programs\Python\Python310\lib\site-packages\praw\reddit.py", line 739, in _objectify_request
self.request(
File "C:\Users\erdtm\AppData\Local\Programs\Python\Python310\lib\site-packages\praw\util\deprecate_args.py", line 43, in wrapped
return func(**dict(zip(_old_args, args)), **kwargs)
File "C:\Users\erdtm\AppData\Local\Programs\Python\Python310\lib\site-packages\praw\reddit.py", line 941, in request
return self._core.request(
File "C:\Users\erdtm\AppData\Local\Programs\Python\Python310\lib\site-packages\prawcore\sessions.py", line 330, in request
return self._request_with_retries(
File "C:\Users\erdtm\AppData\Local\Programs\Python\Python310\lib\site-packages\prawcore\sessions.py", line 266, in _request_with_retries
raise self.STATUS_EXCEPTIONS[response.status_code](response)
prawcore.exceptions.NotFound: received 404 HTTP response

Random NoTransaction in Pyramid

I'm having trouble identifying the source of transaction.interfaces.NoTransaction errors within my Pyramid App. I don't see any patterns to when the error happens, so to me it's quite random.
This app is a (semi-) RESTful API and uses SQLAlchemy and MySQL. I'm currently running within a docker container that connects to an external (bare metal) MySQL instance on the same host OS.
Here's the stack trace for a login attempt within the App. This error happened right after another login attempt that was actually successful.
2020-06-15 03:57:18,982 DEBUG [txn.140501728405248:108][waitress-1] new transaction
2020-06-15 03:57:18,984 INFO [sqlalchemy.engine.base.Engine:730][waitress-1] BEGIN (implicit)
2020-06-15 03:57:18,984 DEBUG [txn.140501728405248:576][waitress-1] abort
2020-06-15 03:57:18,985 ERROR [waitress:357][waitress-1] Exception while serving /auth
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/waitress/channel.py", line 350, in service
task.service()
File "/usr/local/lib/python3.8/site-packages/waitress/task.py", line 171, in service
self.execute()
File "/usr/local/lib/python3.8/site-packages/waitress/task.py", line 441, in execute
app_iter = self.channel.server.application(environ, start_response)
File "/usr/local/lib/python3.8/site-packages/pyramid/router.py", line 270, in __call__
response = self.execution_policy(environ, self)
File "/usr/local/lib/python3.8/site-packages/pyramid_retry/__init__.py", line 127, in retry_policy
response = router.invoke_request(request)
File "/usr/local/lib/python3.8/site-packages/pyramid/router.py", line 249, in invoke_request
response = handle_request(request)
File "/usr/local/lib/python3.8/site-packages/pyramid_tm/__init__.py", line 178, in tm_tween
reraise(*exc_info)
File "/usr/local/lib/python3.8/site-packages/pyramid_tm/compat.py", line 36, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/pyramid_tm/__init__.py", line 135, in tm_tween
userid = request.authenticated_userid
File "/usr/local/lib/python3.8/site-packages/pyramid/security.py", line 381, in authenticated_userid
return policy.authenticated_userid(self)
File "/opt/REDACTED-api/REDACTED_api/auth/policy.py", line 208, in authenticated_userid
result = self._authenticate(request)
File "/opt/REDACTED-api/REDACTED_api/auth/policy.py", line 199, in _authenticate
session = self._get_session_from_token(token)
File "/opt/REDACTED-api/REDACTED_api/auth/policy.py", line 320, in _get_session_from_token
session = service.get(session_id)
File "/opt/REDACTED-api/REDACTED_api/service/__init__.py", line 122, in get
entity = self.queryset.filter(self.Meta.model.id == entity_id).first()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3375, in first
ret = list(self[0:1])
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3149, in __getitem__
return list(res)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3502, in _execute_and_instances
conn = self._get_bind_args(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3517, in _get_bind_args
return fn(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3496, in _connection_from_session
conn = self.session.connection(**kw)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1138, in connection
return self._connection_for_bind(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1146, in _connection_for_bind
return self.transaction._connection_for_bind(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 458, in _connection_for_bind
self.session.dispatch.after_begin(self.session, self, conn)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 322, in __call__
fn(*args, **kw)
File "/usr/local/lib/python3.8/site-packages/zope/sqlalchemy/datamanager.py", line 268, in after_begin
join_transaction(
File "/usr/local/lib/python3.8/site-packages/zope/sqlalchemy/datamanager.py", line 233, in join_transaction
DataManager(
File "/usr/local/lib/python3.8/site-packages/zope/sqlalchemy/datamanager.py", line 89, in __init__
transaction_manager.get().join(self)
File "/usr/local/lib/python3.8/site-packages/transaction/_manager.py", line 91, in get
raise NoTransaction()
transaction.interfaces.NoTransaction
The trace shows that the execution eventually reaches my project, but only my custom authentication policy. And it fails right where the database should be queried for the user.
What intrigues me here is the third line on the stack trace. It seems Waitress somehow aborted the transaction it created? Any clue why?
EDIT: Here's the code where that happens: policy.py:320
def _get_session_from_token(self, token) -> UserSession:
try:
session_id, session_secret = self.parse_token(token)
except InvalidToken as e:
raise SessionNotFound(e)
service = AuthService(self.dbsession, None)
try:
session = service.get(session_id) # <---- Service Class called here
except NoResultsFound:
raise SessionNotFound("Invalid session found Request headers. "
"Session id: %s".format(session_id))
if not service.check_session(session, session_secret):
raise SessionNotFound("Session signature does not match")
now = datetime.now(tz=pytz.UTC)
if session.validity < now:
raise SessionNotFound(
"Current session ID {session_id} is expired".format(
session_id=session.id
)
)
return session
And here is an a view on the that service class method:
class AuthService(ModelService):
class Meta:
model = UserSession
queryset = Query(UserSession)
search_fields = []
order_fields = [UserSession.created_at.desc()]
# These below are from the generic ModelClass father class
def __init__(self, dbsession: Session, user_id: str):
self.user_id = user_id
self.dbsession = dbsession
self.Meta.queryset = self.Meta.queryset.with_session(dbsession)
self.logger = logging.getLogger("REDACTED")
#property
def queryset(self):
return self.Meta.queryset
def get(self, entity_id) -> Base:
entity = self.queryset.filter(self.Meta.model.id == entity_id).first()
if not entity:
raise NoResultsFound(f"Could not find requested ID {entity_id}")
As you can see, the there's already some exception treatment. I really don't see what other exception I could try to catch on AuthService.get

I found the solution to be much simpler than tinkering inside Pyramid or SQLAlchemy.
Debugging my Authentication Policy closely, I found out that my it was keeping a sticky reference for the dbsession. It was stored on the first request ever who used it, and never released.
The first request works as expected, the following one fails: My understanding is that the object is still in memory while the app is running, and after the initial transaction is closed. The second request has a new connection, and a new transaction, but the object in memory still points to the previous one, that when used ultimately causes this.
What I don't understand is why the exception didn't happen sometimes. As I mentioned initially, it was seemingly random.
Another thing that I struggled with was in writing a test case to expose the issue. On my tests, the issue never happens because I have (and I've never seen it done differently) a single connection and a single transaction throughout the entire testing session, as opposed of a new connection/transaction per request, so I have not found no way to actually reproduce.
Please let me know if that makes sense, and if you can shed a light on how to expose the bug on a test case.

Mongodb and Celery "pymongo.errors.OperationFailure: command createIndexes requires authentication"

I am working Celery with Mongodb( as backend and as a broker).
I follow this tutorial on how to set them up: https://skillachie.com/2013/06/15/intro-celery-and-mongodb/
When I have the security enable on the /etc/mongod.conf file like this:
security:
authorization: enabled
and I call the .get() to get the results of the tasks that I have set up I get this error:
Traceback (most recent call last):
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/kombu/utils/__init__.py", line 323, in __get__
return obj.__dict__[self.__name__]
KeyError: 'collection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/result.py", line 169, in get
no_ack=no_ack,
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/base.py", line 229, in wait_for
meta = self.get_task_meta(task_id)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/base.py", line 307, in get_task_meta
meta = self._get_task_meta_for(task_id)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/mongodb.py", line 158, in _get_task_meta_for
obj = self.collection.find_one({'_id': task_id})
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/kombu/utils/__init__.py", line 325, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/mongodb.py", line 246, in collection
collection.ensure_index('date_done', background='true')
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/collection.py", line 2028, in ensure_index
self.__create_index(keys, kwargs, session=None)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/collection.py", line 1894, in __create_index
session=session)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/collection.py", line 250, in _command
user_fields=user_fields)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/pool.py", line 613, in command
user_fields=user_fields)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/network.py", line 167, in command
parse_write_concern_error=parse_write_concern_error)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/helpers.py", line 159, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: command createIndexes requires authentication
But when I disable the security it is working!
I also tried to change the roles on the user on mongodb and nothing happened.
Also, I have tried to log in with that way:
BROKER_URL = 'mongodb://tester:123456#178.128.250.181:27017/test?authSource=admin'
celery = Celery('EOD_TASKS',broker=BROKER_URL)
Is this a security problem or I can ignore it and move on?

if i'm not mistaken, using authorization: enabled means you need to setup a user/pass to login in order to query the mongodb. by default it is left wide open, which is a security consideration

Occasional Flask server error: 'RuntimeError: maximum recursion depth exceeded while calling a Python object'

I'm trying to figure out what causes this error when I run my app using the basic Flask server during development. I start it with this:
from myapp import app
app.run(debug=True, port=5001)
All is well and I'll continue to code and refresh etc, but then after a while I get the recursion error and have to Ctrl-C the server and restart it. Not a big deal, just a little annoying to have to deal with every now and then.
Here's the full traceback, which I tried to use to determine the cause but can't see anything that stands out (possibly something to do with how werkzeug uses Cookie.py?):
Traceback (most recent call last):
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/app.py", line 1701, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/wsgi.py", line 411, in __call__
return self.app(environ, start_response)
(last bit repeated a bunch - trimmed to fit in posting size requirements)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/app.py", line 1685, in wsgi_app
with self.request_context(environ):
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/ctx.py", line 274, in __enter__
self.push()
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/ctx.py", line 238, in push
self.session = self.app.open_session(self.request)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/app.py", line 792, in open_session
return self.session_interface.open_session(self, request)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/sessions.py", line 191, in open_session
secret_key=key)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/contrib/securecookie.py", line 309, in load_cookie
data = request.cookies.get(key)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/utils.py", line 77, in __get__
value = self.func(obj)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/wrappers.py", line 418, in cookies
cls=self.dict_storage_class)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/http.py", line 741, in parse_cookie
cookie.load(header)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 632, in load
self.__ParseString(rawdata)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 665, in __ParseString
self.__set(K, rval, cval)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/_internal.py", line 290, in _BaseCookie__set
morsel = self.get(key, _ExtendedMorsel())
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/_internal.py", line 271, in __init__
Morsel.__init__(self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 438, in __init__
dict.__setitem__(self, K, "")
RuntimeError: maximum recursion depth exceeded while calling a Python object

Since it occurs during your developement process, you could increase recursion limit, before starting your server, using :
sys.setrecursionlimit(2000) # Choose the right figure for you here
# the value on my system is 1000 but this is platform-dependant
However, you should use it very carefully and probably not in production unless you have a good knowledge of it's impacts.
Ref : http://docs.python.org/2/library/sys.html#sys.setrecursionlimit

prestapyt error on edit

I'm using the prestashop API prestapyt
I add a category by hand in my site, but when I'm trying to edit via the API, it fails.
Even the most simple thing, does not work:
ps = PrestaShopWebServiceDict(config.DOMAIN, config.KEY)
xml = ps.get('categories', 35)
ps.edit('categories',35,xml)
The xml (type DICT) loads ok, but the .edit() fails.
Here is the response
Traceback (most recent call last):
File "C:\Users\noukeys\Documents\psapi\main.py", line 31, in <module>
ps.edit('categories',35,xml)
File "C:\Python27\lib\site-packages\prestapyt\prestapyt.py", line 333, in edit
return self.edit_with_url(full_url, content)
File "C:\Python27\lib\site-packages\prestapyt\prestapyt.py", line 452, in edit_with_url
return super(PrestaShopWebServiceDict, self).edit_with_url(url, xml_content)
File "C:\Python27\lib\site-packages\prestapyt\prestapyt.py", line 344, in edit_with_url
return self._parse(self._execute(url, 'PUT', body=unicode_encode.encode(content), add_headers=headers)[2])
File "C:\Python27\lib\site-packages\prestapyt\prestapyt.py", line 171, in _execute
self._check_status_code(status_code)
File "C:\Python27\lib\site-packages\prestapyt\prestapyt.py", line 125, in _check_status_code
% (status_code, message_by_code[status_code]), status_code)
prestapyt.prestapyt.PrestaShopWebServiceError: 'This call to PrestaShop Web Services failed and returned an HTTP status of 400. That means: Bad Request.'

I'm having issues with edits also, but I am getting a 501 error. Turns out my host does not allow PUT requests.

You can turn on debugging in the api.
prestashop.debug = True
And then turn on display_errors in config.inc.php for your shop. This helped me debug why the request was bad. The error messages are pretty clear.