Mongodb and Celery "pymongo.errors.OperationFailure: command createIndexes requires authentication" - python

I am working Celery with Mongodb( as backend and as a broker).
I follow this tutorial on how to set them up: https://skillachie.com/2013/06/15/intro-celery-and-mongodb/
When I have the security enable on the /etc/mongod.conf file like this:
security:
authorization: enabled
and I call the .get() to get the results of the tasks that I have set up I get this error:
Traceback (most recent call last):
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/kombu/utils/__init__.py", line 323, in __get__
return obj.__dict__[self.__name__]
KeyError: 'collection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/result.py", line 169, in get
no_ack=no_ack,
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/base.py", line 229, in wait_for
meta = self.get_task_meta(task_id)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/base.py", line 307, in get_task_meta
meta = self._get_task_meta_for(task_id)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/mongodb.py", line 158, in _get_task_meta_for
obj = self.collection.find_one({'_id': task_id})
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/kombu/utils/__init__.py", line 325, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/mongodb.py", line 246, in collection
collection.ensure_index('date_done', background='true')
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/collection.py", line 2028, in ensure_index
self.__create_index(keys, kwargs, session=None)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/collection.py", line 1894, in __create_index
session=session)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/collection.py", line 250, in _command
user_fields=user_fields)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/pool.py", line 613, in command
user_fields=user_fields)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/network.py", line 167, in command
parse_write_concern_error=parse_write_concern_error)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/helpers.py", line 159, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: command createIndexes requires authentication
But when I disable the security it is working!
I also tried to change the roles on the user on mongodb and nothing happened.
Also, I have tried to log in with that way:
BROKER_URL = 'mongodb://tester:123456#178.128.250.181:27017/test?authSource=admin'
celery = Celery('EOD_TASKS',broker=BROKER_URL)
Is this a security problem or I can ignore it and move on?

if i'm not mistaken, using authorization: enabled means you need to setup a user/pass to login in order to query the mongodb. by default it is left wide open, which is a security consideration

Related

pymongo: Resolver configuration could not be read or specified no nameservers

It's my first time using MongoDB but I can't seem to fix this one issue, my friend who uses MongoDB doesn't know how to use python so he can't really help me.
Here's my code:
import pymongo
# Replace the uri string with your MongoDB deployment's connection string.
conn_str = "mongodb+srv://sqdnoises:{mypass}#sqd.d4kjb.mongodb.net/myFirstDatabase?retryWrites=true&w=majority"
# set a 5-second connection timeout
client = pymongo.MongoClient(conn_str, serverSelectionTimeoutMS=5000)
try:
print(client.server_info())
print('\n\n\n aka connected')
except Exception:
print("Unable to connect to the server.")
Where {mypass} is my MongoDB password.
I keep getting this error:
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 782, in read_resolv_conf
f = stack.enter_context(open(f))
FileNotFoundError: [Errno 2] No such file or directory: '/etc/resolv.conf'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 88, in _resolve_uri
results = _resolve('_' + self.__srv + '._tcp.' + self.__fqdn,
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 41, in _resolve
return resolver.resolve(*args, **kwargs)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 1305, in resolve
return get_default_resolver().resolve(qname, rdtype, rdclass, tcp, source,
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 1278, in get_default_resolver
reset_default_resolver()
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 1290, in reset_default_resolver
default_resolver = Resolver()
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 734, in __init__
self.read_resolv_conf(filename)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 785, in read_resolv_conf
raise NoResolverConfiguration
dns.resolver.NoResolverConfiguration: Resolver configuration could not be read or specified no nameservers.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/storage/emulated/0/! workspace/mongolearn/main.py", line 7, in <module>
client = pymongo.MongoClient(conn_str, serverSelectionTimeoutMS=5000)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 677, in __init__
res = uri_parser.parse_uri(
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/uri_parser.py", line 532, in parse_uri
nodes = dns_resolver.get_hosts()
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 119, in get_hosts
_, nodes = self._get_srv_response_and_hosts(True)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 99, in _get_srv_response_and_hosts
results = self._resolve_uri(encapsulate_errors)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 95, in _resolve_uri
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: Resolver configuration could not be read or specified no nameservers.
How do I fix this?
I am following https://docs.mongodb.com/drivers/pymongo/
indeed, the problem is that dnspython tries to open /etc/resolv.conf
import dns.resolver
dns.resolver.default_resolver=dns.resolver.Resolver(configure=False)
dns.resolver.default_resolver.nameservers=['8.8.8.8']
Just adde this code to the top of your main code, and that should be sufficient to get you past this hurdle..

Twitter app requires python 3.6.13 to run but Dockerfiles' similar alpine version is not working. Running into mismatch of python package

The Twitter bot app requires python 3.6.13 to run. While dockerizing it, Dockerfile has alpines' python versions which is not meeting my apps requirement.
How can I install python 3.6.13 in a Dockerfile?
For example, in my case, I tried initiating a docker file like this
FROM python:3.6-alpine
COPY bots/config.py /bots/
COPY bots/stream.py /bots/
COPY requirements.txt /tmp
RUN pip3 install -r /tmp/requirements.txt
WORKDIR /bots
CMD ["python3", "stream.py"]
After creating the docker container, the app crashes due to pythons mismatch of versions.
Later I tried this with the version I want:
FROM python:3.6.13
COPY bots/config.py /bots/
COPY bots/stream.py /bots/
COPY requirements.txt /tmp
RUN pip3 install -r /tmp/requirements.txt
WORKDIR /bots
CMD ["python3", "stream.py"]
the python package is very large unlike alpine provides.
But When I tried this:
FROM python:3.6.13-alpine
COPY bots/config.py /bots/
COPY bots/stream.py /bots/
COPY requirements.txt /tmp
RUN pip3 install -r /tmp/requirements.txt
WORKDIR /bots
CMD ["python3", "stream.py"]
It looks like it worked as the program started execution but I got another new error which is this:
INFO:tweepy.binder:PARAMS: {}
ERROR:root:Error creating API
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 187, in execute
proxies=self.api.proxy)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 528, in request
prep = self.prepare_request(req)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 466, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 320, in prepare
self.prepare_auth(auth, url)
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 556, in prepare_auth
r = auth(self)
File "/usr/local/lib/python3.6/site-packages/requests_oauthlib/oauth1_auth.py", line 109, in __call__
unicode(r.url), unicode(r.method), None, r.headers
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 351, in sign
('oauth_signature', self.get_oauth_signature(request)))
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 165, in get_oauth_signature
uri, headers, body = self._render(request)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 248, in _render
request.oauth_params, request.headers, realm=realm)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 25, in wrapper
return target(params, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/parameters.py", line 53, in prepare_headers
escaped_value = utils.escape(value)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 50, in escape
'Got {!r} of type {}.'.format(u, type(u)))
ValueError: Only unicode objects are escapable. Got None of type <class 'NoneType'>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/bots/config.py", line 18, in create_api
api.verify_credentials()
File "/usr/local/lib/python3.6/site-packages/tweepy/api.py", line 631, in verify_credentials
)(**kargs)
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 245, in _call
return method.execute()
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 189, in execute
raise TweepError('Failed to send request: %s' % e)
tweepy.error.TweepError: Failed to send request: Only unicode objects are escapable. Got None of type <class 'NoneType'>.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 187, in execute
proxies=self.api.proxy)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 528, in request
prep = self.prepare_request(req)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 466, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 320, in prepare
self.prepare_auth(auth, url)
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 556, in prepare_auth
r = auth(self)
File "/usr/local/lib/python3.6/site-packages/requests_oauthlib/oauth1_auth.py", line 109, in __call__
unicode(r.url), unicode(r.method), None, r.headers
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 351, in sign
('oauth_signature', self.get_oauth_signature(request)))
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 165, in get_oauth_signature
uri, headers, body = self._render(request)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 248, in _render
request.oauth_params, request.headers, realm=realm)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 25, in wrapper
return target(params, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/parameters.py", line 53, in prepare_headers
escaped_value = utils.escape(value)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 50, in escape
'Got {!r} of type {}.'.format(u, type(u)))
ValueError: Only unicode objects are escapable. Got None of type <class 'NoneType'>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/bots/config.py", line 18, in create_api
api.verify_credentials()
File "/usr/local/lib/python3.6/site-packages/tweepy/api.py", line 631, in verify_credentials
)(**kargs)
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 245, in _call
return method.execute()
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 189, in execute
raise TweepError('Failed to send request: %s' % e)
tweepy.error.TweepError: Failed to send request: Only unicode objects are escapable. Got None of type <class 'NoneType'>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "stream.py", line 50, in <module>
main(['485visas', 'extend485', '485visaextension', '485', '485visa',])
File "stream.py", line 42, in main
api = create_api()
File "/bots/config.py", line 21, in create_api
raise Exception
Exception
The config.py file is this:
import tweepy
import logging
import os
logger = logging.getLogger()
def create_api():
# TR485Graduate = tweepy.Stream('CONSUMER_KEY_485', 'CONSUMER_SECRET_485', 'ACCESS_TOKEN_485','ACCESS_TOKEN_SECRET_485' )
consumer_key = os.getenv("CONSUMER_KEY_485")
consumer_secret = os.getenv("CONSUMER_SECRET_485")
access_token = os.getenv("ACCESS_TOKEN_485")
access_token_secret = os.getenv("ACCESS_TOKEN_SECRET_485")
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth,wait_on_rate_limit=True)
try:
api.verify_credentials()
except Exception as e:
logger.error("Error creating API", exc_info=True)
raise Exception
logger.info("API created")
return api
Is there anything that I can do to avoid this exception error?
Later, I tried to add the python version that I want in the requirements.txt file but it didn't work.
I found up a source but couldn't understand but I feel the answer is here, source.
Is there any way I can instantiate the python 3.6.13 version in the Dockerfile so that it uniformly supplies this version to the whole image and the app?

PermissionDenied: 403 error when I'm using Google Cloud Billing Budget API

I'm using Google cloud functions and the Cloud Billing Budget API to get a list with all of my budgets, but I'm having the following error:
Traceback (most recent call last): File "/env/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable return callable_(*args, **kwargs) File "/env/local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/env/local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.PERMISSION_DENIED details = "The caller does not have permission"
debug_error_string = "{"created":"#9627456.9324530376","description":"Error received from peer ipv4:54.128.19.5:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"The caller does not have permission","grpc_status":7}" > The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 449, in run_background_function _function_handler.invoke_user_function(event_object) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 268, in invoke_user_function return call_user_function(request_or_event) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 265, in call_user_function event_context.Context(**request_or_event.context)) File "/user_code/main.py", line 22, in getting_data all_budgets = client.list_budgets(request = {'parent': BILLING_ACCOUNT}) File "/env/local/lib/python3.7/site-packages/google/cloud/billing/budgets_v1/services/budget_service/client.py", line 693, in list_budgets response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,) File "/env/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__ return wrapped_func(*args, **kwargs) File "/env/local/lib/python3.7/site-packages/google/api_core/retry.py", line 290, in retry_wrapped_func on_error=on_error, File "/env/local/lib/python3.7/site-packages/google/api_core/retry.py", line 188, in retry_target return target() File "/env/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable six.raise_from(exceptions.from_grpc_error(exc), exc) File "<string>", line 3, in raise_from google.api_core.exceptions.PermissionDenied: 403 The caller does not have permission
What I've done is give the appropriate permissions (billing.budget.list ... billing.budget.get ... etc ) at the organization level to the service account of the function, but it does not work.
My code is this:
#main.py
import os
import get_budgets
from google.cloud.billing import budgets
def getting_data(data, context):
BILLING_ACCOUNT = 'billingAccounts/XXXXXX-XXXXXX-XXXXXX'
client = budgets.BudgetServiceClient()
all_budgets = client.list_budgets(request = {'parent': BILLING_ACCOUNT})
budget_actions.budget_list(all_budgets)
#get_budgets.py
from google.cloud.billing import budgets
from googleapiclient import discovery
#BUDGET LIST
def budget_list(all_budgets):
print('Budget summary')
for budget in all_budgets:
print(f'Name: {budget.display_name}')
b_amount = budget.amount
if 'specified_amount' in b_amount:
print(f'Specified Amount: {b_amount.specified_amount.units} {b_amount.specified_amount.currency_code}')
if 'last_period_amount' in b_amount:
print('Dynamic spend (based on last period)')
print('')
Is there something that I've forgotten?
The solution to my problem was easier than I expected.
What I've done is add the service account as a member of the billing account.
Here is a quick video from Google of how to do it
https://www.youtube.com/watch?v=Vti0OGQfLHQ
And in the comments of my question, you can find additional details.

Pysyft Federated learning, Error with Websockets

I am trying to run a federated learning from pysyft (https://github.com/OpenMined/PySyft/blob/dev/examples/tutorials/advanced/websockets-example-MNIST-parallel/Asynchronous-federated-learning-on-MNIST.ipynb) that creates remote workers and connect to them via websockets. however I am getting an error in folllowing evaluation step.
future: <Task finished coro=<WebsocketServerWorker._producer_handler() done, defined at C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py:95> exception=AttributeError("'dict' object has no attribute 'owner'")>
Traceback (most recent call last):
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 663, in register_response
register_response_function = register_response_functions[attr_id]
KeyError: 'evaluate'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py", line 113, in _producer_handler
response = self._recv_msg(message)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py", line 124, in _recv_msg
return self.recv_msg(message)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\base.py", line 310, in recv_msg
response = self._message_router[type(msg)](msg.contents)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\base.py", line 457, in execute_command
command_name, response, list(return_ids), self
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 672, in register_response
new_response = register_response_function(response, response_ids=response_ids, owner=owner)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 766, in <lambda>
return lambda x, **kwargs: f(lambdas, x, **kwargs)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 522, in two_fold
return lambdas[0](args[0], **kwargs), lambdas[1](args[1], **kwargs)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 744, in <lambda>
else lambda i, **kwargs: register_tensor(i, **kwargs)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 712, in register_tensor
tensor.owner = owner
AttributeError: 'dict' object has no attribute 'owner'
There are no clear answer from their forum. does anyone have any clue as to what the issue is in this script.
My syft version:
syft : 0.2.3a1
syft-proto : 0.1.1a1.post12
torch : 1.4.0
I came across this problem as well and pushed a fix in https://github.com/OpenMined/PySyft/pull/2948

NotAllowedError when trying to access AppStats

After doing everything from here
http://code.google.com/appengine/docs/python/tools/appstats.html
I get
<class 'google.appengine.api.users.NotAllowedError'>:
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/appstats/ui.py", line 324, in <module>
main()
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/appstats/ui.py", line 320, in main
util.run_bare_wsgi_app(app)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 116, in run_bare_wsgi_app
result = application(env, _start_response)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/appstats/ui.py", line 308, in __call__
users.create_login_url(os.getenv('PATH_INFO', '')))])
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/users.py", line 256, in create_login_url
raise NotAllowedError
when trying to access myapp.appspot.com/_ah/stats or just myapp.appspot.com/stats/
Most likely you have created your app with authentication set to only allow users from your Google Apps domain, but you're attempting to access appstats via the appspot.com subdomain. Try again using the URL for your app on your custom domain.

Categories