We Keep Coding

Twitter app requires python 3.6.13 to run but Dockerfiles' similar alpine version is not working. Running into mismatch of python package

The Twitter bot app requires python 3.6.13 to run. While dockerizing it, Dockerfile has alpines' python versions which is not meeting my apps requirement.
How can I install python 3.6.13 in a Dockerfile?
For example, in my case, I tried initiating a docker file like this
FROM python:3.6-alpine
COPY bots/config.py /bots/
COPY bots/stream.py /bots/
COPY requirements.txt /tmp
RUN pip3 install -r /tmp/requirements.txt
WORKDIR /bots
CMD ["python3", "stream.py"]
After creating the docker container, the app crashes due to pythons mismatch of versions.
Later I tried this with the version I want:
FROM python:3.6.13
COPY bots/config.py /bots/
COPY bots/stream.py /bots/
COPY requirements.txt /tmp
RUN pip3 install -r /tmp/requirements.txt
WORKDIR /bots
CMD ["python3", "stream.py"]
the python package is very large unlike alpine provides.
But When I tried this:
FROM python:3.6.13-alpine
COPY bots/config.py /bots/
COPY bots/stream.py /bots/
COPY requirements.txt /tmp
RUN pip3 install -r /tmp/requirements.txt
WORKDIR /bots
CMD ["python3", "stream.py"]
It looks like it worked as the program started execution but I got another new error which is this:
INFO:tweepy.binder:PARAMS: {}
ERROR:root:Error creating API
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 187, in execute
proxies=self.api.proxy)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 528, in request
prep = self.prepare_request(req)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 466, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 320, in prepare
self.prepare_auth(auth, url)
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 556, in prepare_auth
r = auth(self)
File "/usr/local/lib/python3.6/site-packages/requests_oauthlib/oauth1_auth.py", line 109, in __call__
unicode(r.url), unicode(r.method), None, r.headers
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 351, in sign
('oauth_signature', self.get_oauth_signature(request)))
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 165, in get_oauth_signature
uri, headers, body = self._render(request)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 248, in _render
request.oauth_params, request.headers, realm=realm)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 25, in wrapper
return target(params, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/parameters.py", line 53, in prepare_headers
escaped_value = utils.escape(value)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 50, in escape
'Got {!r} of type {}.'.format(u, type(u)))
ValueError: Only unicode objects are escapable. Got None of type <class 'NoneType'>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/bots/config.py", line 18, in create_api
api.verify_credentials()
File "/usr/local/lib/python3.6/site-packages/tweepy/api.py", line 631, in verify_credentials
)(**kargs)
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 245, in _call
return method.execute()
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 189, in execute
raise TweepError('Failed to send request: %s' % e)
tweepy.error.TweepError: Failed to send request: Only unicode objects are escapable. Got None of type <class 'NoneType'>.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 187, in execute
proxies=self.api.proxy)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 528, in request
prep = self.prepare_request(req)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 466, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 320, in prepare
self.prepare_auth(auth, url)
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 556, in prepare_auth
r = auth(self)
File "/usr/local/lib/python3.6/site-packages/requests_oauthlib/oauth1_auth.py", line 109, in __call__
unicode(r.url), unicode(r.method), None, r.headers
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 351, in sign
('oauth_signature', self.get_oauth_signature(request)))
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 165, in get_oauth_signature
uri, headers, body = self._render(request)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/__init__.py", line 248, in _render
request.oauth_params, request.headers, realm=realm)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 25, in wrapper
return target(params, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/parameters.py", line 53, in prepare_headers
escaped_value = utils.escape(value)
File "/usr/local/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/utils.py", line 50, in escape
'Got {!r} of type {}.'.format(u, type(u)))
ValueError: Only unicode objects are escapable. Got None of type <class 'NoneType'>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/bots/config.py", line 18, in create_api
api.verify_credentials()
File "/usr/local/lib/python3.6/site-packages/tweepy/api.py", line 631, in verify_credentials
)(**kargs)
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 245, in _call
return method.execute()
File "/usr/local/lib/python3.6/site-packages/tweepy/binder.py", line 189, in execute
raise TweepError('Failed to send request: %s' % e)
tweepy.error.TweepError: Failed to send request: Only unicode objects are escapable. Got None of type <class 'NoneType'>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "stream.py", line 50, in <module>
main(['485visas', 'extend485', '485visaextension', '485', '485visa',])
File "stream.py", line 42, in main
api = create_api()
File "/bots/config.py", line 21, in create_api
raise Exception
Exception
The config.py file is this:
import tweepy
import logging
import os
logger = logging.getLogger()
def create_api():
# TR485Graduate = tweepy.Stream('CONSUMER_KEY_485', 'CONSUMER_SECRET_485', 'ACCESS_TOKEN_485','ACCESS_TOKEN_SECRET_485' )
consumer_key = os.getenv("CONSUMER_KEY_485")
consumer_secret = os.getenv("CONSUMER_SECRET_485")
access_token = os.getenv("ACCESS_TOKEN_485")
access_token_secret = os.getenv("ACCESS_TOKEN_SECRET_485")
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth,wait_on_rate_limit=True)
try:
api.verify_credentials()
except Exception as e:
logger.error("Error creating API", exc_info=True)
raise Exception
logger.info("API created")
return api
Is there anything that I can do to avoid this exception error?
Later, I tried to add the python version that I want in the requirements.txt file but it didn't work.
I found up a source but couldn't understand but I feel the answer is here, source.
Is there any way I can instantiate the python 3.6.13 version in the Dockerfile so that it uniformly supplies this version to the whole image and the app?

Pysyft Federated learning, Error with Websockets

I am trying to run a federated learning from pysyft (https://github.com/OpenMined/PySyft/blob/dev/examples/tutorials/advanced/websockets-example-MNIST-parallel/Asynchronous-federated-learning-on-MNIST.ipynb) that creates remote workers and connect to them via websockets. however I am getting an error in folllowing evaluation step.
future: <Task finished coro=<WebsocketServerWorker._producer_handler() done, defined at C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py:95> exception=AttributeError("'dict' object has no attribute 'owner'")>
Traceback (most recent call last):
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 663, in register_response
register_response_function = register_response_functions[attr_id]
KeyError: 'evaluate'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py", line 113, in _producer_handler
response = self._recv_msg(message)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py", line 124, in _recv_msg
return self.recv_msg(message)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\base.py", line 310, in recv_msg
response = self._message_router[type(msg)](msg.contents)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\base.py", line 457, in execute_command
command_name, response, list(return_ids), self
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 672, in register_response
new_response = register_response_function(response, response_ids=response_ids, owner=owner)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 766, in <lambda>
return lambda x, **kwargs: f(lambdas, x, **kwargs)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 522, in two_fold
return lambdas[0](args[0], **kwargs), lambdas[1](args[1], **kwargs)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 744, in <lambda>
else lambda i, **kwargs: register_tensor(i, **kwargs)
File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 712, in register_tensor
tensor.owner = owner
AttributeError: 'dict' object has no attribute 'owner'
There are no clear answer from their forum. does anyone have any clue as to what the issue is in this script.
My syft version:
syft : 0.2.3a1
syft-proto : 0.1.1a1.post12
torch : 1.4.0

I came across this problem as well and pushed a fix in https://github.com/OpenMined/PySyft/pull/2948

msrest.exceptions.ValidationError: Parameter 'resource_group_name' must conform to the following pattern: '^[-\\w\\._\\(\\)]+$'

sathish#Azure:~/quickstart/python-docs-hello-world$ az webapp up -n appname
The command failed with an unexpected error. Here is the traceback:
Parameter 'resource_group_name' must conform to the following pattern: '^[-\\w\\._\\(\\)]+$'.
Traceback (most recent call last):
File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 326, in execute
raise ex
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 384, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 375, in _run_job
cmd_copy.exception_handler(ex)
File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/commands.py", line 54, in _polish_bad_errors
raise ex
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 354, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 145, in __call__
return self.handler(*args, **kwargs)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 451, in default_command_handler
return op(**command_args)
File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/custom.py", line 2313, in webapp_up
_create_new_rg = should_create_new_rg(cmd, default_rg, rg_name, is_linux)
File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/_create_util.py", line 282, in should_create_new_rg
elif (_check_resource_group_exists(cmd, rg_name) and
File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/_create_util.py", line 86, in _check_resource_group_exists
return rcf.resource_groups.check_existence(rg_name)
File "/opt/az/lib/python3.6/site-packages/azure/mgmt/resource/resources/v2018_05_01/operations/resource_groups_operations.py", line 61, in check_existence
'resourceGroupName': self._serialize.url("resource_group_name", resource_group_name, 'str', max_length=90, min_length=1, pattern=r'^[-\w\._\(\)]+$'),
File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 592, in url
data = self.validate(data, name, required=True, **kwargs)
File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 672, in validate
raise ValidationError(key, name, value)
msrest.exceptions.ValidationError: Parameter 'resource_group_name' must conform to the following pattern: '^[-\\w\\._\\(\\)]+$'.
To open an issue, please run: 'az feedback'

I had this problem, but resolved it by removing the quotes around the resource group name. Silly mistake on my behalf, but may be the issue here.
e.g. use az group create -n my-groupname-rg -l northeurope
rather than: e.g. use az group create -n 'my-groupname-rg' -l 'northeurope'

For resource group name, you should take a look at Naming rules and restrictions:

Buildbot MailNotifier Error

I am trying to set up a simple email notification system for every build using Buildbot's reporter.MailNotifier. I have implemented it on two Windows computers, and one linux machine and replicated the same error. Here is the code snippet
from buildbot.plugins import *
c = BuildmasterConfig = {}
#Added workers, protocols, and other configurations
#Test scheduler
c['schedulers'] = [schedulers.Periodic(name="tester", builderNames=["runtest"], periodicBuildTimer=60)]
####### BUILDBOT SERVICES
mn = reporters.MailNotifier(fromaddr='email#gmail.com', sendToInterestedUsers=False,
relayhost="smtp.gmail.com",smtpPort=587, useTls=True,
extraRecipients=["email#gmail.com"],
smtpUser="email#gmail.com", smtpPassword="email_password")
c['services'] = [mn]
However, every time I receive the following error in twistd.log:
2017-06-15 21:20:14-0700 [ESMTPSender,client] SMTP Client retrying server. Retry: 1
2017-06-15 21:20:15-0700 [ESMTPSender,client] Unhandled Error
Traceback (most recent call last):
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\python\log.py", line 103, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\python\log.py", line 86, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\python\context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\python\context.py", line 85, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\internet\selectreactor.py", line 149, in _doReadOrWrite
why = getattr(selectable, method)()
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\internet\tcp.py", line 208, in doRead
return self._dataReceived(data)
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\internet\tcp.py", line 214, in _dataReceived
rval = self.protocol.dataReceived(data)
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\protocols\tls.py", line 330, in dataReceived
self._flushReceiveBIO()
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\protocols\tls.py", line 295, in _flushReceiveBIO
ProtocolWrapper.dataReceived(self, bytes)
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\protocols\policies.py", line 120, in dataReceived
self.wrappedProtocol.dataReceived(data)
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\protocols\basic.py", line 571, in dataReceived
why = self.lineReceived(line)
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\mail\smtp.py", line 995, in lineReceived
why = self._okresponse(self.code, b'\n'.join(self.resp))
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\mail\smtp.py", line 1044, in smtpState_to
return self.smtpState_toOrData(0, b'')
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\mail\smtp.py", line 1062, in smtpState_toOrData
self.sendLine(b'RCPT TO:' + quoteaddr(self.lastAddress))
File "c:\users\me\buildbot\sandbox\lib\site-packages\twisted\mail\smtp.py", line 179, in quoteaddr
res = email.utils.parseaddr(addr)
File "c:\python27\Lib\email\utils.py", line 214, in parseaddr
addrs = _AddressList(addr).addresslist
File "c:\python27\Lib\email\_parseaddr.py", line 457, in __init__
self.addresslist = self.getaddrlist()
File "c:\python27\Lib\email\_parseaddr.py", line 217, in getaddrlist
while self.pos < len(self.field):
exceptions.TypeError: object of type 'module' has no len()
Quick info: Error appears on both Buildbot 0.9.8 and 0.9.1 on Windows 10 (64-bit) and Ubuntu 14.04. The error log is from Python 2.7.13 virtualenv 15.1.0 twisted 17.5.0. Inserting the following code in _parseaddr.py works but I'm looking for a better fix.
if str(type(self.field)) == "<type 'module'>":
return [('',u'email#gmail.com')]

It was a typo in the recent release of Twisted 17.5.0
Around line 1900 of twisted/mail/smtp.py in the constructor of class SMTPSenderFactory:
toEmailFinal.append(email)
should have been
toEmailFinal.append(_email)
The former passed the entire email module instead of passing just the parsed email, which produced the error. The newer releases will probably fix that or you can manually replace the line in the file. The fix (by rodrigc) can be found in this GitHub commit

Strange error adding to Whoosh index

Can anyone help me with this strange error I'm getting when adding a new document to a Whoosh index?
Here's the code:
def add_to_index(self, doc):
ix = index.open_dir(self.index_dir)
writer = AsyncWriter(ix) # use async writer to prevent write lock errors
writer.add_document(**self.get_doc_args(doc))
writer.commit()
def get_doc_args(self, doc):
return {
'id': u""+str(doc['id']),
'org': doc['org__id'],
'created': doc['created_date'],
'date': doc['received_date'],
'from_addr': doc['from_addr'],
'subject': doc['subject'],
'body': doc['messagebody__cleaned_message']
}
I get the following error:
TypeError('ord() expected a character, but string of length 0 found',)
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/celery/execute/trace.py", line 36, in trace
return cls(states.SUCCESS, retval=fun(*args, **kwargs))
File "/usr/local/lib/python2.6/dist-packages/celery/app/task/__init__.py", line 232, in __call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/celery/app/__init__.py", line 172, in run
return fun(*args, **kwargs)
File "/mnt/deploy/prod/chorus/src/chorus/../chorus/search/__init__.py", line 131, in index_message
MessageSearcher().add_to_index(message)
File "/mnt/deploy/prod/chorus/src/chorus/../chorus/search/__init__.py", line 29, in add_to_index
writer.commit()
File "/usr/local/lib/python2.6/dist-packages/whoosh/writing.py", line 423, in commit
self.writer.commit(*args, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filewriting.py", line 501, in commit
new_segments = mergetype(self, self.segments)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filewriting.py", line 78, in MERGE_SMALL
reader = SegmentReader(writer.storage, writer.schema, seg)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filereading.py", line 63, in __init__
self.termsindex = TermIndexReader(tf)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filetables.py", line 590, in __init__
super(TermIndexReader, self).__init__(dbfile)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filetables.py", line 502, in __init__
OrderedHashReader.__init__(self, dbfile)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filetables.py", line 379, in __init__
HashReader.__init__(self, dbfile)
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/filetables.py", line 187, in __init__
self.hashtype = dbfile.read_byte()
File "/usr/local/lib/python2.6/dist-packages/whoosh/filedb/structfile.py", line 219, in read_byte
return ord(self.file.read(1))
Strangely, the exact same code using a standard writer (i.e. not AsyncWriter) works just fine. What am I missing here? Note that in production I have to use AsyncWriter in order to avoid LockErrors.

This error is caused by some kind of index corruption. In my case the machine crashed by another reason while index was being rebuild.
You can easily solve it by deleting whoosh_index folder contents completely and rebuilidng index.

Ended up finding a solution; it's called Solr :-)