I have a Scrapy spider written in python 3 and I want to run it as a cron job on my cloud Linux server (I have root access)
first, I couldn't install using pip3 install scrapy, I faced :
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/tarfile.py", line 1642, in bz2open
import bz2
File "/usr/local/lib/python3.4/bz2.py", line 20, in <module>
from _bz2 import BZ2Compressor, BZ2Decompressor
ImportError: No module named '_bz2'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/local/lib/python3.4/site-packages/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/usr/local/lib/python3.4/site-packages/pip/req.py", line 1197, in prepare_files
do_download,
File "/usr/local/lib/python3.4/site-packages/pip/req.py", line 1375, in unpack_url
self.session,
File "/usr/local/lib/python3.4/site-packages/pip/download.py", line 582, in unpack_http_url
unpack_file(temp_location, location, content_type, link)
File "/usr/local/lib/python3.4/site-packages/pip/util.py", line 625, in unpack_file
untar_file(filename, location)
File "/usr/local/lib/python3.4/site-packages/pip/util.py", line 543, in untar_file
tar = tarfile.open(filename, mode)
File "/usr/local/lib/python3.4/tarfile.py", line 1567, in open
return func(name, filemode, fileobj, **kwargs)
File "/usr/local/lib/python3.4/tarfile.py", line 1644, in bz2open
raise CompressionError("bz2 module is not available")
tarfile.CompressionError: bz2 module is not available
then how can I run it as a cron job ?
Related
As the title says, I'm trying to install pylint with the below command:
pip install pylint
But I get the below exception:
Exception:
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\pip\_vendor\pkg_resources.py", line 2482, in _dep_map
return self.__dep_map
File "C:\Python34\lib\site-packages\pip\_vendor\pkg_resources.py", line 2344, in __getattr__
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\pip\basecommand.py", line 122, in main
status = self.run(options, args)
File "C:\Python34\lib\site-packages\pip\commands\install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "C:\Python34\lib\site-packages\pip\req.py", line 1265, in prepare_files
req_to_install.extras):
File "C:\Python34\lib\site-packages\pip\_vendor\pkg_resources.py", line 2291, in requires
dm = self._dep_map
File "C:\Python34\lib\site-packages\pip\_vendor\pkg_resources.py", line 2484, in _dep_map
self.__dep_map = self._compute_dependencies()
File "C:\Python34\lib\site-packages\pip\_vendor\pkg_resources.py", line 2508, in _compute_dependencies
parsed = next(parse_requirements(distvers))
File "C:\Python34\lib\site-packages\pip\_vendor\pkg_resources.py", line 2605, in parse_requirements
line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
File "C:\Python34\lib\site-packages\pip\_vendor\pkg_resources.py", line 2583, in scan_list
"Expected ',' or end-of-list in",line,"at",line[p:]
ValueError: ("Expected ',' or end-of-list in", 'lazy-object-proxy ==1.4.*', 'at', '*')
Storing debug log for failure in C:\Users\Haiss\pip\pip.log
Anybody have any ideas what's causing it to fail? I have Python 3 installed using Anaconda. I checked the debug log but it doesn't provide any additional details.
try this:
easy_install --upgrade pip
Running pytest through CLI with pytest-xdist runs fine (pytest -n 3 {test}). When running the same set up in PyCharm I get Context version conflict, stating that I need to lower my pytest version. When I lower the version to below 4, pycharm is able to run the pytest script through the pytest runner.
Here is the requirements.txt:
apipkg==1.5
atomicwrites==1.3.0
attrs==19.1.0
execnet==1.5.0
more-itertools==6.0.0
pluggy==0.9.0
py==1.8.0
pytest==4.3.1
pytest-forked==1.0.2
pytest-rerunfailures==6.0
pytest-xdist==1.27.0
selenium==3.141.0
six==1.12.0
urllib3==1.24.1
and the error:
Traceback (most recent call last):
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pluggy/manager.py", line 274, in load_setuptools_entrypoints
plugin = ep.load()
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pkg_resources/init.py", line 2342, in load
self.require(*args, **kwargs)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pkg_resources/init.py", line 2365, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pkg_resources/init.py", line 786, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pytest 4.3.1 (/Users//.pyenv/versions/3.6.7/lib/python3.6/site-packages), Requirement.parse('pytest<4.0.0'), {'pytest-fixture-config'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/helpers/pycharm/_jb_pytest_runner.py", line 31, in <module>
pytest.main(args, plugins_to_load)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/_pytest/config/init.py", line 59, in main
config = _prepareconfig(args, plugins)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/_pytest/config/init.py", line 194, in _prepareconfig
pluginmanager=pluginmanager, args=args
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pluggy/hooks.py", line 289, in call
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pluggy/manager.py", line 68, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pluggy/manager.py", line 62, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pluggy/callers.py", line 203, in _multicall
gen.send(outcome)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/_pytest/helpconfig.py", line 93, in pytest_cmdline_parse
config = outcome.get_result()
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/_pytest/config/init.py", line 653, in pytest_cmdline_parse
self.parse(args)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/_pytest/config/init.py", line 870, in parse
self._preparse(args, addopts=addopts)
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/_pytest/config/init.py", line 815, in _preparse
self.pluginmanager.load_setuptools_entrypoints("pytest11")
File "/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages/pluggy/manager.py", line 280, in load_setuptools_entrypoints
message="Plugin %r could not be loaded: %s!" % (ep.name, e),
pluggy.manager.PluginValidationError: Plugin 'webdriver' could not be loaded: (pytest 4.3.1 (/Users/{name}/.pyenv/versions/3.6.7/lib/python3.6/site-packages), Requirement.parse('pytest<4.0.0'), {'pytest-fixture-config'})!
Currently running through venv on the command line.
I want to be able to run the same setup in pycharm as I do from CLI
Solved the issue: PyCharm had pytest-fixture-config installed. I had to remove the package and running my tests worked. pytest-fixture-config states Pinned pytest<4.0.0 until all deprecation warnings are fixed.
When I run django server, It says "ModuleNotFoundError: No module named 'ws4redis'".
When I try to install ws4redis using the PIP command, I says syntax error.
pip install django-websocket-redis
Here is the tracebacks of the errors:
Collecting django-websocket-redis
Using cached django-websocket-redis-0.5.1.tar.gz
Exception:
Traceback (most recent call last):
File "/anaconda3/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/anaconda3/lib/python3.6/site-packages/pip/commands/install.py", line 335, in run
wb.build(autobuilding=True)
File "/anaconda3/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/anaconda3/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/anaconda3/lib/python3.6/site-packages/pip/req/req_set.py", line 634, in _prepare_file
abstract_dist.prep_for_dist()
File "/anaconda3/lib/python3.6/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
self.req_to_install.run_egg_info()
File "/anaconda3/lib/python3.6/site-packages/pip/req/req_install.py", line 412, in run_egg_info
self.setup_py, self.name,
File "/anaconda3/lib/python3.6/site-packages/pip/req/req_install.py", line 387, in setup_py
import setuptools # noqa
File "/Users/ranjan/.local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/setuptools/__init__.py", line 2, in <module>
from setuptools.extension import Extension, Library
File "/Users/ranjan/.local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/setuptools/extension.py", line 5, in <module>
from setuptools.dist import _get_unpatched
File "/Users/ranjan/.local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/setuptools/dist.py", line 103
except ValueError, e:
^
SyntaxError: invalid syntax
I want put my working app on Heroku. The application using Django and chat based on Twisted websockets. I don't know it's good approach but I installed twisted via heroku bash, but when i want to run my chatserver.py I have an exception because some Twisted module requires OpenSSL which is not installed on my env.
So simply question, how to install Twisted to work with websocket on Heroku?
Update
when I run twistd -n -y chatserver.py:
Unhandled Error
Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/Twisted-13.1.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 652, in run
runApp(config)
File "/app/.heroku/python/lib/python2.7/site-packages/Twisted-13.1.0-py2.7-linux-x86_64.egg/twisted/scripts/twistd.py", line 23, in runApp
_SomeApplicationRunner(config).run()
File "/app/.heroku/python/lib/python2.7/site-packages/Twisted-13.1.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 386, in run
self.application = self.createOrGetApplication()
File "/app/.heroku/python/lib/python2.7/site-packages/Twisted-13.1.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 451, in createOrGetApplication
application = getApplication(self.config, passphrase)
--- <exception caught here> ---
File "/app/.heroku/python/lib/python2.7/site-packages/Twisted-13.1.0-py2.7-linux-x86_64.egg/twisted/application/app.py", line 462, in getApplication
application = service.loadApplication(filename, style, passphrase)
File "/app/.heroku/python/lib/python2.7/site-packages/Twisted-13.1.0-py2.7-linux-x86_64.egg/twisted/application/service.py", line 405, in loadApplication
application = sob.loadValueFromFile(filename, 'application', passphrase)
File "/app/.heroku/python/lib/python2.7/site-packages/Twisted-13.1.0-py2.7-linux-x86_64.egg/twisted/persisted/sob.py", line 210, in loadValueFromFile
exec fileObj in d, d
File "chatserver.py", line 37, in <module>
from twisted.web.websockets import WebSocketsResource, lookupProtocolForFactory
File "/app/.heroku/python/lib/python2.7/site-packages/Twisted-13.1.0-py2.7-linux-x86_64.egg/twisted/web/websockets.py", line 28, in <module>
from twisted.protocols.tls import TLSMemoryBIOProtocol
File "/app/.heroku/python/lib/python2.7/site-packages/Twisted-13.1.0-py2.7-linux-x86_64.egg/twisted/protocols/tls.py", line 40, in <module>
from OpenSSL.SSL import Error, ZeroReturnError, WantReadError
exceptions.ImportError: No module named OpenSSL.SSL
Failed to load application: No module named OpenSSL.SSL
So when I try to install openssl with command: pip install pyopenssl
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_u12732/cryptography/setup.py", line 174, in <module>
"test": PyTest,
File "/app/.heroku/python/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 260, in __init__
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 285, in fetch_build_eggs
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 631, in resolve
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 871, in best_match
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 883, in obtain
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 335, in fetch_build_egg
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 595, in easy_install
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 625, in install_item
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 822, in install_eggs
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1028, in build_and_install
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 1016, in run_setup
distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1
----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_u12732/cryptography
Traceback (most recent call last):
File "/app/.heroku/python/bin/pip", line 9, in <module>
load_entry_point('pip==1.5.6', 'console_scripts', 'pip')()
File "/app/.heroku/python/lib/python2.7/site-packages/pip-1.5.6-py2.7.egg/pip/__init__.py", line 185, in main
return command.main(cmd_args)
File "/app/.heroku/python/lib/python2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 75: ordinal not in range(128)
I try to install distribute with python 3.3 on Ubuntu 12.04 to install pymongo.
I downloaded distribute_setup.py with
curl -O http://python-distribute.org/distribute_setup.py
Afterwards I tried
python3 distribute_setup.py install
But I recive
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.49.tar.gz
Traceback (most recent call last):
File "distribute_setup.py", line 556, in <module>
sys.exit(main())
File "distribute_setup.py", line 552, in main
tarball = download_setuptools(download_base=options.download_base)
File "distribute_setup.py", line 211, in download_setuptools
src = urlopen(url)
File "/usr/local/lib/python3.3/urllib/request.py", line 156, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.3/urllib/request.py", line 475, in open
response = meth(req, response)
File "/usr/local/lib/python3.3/urllib/request.py", line 587, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python3.3/urllib/request.py", line 507, in error
result = self._call_chain(*args)
File "/usr/local/lib/python3.3/urllib/request.py", line 447, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.3/urllib/request.py", line 692, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/local/lib/python3.3/urllib/request.py", line 469, in open
response = self._open(req, data)
File "/usr/local/lib/python3.3/urllib/request.py", line 492, in _open
'unknown_open', req)
File "/usr/local/lib/python3.3/urllib/request.py", line 447, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.3/urllib/request.py", line 1310, in unknown_open
raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: https>
Can someone please help me? It seems as if distribute uses code, that does not compile with python 3.3..
I installed Python 3.3.2 in my homefolder following these steps:
./configure
make
make test
sudo make install
If I try to install pymongo from source I recive:
Extracting in /tmp/tmpth91z4
Traceback (most recent call last):
File "setup.py", line 22, in <module>
from setuptools import setup, Feature
ImportError: No module named 'setuptools'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dev/pymongo/ez_setup.py", line 130, in use_setuptools
import pkg_resources
ImportError: No module named 'pkg_resources'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "setup.py", line 25, in <module>
use_setuptools()
File "/home/dev/pymongo/ez_setup.py", line 132, in use_setuptools
return _do_download(version, download_base, to_dir, download_delay)
File "/home/dev/pymongo/ez_setup.py", line 111, in _do_download
_build_egg(egg, tarball, to_dir)
File "/home/dev/pymongo/ez_setup.py", line 83, in _build_egg
tar = tarfile.open(tarball)
File "/usr/local/lib/python3.3/tarfile.py", line 1571, in open
raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully
Thank you
Are you sure you installed openssl-devel before compiling Python? I have had the same issue, however I fixed it by previously installing openssl-devel and then compiling Python 3 + distribute.