On the command line I'm executing the following simple attempt to invoke scrapy:
scrapy version
I get the following error:
$ scrapy version
Traceback (most recent call last):
File "/Users/nathanielford/virtualenvironments/crawler/bin/scrapy", line 11, in <module>
sys.exit(execute())
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/cmdline.py", line 141, in execute
cmd.crawler_process = CrawlerProcess(settings)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/crawler.py", line 238, in __init__
super(CrawlerProcess, self).__init__(settings)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/crawler.py", line 129, in __init__
self.spider_loader = _get_spider_loader(settings)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/crawler.py", line 325, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/spiderloader.py", line 33, in from_settings
return cls(settings)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/spiderloader.py", line 20, in __init__
self._load_all_spiders()
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/spiderloader.py", line 28, in _load_all_spiders
for module in walk_modules(name):
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/utils/misc.py", line 63, in walk_modules
mod = import_module(path)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
ImportError: No module named 'spiders'
If I leave my project directory, the error no longer occurs:
$ scrapy version
Scrapy 1.2.2
What is causing the ImportError: No module named 'spiders'?
This error was caused by my settings.py file, where the I had the following:
SPIDER_MODULES = ['spiders']
While it is in the default project provided by scrapy, I had moved that module. Values in that list need to be fully qualified module names: even trivial executions of scrapy will load the settings file, and thus try to import modules from your project. If it can find a settings file, it will need to be correct.
Related
I have created my project on Linux mint 19 with python 3.6 installed. I used pytest in conjunction with pytest-qt to test my app which is a PyQt5 program.
Then, I've updated to Linux mint 20. However python 3.8 is installed there instead of 3.6. I don't know whether it's related or not but now I can't run my test suite anymore, when I type pytest in terminal it gives me the following output:
Traceback (most recent call last):
File "/home/kotlin/.local/bin/pytest", line 8, in <module>
sys.exit(console_main())
File "/home/kotlin/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 187, in console_main
code = main()
File "/home/kotlin/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 143, in main
config = _prepareconfig(args, plugins)
File "/home/kotlin/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig
config = pluginmanager.hook.pytest_cmdline_parse(
File "/home/kotlin/.local/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/home/kotlin/.local/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/home/kotlin/.local/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
File "/home/kotlin/.local/lib/python3.8/site-packages/pluggy/callers.py", line 203, in _multicall
gen.send(outcome)
File "/home/kotlin/.local/lib/python3.8/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
config = outcome.get_result() # type: Config
File "/home/kotlin/.local/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File "/home/kotlin/.local/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/home/kotlin/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
self.parse(args)
File "/home/kotlin/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1280, in parse
self._preparse(args, addopts=addopts)
File "/home/kotlin/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1172, in _preparse
self.pluginmanager.load_setuptools_entrypoints("pytest11")
File "/home/kotlin/.local/lib/python3.8/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
plugin = ep.load()
File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/home/kotlin/.local/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 171, in exec_module
exec(co, module.__dict__)
File "/home/kotlin/.local/lib/python3.8/site-packages/pytestqt/plugin.py", line 7, in <module>
from pytestqt.qt_compat import QtCore, QtGui, QtTest
File "/home/kotlin/.local/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 171, in exec_module
exec(co, module.__dict__)
File "/home/kotlin/.local/lib/python3.8/site-packages/pytestqt/qt_compat.py", line 31, in <module>
import PyQt4.QtCore as _QtCore
ModuleNotFoundError: No module named 'PyQt4'
At the end it says ModuleNotFoundError: No module named 'PyQt4' but I don't even use PyQt4, I use PyQt5 instead.
I have pytest and pytest-qt both installed. The program itself works properly.
Thanks.
This part of the error message:
File "/home/kotlin/.local/lib/python3.8/site-packages/pytestqt/qt_compat.py", line 31, in <module>
import PyQt4.QtCore as _QtCore
ModuleNotFoundError: No module named 'PyQt4'
hints at the problem - the used qt-pytest version still references PyQt4, while the current version has these references removed some time ago -- meaning the installed python-qt version is outdated. The version can be updated using
pip3 install -U pytest-qt
as mentioned in the comments. It is still unclear why an old version was installed in the first place - one possibility is that an old version already had been installed by some other package, installing it via pip (without the update option) would not do anything in this case.
I am very new to python and linux, I was trying to create an app from a tutorial and this happend when I tried to run a test of my model.
(venv) kali#kali:~/machinelearning/backend/server$ python manage.py test apps.ml.tests
Traceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 357, in execute
django.setup()
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/apps/registry.py", line 91, in populate
app_config = AppConfig.create(entry)
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/apps/config.py", line 116, in create
mod = import_module(mod_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'apps.endpointsapps'
This is my first question on stackoverflow too
Please add empty __init__.py file in the apps folder so that it can be recognized by Python as a package.
Check this link out. link
Can you please post the project structure you have? Basically just want to see where that apps exists.
You need to add an empty init.py (4 underscores in total) file in the apps folder for it to be recognized by Python as a package.
Have a look at the documentation for more informations.
I am getting below error while running my scrapy project. I tried everything suggested on stackoverflow but yet nothing has solved the problem.
Feel free to ask for more information. Looking forward for any help.
(venv) [kalpesh#localhost scraper]$ scrapy crawl mrdeepfakes -a output=db
Traceback (most recent call last):
File "/home/kalpesh/venv/bin/scrapy", line 8, in <module>
sys.exit(execute())
File "/home/kalpesh/venv/lib/python3.6/site-packages/scrapy/cmdline.py", line 113, in execute
settings = get_project_settings()
File "/home/kalpesh/venv/lib/python3.6/site-packages/scrapy/utils/project.py", line 69, in get_project_settings
settings.setmodule(settings_module_path, priority='project')
File "/home/kalpesh/venv/lib/python3.6/site-packages/scrapy/settings/__init__.py", line 287, in setmodule
module = import_module(module)
File "/home/kalpesh/venv/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'scraper.settings'
Make sure your scrapy.cfg file has the same default and project name as your spider crawler name inside the spiders folder.
I tried changing my spider crawler name in the python file and subsequently modifying the scrapy.cfg file and it works.
Screenshot of scrapy.cfg
When I tried to start my scrapy demo with scrapy crawl getCommodityInfo, the error below occurred.
C:\Users\柘宇\PycharmProjects\GraduationProject\spiders\bin\JDSpider>scrapy crawl getCommodityInfo
Traceback (most recent call last):
File "D:\Anacaonda\Scripts\scrapy-script.py", line 5, in <module>
sys.exit(scrapy.cmdline.execute())
File "D:\Anacaonda\lib\site-packages\scrapy\cmdline.py", line 141, in execute
cmd.crawler_process = CrawlerProcess(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 238, in __init__
super(CrawlerProcess, self).__init__(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 129, in __init__
self.spider_loader = _get_spider_loader(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 325, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 45, in from_settings
return cls(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 23, in __init__
self._load_all_spiders()
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 32, in _load_all_spiders
for module in walk_modules(name):
File "D:\Anacaonda\lib\site-packages\scrapy\utils\misc.py", line 71, in walk_modules
submod = import_module(fullpath)
File "D:\Anacaonda\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "C:\Users\柘宇\PycharmProjects\GraduationProject\spiders\bin\JDSpider\JDSpider\spiders\getCommodityInfo.py", line 12, in <module>
from spiders.bin.JDSpider.JDSpider.items import JDCommodity
ModuleNotFoundError: No module named 'spiders'
It seems that the spider cannot be found, but I don't know why it happened. My whole project hierarchy is here. GraduationProject is the django project. main and spiders are the applications of django. The bin directory stores two demo scrapy projects. When I entered the JDSpider trying to run it, the error occurred. Could you help me fix it?
PS. My spider name: name = "getCommodityInfo"
With the solution applied by PS1212, the scrapy demo could run. However the pycharm warns like this. What happened?
Because it it not recognizing you project module.
Try this:
from JDSpider.items import JDCommodity
I'm running Django 1.8 and Python 3.4 on Ubuntu 14.04 LTS. Just recently, my Django app has been reporting that GEOS is not present. GEOS is installed and libgeos_c.so is where it's supposed to be (/usr/lib/). My code seems fine. It is the source of a docker image which still works. This seems to indicate an os/incompatibility issue. Any help would be much appreciated.
The full traceback is
Traceback (most recent call last):
File "<path/to/my/homedir>/pycharm-4.5.1/helpers/pydev/pydevd.py", line 2358, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "<path/to/my/homedir>/pycharm-4.5.1/helpers/pydev/pydevd.py", line 1778, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "<path/to/my/homedir>/pycharm-4.5.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "<path/to/my/pycharm/project>/manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/core/management/__init__.py", line 312, in execute
django.setup()
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/__init__.py", line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/apps/config.py", line 198, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python3.4/importlib/__init__.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 2231, in _gcd_import
File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1129, in _exec
File "<frozen importlib._bootstrap>", line 1448, in exec_module
File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/contrib/auth/models.py", line 41, in <module>
class Permission(models.Model):
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/db/models/base.py", line 139, in __new__
new_class.add_to_class('_meta', Options(meta, **kwargs))
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/db/models/base.py", line 324, in add_to_class
value.contribute_to_class(cls, name)
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/db/models/options.py", line 250, in contribute_to_class
self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/db/__init__.py", line 36, in __getattr__
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/db/utils.py", line 240, in __getitem__
backend = load_backend(db['ENGINE'])
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/db/utils.py", line 111, in load_backend
return import_module('%s.base' % backend_name)
File "/usr/lib/python3.4/importlib/__init__.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 2231, in _gcd_import
File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1129, in _exec
File "<frozen importlib._bootstrap>", line 1448, in exec_module
File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 8, in <module>
from .features import DatabaseFeatures
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/contrib/gis/db/backends/base/features.py", line 3, in <module>
from django.contrib.gis.db.models import aggregates
File "<path/to/my/virtualenv>/lib/python3.4/site-packages/django/contrib/gis/db/models/__init__.py", line 12, in <module>
"See also https://docs.djangoproject.com/en/%s/ref/contrib/gis/install/geolibs/" % get_docs_version())
django.core.exceptions.ImproperlyConfigured: GEOS is required and has not been detected. Are you sure it is installed? See also https://docs.djangoproject.com/en/1.8/ref/contrib/gis/install/geolibs/
I've tried reinstalling libgdal-dev. All relevant packages (GDAL, GEOS etc) are up to date.
I've tried working my way back through the Python code. it fails in djanjo/contrib/gis/geos/__init__.py on the following. As HAS_GEOS is set to false it reports that GEOS is not present.
try:
from .libgeos import geos_version, geos_version_info # NOQA: flake8 detects only the last __all__
HAS_GEOS = True
__all__ += ['geos_version', 'geos_version_info']
except ImportError:
HAS_GEOS = False
I suspect that a recent OS upgrade/patch has broken GEOS or GDAL. The root cause seems to be something to do with ctypes, but I can't see what it could be.
The following might be useful.
>>> from _ctypes import Union, Structure, Array
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/<path/to/my/homedir>/pycharm-4.5.1/helpers/pydev/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
ImportError: /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so: undefined symbol: _PyTraceback_Add
>>> from django.contrib.gis import gdal
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/<path/to/my/homedir>/pycharm-4.5.1/helpers/pydev/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "/<path/to/my/homedir>/pycharm-4.5.1/helpers/pydev/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "/<path/to/my/virtualenv>/lib/python3.4/site-packages/django/contrib/gis/gdal/__init__.py", line 47, in <module>
from django.contrib.gis.gdal.driver import Driver # NOQA
File "/<path/to/my/homedir>/pycharm-4.5.1/helpers/pydev/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "/<path/to/my/virtualenv>/lib/python3.4/site-packages/django/contrib/gis/gdal/driver.py", line 1, in <module>
from ctypes import c_void_p
File "/<path/to/my/homedir>/pycharm-4.5.1/helpers/pydev/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "/usr/lib/python3.4/ctypes/__init__.py", line 7, in <module>
from _ctypes import Union, Structure, Array
File "/<path/to/my/homedir>/pycharm-4.5.1/helpers/pydev/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
ImportError: /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so: undefined symbol: _PyTraceback_Add
I had the same problem today, though in an unrelated python project. This is is the line I also encountered and which led me here:
ImportError: /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so: undefined symbol: _PyTraceback_Add
It looks like Ubunut has pushed a Python 3.4 update which is not compatible with existing virtual environments. I'm able to fix the problem by recreating the virtualenv. Where this was impractical for the moment I simply replaced the virtualenv's python with the system one:
cd my-virtualenv
cp /usr/bin/python3.4 bin/python3.4
I can't say if this is a very sane thing to do, but it does seem to work in my case.
You should do
cp /usr/bin/python3 /path/to/my-virtualenv/bin/python3
instead, because that's the actual non-symlink file.