I am getting below error while running my scrapy project. I tried everything suggested on stackoverflow but yet nothing has solved the problem.
Feel free to ask for more information. Looking forward for any help.
(venv) [kalpesh#localhost scraper]$ scrapy crawl mrdeepfakes -a output=db
Traceback (most recent call last):
File "/home/kalpesh/venv/bin/scrapy", line 8, in <module>
sys.exit(execute())
File "/home/kalpesh/venv/lib/python3.6/site-packages/scrapy/cmdline.py", line 113, in execute
settings = get_project_settings()
File "/home/kalpesh/venv/lib/python3.6/site-packages/scrapy/utils/project.py", line 69, in get_project_settings
settings.setmodule(settings_module_path, priority='project')
File "/home/kalpesh/venv/lib/python3.6/site-packages/scrapy/settings/__init__.py", line 287, in setmodule
module = import_module(module)
File "/home/kalpesh/venv/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'scraper.settings'
Make sure your scrapy.cfg file has the same default and project name as your spider crawler name inside the spiders folder.
I tried changing my spider crawler name in the python file and subsequently modifying the scrapy.cfg file and it works.
Screenshot of scrapy.cfg
Related
I am very new to python and linux, I was trying to create an app from a tutorial and this happend when I tried to run a test of my model.
(venv) kali#kali:~/machinelearning/backend/server$ python manage.py test apps.ml.tests
Traceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 357, in execute
django.setup()
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/apps/registry.py", line 91, in populate
app_config = AppConfig.create(entry)
File "/home/kali/machinelearning/venv/lib/python3.7/site-packages/django/apps/config.py", line 116, in create
mod = import_module(mod_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'apps.endpointsapps'
This is my first question on stackoverflow too
Please add empty __init__.py file in the apps folder so that it can be recognized by Python as a package.
Check this link out. link
Can you please post the project structure you have? Basically just want to see where that apps exists.
You need to add an empty init.py (4 underscores in total) file in the apps folder for it to be recognized by Python as a package.
Have a look at the documentation for more informations.
I have this new remote job, where I had to clone all the code from a repository, and I have to make an export of the database from MySQL hosted in RDS.
The first problem is that when I set up the configuration to start the app, it raise an error telling me this:
Run Configuration Error: Broken configuration due to unavailable plugin or invalid configuration data.
The other thing is that I already have the data dumped and set up in my local storage (the app works this way, is no longer using AWS Cloud) but when I try to do an python manage.py migrate , this error comes up...
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "C:\Users\Tony-App\Documents\App\venv\lib\site-packages\django\core\management\__init__.py", line 338, in execute_from_command_line
utility.execute()
File "C:\Users\Tony-App\Documents\App\venv\lib\site-packages\django\core\management\__init__.py", line 312, in execute
django.setup()
File "C:\Users\Tony-App\Documents\App\venv\lib\site-packages\django\__init__.py", line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "C:\Users\Tony-App\Documents\App\venv\lib\site-packages\django\apps\registry.py", line 85, in populate
app_config = AppConfig.create(entry)
File "C:\Users\Tony-App\Documents\App\venv\lib\site-packages\django\apps\config.py", line 86, in create
module = import_module(entry)
File "C:\Users\Tony-App\AppData\Local\Programs\Python\Python37-32\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "C:\Users\Tony-App\Documents\App\venv\lib\site-packages\liststyle\__init__.py", line 1, in <module>
from admin import ListStyleAdminMixin
ModuleNotFoundError: No module named 'admin'
First, django-liststyle is a dependency in your project, so make sure it has been installed.
py -m pip install django-liststyle
Second, make sure to add that package to INSTALLED_APPS:
INSTALLED_APPS = {
...
'liststyle',
...
'django.contrib.admin',
...
}
Third, I think the error you are getting is because of the way you are importing ListStyleAdminMixin. This should be the correct way to import it:
from liststyle.admin import ListStyleAdminMixin
When I tried to start my scrapy demo with scrapy crawl getCommodityInfo, the error below occurred.
C:\Users\柘宇\PycharmProjects\GraduationProject\spiders\bin\JDSpider>scrapy crawl getCommodityInfo
Traceback (most recent call last):
File "D:\Anacaonda\Scripts\scrapy-script.py", line 5, in <module>
sys.exit(scrapy.cmdline.execute())
File "D:\Anacaonda\lib\site-packages\scrapy\cmdline.py", line 141, in execute
cmd.crawler_process = CrawlerProcess(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 238, in __init__
super(CrawlerProcess, self).__init__(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 129, in __init__
self.spider_loader = _get_spider_loader(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 325, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 45, in from_settings
return cls(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 23, in __init__
self._load_all_spiders()
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 32, in _load_all_spiders
for module in walk_modules(name):
File "D:\Anacaonda\lib\site-packages\scrapy\utils\misc.py", line 71, in walk_modules
submod = import_module(fullpath)
File "D:\Anacaonda\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "C:\Users\柘宇\PycharmProjects\GraduationProject\spiders\bin\JDSpider\JDSpider\spiders\getCommodityInfo.py", line 12, in <module>
from spiders.bin.JDSpider.JDSpider.items import JDCommodity
ModuleNotFoundError: No module named 'spiders'
It seems that the spider cannot be found, but I don't know why it happened. My whole project hierarchy is here. GraduationProject is the django project. main and spiders are the applications of django. The bin directory stores two demo scrapy projects. When I entered the JDSpider trying to run it, the error occurred. Could you help me fix it?
PS. My spider name: name = "getCommodityInfo"
With the solution applied by PS1212, the scrapy demo could run. However the pycharm warns like this. What happened?
Because it it not recognizing you project module.
Try this:
from JDSpider.items import JDCommodity
On the command line I'm executing the following simple attempt to invoke scrapy:
scrapy version
I get the following error:
$ scrapy version
Traceback (most recent call last):
File "/Users/nathanielford/virtualenvironments/crawler/bin/scrapy", line 11, in <module>
sys.exit(execute())
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/cmdline.py", line 141, in execute
cmd.crawler_process = CrawlerProcess(settings)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/crawler.py", line 238, in __init__
super(CrawlerProcess, self).__init__(settings)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/crawler.py", line 129, in __init__
self.spider_loader = _get_spider_loader(settings)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/crawler.py", line 325, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/spiderloader.py", line 33, in from_settings
return cls(settings)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/spiderloader.py", line 20, in __init__
self._load_all_spiders()
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/spiderloader.py", line 28, in _load_all_spiders
for module in walk_modules(name):
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/site-packages/scrapy/utils/misc.py", line 63, in walk_modules
mod = import_module(path)
File "/Users/nathanielford/virtualenvironments/crawler/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
ImportError: No module named 'spiders'
If I leave my project directory, the error no longer occurs:
$ scrapy version
Scrapy 1.2.2
What is causing the ImportError: No module named 'spiders'?
This error was caused by my settings.py file, where the I had the following:
SPIDER_MODULES = ['spiders']
While it is in the default project provided by scrapy, I had moved that module. Values in that list need to be fully qualified module names: even trivial executions of scrapy will load the settings file, and thus try to import modules from your project. If it can find a settings file, it will need to be correct.
I recently refactored a lot of code and wanted a clean environment, so I deleted and recreated the database schema, created a new venv, and installed dependencies from pip3 one-by-one so I didn't have any superfluous packages left over from the old environment. I quickly installed half a dozen packages, then migrate passed. However, runserver complaines that axes.middleware isn't found (it is installed).
I have django-axes 2.2.0 installed in my virtual environment (Edit: python 3.5, django 1.10).
I verfied the installation was present with pip3 freeze, after
uninstalling and reinstalling just to be sure. django-axes==2.2.0
I have axes listed in INSTALLED_APPS
I have axes.middleware.FailedLoginMiddleware listed in MIDDLEWARE_CLASSES. Note that if I comment out this line, django doesn't attempt to import axes.middleware and consequently runserver succeeds.
I can do import axes; axes.get_version() and also from axes.decorators import watch_login on the shell, so clearly axes is available to the environment.
What is going wrong here?
Traceback from ./manage.py runserver:
Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x7f2d43a381e0>
Traceback (most recent call last):
File "/webapps/my_app/lib/python3.5/site-packages/django/core/servers/basehttp.py", line 49, in get_internal_wsgi_application
return import_string(app_path)
File "/webapps/my_app/lib/python3.5/site-packages/django/utils/module_loading.py", line 20, in import_string
module = import_module(module_path)
File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 665, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/webapps/my_app/wsgi.py", line 14, in <module>
application = get_wsgi_application()
File "/webapps/my_app/lib/python3.5/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application
return WSGIHandler()
File "/webapps/my_app/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 153, in __init__
self.load_middleware()
File "/webapps/my_app/lib/python3.5/site-packages/django/core/handlers/base.py", line 56, in load_middleware
mw_class = import_string(middleware_path)
File "/webapps/my_app/lib/python3.5/site-packages/django/utils/module_loading.py", line 20, in import_string
module = import_module(module_path)
File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 956, in _find_and_load_unlocked
ImportError: No module named 'axes.middleware'
As of 2.0.0, django-axes has default_app_config so you can just use axes in INSTALLED_APPS without installing middleware. Hence, delete the relevant MIDDLEWARE_CLASSES line in settings.py