When I tried to start my scrapy demo with scrapy crawl getCommodityInfo, the error below occurred.
C:\Users\柘宇\PycharmProjects\GraduationProject\spiders\bin\JDSpider>scrapy crawl getCommodityInfo
Traceback (most recent call last):
File "D:\Anacaonda\Scripts\scrapy-script.py", line 5, in <module>
sys.exit(scrapy.cmdline.execute())
File "D:\Anacaonda\lib\site-packages\scrapy\cmdline.py", line 141, in execute
cmd.crawler_process = CrawlerProcess(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 238, in __init__
super(CrawlerProcess, self).__init__(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 129, in __init__
self.spider_loader = _get_spider_loader(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 325, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 45, in from_settings
return cls(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 23, in __init__
self._load_all_spiders()
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 32, in _load_all_spiders
for module in walk_modules(name):
File "D:\Anacaonda\lib\site-packages\scrapy\utils\misc.py", line 71, in walk_modules
submod = import_module(fullpath)
File "D:\Anacaonda\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "C:\Users\柘宇\PycharmProjects\GraduationProject\spiders\bin\JDSpider\JDSpider\spiders\getCommodityInfo.py", line 12, in <module>
from spiders.bin.JDSpider.JDSpider.items import JDCommodity
ModuleNotFoundError: No module named 'spiders'
It seems that the spider cannot be found, but I don't know why it happened. My whole project hierarchy is here. GraduationProject is the django project. main and spiders are the applications of django. The bin directory stores two demo scrapy projects. When I entered the JDSpider trying to run it, the error occurred. Could you help me fix it?
PS. My spider name: name = "getCommodityInfo"
With the solution applied by PS1212, the scrapy demo could run. However the pycharm warns like this. What happened?
Because it it not recognizing you project module.
Try this:
from JDSpider.items import JDCommodity
Related
I´m having a problem running my old django project. Its giving an error after i run py manage.py.
I tryed to use the command py manage.py makemigrations
Traceback (most recent call last):File "C:\Users\ricardo martins\Downloads\LES-fosquito\LES-fosquito\manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\core\management\__init__.py", line 446, in execute_from_command_line
utility.execute()
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\core\management\__init__.py", line 420, in execute
django.setup()
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\apps\registry.py", line 124, in populate
app_config.ready()
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\contrib\admin\apps.py", line 27, in ready
self.module.autodiscover()
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\contrib\admin\__init__.py", line 50, in autodiscover
autodiscover_modules("admin", register_to=site)
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\utils\module_loading.py", line 58, in autodiscover_modules
import_module("%s.%s" % (app_config.name, module_to_search))
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "C:\Users\ricardo martins\Downloads\LES-fosquito\LES-fosquito\processes\admin.py", line 4, in <module>
from tinymce.widgets import TinyMCE
ModuleNotFoundError: No module named 'tinymce'
Are you using the TinyMCE library in your code? If you are trying to run it locally, try to see if you have the correct dependencies installed, and if it is not needed you can simply remove the import statemnt.
I am new to Django and was trying to make a Django project inside the virtualenv to learn but getting following error on python manage.py startapp Authentication:
python manage.py startapp Authentication
Traceback (most recent call last):
File "C:\Users\Prompt\Desktop\Django\manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "D:\anaconda\lib\site-packages\django\core\management\__init__.py", line 363, in execute_from_command_line
utility.execute()
File "D:\anaconda\lib\site-packages\django\core\management\__init__.py", line 337, in execute
django.setup()
File "D:\anaconda\lib\site-packages\django\__init__.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "D:\anaconda\lib\site-packages\django\apps\registry.py", line 85, in populate
app_config = AppConfig.create(entry)
File "D:\anaconda\lib\site-packages\django\apps\config.py", line 94, in create
module = import_module(entry)
File "D:\anaconda\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "D:\anaconda\lib\site-packages\django\contrib\admin\__init__.py", line 4, in <module>
from django.contrib.admin.filters import (
File "D:\anaconda\lib\site-packages\django\contrib\admin\filters.py", line 10, in <module>
from django.contrib.admin.options import IncorrectLookupParameters
File "D:\anaconda\lib\site-packages\django\contrib\admin\options.py", line 12, in <module>
from django.contrib.admin import helpers, widgets
File "D:\anaconda\lib\site-packages\django\contrib\admin\widgets.py", line 151
'%s=%s' % (k, v) for k, v in params.items(),
^
SyntaxError: Generator expression must be parenthesized
I have an error when i use django.
Traceback (most recent call last):
File "manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/Users/ghk/djangoprojects/myvenv/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/Users/ghk/djangoprojects/myvenv/lib/python3.6/site-packages/django/core/management/__init__.py", line 308, in execute
settings.INSTALLED_APPS
File "/Users/ghk/djangoprojects/myvenv/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
self._setup(name)
File "/Users/ghk/djangoprojects/myvenv/lib/python3.6/site-packages/django/conf/__init__.py", line 41, in _setup
self._wrapped = Settings(settings_module)
File "/Users/ghk/djangoprojects/myvenv/lib/python3.6/site-packages/django/conf/__init__.py", line 110, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/ghk/djangoprojects/lab-in-the-wild/labinthewild/settings.py", line 18, in <module>
from config import *
ModuleNotFoundError: No module named 'config'
If I command 'python manage.py runserver', it show this error.
As you said in your comments, you cloned this project from git.
Perhaps config is a local settings file you are meant to create yourself. Check the README and also check for a config example of some sorts
I can't figure out how to do python manage.py createsuperuser in my PyDev Django shell.
I was hoping something below would work, but nothing did. I've never really used these shells before, and hope there's a way to create this superuser with a shell.
import manage
python manage.py createsuperuser
File "<ipython-input-15-fcfe38f02d6e>", line 1
python manage.py createsuperuser
^
SyntaxError: invalid syntax
I tried doing it with the Windows cmd, which led to this error:
C:\Users\Rasmus\workspace\Crowd\src>python manage.py createsuperuser
Traceback (most recent call last):
File "C:\Anaconda3\lib\site-packages\django\db\backends\mysql\base.py", line 25, in <module>
import MySQLdb as Database
ImportError: No module named 'MySQLdb'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "C:\Anaconda3\lib\site-packages\django\core\management\__init__.py", line 353, in execute_from_command_line
utility.execute()
File "C:\Anaconda3\lib\site-packages\django\core\management\__init__.py", line 327, in execute
django.setup()
File "C:\Anaconda3\lib\site-packages\django\__init__.py", line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "C:\Anaconda3\lib\site-packages\django\apps\registry.py", line 108, in populate
app_config.import_models(all_models)
File "C:\Anaconda3\lib\site-packages\django\apps\config.py", line 202, in import_models
self.models_module = import_module(models_module_name)
File "C:\Anaconda3\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 662, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "C:\Users\Rasmus\workspace\Crowd\src\Cr\models.py", line 3, in <module>
class User(models.Model):
File "C:\Anaconda3\lib\site-packages\django\db\models\base.py", line 108, in __new__
new_class.add_to_class('_meta', Options(meta, app_label))
File "C:\Anaconda3\lib\site-packages\django\db\models\base.py", line 307, in add_to_class
value.contribute_to_class(cls, name)
File "C:\Anaconda3\lib\site-packages\django\db\models\options.py", line 263, in contribute_to_class
self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "C:\Anaconda3\lib\site-packages\django\db\__init__.py", line 36, in __getattr__
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "C:\Anaconda3\lib\site-packages\django\db\utils.py", line 212, in __getitem__
backend = load_backend(db['ENGINE'])
File "C:\Anaconda3\lib\site-packages\django\db\utils.py", line 116, in load_backend
return import_module('%s.base' % backend_name)
File "C:\Anaconda3\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 662, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "C:\Anaconda3\lib\site-packages\django\db\backends\mysql\base.py", line 28, in <module>
raise ImproperlyConfigured("Error loading MySQLdb module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: No module named 'MySQLdb'
Which I find weird seeing as my Django is connected to my database and that I can store and retrieve objects from the database within PyDev.
Are there other ways to createsuperuser than using shell or cmd? Like doing it with a method?
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")
from django.core.management import execute_from_command_line
execute_from_command_line(["manage.py","createsuperuser"])
Based on the error information "loading MySQLdb module: No module named 'MySQLdb'", you need to install MySQL-python
I'm having the following error when trying to use Scrapy in Python 3.
I'm using Ubuntu 16.04 and installing python3-twisted didn't solve the issue.
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/crawler.py", line 150, in crawl
crawler = self._create_crawler(crawler_or_spidercls)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/crawler.py", line 166, in _create_crawler
return Crawler(spidercls, self.settings)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/crawler.py", line 46, in __init__
self.extensions = ExtensionManager.from_crawler(self)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/middleware.py", line 56, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/middleware.py", line 32, in from_settings
mwcls = load_object(clspath)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/utils/misc.py", line 44, in load_object
mod = import_module(module)
File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 662, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/extensions/memusage.py", line 16, in <module>
from scrapy.mail import MailSender
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/mail.py", line 23, in <module>
from twisted.mail.smtp import ESMTPSenderFactory
ImportError: No module named 'twisted.mail.smtp'
Scrapy's Python 3 support is still in a beta stage and (according to the release notes)
Sending emails is not supported
so it's unlikely you can do anything to fix this.
Scrapy does not support sending mail with Python 3.
It's a limitation of Twisted, which has not ported it's twisted.mail module yet!
(Unfortunately, Pull request is pending)
See:
Twisted #8770
[#8770] Port twisted.mail.smtp to Python 3 #509