I'm having the following error when trying to use Scrapy in Python 3.
I'm using Ubuntu 16.04 and installing python3-twisted didn't solve the issue.
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/crawler.py", line 150, in crawl
crawler = self._create_crawler(crawler_or_spidercls)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/crawler.py", line 166, in _create_crawler
return Crawler(spidercls, self.settings)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/crawler.py", line 46, in __init__
self.extensions = ExtensionManager.from_crawler(self)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/middleware.py", line 56, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/middleware.py", line 32, in from_settings
mwcls = load_object(clspath)
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/utils/misc.py", line 44, in load_object
mod = import_module(module)
File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 662, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/extensions/memusage.py", line 16, in <module>
from scrapy.mail import MailSender
File "/home/fran/.local/lib/python3.5/site-packages/scrapy/mail.py", line 23, in <module>
from twisted.mail.smtp import ESMTPSenderFactory
ImportError: No module named 'twisted.mail.smtp'
Scrapy's Python 3 support is still in a beta stage and (according to the release notes)
Sending emails is not supported
so it's unlikely you can do anything to fix this.
Scrapy does not support sending mail with Python 3.
It's a limitation of Twisted, which has not ported it's twisted.mail module yet!
(Unfortunately, Pull request is pending)
See:
Twisted #8770
[#8770] Port twisted.mail.smtp to Python 3 #509
Related
Dependencies on html5lib are pretty broad across developer tools on Linux. I uncovered this issue while attempting to implement the selenium test suite for my Django app. When I attempt to pip install, I receive the below error. How do I adjust my system to grab and install html5lib?
Traceback (most recent call last):
File "/home/satch/.local/bin/pip", line 8, in <module>
sys.exit(main())
File "/usr/lib/python3/dist-packages/pip/_internal/cli/main.py", line 73, in main
command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
File "/usr/lib/python3/dist-packages/pip/_internal/commands/__init__.py", line 104,
in create_command
module = importlib.import_module(module_path)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 24,
in <module>
from pip._internal.cli.req_command import RequirementCommand, with_cleanup
File "/usr/lib/python3/dist-packages/pip/_internal/cli/req_command.py", line 16,
in <module>
from pip._internal.index.package_finder import PackageFinder
File "/usr/lib/python3/dist-packages/pip/_internal/index/package_finder.py", line 21,
in <module>
from pip._internal.index.collector import parse_links
File "/usr/lib/python3/dist-packages/pip/_internal/index/collector.py", line 14,
in <module>
from pip._vendor import html5lib, requests
ImportError: cannot import name 'html5lib' from 'pip._vendor'
(/usr/lib/python3/dist-packages/pip/_vendor/__init__.py)
I´m having a problem running my old django project. Its giving an error after i run py manage.py.
I tryed to use the command py manage.py makemigrations
Traceback (most recent call last):File "C:\Users\ricardo martins\Downloads\LES-fosquito\LES-fosquito\manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\core\management\__init__.py", line 446, in execute_from_command_line
utility.execute()
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\core\management\__init__.py", line 420, in execute
django.setup()
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\apps\registry.py", line 124, in populate
app_config.ready()
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\contrib\admin\apps.py", line 27, in ready
self.module.autodiscover()
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\contrib\admin\__init__.py", line 50, in autodiscover
autodiscover_modules("admin", register_to=site)
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\site-packages\django\utils\module_loading.py", line 58, in autodiscover_modules
import_module("%s.%s" % (app_config.name, module_to_search))
File "C:\Users\ricardo martins\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "C:\Users\ricardo martins\Downloads\LES-fosquito\LES-fosquito\processes\admin.py", line 4, in <module>
from tinymce.widgets import TinyMCE
ModuleNotFoundError: No module named 'tinymce'
Are you using the TinyMCE library in your code? If you are trying to run it locally, try to see if you have the correct dependencies installed, and if it is not needed you can simply remove the import statemnt.
So I have received a code that is not mine. My aim was to run the Django website I received as they told me all was working fine. But when I tried to run Runserver I got this error:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "C:\Users\eneko\AppData\Local\Programs\Python\Python38-32\lib\site-
packages\django\core\management\__init__.py", line 353, in execute_from_command_line
utility.execute()
File "C:\Users\eneko\AppData\Local\Programs\Python\Python38-32\lib\site-
packages\django\core\management\__init__.py", line 327, in execute
django.setup()
File "C:\Users\eneko\AppData\Local\Programs\Python\Python38-32\lib\site-packages\django\__init__.py",
line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "C:\Users\eneko\AppData\Local\Programs\Python\Python38-32\lib\site-
packages\django\apps\registry.py", line 108, in populate
app_config.import_models(all_models)
File "C:\Users\eneko\AppData\Local\Programs\Python\Python38-32\lib\site-
packages\django\apps\config.py", line 202, in import_models
self.models_module = import_module(models_module_name)
File "C:\Users\eneko\AppData\Local\Programs\Python\Python38-32\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "C:\Users\eneko\AppData\Local\Programs\Python\Python38-32\lib\site-
packages\django\contrib\auth\models.py", line 4, in <module>
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
File "C:\Users\eneko\AppData\Local\Programs\Python\Python38-32\lib\site-
packages\django\contrib\auth\base_user.py", line 49, in <module>
class AbstractBaseUser(models.Model):
RuntimeError: __class__ not set defining 'AbstractBaseUser' as <class
'django.contrib.auth.base_user.AbstractBaseUser'>. Was __classcell__ propagated to type.__new__?
These are my package versions:
Django 1.9.13
django-autoslug 1.9.7
pip 20.1.1
My python version is 3.8.2
Help would be very much appreciated
Django 1.9 requires Python 2.7, 3.4, or 3.5 as per official release note
so try with python version 3.5
When I tried to start my scrapy demo with scrapy crawl getCommodityInfo, the error below occurred.
C:\Users\柘宇\PycharmProjects\GraduationProject\spiders\bin\JDSpider>scrapy crawl getCommodityInfo
Traceback (most recent call last):
File "D:\Anacaonda\Scripts\scrapy-script.py", line 5, in <module>
sys.exit(scrapy.cmdline.execute())
File "D:\Anacaonda\lib\site-packages\scrapy\cmdline.py", line 141, in execute
cmd.crawler_process = CrawlerProcess(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 238, in __init__
super(CrawlerProcess, self).__init__(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 129, in __init__
self.spider_loader = _get_spider_loader(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\crawler.py", line 325, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 45, in from_settings
return cls(settings)
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 23, in __init__
self._load_all_spiders()
File "D:\Anacaonda\lib\site-packages\scrapy\spiderloader.py", line 32, in _load_all_spiders
for module in walk_modules(name):
File "D:\Anacaonda\lib\site-packages\scrapy\utils\misc.py", line 71, in walk_modules
submod = import_module(fullpath)
File "D:\Anacaonda\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "C:\Users\柘宇\PycharmProjects\GraduationProject\spiders\bin\JDSpider\JDSpider\spiders\getCommodityInfo.py", line 12, in <module>
from spiders.bin.JDSpider.JDSpider.items import JDCommodity
ModuleNotFoundError: No module named 'spiders'
It seems that the spider cannot be found, but I don't know why it happened. My whole project hierarchy is here. GraduationProject is the django project. main and spiders are the applications of django. The bin directory stores two demo scrapy projects. When I entered the JDSpider trying to run it, the error occurred. Could you help me fix it?
PS. My spider name: name = "getCommodityInfo"
With the solution applied by PS1212, the scrapy demo could run. However the pycharm warns like this. What happened?
Because it it not recognizing you project module.
Try this:
from JDSpider.items import JDCommodity
Python 2.6 and Django 1.6 was installed on the server (Suse).
I installed Python 3.4 and Django 1.7.
I can run
django-admin.py startproject hi
Once I run any command related to manage.py like
python manage.py syncdb
I will get this error
File "/usr/local/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 33, in <module>
from pysqlite2 import dbapi2 as Database
ImportError: No module named 'pysqlite2'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 35, in <module>
from sqlite3 import dbapi2 as Database
File "/usr/local/lib/python3.4/sqlite3/__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "/usr/local/lib/python3.4/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: No module named '_sqlite3'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.4/site-packages/django/core/management/__init__.py", line 354, in execute
django.setup()
File "/usr/local/lib/python3.4/site-packages/django/__init__.py", line 21, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.4/site-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/usr/local/lib/python3.4/site-packages/django/apps/config.py", line 202, in import_models
self.models_module = import_module(models_module_name)
File "/usr/local/lib/python3.4/importlib/__init__.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1129, in _exec
File "<frozen importlib._bootstrap>", line 1471, in exec_module
File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
File "/usr/local/lib/python3.4/site-packages/django/contrib/auth/models.py", line 40, in <module>
class Permission(models.Model):
File "/usr/local/lib/python3.4/site-packages/django/db/models/base.py", line 122, in __new__
new_class.add_to_class('_meta', Options(meta, **kwargs))
File "/usr/local/lib/python3.4/site-packages/django/db/models/base.py", line 297, in add_to_class
value.contribute_to_class(cls, name)
File "/usr/local/lib/python3.4/site-packages/django/db/models/options.py", line 166, in contribute_to_class
self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "/usr/local/lib/python3.4/site-packages/django/db/__init__.py", line 40, in __getattr__
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "/usr/local/lib/python3.4/site-packages/django/db/utils.py", line 242, in __getitem__
backend = load_backend(db['ENGINE'])
File "/usr/local/lib/python3.4/site-packages/django/db/utils.py", line 108, in load_backend
return import_module('%s.base' % backend_name)
File "/usr/local/lib/python3.4/importlib/__init__.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1129, in _exec
File "<frozen importlib._bootstrap>", line 1471, in exec_module
File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
File "/usr/local/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 38, in <module>
raise ImproperlyConfigured("Error loading either pysqlite2 or sqlite3 modules (tried in that order): %s" % exc)
django.core.exceptions.ImproperlyConfigured: Error loading either pysqlite2 or sqlite3 modules (tried in that order): No module named '_sqlite3'
according to Install pysqlite in virtualenv with python3 support
There is no public version of pysqlite for Python 3.x.
Also, I installed sqlite-autoconf-3080704. and still I get the same error message.
how can I solve it?
You seem to have only a minimum installation of Python 3. If you're using Debian or a Debian-based system like Ubuntu, you can do apt-get install python3 to install the full version.