I have successfully installed Scrapy and all it's dependencies into Python.org 64 shell on a Windows Vista 64 bit platform. This is installed as Site Package. I have tested Scrapy works ok at a basic level by typing:
Import Scrapy
Into a Python Shell and got no errors. I've just got started with tutorials on how to use Scrapy. Most of these suggest setting up a new project using the following Command Prompt line:
C:\Python27>scrapy startproject myproject
However this throws up the error message in the title about Scrapy being an invalid command. I have tried amending my environment variables to allow me to use this command from any file location within Command Prompt. I have added the following to the end of my environment variables:
C:\Python27;C:\Python27\Scripts;C:\Python27\Lib\site-packages\scrapy;C:\Python27\Scrapy
This does not seem to resolve the problem however as I still get the same error message. I have also tried running the 'scrapy startproject myproject' command from the following locations with Command Prompt:
C:\Python27
C:\Python27\Scripts
C:\Python27\Scrapy
This again still throws up the same error message. I feel like I am quite close to having the solution, but I just cant quite seem to make it work.
Can anyone spot where I am going wrong?
Thanks
EDIT:
In response to the last comment above from MattDMo, the command now works, however it now throws up the following error message, which I'm not even sure where to start with:
C:\Python27\Scripts>scrapy startprogram mrscript
Traceback (most recent call last):
File "c:\Python27\Scripts\scrapy", line 3, in <module>
from scrapy.cmdline import execute
File "c:\Python27\lib\site-packages\scrapy\cmdline.py", line 9, in <module>
from scrapy.crawler import CrawlerProcess
File "c:\Python27\lib\site-packages\scrapy\crawler.py", line 3, in <module>
from twisted.internet import reactor, defer
File "c:\Python27\lib\site-packages\twisted\internet\reactor.py", line 38, in
<module>
from twisted.internet import default
File "c:\Python27\lib\site-packages\twisted\internet\default.py", line 56, in
<module>
install = _getInstallFunction(platform)
File "c:\Python27\lib\site-packages\twisted\internet\default.py", line 50, in
_getInstallFunction
from twisted.internet.selectreactor import install
File "c:\Python27\lib\site-packages\twisted\internet\selectreactor.py", line 1
8, in <module>
from twisted.internet import posixbase
File "c:\Python27\lib\site-packages\twisted\internet\posixbase.py", line 24, i
n <module>
from twisted.internet import error, udp, tcp
File "c:\Python27\lib\site-packages\twisted\internet\tcp.py", line 29, in <mod
ule>
from twisted.internet._newtls import (
File "c:\Python27\lib\site-packages\twisted\internet\_newtls.py", line 21, in
<module>
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
File "c:\Python27\lib\site-packages\twisted\protocols\tls.py", line 41, in <mo
dule>
from OpenSSL.SSL import Error, ZeroReturnError, WantReadError
File "c:\Python27\lib\site-packages\OpenSSL\__init__.py", line 8, in <module>
from OpenSSL import rand, crypto, SSL
File "c:\Python27\lib\site-packages\OpenSSL\rand.py", line 11, in <module>
from OpenSSL._util import (
File "c:\Python27\lib\site-packages\OpenSSL\_util.py", line 4, in <module>
binding = Binding()
File "c:\Python27\lib\site-packages\cryptography\hazmat\bindings\openssl\bindi
ng.py", line 87, in __init__
self._ensure_ffi_initialized()
File "c:\Python27\lib\site-packages\cryptography\hazmat\bindings\openssl\bindi
ng.py", line 106, in _ensure_ffi_initialized
libraries=libraries,
File "c:\Python27\lib\site-packages\cryptography\hazmat\bindings\utils.py", li
ne 80, in build_ffi
extra_link_args=extra_link_args,
File "c:\Python27\lib\site-packages\cffi\api.py", line 341, in verify
lib = self.verifier.load_library()
File "c:\Python27\lib\site-packages\cffi\verifier.py", line 75, in load_librar
y
return self._load_library()
File "c:\Python27\lib\site-packages\cffi\verifier.py", line 151, in _load_libr
ary
return self._vengine.load_library()
File "c:\Python27\lib\site-packages\cffi\vengine_cpy.py", line 138, in load_li
brary
raise ffiplatform.VerificationError(error)
cffi.ffiplatform.VerificationError: importing 'c:\\Python27\\lib\\site-packages\
\cryptography\\_Cryptography_cffi_444d7397xa22f8491.pyd': DLL load failed: The s
pecified module could not be found.
Is this actually an error log or just a standard print out? Also, could someone explain please why adding a .bat file to my Python scripts causes this to work?
Thanks
pip install scrapy installs a file called scrapy in Python's Scripts directory on Windows, the default location for which is C:\Python27\Scripts. However, C:\Python27\Scripts\scrapy is a Python file, even though it lacks the .py suffix. There are two ways of running it. My preferred method is to use the Git Shell, as scrapy has its executable bit set, and can be run without further ado.
The other method, if you want to keep using cmd.exe, is to make a batch file with the following contents:
#echo off
c:\Python27\python c:\Python27\Scripts\scrapy %*
Save it in the Scripts directory as scrapy.bat (assuming your PATH now includes C:\Python27;C:\Python27\Scripts), restart cmd.exe, and you should now be able to run it from the command line.
I experienced a similar problem (I'm using a virtual environment in Windows 7).
I noticed that for older versions of scrapy, e.g. pip install scrapy==0.18.4, it automatically creates a scrapy.bat in my virtual environment: {VENV_ROOT_FOLDER}\Scripts\scrapy.bat
But if I use the latest scrapy (currently version 0.24.2), the scrapy.bat is missing.
I wanted to use the latest scrapy version, so what I did is to put a copy of scrapy.bat to its Scripts folder and it solves the problem.
Related
I had this project running on my Mac here under python3. It took a while after doing a 2to3 to update some other bits and pieces to it to finally get it to run. Now though the project fails and all I recall of doing to my mac is:
Upgrading the OS
Upgrading Xcode and other pieces so that a 'brew install root' would work.
Not sure what else it touched but it is like it lost all the modules I had done a pip install on. So i redid a pip install on several modules
astropy
matplotlib
scipy
now the project fails when I run it:
File "/Users/shathoma/TUI3/TUI/Guide/GuideImage.py", line 26, in <module>
import astropy.io.fits as pyfits
File "/usr/local/lib/python3.9/site-packages/astropy/io/fits/__init__.py", line 70, in <module>
from . import convenience
File "/usr/local/lib/python3.9/site-packages/astropy/io/fits/convenience.py", line 63, in <module>
from .diff import FITSDiff, HDUDiff
File "/usr/local/lib/python3.9/site-packages/astropy/io/fits/diff.py", line 25, in <module>
from .header import Header
File "/usr/local/lib/python3.9/site-packages/astropy/io/fits/header.py", line 11, in <module>
from .file import _File
File "/usr/local/lib/python3.9/site-packages/astropy/io/fits/file.py", line 29, in <module>
from astropy.utils.compat.optional_deps import HAS_BZ2
File "/usr/local/lib/python3.9/site-packages/astropy/utils/compat/__init__.py", line 19, in <module>
from .numpycompat import * # noqa
File "/usr/local/lib/python3.9/site-packages/astropy/utils/compat/numpycompat.py", line 14, in <module>
NUMPY_LT_1_19 = not minversion('numpy', '1.19')
File "/usr/local/lib/python3.9/site-packages/astropy/utils/decorators.py", line 546, in wrapper
return function(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/astropy/utils/introspection.py", line 167, in minversion
return Version(module_version) >= Version(version)
File "/usr/local/lib/python3.9/site-packages/packaging/version.py", line 264, in __init__
match = self._regex.search(version)
TypeError: expected string or bytes-like object
I have this project running fine still under a windows machine and the versions of the installed modules like astropy etc are the same. Still fails on the mac with the above error. I could go in and fix packaging/version.py to deal with a self._regex.search(str(version)) which is a bad Idea I know (I tried that and since reverted it gave an error on how version = none somewhere.
it is almost like I am missing a module but nothing knows it? Not sure, and not sure how after working this broke with just probably the brew install command of the root (https://root.cern.ch/) package. Out of ideas what to look for or try next to get this running again?
I have a Django project running under Django 1.8.4 that I'm trying to upgrade to latest version of Django. Since I was still using Python 2.7 on a Debian 8 server, I first upgraded my server to Debian 9 and then I upgraded my Django project's virtual environment to Python 3.5.3 (the default version of Python 3 on Debian 9).
After I rebuilt the virtual environment to use Python 3, I ran the command 2to3 -w . in my project's root folder. 2to3 found and fixed 62 problems without any difficulties. I also have 216 unit tests and only two of them failed under Python 3.5.
I then went to my Django website and the site appeared to be working except that the forms that contain pull-down elements show "SomeThingobject" instead of the values from the SomeThing table. I ran a few SQL queries directly against the database and confirmed that my SomeThing lookup table is populated.
Now I want to run a Django queryset command in the Python shell like the following to see why my pulldowns are not showing my column values:
queryset=SomeThing.objects.all()
However, if I activate my virtual environment and try to run the command ./manage.py shell, I get the following error:
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/django/core/management/base.py", line 393, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/django/core/management/base.py", line 444, in execute
output = self.handle(*args, **options)
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/django/core/management/commands/shell.py", line 69, in handle
self.run_shell(shell=options['interface'])
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/django/core/management/commands/shell.py", line 58, in run_shell
return getattr(self, shell)()
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/django/core/management/commands/shell.py", line 41, in ipython
ip()
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/django/core/management/commands/shell.py", line 34, in _ipython
from IPython import start_ipython
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/IPython/__init__.py", line 48, in <module>
from .terminal.embed import embed
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/IPython/terminal/embed.py", line 19, in <module>
from IPython.terminal.ipapp import load_default_config
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/IPython/terminal/ipapp.py", line 30, in <module>
from IPython.core.magics import ScriptMagics
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/IPython/core/magics/__init__.py", line 22, in <module>
from .execution import ExecutionMagics
File "/srv/https/my_project.com/venvs/0bc664fa2a7b562d0b7e42a7acb3f095fe5df3cf/lib/python3.5/site-packages/IPython/core/magics/execution.py", line 21, in <module>
import cProfile as profile
File "/usr/lib/python3.5/cProfile.py", line 22, in <module>
run.__doc__ = _pyprofile.run.__doc__
AttributeError: module 'profile' has no attribute 'run'
Now I do have an application named 'profile' in my INSTALLED_APPS. If I try to open the Python shell on my production server which is still using Python 2.7, the shell executes without a problem. It's only having this problem with Python 3.5. What's interesting is that my Django 1.8/Python 2.7 project contains .pyc files but now under Python 3, they're replaced with __pycache__/<filename>.cpython-35.pyc files.
I read the Django document Porting to Python 3 but it doesn't seem to allude to this problem. What do I have to do to make this Django project operate properly under Python 3.5? Is there perhaps something new in Python 3.5 that is having a conflict with my profile application or is it something else?
After much digging around in the Python code I surmised that the problem was being caused by the fact that one of my applications inside the Django project was called "profile" and this was conflicting with a profile module in the newer version of cPython in Python 3.5. After I changed the name of the application to "userprofile" I could start the shell without any problems. These two articles help me change the app's name:
Stackoverflow: How to change the name of a Django app?
How to Change Name of Django Application
I have a cloud instance of a Linux machine (openSuSE) with multiple users.
I have created a virtual environment and installed all my required libraries (including Klein).
I have two users "a" and "b".
While logged in as "a" and inside virtualenv, when I open python shell at home directory and type
import klein
it imports normally.
Now when I change directory to
/home/b/
and run the same (open python shell, import klein) while being in the same virtualenv, it gives me an error.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/a/.local/lib/python3.6/site-packages/klein/__init__.py", line 3, in <module>
from klein._plating import Plating
File "/home/a/.local/lib/python3.6/site-packages/klein/_plating.py", line 16, in <module>
from .app import _call
File "/home/a/.local/lib/python3.6/site-packages/klein/app.py", line 19, in <module>
from twisted.internet import endpoints, reactor
File "/home/a/.local/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 58, in <module>
from twisted.protocols.tls import TLSMemoryBIOFactory
File "/home/a/.local/lib/python3.6/site-packages/twisted/protocols/tls.py", line 63, in <module>
from twisted.internet._sslverify import _setAcceptableProtocols
File "/home/a/.local/lib/python3.6/site-packages/twisted/internet/_sslverify.py", line 158, in <module>
verifyHostname, VerificationError = _selectVerifyImplementation()
File "/home/a/.local/lib/python3.6/site-packages/twisted/internet/_sslverify.py", line 141, in _selectVerifyImplementation
from service_identity import VerificationError
File "/home/a/.conda/envs/mm/lib/python3.6/site-packages/service_identity/__init__.py", line 7, in <module>
from . import cryptography, pyopenssl
File "/home/a/.conda/envs/mm/lib/python3.6/site-packages/service_identity/cryptography.py", line 16, in <module>
from .exceptions import SubjectAltNameWarning
File "/home/a/.conda/envs/mm/lib/python3.6/site-packages/service_identity/exceptions.py", line 21, in <module>
#attr.s
AttributeError: module 'attr' has no attribute 's'
Command "which python" gives same address at both location which is my virtualenv python address and that should be expected.
But what causes this weird python shell behavior.
Thank you
I solved it and a very shameful reason caused the error.
One of the modules Twisted uses is "attr" module. I had named one of my files attr.py and that is what was causing all the error.
I myself am not deleting this question if moderation has no problem, maybe somebody like me might be stuck at the same situation. It may help them.
Never name your python files same as that of any standard module unless overriding.
Also if your issue persists, then Jean's answer will definitely resolve it.
There can be multiple different Python packages that provide the same Python module. For example, there are at least two packages that provide the attr module:
https://pypi.org/project/attr/
https://pypi.org/project/attrs/
It's possible you've installed the wrong package based on the requirements. You can check what you have installed with pip freeze.
UPDATE: if I change the name of the select module (in which the import error occurs) to select1, the problem goes away! WHAT the?
First off: everything works fine when I try to import my package from the command line, or run pytest in the project directory.
But in PyCharm I'm getting the "attempted relative import beyond top-level package" problem when I try to run my tests, and ONLY while in PyCharm. I can't figure out why.
I have even deleted and re-cloned the repository and the problem still occurs.
Here is the link to the problem branch.
And here is the tb:
Testing started at 4:02 PM ...
C:\Users\ricky\AppData\Local\Programs\Python\Python37\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.2\helpers\pycharm\_jb_pytest_runner.py" --path C:/Users/ricky/PycharmProjects/candejar/tests
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.2\helpers\pycharm\_jb_pytest_runner.py", line 10, in <module>
from pkg_resources import iter_entry_points
File "C:\Users\ricky\AppData\Local\Programs\Python\Python37\lib\site-packages\pkg_resources\__init__.py", line 36, in <module>
import email.parser
File "C:\Users\ricky\AppData\Local\Programs\Python\Python37\lib\email\parser.py", line 12, in <module>
from email.feedparser import FeedParser, BytesFeedParser
File "C:\Users\ricky\AppData\Local\Programs\Python\Python37\lib\email\feedparser.py", line 27, in <module>
from email._policybase import compat32
File "C:\Users\ricky\AppData\Local\Programs\Python\Python37\lib\email\_policybase.py", line 9, in <module>
from email.utils import _has_surrogates
File "C:\Users\ricky\AppData\Local\Programs\Python\Python37\lib\email\utils.py", line 29, in <module>
import socket
File "C:\Users\ricky\AppData\Local\Programs\Python\Python37\lib\socket.py", line 52, in <module>
import os, sys, io, selectors
File "C:\Users\ricky\AppData\Local\Programs\Python\Python37\lib\selectors.py", line 12, in <module>
import select
File "C:\Users\ricky\PycharmProjects\candejar\candejar\select.py", line 24, in <module>
from . import exc
ImportError: attempted relative import with no known parent package
Process finished with exit code 1
I'm sure there is something simple going on here but I just can't figure it out. NOTE: I'm using Python 3.7 RC 1.
The sys.path in pycharm and the command line come out to be the same, except pycharm also includes the pycharm project main package directory (which for some reason appears twice).
I was able to solve this by deleting the .idea directory from the cloned repository and re-opening PyCharm. I still don't know what was wrong, but it is fixed now. Very weird.
UPDATE: it has something to do with the following line in the project .iml file in the .idea folder:
<sourceFolder url="file://$MODULE_DIR$/candejar" isTestSource="false" />
Removing that xml code solves the issue. But I do not know why, and I don't know how it got there in the first place (I do not recall messing around with any of my settings prior to this error).
I'm trying use py2exe to compile an eye tracking experiment written in Python 2.7 (32-bit). The experiment uses the psychopy library. I wrote the experiment using the PyCharm IDE, and the experiment runs when I run it through the PyCharm IDE, using an interpreter in a virtual environment located at C:\Users\phil\Python_2.7_32-bit.
The experiment compiles without generating any errors when I enter the following command into the command prompt: C:\Users\phil\Python_2.7_32-bit\Scripts\python.exe C:\Users\phil\PycharmProjects\iTRAC\VisSearch\setup.py py2exe.
When I run the executable generated by the above py2exe command, I get this error:
Traceback (most recent call last):
File "VisualSearch.py", line 3, in <module>
File "psychopy\__init__.pyc", line 39, in <module>
File "psychopy\preferences\__init__.pyc", line 5, in <module>
File "psychopy\preferences\preferences.pyc", line 172, in <module>
File "psychopy\preferences\preferences.pyc", line 33, in __init__
File "psychopy\preferences\preferences.pyc", line 98, in loadAll
File "psychopy\preferences\preferences.pyc", line 146, in loadAppData
File "psychopy\preferences\configobj.pyc", line 583, in __getitem__
KeyError: 'builder'
My setup.py script is as follows:
from distutils.core import setup
import py2exe
setup(windows =['C:\Users\phil\PycharmProjects\iTRAC\VisSearch\VisualSearch.py'])
I've also tried using the following setup.py script with the same results:
from distutils.core import setup
import py2exe
setup(windows = [{'script':'C:\Users\phil\PycharmProjects\iTRAC\VisSearch\VisualSearch.py',
'options' : {'py2exe':{'includes':['psychopy'],
'compressed': True,
'bundle_files': 1,}}}])
I googled the error and came up with 0 results.
Can anybody tell me why I am running into this error?
This is probably a missing config/prefs file. PsychoPy uses the configobj library to read and validate preferences but my guess is that py2exe is only automatically packaging py/pyc files and needs to include the .spec files in the psychopy/preferences folder.