I created a new environment with
conda create -n scraping python=2.7
I activate this environment with
source activate scraping
Then I proceed to install scrapy with (-n scraping doesn't seem necessary anymore, but I added it just in case)
conda install scrapy -n scraping
I now have the following installed
% conda list
# packages in environment at /Users/alexis/anaconda3/envs/scraping:
#
cffi 1.1.2 py27_0
cryptography 0.9.2 py27_0
cssselect 0.9.1 py27_0
enum34 1.0.4 py27_0
idna 2.0 py27_0
ipaddress 1.0.7 py27_0
ipython 3.2.1 py27_0
libxml2 2.9.2 0
libxslt 1.1.28 2
lxml 3.4.4 py27_0
nose 1.3.7 py27_0
openssl 1.0.1k 1
pip 7.1.0 py27_0
pyasn1 0.1.7 py27_0
pycparser 2.14 py27_0
pyopenssl 0.14 py27_0
python 2.7.10 0
python.app 1.2 py27_4
queuelib 1.2.2 py27_0
readline 6.2 2
scrapy 0.24.4 py27_0
setuptools 18.0.1 py27_0
six 1.9.0 py27_0
sqlite 3.8.4.1 1
tk 8.5.18 0
twisted 15.2.1 py27_0
w3lib 1.8.1 py27_1
zlib 1.2.8 0
zope.interface 4.1.2 py27_1
(I also installed ipython)
Now when I try to start a project I get
% scrapy startproject megadeluxe
Traceback (most recent call last):
File "/Users/alexis/anaconda3/envs/scraping/bin/scrapy", line 4, in <module>
from scrapy.cmdline import execute
File "/Users/alexis/anaconda3/envs/scraping/lib/python2.7/site- packages/scrapy/__init__.py", line 48, in <module>
from scrapy.spiders import Spider
ImportError: No module named spiders
Looking at the content of scraping/bin/scrapy I see no file named spiders.py and if I grep for class Spider I get no such class anywhere.
Did I do something wrong with the conda install at first? I install python 2.7, which is the requirement.
which openssl returns (I think that it's fine)
/Users/alexis/anaconda3/envs/scraping/bin/openssl
lxml is installed as we can see from conda list
I'm a little clueless here.
[Edit]
As mentioned in the comments, I was not using the latest version of scrap. This does not explain the error, but I proceeded to install the latest version (1.0) with pip (as of this writing conda only install version 0.24).
After doing so, starting a project would spit something out, ending with Library not loaded: libssl.1.0.0.dylib
Looking this up on stack overflow returned me this link
Cannot Set Up a Scrapy Project
The answer given by Joe Hooper solved my problem
https://stackoverflow.com/a/26961576/2883980
After all this I now have a solution, but have no clue as to the root reason.
I will put this here so other may find this useful (because I had no idea what DYLD_LIBRARY_PATH was used for)
man dyld
DESCRIPTION
The dynamic linker uses the following environment variables.
They affect any program that uses the dynamic linker.
DYLD_FRAMEWORK_PATH
This is a colon separated list of directories that contain
frameworks. The dynamic linker searches these
directories before it searches for the framework by its
install name. It allows you to test new versions of
existing frameworks. (A framework is a library install
name that ends in the form XXX.framework/Versions/YYY/XXX
or XXX.framework/XXX, where XXX and YYY are any name.)
For each framework that a program uses, the dynamic linker
looks for the framework in each directory in
DYLD_FRAMEWORK_PATH in turn. If it looks in all the
directories and can't find the framework, it searches the
directories in DYLD_LIBRARY_PATH in turn. If it still
can't find the framework, it then searches
DYLD_FALLBACK_FRAMEWORK_PATH and DYLD_FALL-
BACK_LIBRARY_PATH in turn.
Use the -L option to otool(1). to discover the frameworks
and shared libraries that the executable is linked
against.
I noticed that you have the scrapy version 0.24.4 installed. Any reasons you're not running the new version 1.0 ?
I believe Scrapy.spiders is a 1.0 class, and not a 0.24 one. I would try to get the really last version installed in your environment, and see if this works.
I was working on exactly the same task as you do.
I followed the steps in here to create an environment:
Create the environment
And here to activate my environment:
Activate the newly created environment
I was working these out with anaconda command prompt.
I installed Scrapy a long time ago and therefore when I create the environment, scrapy is already there.
Currently I am having problems with importing items.py :
My question
Let me know if I could help further.
Related
I'm using the Python launcher on Windows to simplify version management and avoid modifying system path.
I've successfully installed a few packages:
> py -m pip list
Package Version
------------------ ---------
Brotli 1.0.9
certifi 2021.10.8
charset-normalizer 2.0.12
gallery-dl 1.21.1
idna 3.3
mutagen 1.45.1
pip 22.0.4
pycryptodomex 3.14.1
requests 2.27.1
setuptools 58.1.0
urllib3 1.26.9
websockets 10.3
yt-dlp 2022.4.8
They're all listed under Program Files/Python310/Scripts:
but when I try to call them I get the following error:
> py -m gallery-dl
C:\Program Files\Python310\python.exe: No module named gallery-dl
unless I replace hyphens with underscores:
> py -m gallery_dl --help
usage: __main__.py [OPTION]... URL...
General Options:
...
This is quite different from when I wasn't using the launcher and had python in PATH and could just call gallery-dl inside Powershell without:
prepending py -m - which is fine
replacing hyphens with underscores - which is less fine because it sort of feels like I'm calling a raw function
having the __main__.py: bit - which doesn't feel right either.
Is this the expected and proper behavior, or [am I doing / did I do] something wrong?
I am working with a Python virtualenv named env to execute Odoo. In the virtualenv bin directory, I got this:
pip
pip3
pip3.8
python -> /usr/bin/python3
python3 -> python
python3.8 -> python
And the Odoo service is running this command to execute Odoo (as you can see, using the python3.8 of the virtualenv):
ExecStart=/opt/odoo/env/bin/python3.8 /opt/odoo/odoo_13/src/oca/OCB/odoo-bin -c /opt/odoo/odoo_13/config/.odoo.conf
The problem is that every now and again, Odoo gives the following critical error and the instance stops working:
ModuleNotFoundError: No module named 'werkzeug'
It seems that the werkzeug package is not installed, but, if I do the following:
anubia#my-server:/opt/odoo$ . env/bin/activate
(env) anubia#my-server:/opt/odoo$ which pip
/opt/odoo/env/bin/pip
(env) anubia#my-server:/opt/odoo$ which pip3
/opt/odoo/env/bin/pip3
(env) anubia#my-server:/opt/odoo$ which pip3.8
/opt/odoo/env/bin/pip3.8
(env) anubia#my-server:/opt/odoo$ pip3.8 install werkzeug
Requirement already satisfied: werkzeug in ./env/lib/python3.8/site-packages (0.11.15)
(env) anubia#my-server:/opt/odoo$ cd env/lib/python3.8/site-packages/
Display all 125 possibilities? (y or n)
Babel-2.9.1.dist-info/ attrs-21.2.0.dist-info/ docutils/ numpy-1.21.0.dist-info/ python_stdnum-1.16.dist-info/ urllib3-1.26.6.dist-info/
Crypto/ babel/ docutils-0.17.1.dist-info/ numpy.libs/ pytz/ validate_email-1.3.dist-info/
Jinja2-3.0.1.dist-info/ bcrypt/ gevent/ paramiko/ pytz-2021.1.dist-info/ vatnumber/
MarkupSafe-2.0.1.dist-info/ bcrypt-3.2.0.dist-info/ gevent-21.1.2.dist-info/ paramiko-2.7.2.dist-info/ reportlab/ vatnumber-1.2.dist-info/
OpenSSL/ cached_property-1.5.2.dist-info/ greenlet/ passlib/ reportlab-3.5.68.dist-info/ werkzeug/
PIL/ cachetools/ greenlet-1.1.0.dist-info/ passlib-1.7.4.dist-info/ reportlab.libs/ wheel/
Pillow-8.3.1.dist-info/ cachetools-4.2.2.dist-info/ html2text/ phonenumbers/ requests/ wheel-0.36.2.dist-info/
Pillow.libs/ certifi/ html2text-2020.1.16.dist-info/ phonenumbers-8.12.27.dist-info/ requests-2.25.1.dist-info/ xlrd/
PyNaCl-1.4.0.dist-info/ certifi-2021.5.30.dist-info/ idna/ pip/ requests_file-1.5.1.dist-info/ xlrd-2.0.1.dist-info/
PyPDF2/ cffi/ idna-2.10.dist-info/ pip-21.3.1.dist-info/ requests_toolbelt/ xlsxwriter/
PyPDF2-1.26.0.dist-info/ cffi-1.14.5.dist-info/ isodate/ pkg_resources/ requests_toolbelt-0.9.1.dist-info/ xlwt/
PyYAML-3.12.dist-info/ cffi.libs/ isodate-0.6.0.dist-info/ polib-1.1.1.dist-info/ sassutils/ xlwt-1.3.0.dist-info/
Unidecode-1.2.0.dist-info/ chardet/ jinja2/ psutil/ setuptools/ xmlsig/
Werkzeug-0.11.15.dist-info/ chardet-4.0.0.dist-info/ libsass-0.21.0.dist-info/ psutil-5.8.0.dist-info/ setuptools-57.1.0.dist-info/ xmlsig-0.1.5.dist-info/
XlsxWriter-1.4.4.dist-info/ cryptography/ lxml/ psycopg2/ six-1.16.0.dist-info/ yaml/
__pycache__/ cryptography-3.4.7.dist-info/ lxml-4.6.3.dist-info/ psycopg2-2.9.1.dist-info/ stdnum/ zeep/
_distutils_hack/ dateutil/ markupsafe/ pyOpenSSL-20.0.1.dist-info/ suds/ zeep-4.0.0.dist-info/
appdirs-1.4.4.dist-info/ decorator-5.0.9.dist-info/ nacl/ pycparser/ suds_jurko-0.6.dist-info/ zope/
asn1crypto/ defusedxml/ num2words/ pycparser-2.20.dist-info/ tests/ zope.event-4.5.0.dist-info/
asn1crypto-1.4.0.dist-info/ defusedxml-0.7.1.dist-info/ num2words-0.5.10.dist-info/ pycryptodome-3.10.1.dist-info/ unidecode/ zope.interface-5.4.0.dist-info/
attr/ docopt-0.6.2.dist-info/ numpy/ python_dateutil-2.8.1.dist-info/ urllib3/
So werkzeug seems to be installed. What is the problem?
EDIT
After trying with #ChesuCR answer two weeks, the Odoo server stopped working again. This time the error is:
ModuleNotFoundError: No module named 'PyPDF2'
However, if I check the virtualenv:
myuser#myserver:/opt/odoo$ . venv3.7/bin/activate
(venv3.7) myuser#myserver:/opt/odoo$ which python3.7
/opt/odoo/venv3.7/bin/python3.7
(venv3.7) myuser#myserver:/opt/odoo$ python3.7 -m pip install PyPDF2
Requirement already satisfied: PyPDF2 in ./venv3.7/lib/python3.7/site-packages (1.26.0)
(venv3.7) myuser#myserver:/opt/odoo$ python3.7 -m pip list
Package Version
----------------- ---------
appdirs 1.4.4
attrs 21.2.0
Babel 2.6.0
beautifulsoup4 4.10.0
cached-property 1.5.2
certifi 2021.10.8
cffi 1.15.0
chardet 3.0.4
cryptography 36.0.0
decorator 4.3.0
defusedxml 0.7.1
docutils 0.14
ebaysdk 2.1.5
gevent 1.5.0
greenlet 0.4.15
html2text 2018.1.9
idna 2.8
isodate 0.6.0
Jinja2 2.10.1
libsass 0.17.0
lxml 4.3.2
Mako 1.0.7
MarkupSafe 1.1.0
mock 2.0.0
num2words 0.5.6
ofxparse 0.19
passlib 1.7.1
pbr 5.8.0
Pillow 6.1.0
pip 21.3.1
polib 1.1.0
psutil 5.6.6
psycopg2 2.7.7
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.21
pydot 1.4.1
pyOpenSSL 21.0.0
pyparsing 2.2.0
PyPDF2 1.26.0
pyserial 3.4
python-dateutil 2.7.3
python-stdnum 1.17
pytz 2019.1
pyusb 1.0.2
qrcode 6.1
reportlab 3.5.13
requests 2.21.0
requests-toolbelt 0.9.1
setuptools 57.5.0
six 1.16.0
soupsieve 2.3.1
urllib3 1.24.3
vatnumber 1.2
vobject 0.9.6.1
Werkzeug 0.14.1
wheel 0.37.0
xlrd 1.1.0
XlsxWriter 1.1.2
xlwt 1.3.0
zeep 3.2.0
And of course Odoo is running with the Python interpreter of the virtualenv:
ExecStart=/opt/odoo/venv3.7/bin/python3.7 /opt/odoo/odoo_13/src/oca/OCB/odoo-bin -c /opt/odoo/odoo_13/config/.odoo.conf
I don't know what Odoo is, but in order to find the packages, you usually need to "activate" the virtualenv. Just like you did manually, but you also need that when Odoo is running the program. It is not enough to just call the correct python binary. So you can either try to have Odoo activate the venv before it starts the program, or if not possible, emulate doing that (it's probably mostly setting PATH and PYTHONPATH environment variables)
Odoo 13 is not compatible with the Python 3.8 version. You can try with Python 3.7 or 3.6 versions. Anyway, many issues have been fixed, so I am not sure about the Python 3.8 incompatibility. But, in principle, the branch Odoo v13 was created for Python 3.6, as you can check in the setup.py file
If the problem still bothers you, try to install the exact versions that Odoo developers posted in the requirements.txt file, where dependencies should have been tested. So you can install the right Werzeug version:
pip install Werkzeug==0.14.1
Ah, and if you want to show all the installed packages you can do:
pip list
For me, I had to fix it like this:
First of all and a key point is that I had to use root user, not ubuntu not odoo nothing, only root
I activated my venv in root user
I installed required packages: pip3 install paramiko (or any other packages: boto3, or pretty-bad-protocol)
and then I could install the module from the odoo admin backend, and it's working just fine
I'm on Ubuntu 20.04 trying to load and run an FMU exported from OpenModelica with PyFMI. When I run the code:
from pyfmi import load_fmu
model = load_fmu('Model.fmu')
The following error is produced:
File "src/pyfmi/fmi.pyx", line 7943, in pyfmi.fmi.load_fmu
File "src/pyfmi/fmi.pyx", line 7086, in pyfmi.fmi.FMUModelME2.__init__
File "src/pyfmi/fmi.pyx", line 3761, in pyfmi.fmi.FMUModelBase2.__init__
File "src/pyfmi/fmi.pyx", line 45, in pyfmi.fmi.encode
TypeError: latin_1_encode() argument 1 must be str, not bytes
The result is the same regardless of which I use the version of PyFMI available through pip or conda.
I've spent hours trying to find a solution, but to no avail. On pip, he package information returned from pyfmi.check_packages() is:
PyFMI version ................ 2.5
Platform ..................... linux
Python version ............... 3.8.5
Dependencies:
Package Version
------- -------
assimulo...................... 3.0
Cython........................ 0.29.22
lxml.......................... 4.5.0
matplotlib.................... 3.1.2
numpy......................... 1.17.4
scipy......................... 1.6.1
and on conda it's
PyFMI version ................ 2.5
Platform ..................... linux
Python version ............... 3.8.5
Dependencies:
Package Version
------- -------
assimulo...................... 3.0
Cython........................ 0.29.22
lxml.......................... 4.6.2
matplotlib.................... 3.3.4
numpy......................... 1.20.1
scipy......................... 1.6.1
Note: I've tried install version PyFMI 2.8.5 via conda, but I always seem to end up with 2.5 regardless of what I try.
Does anyone have any suggestions? Many thanks for any help!
I realise now why the wrong version of the package was being loaded. After uninstalling the pip version (2.5) the conda version (2.8.5) became visible, see below (updated after fixing second issue). Of course, now I have a different problem, but I'll document that in a different thread... it turned out to be a similar issue. I had an outdated Assimulo package installed in pip, which when uninstalled, allowed the conda version to be used correctly. I'm leaving the record of my solution visible in the hope that someone else in my position might benefit from it.
Performing pyfmi package check
==============================
PyFMI version ................ 2.8.5
Platform ..................... linux
Python version ............... 3.8.5
Dependencies:
Package Version
------- -------
assimulo...................... 3.2.3
Cython........................ 0.29.22
lxml.......................... 4.6.2
matplotlib.................... 3.3.4
numpy......................... 1.20.1
scipy......................... 1.6.1
Environment
Pythonnet version: 2.4.0.dev0, installed from Master
Python version: Python 3.6.6, Anaconda 3
Operating System: Window 10, 64 bit
Using virtual studio 2017 Community
Details
I had virtual environment set up.
1) pip list
(py36) C:\Users\User\Desktop>pip list
Package Version
--------------- ----------
certifi 2018.8.13
click 6.7
cycler 0.10.0
kiwisolver 1.0.1
matplotlib 2.2.3
mkl-fft 1.0.4
mkl-random 1.0.1
numpy 1.15.0
Pillow 5.2.0
pip 18.0
pyparsing 2.2.0
PyQt5 5.11.2
PyQt5-sip 4.19.12
python-dateutil 2.7.3
pythonnet 2.4.0.dev0
pytz 2018.5
scipy 1.1.0
setuptools 40.0.0
six 1.11.0
visbrain 0.4.2
vispy 0.5.3
wheel 0.31.1
wincertstore 0.2
2) path and pythonpath before python initialize.
string path = #"D:\Anaconda3\envs\py36;" + Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine);
Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONHOME", #"D:\Anaconda3\envs\py36", EnvironmentVariableTarget.Process);
// initialize python engine
PythonEngine.Initialize();
3) set Amaconda in %PATH%
4) python path
(py36) C:\Users\User\Desktop>python --version
Python 3.6.6 :: Anaconda, Inc.
ReadMe works fine.
but when i typed like this in my C# program
using (Py.GIL())
{
// import vispy
dynamic myVispy = Py.Import("vispy");
Console.WriteLine("hi vispy");
}
error message below
Python.Runtime.PythonException: 'ModuleNotFoundError : No module named 'vispy''
stack trace :
Python.Runtime.PythonException
HResult=0x80131500
Message=ModuleNotFoundError : No module named 'vispy'
Source=Python.Runtime
StackTrace:
would like to know how to solve this ?
Thank You.
See this answer: Python.NET does not work with Anaconda or Miniconda. You need a fresh python install.
(I'm aware of this question Cannot install ggplot with anaconda but that is aimed at Windows, and I'm running a Linux OS)
I'm attempting to install the ggplot package in a python3 (v3.6.0) Anaconda environment:
$ conda install ggplot
Fetching package metadata .............
PackageNotFoundError: Package missing in current linux-64 channels:
- ggplot
Close matches found; did you mean one of these?
ggplot: r-ggplot2, r-gplots
If I use conda search I get:
$ conda search ggplot
Fetching package metadata .............
r-ggplot2 1.0.0 0 defaults
1.0.0 0a defaults
1.0.1 r3.2.2_0 defaults
1.0.1 r3.2.0_0 defaults
1.0.1 r3.2.1_0 defaults
1.0.1 r3.2.1_0a defaults
1.0.1 r3.2.2_0a defaults
1.0.1 r3.2.0_0a defaults
2.1.0 r3.3.1_0 defaults
2.2.0 r3.3.1_0 defaults
2.2.0 r3.3.2_0 defaults
but if I search https://anaconda.org/search for ggplot I get lots of results.
The questions: why am I not seeing those results when using conda search? What is the difference between ggplot and r-ggplot2 (the package it offers to install when I search for ggplot)?
Why am I not seeing those results when using conda search?
The difference for search is that conda search only searches in your channels, anaconda search or the search on anaconda.org includes all (public) channels. The name in front of the package name is the channel, for example xyz/ggplot the xyz is the channel.
What is the difference between ggplot and r-ggplot2
It's probably a naming convention. anaconda has several R based packages and it's likely they seperate them using the r- prefix from more regular python packages. So if you don't plan to use it with "R" you should probably look for a suitable candidate without the r-.