I would like to know if there's a way that I can put Scrapy into a subdirectory and import it. I did this with BeautifulSoup, rather than installing it, I just drop the bs4 directory into the directory of my app, and import it:
from bs4 import BeautifulSoup
In the source that I downloaded from scrapy.org there is no scrapy.py so I tried importing
from scrapy import *
This returned a bunch of errors.
Traceback (most recent call last):
File "C:\Users\Kat\Desktop\linkscrape\cookie.py", line 1, in <module> from scrapy import *
File "C:\Users\Kat\Desktop\linkscrape\scrapy\__init__.py", line 27, in <module>
from . import _monkeypatches
File "C:\Users\Kat\Desktop\linkscrape\scrapy\_monkeypatches.py", line 2, in <module>
from six.moves import copyreg
ImportError: No module named six.moves
Is there any way that I can simply include this to make it easy to migrate the app from computer to computer, or does this have to be installed? Thanks.
Don't do that. You will hurt yourself.
To achieve easy portability, use virtualenv - it's a golden standard of Python development.
Create a file named requirements.txt in a root directory of your project, and write all necessary dependencies, like this:
python-dateutil==2.4.2
Scrapy==1.0.4
XlsxWriter==0.7.7
When you are setting up the environment, create a fresh virtualenv and then simply:
pip install -r requirements.txt
Voilà! You have a working environment.
In case of Scrapy, it's even more important because in production Scrapy is typically deployed using scrapyd, where having a correct package with a version number and fixed requirements is absolutely necessary.
Related
Running a locust (locust.io) script from the command line.
locust calls main.py which has the following imports:
from locust import HttpUser, between, task
from StreamLoader.stream_generator import * # thought this brings in everything
Packer.py has these imports:
from multipledispatch import dispatch
from PackedItem import PackedItem
StreamGenerator.py has:
import hashlib
from StreamLoader.Packer import Packer
from aes_encryption import AesEncryption
I used pip to install multipledispatch and when I run from within PyCharm it works fine, but from the command line I get the following.
File "C:\Users\guyl\PycharmProjects\engine-load-tests\engine_load_tester_locust\main.py", line 2, in <module>
from StreamLoader.stream_generator import *
File "C:\Users\guyl\PycharmProjects\engine-load-tests\StreamLoader\stream_generator.py", line 2, in <module>
from StreamLoader.Packer import Packer
File "C:\Users\guyl\PycharmProjects\engine-load-tests\StreamLoader\Packer.py", line 1, in <module>
from multipledispatch import dispatch
ModuleNotFoundError: No module named 'multipledispatch'
Here is what I have tried so far:
Add directories to the PYTHONPATH environment variable
Add empty __init__.py files in each package
This all seems unnecessary if I actually pip installed the module, though.
Answer below caused me to no longer see the error with multipledispatch. However, I now see a missing module error:
File "C:\Users\guyl\PycharmProjects\engine-load-tests\engine_load_tester_locust\main.py", line 2, in <module>
from StreamLoader.stream_generator import *
File "C:\Users\guyl\PycharmProjects\engine-load-tests\StreamLoader\stream_generator.py", line 2, in <module>
from Packer import Packer
ModuleNotFoundError: No module named 'Packer'
For clarity, I am running the code from locust which calls the Python code as depicted here. [Moderators - this question is getting quite long. Is that all right?]
PyCharm uses the virtual environment automatically but when you run from the command line the virtual environment isn't turned on.
Just follow the steps:
cd your_working_directory
environment_name/Scripts/activate if on Windows where environment_name is the name of the PyCharm virtual environment
Or environment_name/bin/activate if on MacOS
-------------EDIT------------------------
To answer the new question, try using pip freeze then see which packages are installed. Then install any dependencies which you want which are missing.
On Ubuntu 16.04, am suddenly getting import errors from the local GAE development server.
The local dev server starts up, including the admin interface, but app no longer loads.
Native python imports of the same library on same machine (in this case "from google.cloud import datastore") work fine.
The GAE standard app does run when deployed, but development just got a little challenging.
google-cloud is version 0.27.0
gcloud components is 172.0.1
python is Anaconda 2.7.13
GAE is standard
I have confirmed to the best of my middling abilities that $PATH is correct for all named libraries.
I have removed and re-added all the named libraries to no effect.
cachetools(2.0.1) it should probably be noted, is installed as a dependency of the google cloud libraries, so I don't think this is addressable through requirements.txt or "libraries" in app.yaml.
I did recently go through a cycle of removing and adding libraries to fix a problem with apache_beam 2.0.1, so I may have jacked up something else, but am not sure where to look.
Suggestions deeply appreciated. Full traceback (from admin, same as from app):
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 232, in handle_interactive_request
exec(compiled_code, self._command_globals)
File "<string>", line 1, in <module>
File "/home/brian/anaconda3/lib/python2.7/site-packages/google/cloud/datastore/__init__.py", line 61, in <module>
from google.cloud.datastore.client import Client
File "/home/brian/anaconda3/lib/python2.7/site-packages/google/cloud/datastore/client.py", line 23, in <module>
from google.cloud.client import ClientWithProject
File "/home/brian/anaconda3/lib/python2.7/site-packages/google/cloud/client.py", line 27, in <module>
from google.oauth2 import service_account
File "/home/brian/anaconda3/lib/python2.7/site-packages/google/oauth2/service_account.py", line 79, in <module>
from google.auth import jwt
File "/home/brian/anaconda3/lib/python2.7/site-packages/google/auth/jwt.py", line 49, in <module>
import cachetools
ImportError: No module named cachetools
The stacktrace indicates you're running the libraries from the local system installation (the site-packages dir), not from your app.
For standard env GAE apps you need to install the dependencies inside your app and they will be uploaded to GAE together with your app code.
More specifically you need to use the -t <your_app_lib_dir> option for the pip installation. From Installing a third-party library:
Use pip (version 6 or later) with the -t <directory> flag to copy the libraries into the folder you created in the previous
step. For example:
pip install -t lib/ <library_name>
I addressed my problem through the requirements.txt file in app root directory.
I had: google-cloud==0.22.0
and changed it to: google-cloud==0.27.0
which fixed it.
I am working in Ubuntu 14.04 and I have multiple versions of Python on my machine (they include python2.7 and python3.4). Few days back, I installed simplejson on my system. I don't remember how I did that but I guess it must be similar to pip install simplejson. However, now a strange problem has started appearing when I try installing any python package. For example, just now I tried installing Tkinter using sudo pip3.4 install Tkinter and it throws the following error:
Traceback (most recent call last):
File "/usr/local/bin/pip3.4", line 9, in <module>
load_entry_point('pip==1.5.4', 'console_scripts', 'pip3.4')()
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 351, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2363, in load_entry_point
return ep.load()
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2088, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/usr/lib/python3/dist-packages/pip/__init__.py", line 61, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/lib/python3/dist-packages/pip/vcs/subversion.py", line 4, in <module>
from pip.index import Link
File "/usr/lib/python3/dist-packages/pip/index.py", line 15, in <module>
from pip.wheel import Wheel, wheel_ext
File "/usr/lib/python3/dist-packages/pip/wheel.py", line 25, in <module>
from distlib.scripts import ScriptMaker
File "/usr/share/python-wheels/distlib-0.1.8-py2.py3-none-any.whl/distlib/scripts.py", line 15, in <module>
File "/usr/share/python-wheels/distlib-0.1.8-py2.py3-none-any.whl/distlib/resources.py", line 20, in <module>
File "/usr/share/python-wheels/distlib-0.1.8-py2.py3-none-any.whl/distlib/util.py", line 11, in <module>
ImportError: No module named 'json'
Sometimes I can fix this if the error tells me that in one of the files I have:
import json
which I simply convert to
import simplejson as json
I tried uninstalling simplejson:
sudo pip uninstall simplejson
but it gives me the same error: json not found.
Can anybody please help me fix this so that I would happily be able to install python packages? Thanks in advance.
Note: I do not have a definitive answer but will offer a series of steps you can try:
The first thing is see if you can import json from the usual python interpreter:
import json
print(json.__file__) #this would be important to know if it works
If that does work (as well as commenting what json.__file__ is) you would then want to try to use pip from the interpreter.
If you can't import json normally:
This is not surprising, I did not expect pip to be looking in a non-standard place for modules. You will want to figure out where the json package should be located on your computer, you can do this by importing another module from the standard library and looking at it's __file__:
>>> import fractions
>>> fractions.__file__
'/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/fractions.py'
This will obviously be different for you but I'd expect there to be a json folder in the same folder as fractions.py
if you can't import fractions or queue or datetime etc.
If you can't import anything from the standard library you will probably want to just reinstall python.
If the json folder is there and contains an __init__.py
Use the rename function of your file browser to make sure there are no weird special characters, but other then that I'm not sure, if you can import fractions.py but not a package from the same folder that would imply there is something very wrong with the import mechanics of your python version.
If the json folder is not with the rest of the standard library
It is possible that your python distribution has a different structure then I'd expect, it at least can't hurt to take a look for it.
You can search for the json folder amongst your various python files using the find command, not really sure how it works but just another thing to try. If you do find it with the __init__.py, encode.py, decode.py, scanner.py, and tool.py (at least those are the ones in my version) you'll probably want to figure out how it got there, but maybe just move it to the same folder as the rest of the standard library.
If you can't find the json package or you find it and it is corrupted
Well then you will need to replace it! Don't worry, this isn't too hard, just grab a source release of python from the site and extract the json package from it, once it is uncompressed the json folder should be in the Lib folder. Simply copy/move it to the rest of the standard library and you should be good to go!
I hope this helps you debug what is going on, This covers all the scenarios I could imagine happening and I would be interested in which one fixed your issue (or what you were able to figure out so I can come up with more options)
I am guessing that you install it using either pip install simplejson to download from PyPI or using apt-get install python-simplejson to download from ubuntu repositories.
It is possible that you downlaoded the library for the Python2 if you used any of the commands above and it won't be available for Python3 (which pip3.4 will use). Can you try these commands and help debug yourself?
$ python -c "import simplejson"
$ python3.4 -c "import simplejson"
This would tell you which version of the python did you install simplejson for last time (my guess in python2). If the 2nd command errors out with ImportError try:
$ pip3.4 install simplejson
and then install your libraries.
Dreamhost upgraded a number of servers this weekend, including the one I was on. It broke my configuration, so as recommended I attempted to delete the virtual environment it was running on and attempted to re-set it up. However, when I try to navigate to the site, I get this:
Traceback (most recent call last):
File "/home/thesp/mysite.com/env/lib/python2.7/site-packages/site.py", line 74, in <module>
__boot()
File "/home/thesp/mysite.com/env/lib/python2.7/site-packages/site.py", line 2, in __boot
import sys, os, os.path
File "/home/thesp/lib/python2.7/os.py", line 400, in <module>
import UserDict
File "/home/thesp/lib/python2.7/UserDict.py", line 83, in <module>
import _abcoll
File "/home/thesp/lib/python2.7/_abcoll.py", line 11, in <module>
from abc import ABCMeta, abstractmethod
File "/home/thesp/lib/python2.7/abc.py", line 8, in <module>
from _weakrefset import WeakSet
ImportError: No module named _weakrefset
I've got Python 2.7.8 installed and running, and from shell access, both in and out of my virtual environment when I run Python, I'm pulling up the correct version (which is different from the native version installed, so it's finding my setup). Other posts which reference this error message seem to think it's a problem with not having an upgraded version of virtualenv, but its version is higher than the troublesome version. (I'm running 1.11.6.)
Weirder still, I can shell into Python, type from _weakrefset import WeakSet, and I don't get import errors. I'm running Django 1.6 and I can python manage.py runserver with no errors, but the web server is throwing up before it ever sees Django.
In the traceback, the first two lines are pulling from my virtual environment, but the remaining ones don't seem to be, and I have no idea why, or even if that's relevant.
Any advice on what I should do next? I've about pulled my hair out on this one! I can post any additional information that would help troubleshoot. Thanks!
Well, I feel silly now. I went to the /home/thesp/lib/python2.7/ directory and downloaded _weakrefset.py, like so:
wget http://svn.python.org/projects/python/trunk/Lib/_weakrefset.py
...and now everything seems to be running fine. There was a _weakrefset.pyo file in the directory, so I'm not sure why _weakrefset.py never made it in, but this seems to have done the trick.
Now, that doesn't solve the mystery of why the stack trace switches directories like it does, but it's running now, so I'll take it for now!
I'm trying to install Plone 3.3rc4 with plone.app.blob and repoze but nothing I've tried has worked so far. For one attempt I've pip-installed repoze.zope2, Plone, and plone.app.blob into a virtualenv. I have this version of DocumentTemplate in the virtualenv's site-packages directory and I'm trying to get it running in RHEL5.
For some reason when I try to run paster serve etc/zope2.ini in this environment way Python gives the message ImportError: No module named DT_Util? DT_Util.py exists in the directory, __init__.py is there too, and the C module it depends on is there. I suspect there's some circular dependency or failure when importing the C extension. Of course this module would work in a normal Zope install...
>>> import DocumentTemplate
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "DocumentTemplate/__init__.py", line 21, in ?
File ".../lib/python2.4/site-packages/DocumentTemplate/DocumentTemplate.py", line 112, in ?
from DT_String import String, File
File ".../lib/python2.4/site-packages/DocumentTemplate/DT_String.py", line 19, in ?
from DocumentTemplate.DT_Util import ParseError, InstanceDict
ImportError: No module named DT_Util
I must say I doubt DocumentTemplate from Zope will work standalone. You are welcome to try though. :-)
Note that DT_Util imports C extensions:
from DocumentTemplate.cDocumentTemplate import InstanceDict, TemplateDict
from DocumentTemplate.cDocumentTemplate import render_blocks, safe_callable
from DocumentTemplate.cDocumentTemplate import join_unicode
You'll need to make sure those are compiled. My guess is that importing the cDocumentTemplate module fails and thus the import of DT_Util fails.