How to add third party python libraries in Google App Engine, which are not provided by Google? I am trying to use BeautifulSoup in Google App Engine and unable to do so. But my question is for any library I want to use in Google App Engine.
Google has provided a documented way for included third-party libraries in your GAE project.
See the "Adding Third-party Packages to the Application" section of the Libraries in Python 2.7 docs.
If you want to include additional pure-python third-party packages, you can do so by setting up vendoring. Vendoring allows you to install packages to a subdirectory of your project and include them in your code. To use vendoring, create (or modify) appengine_config.py in the root of your project.
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
And then just put all your libs' source code in your lib dir
> pip install beautifulsoup4 -t lib
So your project directory structure looks like this:
project
- lib
- bs4
- your_code.py
This will allow your project's source files to import libs' packages/modules as if they were added to your PYTHON_PATH. For example:
# file: your_code.py
import bs4 # no need for 'from lib import bs4'
# do stuff with bs4...
You can also easily install everything from a requirements.txt file by doing the following command
> pip install -t lib -r requirements.txt
Actually I think this answer fits better here.
If you want to use 3rd party libraries that are not included in this list, then you'll have to add them manually.
In order to include manually any other library you have to have them inside the directory where the app.yaml lives. So for example if you have the following structure:
hello
├── libs
│ └── bs4
├── hello.py
└── app.yaml
then in your hello.py you have to put these two lines in the beginning of the file:
import sys
sys.path.insert(0, 'libs')
After doing that you'll be able to use any 3rd party library that you're going to put in that libs directory.
For example:
from bs4 import BeautifulSoup
You simply copy the folder containing the library you want to use into your app engine project.
Then when you deploy it's uploaded with your application and is available for use.
EDIT: Jesse's answer is how I now do this. So do that!
The way it worked here is:
import sys
# sys.path.insert(0, 'libs') #"Old" way, not working for me.
sys.path.append(os.path.join(os.path.dirname(__file__), "libs")) # This works!
Then import normally:
from bs4 import BeautifulSoup
Just put Beautifulsoup in the root of your project and upload it all
pip install -t lib package_name
lib: the location for third party libraries
Then you are good to use this package like a normal library you use from ipython or terminal.
Related
tl;dr - My python package requires import package.package instead of working with just import package. How do I get it to work with the latter?
I am trying to set up my first python package and am running into some issues with the import part of the process.
My python package file setup looks like this on my computer:
my-package
- build
- dist
- package
- package.egg-info
- LICENSE
- README.md
- Setup.py
Inside package is the following:
__init__.py
package.py
__init__.py reads name = 'package', and package.py contains all of the content of the package.
EDIT: I have attempted using various different versions of __init__.py, including adding import package, import package.package, or import package.package as package below the line name = 'package', but all of resulted in the same issue.
Using the Packaging Python Projects tutorial, I've been able to get my package upload to TestPyPi, but when I install the package to my computer, none of the functions/methods are available, and when I run "import package" and do help(package), I get the following:
Help on package package:
NAME
package
PACKAGE CONTENTS
package
DATA
name = 'package'
FILE
url/to/package
When I run import package.package and help(package), I can access the methods/functions, and get the expected help text for the package's content.
My question is, how to I configure the package file on my computer in such a way that once it is upload to TestPyPi and then downloaded, import package works, instead of needing to run import package.package?
When you write import package, you can access names in package/__init__.py as package.foo.
So, inside __init__.py, if you import all the necessary functions/variables/etc from package.py, those names will be visible to clients that just import package.
So, if you have this in package/__init__.py:
from .package import (foo, bar, baz)
Then in your other code you can do this:
from package import foo
And you dont have to worry about from package.package import foo.
As the title says.
So I've added the following to appengine_config.py with no luck:
# appengine_config.py
from google.appengine.ext import vendor
# Add any libraries install in the "lib" folder.
vendor.add('lib')
I did a print sys.path and verified the lib dir contains google/cloud/bigquery
I can import it if I run python myself:
from google.cloud import bigquery
print bigquery.__path__
['/usr/local/lib/python2.7/dist-packages/google/cloud/bigquery']
From a Google App Engine end points:
import google.cloud;print google.cloud.__path__
['/usr/local/lib/python2.7/dist-packages/google/cloud']
Big Query is at that system location. /usr/local/lib/python2.7/dist-packages/google/cloud/biqguery/ exists. However if I try the following from app engine end point:
from google.cloud import bigquery
ImportError: No module named google.cloud.bigquery
The file in question includes and that does not appear to help:
from __future__ import absolute_import
Update
I setup a venv and install everything like this: pip install -t lib/ -r requirements.txt --upgrade. From there if I try import google; print google.__path__ I get:
['lib/google', '/usr/lib/google-cloud-sdk/platform/google_appengine/google']
I think the second path might be the cause.
I've reviewed the following with no success:
Error importing Google Cloud Bigquery api module in python app
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2366
How to import BigQuery in AppEngine for Python
You need to install google-cloud-bigquery into you app's lib dir, that's where the development server is looking at, not on your system's libs. From Installing a third-party library:
Create a directory to store your third-party libraries, such as lib/.
mkdir lib
Use pip (version 6 or later) with the -t <directory> flag to copy the libraries into the folder you created in the previous
step. For example:
pip install -t lib/ <library_name>
I am trying to develop a simple forum site for my udacity assignment. It is not a strict requirement to use the bcrypt for password hashing, but I'd like to do it because I also like to know how to use third party libraries which are not provided by Google.
Following instructions provided here (installing a third-party library), I have created a folder named lib, and installed bcrypt library with following command:
python -m pip install -t lib/ bcrypt
I have the lib folder automatically structred like this:
I also created an appengine_config.py file with following content, as per instructions in above manual:
# appengine_config.py
from google.appengine.ext import vendor
# add lib folder as vendor directory
vendor.add('lib')
At this point, I am unable to import the bcrypt to my scripts. The import commands I tried so far are as follows:
from lib import bcrypt
ImportError: No module named lib
import bcrypt
ImportError: No module named bcrypt._bcrypt
from lib.bcrypt import bcrypt
ImportError: No module named lib.bcrypt
What am I missing?
As Avinash Raj pointed out, and as already pointed out in referenced manual, one cannot use python libraries with c extensions. So I downloaded the py-bcrypt, it worked like a charm.
For any newbie like me who needs it, here is the steps you have to take:
Inside your project folder, create a folder called "lib"
Extract the zip downloaded from github above, to the folder 'lib'. Do not use
- in your folder name. Name it something like pybcrypt
Create the appengine_config.pyfile, as outlined in here
Import the library to your script, like so: from pybcrypt import bcrypt
Pat yourself on the back.
Here is another option, you need to setup Wheel package before you can import bcrypt
pip install wheel
pip install bcrypt
from flask_bcrypt import Bcrypt
I'm using the following python project structure in order to allow packaging (based on this):
In each package I import all the submodules using
__init__.py
When i'm inside config.py and trying to
import settings
it does not identify settings as a package of modules..
How should I refer to that so it will recognise it?
So the problem was that because everything was inside as_manager package, I should have import as_manager.settings ...
For my gae python project, I'd like to import an external library named 'vobject'. What's the correct way to import it in my .py files?
The project's readme says that in order to install it, you need to run
python setup.py install
Additionally, vobject requires the 'dateutil' package.
Since this is going to run on GAE, I thought I should copy both libs over into my project instead of running the install script to make use of it in my code.
But I'm getting a bunch of import errors and I'm not sure what the correct convention is for external gae/python libs.
utc = dateutil.tz.tzutc()
## error produced:
File "myGaeProject/external/vobject/icalendar.py", line 47, in <module>
NameError: name 'dateutil' is not defined
Because of the way I've structured my project, I changed icalendar.py's import structure from:
import dateutil.rrule
import dateutil.tz
to:
import external.dateutil.rrule
import external.dateutil.tz
I also tried:
from external.dateutil import *
What's the correct import mechanism for a project structured like so:
-myGaeProject
--external
----__init__.py
----dateutil
------__init__.py
------tz.py
------rrule.py
------[more dateutil files]
----vobject
------__init__.py
------base.py
------icalendar.py
--handlers
------__init__.py
------mainHandler.py
Don't modify the library. If you want to put all your libraries in external, you need to add external to your python path before you attempt to import libraries from there:
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), 'external'))
import some_external_library
You can't do from external import dateutil if external is missing an __init__.py file.
The good way is to use zipimport, you can check the project jaikuengine they are a lot of good things about that.
http://code.google.com/p/jaikuengine/source/browse/trunk/build.py
In Jaiku, all external libs are stocked in the directory vendor but if you see the app.yaml, all files in vendor are skipped.
Jaiku uses a script to build a zip of each libs in vendor and put it to the root of the project before the deployment or when the dev_server is launched.
With that, you don't need to fix the path of your libs.
EDIT an example to load all zipped archives
Largely inspired from jaikuengine:
def load_zipped(path='.'):
for x in os.listdir(path):
if x.endswith('.zip'):
if not any([y.endswith(x) for y in sys.path]):
sys.path.append(os.path.abspath('%s/%s') % (path, x))