I want to use two Python libraries (Google's Cloud Library, and their Cloud SDK) in a single application, but they have conflicting names (they both use google in their base import names and do not use relative imports internally). How can I use them in a single app?
Changing the library's code to use proper relative imports is not practical. Also, I know I can use virtualenv to access these libraries from separate python applications, but how do I access them from within the same python app?
Details of the Naming Conflict
Here are some of the details on the import. When I import a module from the Cloud Library (I run import google.cloud.datastore), there is an exception about another import within that library:
>>> import libs.google.cloud.datastore
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "C:\[ProjectDIR]\libs\google\cloud\datastore\__init__.py", line 52, in <module>
from google.cloud.datastore.batch import Batch
ImportError: No module named cloud.datastore.batch
The library is trying to do an absolute import, rather than a relative one. The reason that the Google Cloud Library cannot import google.cloud.datastore.batch is because google is already defined in the SDK, there is a naming conflict:
>>> print google.__path__
['C:\\Program Files (x86)\\Google\\Cloud SDK\\google-cloud-sdk\\platform\\google_appengine\\google']
Because the Cloud Library uses absolute imports, and the name google is already defined in the SDK, then the import fails.
The google packages take care to register themselves as a namespace package. With a properly set up sys.path there is no conflict here.
You need to set up your library environment correctly. Add a appengine_config.py file in the root of your project with:
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
This adds the lib subdirectory in the right location of sys.path. See the Installing a third-party library section in the Developing Python Apps on App Engine How-To.
From here on out imports of google.cloud just work:
$ ls -1d lib *.py *.yaml
app.yaml
appengine_config.py
lib
main.py
$ pip install -t lib google-cloud
# installing into the lib subdirectory
$ cat main.py
import google
from google.cloud import datastore
from google.appengine.api import memcache
import os.path
here = os.path.dirname(os.path.abspath(__file__))
def app(*args, **kwargs):
return '''
google: {}<br />
google.cloud.datastore: {}<br />
google.appengine.api.memcache: {}'''.format(
os.path.relpath(google.__file__, here),
os.path.relpath(datastore.__file__, here),
os.path.relpath(memcache.__file__, here))
and in the browser I am served:
google: ../google-cloud-sdk/platform/google_appengine/google/__init__.py
google.cloud.datastore: lib/google/cloud/datastore/__init__.pyc
google.appengine.api.memcache: ../google-cloud-sdk/platform/google_appengine/google/appengine/api/memcache/__init__.pyc
Related
I have a few seperate pythone file and I am using them to import another py file. Modules that trying to import them are in seperate folder I code sample is below
from tez.library.image_crop import ImageCrop
from tez.library.image_process import ImageProcess
from tez.library.image_features import ImageFeatures
from tez.const.application_const import ApplicationConst
from tez.library.file_operation import FileOperation
this code is in where I want to start the py file using commond line as "python samples1.py" and thrown an error as below
Traceback (most recent call last): File "samples1.py", line 1, in
from tez.library.image_crop import ImageCrop ModuleNotFoundError: No module named 'tez'
folder structure :
.tez
-- library
---- image_crop.py
---- image_process.py
---- image_features.py
--src
---- samples1.py
Python version : 3.8
Pip : 20.0.2
Windows 10 Pro 1909
If you are building a package called tez (and since you tried to import it I think you are). Then everything with tez needs to refer to itself locally. All the files in the tez package need to refer to each other with the "." and ".." imports.
In samples1.py:
from ..library.image_crop import <something>
EDIT:
It sounds like you are misunderstanding how python imports things. When you run "import X" in a python script, then python looks for a package named X under sys.path. You can append to sys.path at the top of your script if you have a custom package to look for.
import sys
sys.path.append(<directory of tez>)
import tez
However, it is strongly recommended that you should not be importing from a file that is under the directory structure of the package name. If "examples" is a directory of examples that use the package "tez" then "examples" should be located outside the package "tez". If "examples" is inside the package "tez", then "examples" should be doing local imports "with-in" the package.
Getting a handle on package use can be tricky.
sample.py can't see above of src folder, but you can tell Python to do this.:
import sys
import os
tez = os.path.dirname(os.path.dirname(__file__))
# __file__ is path of our file (samples.py)
# dirname of __file__ is "src" in our state
# dirname of "src" is "tez" in our state
sys.path.append(tez) # append tez to sys.path, python will look at here when you try import something
import library.image_crop # dont write "tez"
But this is not a very good design I think.
My file structure is as follows:
monitor/
core/
database.py
processor.py
timekeeper.py
jobs/
jobA.py
jobB.py
setup.py
From jobA.py I import like this:
from core.database import Database
from core.timekeeper import Timekeeper
from core.processor import Processor
While at database.py, processor.py and timekeeper.py I import setup.py.
Get the following error when I run jobA.py:
root#test:/var/www/python/monitor# python3 jobs/jobA.py
Traceback (most recent call last):
File "jobs/jobA.py", line 2, in <module>
from core.database import Database
ModuleNotFoundError: No module named 'core'
To allow import core or import core.database (without the relative dots or double-dots) the parent directory of core should either be the current directory, or be included on sys.path. You appear to have a setup.py. Conventionally that means a file that performs installation and packaging tasks via the setuptools or distutils packages. If that is indeed the role it performs, perhaps you need to run it. One way to run it would be to issue (from the command-line outside Python) the command pip install -e /path/to/monitor. Assuming setup.py was written correctly, this will ensure that the core package, in its current location, is lastingly made available for the default Python distribution. Next time you launch Python, /path/to/monitor will be on sys.path and import core will work from (almost) anywhere.
Add
import os, sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
to the top of your jobA.py file. If you are using python 3.3+, you do not need an __init__.py file. It needs to be above your other import statements.
From this answer you can use 2 dots to import from a directory above. So you could potentially use:
from .core.database import Database
from .core.timekeeper import Timekeeper
from .core.processor import Processor
Python 3.3+ you don’t need an __init__.py file so I don’t believe just adding one will help.
What module are you trying to use? Maybe your module is not compatible with Python 3.
I am using google-auth to allow firebase authentication in my GAE project.
Everything works fine when I run the code locally using dev_appserver.py or when I deploy it to google app engine.
But I get this ImportError exceptions when I try to use Django's manage.py script to create/run migrations.
ImportError: Could not import 'firebase.authentication.FirebaseAuthentication' for API setting 'DEFAULT_AUTHENTICATION_CLASSES'. ImportError: No module named auth.transport.requests.
The google-auth module is installed under lib directory and has this structure:
- lib
- google
- auth
- oauth2
These import cause the ImportErrors:
import google.auth.transport.requests
from google.oauth2 import id_token
My guess is that there might be naming conflicts as other imports work fine.
Please help!
If you want to use 3rd party libraries that are not included in this list, then you'll have to add them manually since you have done that by adding lib folder and including all the package folder follow these steps.
Create your_app_directory/appengine_config.py file.
Add these following lines inside that file
from google.appengine.ext import vendor
vendor.add('lib')
This should solve the import error problem.
I have a Google App Engine project named gaend. I am attempting to build better documentation for it using readthedocs. I have used
> sphinx-quickstart
to build my base sphinx project. Followed by
> sphinx-apidoc -o . ../gaend
to generate a API list of the project. I then run
sphinx-autobuild . _build/html
+--------- manually triggered build ---------------------------------------------
| Running Sphinx v1.5.2
| loading pickled environment... done
| building [mo]: targets for 0 po files that are out of date
| building [html]: targets for 0 source files that are out of date
| updating environment: 0 added, 1 changed, 0 removed
| reading sources... [100%] gaend
/Users/stephen/gaend/docs/gaend.rst:10: WARNING: autodoc: failed to import module u'gaend.bigquery'; the following exception was raised:
Traceback (most recent call last):
File "/Users/stephen/.virtualenvs/gaend/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
__import__(self.modname)
File "/Users/stephen/gaend/gaend/bigquery.py", line 1, in <module>
from google.cloud import bigquery
ImportError: No module named google.cloud
...
ImportError: No module named google.appengine.ext
I recognize that the problem is that it does not have access to the Google Cloud SDK. I have had similar issues with running my unit test, and the work around to doing this is in runner.py. However, this relies on having the Google Cloud SDK (with the Python Google App Engine module installed) somewhere on my system. How am I going to get the Google Cloud SDK (and Python GAE) on the readthedocs server that build my documentation?
I was the same problem, I don't remember where I found the solution and maybe don't fix your problem but I solved mine adding the path of google_appengine folder and google_appengine/lib/yam/lib folder in conf.py file in your source folder inside of Sphinx doc folder, something like this:
In conf.py:
sys.path.insert(0, os.path.abspath('../../'))
sys.path.insert(1, '<your local full path>/google_appengine')
sys.path.insert(1, '<your local full path>/google_appengine/lib/yaml/lib')
if 'google' in sys.modules:
del sys.modules['google']
I know that it would be better with relative path but it worked for me.
That Sphinx need run the program that you are documenting is something that I don't understand fine but anyway, is a good tool.
I just solved the same issue. There are not many questions asking about Sphinx + GAE, so I'm posting the conf.py code that got it working for me in the hopes it helps others:
import os
import sys
project_id = 'gae-project-id'
# environment variables that several utils are assuming are present
if not os.environ.get('SERVER_SOFTWARE', None):
os.environ['SERVER_SOFTWARE'] = 'Development-'+project_id
if not os.environ.get('APPLICATION_ID', None):
os.environ['APPLICATION_ID'] = 'dev~'+project_id
# project root
sys.path.insert(0, os.path.dirname(__file__))
# downloaded third party libs
sys.path.insert(0, os.path.dirname(__file__)+"/lib")
# path to gae sdk
sdk_path = os.path.join('~/google-cloud-sdk', 'platform/google_appengine')
try:
import google
google.__path__.append("{0}/google".format(sdk_path))
except ImportError:
pass
sys.path.insert(0, os.path.expanduser(sdk_path))
import dev_appserver
dev_appserver.fix_sys_path()
For my gae python project, I'd like to import an external library named 'vobject'. What's the correct way to import it in my .py files?
The project's readme says that in order to install it, you need to run
python setup.py install
Additionally, vobject requires the 'dateutil' package.
Since this is going to run on GAE, I thought I should copy both libs over into my project instead of running the install script to make use of it in my code.
But I'm getting a bunch of import errors and I'm not sure what the correct convention is for external gae/python libs.
utc = dateutil.tz.tzutc()
## error produced:
File "myGaeProject/external/vobject/icalendar.py", line 47, in <module>
NameError: name 'dateutil' is not defined
Because of the way I've structured my project, I changed icalendar.py's import structure from:
import dateutil.rrule
import dateutil.tz
to:
import external.dateutil.rrule
import external.dateutil.tz
I also tried:
from external.dateutil import *
What's the correct import mechanism for a project structured like so:
-myGaeProject
--external
----__init__.py
----dateutil
------__init__.py
------tz.py
------rrule.py
------[more dateutil files]
----vobject
------__init__.py
------base.py
------icalendar.py
--handlers
------__init__.py
------mainHandler.py
Don't modify the library. If you want to put all your libraries in external, you need to add external to your python path before you attempt to import libraries from there:
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), 'external'))
import some_external_library
You can't do from external import dateutil if external is missing an __init__.py file.
The good way is to use zipimport, you can check the project jaikuengine they are a lot of good things about that.
http://code.google.com/p/jaikuengine/source/browse/trunk/build.py
In Jaiku, all external libs are stocked in the directory vendor but if you see the app.yaml, all files in vendor are skipped.
Jaiku uses a script to build a zip of each libs in vendor and put it to the root of the project before the deployment or when the dev_server is launched.
With that, you don't need to fix the path of your libs.
EDIT an example to load all zipped archives
Largely inspired from jaikuengine:
def load_zipped(path='.'):
for x in os.listdir(path):
if x.endswith('.zip'):
if not any([y.endswith(x) for y in sys.path]):
sys.path.append(os.path.abspath('%s/%s') % (path, x))