How can I tell readthedocs not to load any Python files located at top root folder?
Background: I would like readthedocs to build Sphinx documentation for a Blender Python add-on, hosted on github. I have created docs folder in the repository for Sphinx files. I can build Sphinx docs OK on my local computer (in docs directory), but the build on readthedocs fails on error
File "/home/docs/checkouts/readthedocs.org/user_builds/bvtknodes/envs/latest/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 20, in <module>
from pip._vendor import pkg_resources
File "/home/docs/checkouts/readthedocs.org/user_builds/bvtknodes/envs/latest/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 41, in <module>
import inspect
File "/home/docs/checkouts/readthedocs.org/user_builds/bvtknodes/checkouts/latest/inspect.py", line 1, in <module>
import bpy
ModuleNotFoundError: No module named 'bpy'
The reason for this error was that there was inspect.py module located at root directory, so readthedocs setup process loaded that instead of system version of inspect module. When I renamed my inspect.py to b_inspect.py, then the docs building process finished properly at readthedocs. However this is just a workaround, is there a better solution?
Original wrong interpretation: It seems that readthedocs is trying to inspect __init__.py (located at top folder level and can't be moved from there to a subfolder) contents and build fails when it can't import bpy. AFAIK there is no way to install bpy module via pip (but I wouldn't want to do that anyway). I'd like readthedocs to ignore all python files. language = None and extensions list in docs/conf.py is empty. I've also set "Python configuration file" setting in readthedocs Advanced Settings to docs/conf.py. Thanks!
Here is link to readthedocs build which failed.
Related
I am experimenting with Google Cloud Platform buildpacks, specifically for Python. I started with the Sample Functions Framework Python example app, and got that running locally, with commands:
pack build --builder=gcr.io/buildpacks/builder sample-functions-framework-python
docker run -it -ePORT=8080 -p8080:8080 sample-functions-framework-python
Great, let's see if I can apply this concept on a legacy project (Python 3.7 if that matters).
The legacy project has a structure similar to:
.gitignore
source/
main.py
lib
helper.py
requirements.txt
tests/
<test files here>
The Dockerfile that came with this project packaged the source directory contents without the "source" directory, like this:
COPY lib/ /app/lib
COPY main.py /app
WORKDIR /app
... rest of Dockerfile here ...
Is there a way to package just the contents of the source directory using the buildpack?
I tried to add this config to the project.toml file:
[[build.env]]
name = "GOOGLE_FUNCTION_SOURCE"
value = "./source/main.py"
But the Python modules/imports aren't set up correctly for that, as I get this error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
Putting both main.py and /lib into the project root dir would make this work, but I'm wondering if there is a better way.
Related question, is there a way to see what project files are being copied into the image by the buildpack? I tried using verbose logging but didn't see anything useful.
Update:
The python module error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
was happening because I moved the lib dir into source in my test project, and when I did this, Intellij updated the import statement in main.py without me catching it. I fixed the import, then applied the solution listed below and it worked.
I had been searching the buildpack and Google cloud function documentation, but I discovered the option I need on the pack build documentation page: option --path.
This command only captures the source directory contents:
pack build --builder=gcr.io/buildpacks/builder --path source sample-functions-framework-python
If changing the path, the project.toml descriptor needs to be in that directory too (or specify with --descriptor on command line).
My Apache beam pipeline implements custom Transforms and ParDo's python modules which further imports other modules written by me. On Local runner this works fine as all the available files are available in the same path. In case of Dataflow runner, pipeline fails with module import error.
How do I make custom modules available to all the dataflow workers? Please advise.
Below is an example:
ImportError: No module named DataAggregation
at find_class (/usr/lib/python2.7/pickle.py:1130)
at find_class (/usr/local/lib/python2.7/dist-packages/dill/dill.py:423)
at load_global (/usr/lib/python2.7/pickle.py:1096)
at load (/usr/lib/python2.7/pickle.py:864)
at load (/usr/local/lib/python2.7/dist-packages/dill/dill.py:266)
at loads (/usr/local/lib/python2.7/dist-packages/dill/dill.py:277)
at loads (/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py:232)
at apache_beam.runners.worker.operations.PGBKCVOperation.__init__ (operations.py:508)
at apache_beam.runners.worker.operations.create_pgbk_op (operations.py:452)
at apache_beam.runners.worker.operations.create_operation (operations.py:613)
at create_operation (/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py:104)
at execute (/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py:130)
at do_work (/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py:642)
The issue is probably that you haven't grouped your files as a package. The Beam documentation has a section on it.
Multiple File Dependencies
Often, your pipeline code spans multiple files. To run your project remotely, you must group these files as a Python package and specify the package when you run your pipeline. When the remote workers start, they will install your package. To group your files as a Python package and make it available remotely, perform the following steps:
Create a setup.py file for your project. The following is a very basic setup.py file.
setuptools.setup(
name='PACKAGE-NAME'
version='PACKAGE-VERSION',
install_requires=[],
packages=setuptools.find_packages(),
)
Structure your project so that the root directory contains the setup.py file, the main workflow file, and a directory with the rest of the files.
root_dir/
setup.py
main.py
other_files_dir/
See Juliaset for an example that follows this required project structure.
Run your pipeline with the following command-line option:
--setup_file /path/to/setup.py
Note: If you created a requirements.txt file and your project spans multiple files, you can get rid of the requirements.txt file and instead, add all packages contained in requirements.txt to the install_requires field of the setup call (in step 1).
I ran into the same issue and unfortunately, the docs are not as verbose as they need to be.
So, the problem as it turns out is that both the root_dir and the other_files_dir must contain an __init__.py file. When a directory contains an __init__.py file (even if it's empty) python will treat that directory as a package, which in this instance is what we want. So, your final folder structure should look something like this:
root_dir/
__init__.py
setup.py
main.py
other_files_dir/
__init__.py
module_1.py
module_2.py
And what you'll find is that python will build an .egg-info folder that describes your package including all pip dependencies. It will also contain the top_level.txt file which contains the name of the directory that holds the modules (i.e other_files_dir)
Then you would simply call the modules in main.py as below
from other_files_dir import module_1
I have a Google App Engine project named gaend. I am attempting to build better documentation for it using readthedocs. I have used
> sphinx-quickstart
to build my base sphinx project. Followed by
> sphinx-apidoc -o . ../gaend
to generate a API list of the project. I then run
sphinx-autobuild . _build/html
+--------- manually triggered build ---------------------------------------------
| Running Sphinx v1.5.2
| loading pickled environment... done
| building [mo]: targets for 0 po files that are out of date
| building [html]: targets for 0 source files that are out of date
| updating environment: 0 added, 1 changed, 0 removed
| reading sources... [100%] gaend
/Users/stephen/gaend/docs/gaend.rst:10: WARNING: autodoc: failed to import module u'gaend.bigquery'; the following exception was raised:
Traceback (most recent call last):
File "/Users/stephen/.virtualenvs/gaend/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
__import__(self.modname)
File "/Users/stephen/gaend/gaend/bigquery.py", line 1, in <module>
from google.cloud import bigquery
ImportError: No module named google.cloud
...
ImportError: No module named google.appengine.ext
I recognize that the problem is that it does not have access to the Google Cloud SDK. I have had similar issues with running my unit test, and the work around to doing this is in runner.py. However, this relies on having the Google Cloud SDK (with the Python Google App Engine module installed) somewhere on my system. How am I going to get the Google Cloud SDK (and Python GAE) on the readthedocs server that build my documentation?
I was the same problem, I don't remember where I found the solution and maybe don't fix your problem but I solved mine adding the path of google_appengine folder and google_appengine/lib/yam/lib folder in conf.py file in your source folder inside of Sphinx doc folder, something like this:
In conf.py:
sys.path.insert(0, os.path.abspath('../../'))
sys.path.insert(1, '<your local full path>/google_appengine')
sys.path.insert(1, '<your local full path>/google_appengine/lib/yaml/lib')
if 'google' in sys.modules:
del sys.modules['google']
I know that it would be better with relative path but it worked for me.
That Sphinx need run the program that you are documenting is something that I don't understand fine but anyway, is a good tool.
I just solved the same issue. There are not many questions asking about Sphinx + GAE, so I'm posting the conf.py code that got it working for me in the hopes it helps others:
import os
import sys
project_id = 'gae-project-id'
# environment variables that several utils are assuming are present
if not os.environ.get('SERVER_SOFTWARE', None):
os.environ['SERVER_SOFTWARE'] = 'Development-'+project_id
if not os.environ.get('APPLICATION_ID', None):
os.environ['APPLICATION_ID'] = 'dev~'+project_id
# project root
sys.path.insert(0, os.path.dirname(__file__))
# downloaded third party libs
sys.path.insert(0, os.path.dirname(__file__)+"/lib")
# path to gae sdk
sdk_path = os.path.join('~/google-cloud-sdk', 'platform/google_appengine')
try:
import google
google.__path__.append("{0}/google".format(sdk_path))
except ImportError:
pass
sys.path.insert(0, os.path.expanduser(sdk_path))
import dev_appserver
dev_appserver.fix_sys_path()
I'm following this tutorial
https://developers.google.com/bigquery/articles/dashboard#downloadinstalllibraries
Here they say that
cd source/
enable-app-engine-project hellodashboard/
which will copy some 60 files into the project directory.But i saw only some 10 files copied.And the oauth2client folder in project directory only has __init__.pyfile
Since i'm using
from oauth2client.appengine import oauth2decorator_from_clientsecrets
When i run my program it shows an error
from oauth2client.appengine import oauth2decorator_from_clientsecrets
ImportError: No module named appengine
So i manually copied all files from oauth2client zip to the project oauth2client directory.
Now when i run my program it doesn't show any error and seems to run fine.Is it some thing messy with the enable-app-engine-project command or am i doing some thing wrong.
enable-app-engine-project tries to locate sources based on the path(s) where the modules are installed on your local system. Have you installed these modules locally?
As an alternative, you can simply copy the following directories into your App Engine project manually:
SOURCES = [
'gflags',
'gflags_validators',
'httplib2',
'oauth2client',
'oauth2',
'apiclient',
'uritemplate',
]
I have a Python project in which I am using many non-code files. Currently these are all images, but I might use other kinds of files in the future. What would be a good scheme for storing and referencing these files?
I considered just making a folder "resources" in the main directory, but there is a problem; Some images are used from within sub-packages of my project. Storing these images that way would lead to coupling, which is a disadvantage.
Also, I need a way to access these files which is independent on what my current directory is.
You may want to use pkg_resources library that comes with setuptools.
For example, I've made up a quick little package "proj" to illustrate the resource organization scheme I'd use:
proj/setup.py
proj/proj/__init__.py
proj/proj/code.py
proj/proj/resources/__init__.py
proj/proj/resources/images/__init__.py
proj/proj/resources/images/pic1.png
proj/proj/resources/images/pic2.png
Notice how I keep all resources in a separate subpackage.
"code.py" shows how pkg_resources is used to refer to the resource objects:
from pkg_resources import resource_string, resource_listdir
# Itemize data files under proj/resources/images:
print resource_listdir('proj.resources.images', '')
# Get the data file bytes:
print resource_string('proj.resources.images', 'pic2.png').encode('base64')
If you run it, you get:
['__init__.py', '__init__.pyc', 'pic1.png', 'pic2.png']
iVBORw0KGgoAAAANSUhE ...
If you need to treat a resource as a fileobject, use resource_stream().
The code accessing the resources may be anywhere within the subpackage structure of your project, it just needs to refer to subpackage containing the images by full name: proj.resources.images, in this case.
Here's "setup.py":
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='proj',
packages=find_packages(),
package_data={'': ['*.png']})
Caveat:
To test things "locally", that is w/o installing the package first, you'll have to invoke your test scripts from directory that has setup.py. If you're in the same directory as code.py, Python won't know about proj package. So things like proj.resources won't resolve.
The new way of doing this is with importlib. For Python versions older than 3.7 you can add a dependency to importlib_resources and do something like
from importlib_resources import files
def get_resource(module: str, name: str) -> str:
"""Load a textual resource file."""
return files(module).joinpath(name).read_text(encoding="utf-8")
If your resources live inside the foo/resources sub-module, you would then use get_resource like so
resource_text = get_resource('foo.resources', 'myresource')
You can always have a separate "resources" folder in each subpackage which needs it, and use os.path functions to get to these from the __file__ values of your subpackages. To illustrate what I mean, I created the following __init__.py file in three locations:
c:\temp\topp (top-level package)
c:\temp\topp\sub1 (subpackage 1)
c:\temp\topp\sub2 (subpackage 2)
Here's the __init__.py file:
import os.path
resource_path = os.path.join(os.path.split(__file__)[0], "resources")
print resource_path
In c:\temp\work, I create an app, topapp.py, as follows:
import topp
import topp.sub1
import topp.sub2
This respresents the application using the topp package and subpackages. Then I run it:
C:\temp\work>topapp
Traceback (most recent call last):
File "C:\temp\work\topapp.py", line 1, in
import topp
ImportError: No module named topp
That's as expected. We set the PYTHONPATH to simulate having our package on the path:
C:\temp\work>set PYTHONPATH=c:\temp
C:\temp\work>topapp
c:\temp\topp\resources
c:\temp\topp\sub1\resources
c:\temp\topp\sub2\resources
As you can see, the resource paths resolved correctly to the location of the actual (sub)packages on the path.
Update: Here's the relevant py2exe documentation.
# pycon2009, there was a presentation on distutils and setuptools. You can find all of the videos here
Eggs and Buildout Deployment in Python - Part 1
Eggs and Buildout Deployment in Python - Part 2
Eggs and Buildout Deployment in Python - Part 3
In these videos, they describe how to include static resources in your package. I believe its in part 2.
With setuptools, you can define dependancies, this would allow you to have 2 packages that use resources from 3rd package.
Setuptools also gives you a standard way of accessing these resources and allows you to use relative paths inside of your packages, which eliminates the need to worry about where your packages are installed.