We Keep Coding

Trying to read JSON file within a Python package

I am in the process of packaging up a python package that I'll refer to as MyPackage.
The package structure is:
MyPackage/
script.py
data.json
The data.json file comprises cached data that is read in script.py.
I have figured out how to include data files (use of setuptools include_package_data=True and to also include path to data file in the MANIFEST.in file) but now when I pip install this package and import the installed MyPackage (currently testing install by pip from the GitHub repository) I get a FileNotFound exception (data.json) in the script that is to utilize MyPackage. However, I see that the data.json file is indeed installed in Lib/site-packages/MyPackage.
Am I doing something wrong here by trying to read in a json file in a package?
Note that in script.py I am attempting to read data.json as open('data.json', 'r')
Am I screwing up something regarding the path to the data file?

You're not screwing something up, accessing package resources is just a little tricky - largely because they can be packaged in formats where your .json might strictly speaking not exist as an actual file on the system where your package is installed (e.g. as zip-app). The right way to access your data file is not by specifying a path to it (like "MyPackage/data.json"), but by accessing it as a resource of your installed package (like "MyPackage.data.json"). The distinction might seem pedantic, but it can matter a lot.
Anyway, the access should be done using the builtin importlib.resources module:
import importlib.resources
import json
with importlib.resources.open_text("MyPackage", "data.json") as file:
data = json.load(file)
# you should be able to access 'data' like a dictionary here
If you happen to work on a python version lower than 3.7, you will have to install it as importlib_resources from pyPI.

I resolved the issue by getting the 'relative path' to where the package is.
self.data = self.load_data(path=os.path.join(
os.path.dirname(os.path.abspath(__file__)),
'data.json'))
load_data just reads the data file
Any constructive criticism is still very much welcome. Not trying to write stupid code if I can't help it :)

What is Python looking for when it tries to "import"?

Imagine some hypothetical python code:
from turboencabulator import panametricfan
If i try to run this on my computer (Windows, 64-bit, Windows 10), it fails to find the turboencabulator module:
What is it looking for?
When Python is attempting to load a module, what is it looking for? Is it a .py file? Is it a .dll (since this is Windows). Is it a folder?
Windows has modules; they're called assemblies
Windows has the notion of assemblies. If you declare you want to use an assembly called turboencabulator, it will look for an assembly manifest:
turboencabulator.manifest
You can also create a folder called turboencabulator, where it will look for an assemebly manifest there:
turboencabulator
|--- turboencabulator.manifest
That manifest will then in turn declare all the files that make up that assembly.
For example, if i need to use the Microsoft Visual C Runtime 8.0, i would declare a dependency on the module
Microsoft.VC80.CRT
Then Windows will go hunting for:
Microsoft.VC80.CRT.manifest file in the folder
Microsoft.VC80.CRT folder
which then must contain Microsoft.VC80.CRT.manifest
Some other places (like WinSxS, maybe the path, who knows)
And then the manifest is simply a text file that lets you know that msvcrt is comprised of 3 dlls:
<file name="msvcr80.dll" />
<file name="msvcp80.dll" />
<file name="msvcm80.dll" />
What is python looking for?
When i specify i want to import a module:
from RetroEncabulator import GirdleSpring
what is python looking for?
a binary?
a zip?
is it a named item that has been registered in the registry?
the name of a module manifest file that declares all the files that make up the module?
What, exactly, is python looking for?
Bonus Questions
where does it look
can it be configured to look places
can modules be registered somewhere, so it doesn't have to go hunting? (as correctly written Windows applications are supposed to do)
Bonus Chatter
in reality i'm from numpy
in reality i'm from cntk
And neither of these "things" are "found".
Bonus Reading
Python - Modules
Additional Investigation
To contrast the heresay so far, i decided to actually spy on python to see what it goes and looks for:
HKEY_CURRENT_USER/Software/Python/PythonCore/2.7/Modules/turboencabulator
HKEY_LOCAL_MACHINE/Software/Python/PythonCore/2.7/Modules/turboencabulator
D:\DevelopPy\Bunny\turboencabulator
D:\DevelopPy\Bunny\turboencabulator.pyd
D:\DevelopPy\Bunny\turboencabulator.py
D:\DevelopPy\Bunny\turboencabulator.pyw
D:\DevelopPy\Bunny\turboencabulator.pyc
D:\Programs\Python27\DLLs\turboencabulator
D:\Programs\Python27\Lib\turboencabulator
D:\Programs\Python27\Lib\site-packages\turboencabulator
I see it looking in the registry. Does that mean it can be registered?
I see it looking for files besides py (pyc, pyw, pyd). Are those compiled?
I see it looking in the Python\DLLs folder. Does that means that there can be DLLs?
I see it looking in Lib? Are those for intermediate object binaries?
Man, it really has been four years without being able to figure it out. Sure would be super if the SO community was helpful. Well-earned reputation.

When you include import statement in your code, python tries to find the python file or package directory with __init__.py file in it the following order
current working directory
local user site packages
common site packages
You can see the path by the following code
import sys
print(sys.path)
You can also add your own path.
Note: You must add this line before importing other modules
import sys
sys.path.append("your/complete/path/to/directory")

Including a Python Library (suds) in a portable way

I'm using suds (brilliant library, btw), and I'd like to make it portable (so that everyone who uses the code that relies on it, can just checkout the files and run it).
I have tracked down 'suds-0.4-py2.6.egg' (in python/lib/site-packages), and put it in with my files, and I've tried:
import path.to.egg.file.suds
from path.to.egg.file.suds import *
import path.to.egg.file.suds-0.4-py2.6
The first two complain that suds doesn't exist, and the last one has invalid syntax.
In the __init__.py file, I have:
__all__ = [ "FileOne" ,
"FileTwo",
"suds-0.4-py2.6"]
and have previously tried
__all__ = [ "FileOne" ,
"FileTwo",
"suds"]
but neither work.
Is this the right way of going about it? If so, how can I get my imports to work. If not, how else can I achieve the same result?
Thanks

You must add your egg file to sys.path, like this:
import sys
# insert at 0 instead of appending to end to take precedence
# over system-installed suds (if there is one).
sys.path.insert(0, "suds-0.4-py2.6.egg")
import suds

.egg files are zipped archives; hence you cannot directly import them as you have discovered.
The easy way is to simply unzip the archive, and then copy the suds directory to your application's source code directory. Since Python will stop at the first module it discovers; your local copy of suds will be used even if it is not installed globally for Python.
One step up from that, is to add the egg to your path by appending it to sys.path.
However, the proper way would be to package your application for distribution; or provide a requirements file that lets other people know what external packages your program depends on.

Usually I distribute my program with a requirements.txt file that contain all dependencies and their version.
The users can then install these libraries with:
pip install -r requirements.txt
I don't think including eggs with your code is a good idea, what if the user use python2.7 instead of python2.6
More info about requirement file: http://www.pip-installer.org/en/latest/requirements.html

Adding Version Control / Numbering (?) to Python Project

With my Java projects at present, I have full version control by declaring it as a Maven project. However I now have a Python project that I'm about to tag 0.2.0 which has no version control. Therefore should I come accross this code at a later date, I won't no what version it is.
How do I add version control to a Python project, in the same way Maven does it for Java?

First, maven is a build tool and has nothing to do with version control. You don't need a build tool with Python -- there's nothing to "build".
Some folks like to create .egg files for distribution. It's as close to a "build" as you get with Python. This is a simple setup.py file.
You can use SVN keyword replacement in your source like this. Remember to enable keyword replacement for the modules that will have this.
__version__ = "$Revision$"
That will assure that the version or revision strings are forced into your source by SVN.
You should also include version keywords in your setup.py file.

Create a distutils setup.py file. This is the Python equivalent to maven pom.xml, it looks something like this:
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
If you want dependency management like maven, take a look at setuptools.

Ants's answer is correct, but I would like to add that your modules can define a __version__ variable, according to PEP 8, which can be populated manually or via Subversion or CVS, e.g. if you have a module thingy, with a file thingy/__init__.py:
___version___ = '0.2.0'
You can then import this version in setup.py:
from distutils.core import setup
import thingy
setup(name='thingy',
version=thingy.__version__,
py_modules=['thingy'],
)

Finding a file in a Python module distribution [duplicate]

This question already has answers here:
How to read a (static) file from inside a Python package?
(6 answers)
Closed 2 years ago.
I've written a Python package that includes a bsddb database of pre-computed values for one of the more time-consuming computations. For simplicity, my setup script installs the database file in the same directory as the code which accesses the database (on Unix, something like /usr/lib/python2.5/site-packages/mypackage/).
How do I store the final location of the database file so my code can access it? Right now, I'm using a hack based on the __file__ variable in the module which accesses the database:
dbname = os.path.join(os.path.dirname(__file__), "database.dat")
It works, but it seems... hackish. Is there a better way to do this? I'd like to have the setup script just grab the final installation location from the distutils module and stuff it into a "dbconfig.py" file that gets installed alongside the code that accesses the database.

Try using pkg_resources, which is part of setuptools (and available on all of the pythons I have access to right now):
>>> import pkg_resources
>>> pkg_resources.resource_filename(__name__, "foo.config")
'foo.config'
>>> pkg_resources.resource_filename('tempfile', "foo.config")
'/usr/lib/python2.4/foo.config'
There's more discussion about using pkg_resources to get resources on the eggs page and the pkg_resources page.
Also note, where possible it's probably advisable to use pkg_resources.resource_stream or pkg_resources.resource_string because if the package is part of an egg, resource_filename will copy the file to a temporary directory.

Use pkgutil.get_data. It’s the cousin of pkg_resources.resource_stream, but in the standard library, and should work with flat filesystem installs as well as zipped packages and other importers.

That's probably the way to do it, without resorting to something more advanced like using setuptools to install the files where they belong.
Notice there's a problem with that approach, because on OSes with real a security framework (UNIXes, etc.) the user running your script might not have the rights to access the DB in the system directory where it gets installed.

Use the standard Python-3.7 library's importlib.resources module,
which is more efficient than setuptools:pkg_resources
(on previous Python versions, use the backported importlib_resources library).
Attention: For this to work, the folder where the data-file resides must be a regular python-package. That means you must add an __init__.py file into it, if not already there.
Then you can access it like this:
try:
import importlib.resources as importlib_resources
except ImportError:
# In PY<3.7 fall-back to backported `importlib_resources`.
import importlib_resources
## Note that the actual package could have been used,
# not just its (string) name, with something like:
# from XXX import YYY as data_pkg
data_pkg = '.'
fname = 'database.dat'
db_bytes = importlib_resources.read_binary(data_pkg, fname)
# or if a file-like stream is needed:
with importlib_resources.open_binary(data_pkg, fname) as db_file:
...