We Keep Coding

load static config files in python package created by setuptools [duplicate]

I have a project like this:
├── CHANGES.txt
├── LICENSE
├── MANIFEST.in
...
├── docs
│   └── index.rst
├── negar
│   ├── Negar.py
│   ├── Virastar.py
│   ├── Virastar.pyc
│   ├── __init__.py
│   ├── data
│   │   ├── __init__.py
│   │   └── untouchable.dat
│   ├── gui.py
│   ├── gui.pyc
│   ├── i18n
│   │   ├── fa_IR.qm
│   │   └── fa_IR.ts
│   └── negar.pro
├── setup.py
...
and inside that my file Virastar.py need some data from data.untouchable.dat. it works fine until I install the project with this setup.py:
setup(
...
include_package_data=True,
packages = find_packages() + ['negar'],
package_dir={'negar': 'negar'},
package_data={'negar': ['data/*.dat']},
entry_points={
'console_scripts': [
'negar = negar.Negar:main',
],
},
...
)
after that when I start my program and when it needs that data file it return this error:
IOError: [Errno 2] No such file or directory: 'data/untochable.dat'
even in my egg-info sources I can't find any data file:
...
negar/Negar.py
negar/Virastar.py
negar/__init__.py
negar/gui.py
negar/data/__init__.py
have I missed something here?
Thank you all.
EDIT: Do I have to add any special thing in init.py?
and I have to add this: I used untouchable.dat just like this:
f = codecs.open('data/untouchable.dat', encoding="utf-8")

I used data_files
data_files = [('', ['negar/data/untouchable.dat'])],

The first problem is that I didn't import my data file into the package with MANIFEST.in file. I imported it like this:
include negar/data/*.dat
After that my data file already imported with my package install. but because I had mistakes in open my data files, python couldn't find it. this question helped me to find the right way
Python Access Data in Package Subdirectory and now I use something like this:
import os
this_dir, this_filename = os.path.split(__file__)
DATA_PATH = os.path.join(this_dir, "data", "data.txt")
print open(DATA_PATH).read()

Maybe try:
package_data={'negar/data': ['data/*.dat']},

A solution that does not require any of:
MANIFEST.in
include_package_data=True
package_dir={}
and neither the __init__.py file in the folder!
Having the project like:
├── CHANGES.txt
├── LICENSE
...
├── negar
│ ├── Negar.py
│ ├── Virastar.py
│ ├── Virastar.pyc
│ ├── __init__.py
│ ├── data
│ │ ├── __init__.py
│ │ └── untouchable.dat
│ ├── subfolder
│ │ ├── __init__.py
│ │ ├── data_NOT_included
│ │ │ └── garbage.toml
│ │ └── data_with_no_init_file
│ │ ├── config.ini
│ │ └── options.yml
│ ├── gui.py
│ ├── gui.pyc
│ ├── i18n
│ │ ├── fa_IR.qm
│ │ └── fa_IR.ts
│ └── negar.pro
├── setup.py
...
The setup file:
setup(
...
packages = find_packages()
package_data= {
# all .dat files at any package depth
'': ['**/*.dat'],
# into the data folder (being into a module) but w/o the init file
'negar.subfolder': [ '**/*.ini', '**/*.yml', ]
},
entry_points={
'console_scripts': [
'negar = negar.Negar:main',
],
},
...
)
This worked great in my case and I avoided maintaining an additional MANIFEST.in file.
caveat: the "negar.subfolder" has to be a python module.
To the access the file:
from importlib.resources import files
config = files('negar.subfolder').joinpath('data_with_no_init_file').joinpath('config.ini').read_text()
from accessing-data-files-at-runtime

Sphinx: unable to import internal modules

My project (written in python 2.7) has a complex structure and most modules are interlinked. There is no direct entry or link to this project to execute. It works as toolbox for other project.
When I tried to use sphinx to create the documentation it is giving error related to "unable to import module.
sample structure:
<workspace>
└── toolbox (main folder)
├── __init__.py
│
├── sub
│ ├── __init__.py
│ ├── sub1.py
│ └── sub2.py
│
├── subpackageA
│ ├── __init__.py
│ ├── submoduleA1.py
│ └── submoduleA2.py
│
└── subpackageB
├── __init__.py
├── submoduleB1.py
└── submoduleB2.py code[from sub import sub1
from subpackageA import submoduleA2 and so on]
Is there a way to configure the index.rst or the config.rst to ignore the import module error and give a output document directory like below:
└── toolbox
│
├── sub
│ ├── index
│ ├── sub1.m
│ └── sub2.m
│
├── subpackageA
│ ├── index
│ ├── submoduleA1.m
│ └── submoduleA2.m
│
└── subpackageB
├── index
├── submoduleB1.m
└── submoduleB2.m
I tried adding system path in config.rst
import os
import sys
sys.path.insert(0, os.path.abspath('../'))
tried ('../..') or ('..')
even hardcoded the project path.
even tried to use the sphinx.ext.autodoc but getting the same import error.
commands used:
sphinx-apidoc -o doc project/toolbox
make html

Is there a way to include extra files when building a PEX (i.e. with MANIFEST.in)?

I have a directory structure similar to the following:
├── myproj
│ ├── utils.py
│ ├── __init__.py
│ ├── routes
│ │ ├── __init__.py
│ │ ├── auth.py
│ │ └── stuff.py
├── html
│ ├── index.html
│ └── about.html
├── MANIFEST.in
├── setup.cfg
└── setup.py
The contents of MANIFEST.in are:
graft html
The following post alludes to being able to use MANIFEST.in with PEX (Python PEX: Pack a package with its sub-packages) but when I run either pex . -o myproject or python setup.py bdist_pex the html/ directory is not included, verified via unzip -Z1 myproject on the resulting output, but it is included when running python setup.py sdist.
How do I include these extra html files when building a PEX binary?

Defining a MANIFEST.in alone isn't enough. You also need to set the include_package_data option to True in setup.cfg.
This option will include extra files found in the package so you must also move the html directory inside the myproj package.
So the directory structure looks like:
├── myproj
│ ├── utils.py
│ ├── __init__.py
│ ├── routes
│ │ ├── __init__.py
│ │ ├── auth.py
│ │ └── stuff.py
│ ├── html
│ │ ├── index.html
│ │ └── about.html
├── MANIFEST.in
├── setup.cfg
└── setup.py
The contents of MANIFEST.in are:
graft myproj/html
And setup.cfg contains in the [options] section:
include_package_data = True

Fix pylance import error by shifting working directory one level up to cover the imported code

I have this structure for my project:
├── Dockerfile
├── app
│ ├── __init__.py
│ ├── __pycache__
│ ├── config
│ ├── database
│ ├── logging.py
│ ├── main.py
│ ├── routers
│ ├── services
│ ├── static
│ ├── templates
│ ├── utils
│ └── worker
├── k6.js
├── poetry.lock
├── prestart.sh
├── pyproject.toml
├── pytest.ini
└── run.py
Inside app, I have this worker folder that I also open as a kind of separate project.
├── __init__.py
├── database
│ ├── __init__.py
│ └── conn.py
├── engine
│ ├── __init__.py
│ ├── core
│ ├── data
│ ├── main.py
│ └── utils
├── main.py
├── poetry.lock
├── pyproject.toml
└── run.sh
The issue that I have when I open worker project which uses code from upper directory, pylance gives me an error of an import that could not be resolved. However, this code runs fine and perfect.
I created .vscode/settings.json for the worker project and add these options:
"python.analysis.extraPaths": ["../../app"],
"python.autoComplete.extraPaths": ["../../app"]
But I am still getting these errors! How can I fix this?

These paths fixed my issue:
"python.analysis.extraPaths": ["${workspaceFolder}\\..\\.."],
"python.autoComplete.extraPaths": ["${workspaceFolder}\\..\\.."]

subpackages can't be imported after package installed with pip

I have a project called build-your-own-computer with a structure that looks like this:
build-your-own-computer
├── computer
│   ├── arithmetic
│   ├── __init__.py
│   ├── logic
│   └── memory
├── README.md
├── setup.py
├── setup.py~
├── solutions
│   ├── arithmetic
│   │   ├── half_adder.py
│   │   ├── __init__.py
│   │   └── __init__.py~
│   ├── __init__.py
│   ├── __init__.py~
│   ├── logic
│   │   ├── _and.py
│   │   ├── __init__.py
│   │   ├── __init__.py~
│   │   ├── _not.py
│   │   ├── _or.py
│   │   └── xor.py
│   └── memory
│   └── __init__.py
└── tests
├── arithmetic
│   └── test_half_adder.py
├── logic
│   ├── test_and.py
│   ├── test_not.py
│   ├── test_or.py
│   └── test_xor.py
└── memory
My goal is to be able to install this project/package using pip and then to use it from anywhere on my system. I want to import the packages/modules like this:
from byoc.solutions.logic import _and
from byoc.computer.arithmetic import half_adder
As I understand it, build-your-own-computer itself can be considered a package, build-your-own-computer\computer a subpackage, and build-your-own-computer\computer\logic\ a sub-sub-package. All .py files besides init's and setup.py are modules. Is this correct, and does the import scheme above correspond with this project structure?
All of the __init__.py files are empty.
setup.py contains the following:
from setuptools import setup, find_packages
setup(
name='byoc',
packages=find_packages()
)
When I install this using pip and then try importing the submodules, I run into problems:
>>> from byoc.solutions.logic import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'byoc.solutions'
I can import byoc without errors.
I'm obviously doing something wrong, but what?

You can check here the suggested layout by setuptools. Also here is a good guide.
Here is my suggestion on your layout:
build-your-own-computer
├── README.md
├── setup.py
├── setup.cfg
├── src
│ └── byoc
│ ├── __init__.py
│ ├── computer
│ │ ├── arithmetic
│ │ ├── __init__.py
│ │ ├── logic
│ │ └── memory
│ └── solutions
│ ├── arithmetic
│ │ ├── half_adder.py
│ │ └── __init__.py
│ ├── __init__.py
│ ├── logic
│ │ ├── _and.py
│ │ ├── __init__.py
│ │ ├── __init__.py~
│ │ ├── _not.py
│ │ ├── _or.py
│ │ └── xor.py
│ └── memory
│ └── __init__.py
└── tests
├── arithmetic
│ └── test_half_adder.py
├── logic
│ ├── test_and.py
│ ├── test_not.py
│ ├── test_or.py
│ └── test_xor.py
└── memory
And then your setup.py should be:
from setuptools import setup, find_packages
setup(
name='byoc',
packages=find_packages('src/')
package_dir={'': 'src/'},
)
This way you should then be able to import it anywhere, e.g.:
from byoc.solutions import logic