We Keep Coding

load static config files in python package created by setuptools [duplicate]

I have a project like this:
├── CHANGES.txt
├── LICENSE
├── MANIFEST.in
...
├── docs
│   └── index.rst
├── negar
│   ├── Negar.py
│   ├── Virastar.py
│   ├── Virastar.pyc
│   ├── __init__.py
│   ├── data
│   │   ├── __init__.py
│   │   └── untouchable.dat
│   ├── gui.py
│   ├── gui.pyc
│   ├── i18n
│   │   ├── fa_IR.qm
│   │   └── fa_IR.ts
│   └── negar.pro
├── setup.py
...
and inside that my file Virastar.py need some data from data.untouchable.dat. it works fine until I install the project with this setup.py:
setup(
...
include_package_data=True,
packages = find_packages() + ['negar'],
package_dir={'negar': 'negar'},
package_data={'negar': ['data/*.dat']},
entry_points={
'console_scripts': [
'negar = negar.Negar:main',
],
},
...
)
after that when I start my program and when it needs that data file it return this error:
IOError: [Errno 2] No such file or directory: 'data/untochable.dat'
even in my egg-info sources I can't find any data file:
...
negar/Negar.py
negar/Virastar.py
negar/__init__.py
negar/gui.py
negar/data/__init__.py
have I missed something here?
Thank you all.
EDIT: Do I have to add any special thing in init.py?
and I have to add this: I used untouchable.dat just like this:
f = codecs.open('data/untouchable.dat', encoding="utf-8")

I used data_files
data_files = [('', ['negar/data/untouchable.dat'])],

The first problem is that I didn't import my data file into the package with MANIFEST.in file. I imported it like this:
include negar/data/*.dat
After that my data file already imported with my package install. but because I had mistakes in open my data files, python couldn't find it. this question helped me to find the right way
Python Access Data in Package Subdirectory and now I use something like this:
import os
this_dir, this_filename = os.path.split(__file__)
DATA_PATH = os.path.join(this_dir, "data", "data.txt")
print open(DATA_PATH).read()

Maybe try:
package_data={'negar/data': ['data/*.dat']},

A solution that does not require any of:
MANIFEST.in
include_package_data=True
package_dir={}
and neither the __init__.py file in the folder!
Having the project like:
├── CHANGES.txt
├── LICENSE
...
├── negar
│ ├── Negar.py
│ ├── Virastar.py
│ ├── Virastar.pyc
│ ├── __init__.py
│ ├── data
│ │ ├── __init__.py
│ │ └── untouchable.dat
│ ├── subfolder
│ │ ├── __init__.py
│ │ ├── data_NOT_included
│ │ │ └── garbage.toml
│ │ └── data_with_no_init_file
│ │ ├── config.ini
│ │ └── options.yml
│ ├── gui.py
│ ├── gui.pyc
│ ├── i18n
│ │ ├── fa_IR.qm
│ │ └── fa_IR.ts
│ └── negar.pro
├── setup.py
...
The setup file:
setup(
...
packages = find_packages()
package_data= {
# all .dat files at any package depth
'': ['**/*.dat'],
# into the data folder (being into a module) but w/o the init file
'negar.subfolder': [ '**/*.ini', '**/*.yml', ]
},
entry_points={
'console_scripts': [
'negar = negar.Negar:main',
],
},
...
)
This worked great in my case and I avoided maintaining an additional MANIFEST.in file.
caveat: the "negar.subfolder" has to be a python module.
To the access the file:
from importlib.resources import files
config = files('negar.subfolder').joinpath('data_with_no_init_file').joinpath('config.ini').read_text()
from accessing-data-files-at-runtime

Fix pylance import error by shifting working directory one level up to cover the imported code

I have this structure for my project:
├── Dockerfile
├── app
│ ├── __init__.py
│ ├── __pycache__
│ ├── config
│ ├── database
│ ├── logging.py
│ ├── main.py
│ ├── routers
│ ├── services
│ ├── static
│ ├── templates
│ ├── utils
│ └── worker
├── k6.js
├── poetry.lock
├── prestart.sh
├── pyproject.toml
├── pytest.ini
└── run.py
Inside app, I have this worker folder that I also open as a kind of separate project.
├── __init__.py
├── database
│ ├── __init__.py
│ └── conn.py
├── engine
│ ├── __init__.py
│ ├── core
│ ├── data
│ ├── main.py
│ └── utils
├── main.py
├── poetry.lock
├── pyproject.toml
└── run.sh
The issue that I have when I open worker project which uses code from upper directory, pylance gives me an error of an import that could not be resolved. However, this code runs fine and perfect.
I created .vscode/settings.json for the worker project and add these options:
"python.analysis.extraPaths": ["../../app"],
"python.autoComplete.extraPaths": ["../../app"]
But I am still getting these errors! How can I fix this?

These paths fixed my issue:
"python.analysis.extraPaths": ["${workspaceFolder}\\..\\.."],
"python.autoComplete.extraPaths": ["${workspaceFolder}\\..\\.."]

Run Django tests with nosetests

My python apps testing is performed on the remote server with command nosetests. I cannot modify the way tests are started nor can I add options to it. I have Django app with tests, but tests are not working properly.
My project structure:
project
├── README.md
├── setup.py
├── mysite
│   ├── blog
│   │   ├── __init__.py
│   │   ├── models.py
│   │   ├── tests.py
| | ├── ...
│   ├── db.sqlite3
│   ├── manage.py
│   ├── mysite
│   │   ├── __init__.py
│   │   ├── settings.py
| | ├── ...
Command nosetests is executed in project directory. I want it to properly run tests.py which has 2 Django testcases. I tried creating tests directory in project root and invoke tests with DiscoverRunner):
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
test_dir = os.path.dirname(os.path.dirname(__file__)) # one level up
sys.path.insert(0, os.path.join(test_dir, 'mysite'))
class ServerTest(unittest.TestCase):
def test_runtests(self):
django.setup()
self.test_runner = DiscoverRunner(verbosity=1, interactive=True, failfast=False)
failures = self.test_runner.run_tests(['mysite'])
self.assertEqual(failures, 0)
It works but the problem is all the tests are considered as a single test and wrong reports are produced by the server.
Another solution: if I add empty __init__.py to project/mysite nose discovers tests.py but the tests fail because 'Apps are not loaded yet' which probably means I have to invoke django.setup() earlier but I don't know how to do it. I found a plugin for the nose which does it but I cannot install plugins or alter options on the remote machine.
Any ideas how to make any of my approaches solve the problem?

Firsts things first, you should install and configure django-nose if you haven't done so already:
pip install django-nose
Then add it on your INSTALLED_APPS:
INSTALLED_APPS = (
...
'django_nose'
)
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
Now for the nosetests command to run correctly, you can create a folder named tests with the following structure:
project
├── mysite
│ ├── blog
│ │ ├── __init__.py
│ │ ├── models.py
| | ├── ...
│ ├── mysite
│ │ ├── __init__.py
│ │ ├── settings.py
| | ├── ...
│ ├── tests
│ │ ├── __init__.py
│ │ ├── test_a_whole_class_of_methods.py
│ │ ├── test_a_whole_view.py
│ │ ├── test_one_of_my_methods.py
│ │ ├── test_another_one_of_my_methods.py
| | ├── ...
│ ├── db.sqlite3
│ ├── manage.py
├── README.md
├── setup.py
DON'T FORGET THE __init__.py FILE INSIDE THE tests FOLDER!!!
Now when you run nosetests from the root of your project, it will run every test in the tests folder:
~ $ cd path/to/project
path/to/project $ nosetests

Since you can add files to the root of your project, you could try adding a setup.cfg file there that nose will look for when executing, and in it specify where to look for tests:
# setup.cfg
[nosetests]
where=mysite/blog/
(See the documentation for what parameters you can put here).
I'm not sure that this will work (it is possible that the command that starts nose has already specified a different configuration file to use), but it seems worth a shot.

scrapy import .py files from django has import error

This is my structure,I have a django project named kkpro
in kkpro,I have a django app called myapp .
And a scrapy project called project1 inder the kkpro/scrapy/music
├── kkpro
├── manage.py
├── savepage.py
├── myapp
│ ├── __init__.py
│ ├── models.py
│ ├── tests.py
│ └── views.py
└── kkpro
├── __init__.py
├── settings.py
├── urls.py
└── wsgi.py
│── ── scrapy
│ ├──music
│ ├──project1
│ ├── scrapy.cfg
│ ├──project1
│ ├── __init__.py
│ ├── items.py
│ ├── pipelines.py
│ ├── settings.py
this is my project1/settings.py
import sys, os
path_django_site = os.path.join(os.path.dirname(__file__), "../../../../")
sys.path.append(path_django_site)
os.environ['DJANGO_SETTINGS_MODULE'] = 'kkpro.settings'
import save_page
in my computer it works well to used kkpro setting,
but when I deploy to aws eb , import save_page has error
I edit to from kali import save_page. still has error ImportError: cannot import name save_page
What should I do ?
Please help me,thank you very much

Common folder/file structure in Flask app [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 months ago.
The community reviewed whether to reopen this question 3 months ago and left it closed:
Original close reason(s) were not resolved
Improve this question
I have just created a flask application and so far I have a router for my "Hello world!" template.
I would like to add a little (a lot) more functionality, but I wonder how I should structure the app directory.
What's the most common way of structuring a Flask app?
For instance, should I create a routes.py for all my routes?
Where does the SQLAlchemy stuff go?
Should models be in models.py?

You should check out the Larger Applications page in the Patterns section of the Flask docs: http://flask.pocoo.org/docs/patterns/packages/. It seems to be the model that most people follow when their application calls for a package instead of a module.
I believe views.py is what you are calling routes.py. After that, models would go in models.py, forms would go in forms.py, etc.

An example of a FlaskApp directory:
/yourapp
/run.py
/config.py
/app
/__init__.py
/views.py
/models.py
/static/
/main.css
/templates/
/base.html
/requirements.txt
/yourappenv
run.py - contains the actual python code that will import the app and start the development server.
config.py - stores configurations for your app.
__init__.py - initializes your application creating a Flask app instance.
views.py - this is where routes are defined.
models.py - this is where you define models for your application.
static - contains static files i.e. CSS, Javascript, images
templates - this is where you store your html templates i.e. index.html, layout.html
requirements.txt - this is where you store your package dependancies, you can use pip
yourappenv - your virtual environment for development

I would say if you split the application use divisional rather than functional structure.
I advocate this because you are more likely to work on 1 of these divisional components at any one time.
This type of structure lends itself well on marketplace or SaaS apps where different user groups use a different type of views. API only flask app I might use functional splitting.
Here are examples from Flask Blueprints. Blueprints are essentially documented advice how to split Flask application for more manageable pieces. More on this at : http://exploreflask.com/en/latest/blueprints.html
Here is an example of divisional splitting. See how each feature is grouped together.
yourapp/
__init__.py
admin/
__init__.py
views.py
static/
templates/
home/
__init__.py
views.py
static/
templates/
control_panel/
__init__.py
views.py
static/
templates/
models.py
Here is the functional example >
yourapp/
__init__.py
static/
templates/
home/
control_panel/
admin/
views/
__init__.py
home.py
control_panel.py
admin.py
models.py

I think flask is micro framework and now you must decide how create files and folders.
i use this way :
flask folders and files structure -> https://gist.github.com/4545740
this is near Django structure
i suggest you see some project to give you what you want
danjac / newsmeme — Bitbucket -> https://bitbucket.org/danjac/newsmeme/overview
sean-/flask-skeleton · GitHub -> https://github.com/sean-/flask-skeleton

Anyone looking for a simple beginner-friendly structure for the flask project may find this helpful:
|__movies
|__run.py
|__app
├── templates
│ └── index.html
│ └── signup.html
└── __init__.py
└── routes.py
Here 'movies' is the name given for the main application. It contains 'run.py' and a folder called 'app'.
'app' folder contains all necessary flask files such as 'templates' folder, '__init __.py', and 'routes.py'.
Contents of:
run.py:
from app import app
__init.py__:
from flask import Flask
app = Flask(__name__)
from app import routes
app.run(debug=True)
routes.py:
from app import app
#app.route('/')
#app.route('/index')
def index():
return "Hello, World!"

Beauty of flask lies in its flexibility. You can build django like project-structure easily. Django popularized abstraction of features in apps and making them reusable but it can be a overkill for many projects.
But with flask you can go either way. Write reusable apps or write simple apps. Check these cookiecutter skeletons -
minimal skeleton
myproject
├── config.py
├── instance
│   └── config.py
├── myproject
│   ├── commands.py
│   ├── controllers.py
│   ├── extensions.py
│   ├── forms.py
│   ├── __init__.py
│   ├── models.py
│   ├── routes.py
│   └── ui
│   ├── static
│   │   ├── css
│   │   │   └── styles.css
│   │   └── js
│   │   └── custom.js
│   └── templates
│   └── index.html
├── README.md
├── requirements.txt
└── wsgi.py
django like skeleton
myproject
├── config.py
├── development.py
├── instance
│   └── config.py
├── myproject
│   ├── auth
│   │   ├── controllers.py
│   │   ├── forms.py
│   │   ├── __init__.py
│   │   ├── models.py
│   │   └── routes.py
│   ├── helpers
│   │   ├── controllers.py
│   │   ├── __init__.py
│   │   └── models.py
│   ├── __init__.py
│   └── ui
│   └── templates
│   ├── 404.html
│   ├── 500.html
│   └── base.html
├── README.md
├── requirements.txt
├── tests
│   ├── auth
│   │   ├── __init__.py
│   │   └── test_controllers.py
│   └── __init__.py
└── wsgi.py
This is an excellent article on this.

You could get inspired by the cookiecutter templates here to jumpstart your app development
https://github.com/imwilsonxu/fbone
https://github.com/cookiecutter-flask/cookiecutter-flask

Here is the basic file structure for flask I use regularly.
yourapp/
static/
js
css
img
templates/
home.html
index.html
app.py
static folder contains all the static files of the website.
The templates folder contains all the HTML pages.
and app.py contains your python views.

I decided to nest the source code folder under src/.
my_flask_app (project folder)
├── app.py
├── setup.py
├── src/
│ ├── my_flask_app/ (source code folder)
│ │ ├── config.py
│ │ ├── errors.py
│ │ ├── forms.py
│ │ ├── __init__.py
│ │ ├── models.py
│ │ ├── routes.py
│ │ ├── static/
│ │ └── templates/
│ └── my_flask_app.egg-info/
|
└── tests/ (test folder)
Because of this, I added package_dir={'': 'src'} in setup.py.

good idea is to goole for sceletons/template projects on github
for example you can look at this
https://github.com/xen/flask-project-template
https://github.com/upperlinecode/flaskproject
https://github.com/alisezer/flask-template

In flask we can maintain the mvc structure like as separate all thinks
for example
1 Templets folder contains all html files
2 Static folder contains all css and boostrap related files
3 User defined db folder for db connection and crud operations
4 User defined Model folder for accessing/ binding the data from Templets/front- end to db/back-end connectivity
and after the main activity file
for reference flask file structure link as follow
https://flask.palletsprojects.com/en/1.1.x/tutorial/layout/

For larger projects here's what my directory structure looks like
app_name(root folder)
│ .env
│ .gitignore
│ .gitattributes
│ README.MD
│ pyproject.toml
│
└── app(source code)
│ │ __init__.py
│ │ __main__.py
│ │ db.py
│ │ auth.py
│ │ forms.py
│ │ utils.py
│ │ config.py
│ │
│ └─── routes
│ __init__.py
│ views.py
│ api.py
│ auth.py
│
└─ resources
│ |─── static
│ │ css
│ │ js
│ │ images
│ │
│ └─── templates
│ base
| components
│
└─ tests
└─ venv
└─ docs