We Keep Coding

Fix pylance import error by shifting working directory one level up to cover the imported code

I have this structure for my project:
├── Dockerfile
├── app
│ ├── __init__.py
│ ├── __pycache__
│ ├── config
│ ├── database
│ ├── logging.py
│ ├── main.py
│ ├── routers
│ ├── services
│ ├── static
│ ├── templates
│ ├── utils
│ └── worker
├── k6.js
├── poetry.lock
├── prestart.sh
├── pyproject.toml
├── pytest.ini
└── run.py
Inside app, I have this worker folder that I also open as a kind of separate project.
├── __init__.py
├── database
│ ├── __init__.py
│ └── conn.py
├── engine
│ ├── __init__.py
│ ├── core
│ ├── data
│ ├── main.py
│ └── utils
├── main.py
├── poetry.lock
├── pyproject.toml
└── run.sh
The issue that I have when I open worker project which uses code from upper directory, pylance gives me an error of an import that could not be resolved. However, this code runs fine and perfect.
I created .vscode/settings.json for the worker project and add these options:
"python.analysis.extraPaths": ["../../app"],
"python.autoComplete.extraPaths": ["../../app"]
But I am still getting these errors! How can I fix this?

These paths fixed my issue:
"python.analysis.extraPaths": ["${workspaceFolder}\\..\\.."],
"python.autoComplete.extraPaths": ["${workspaceFolder}\\..\\.."]

Cant import my python modules despite being at the same level in the hierarchy

Im writing a library for internal use,its called "etllib", and I have the following structure:
etl-lib
├── README.md
├── etllib
│ ├── __init__.py
│ ├── client
│ │ ├── __init__.py
│ │ ├── elastic.py
│ │ └── qradar.py
│ ├── etl
│ │ ├── __init__.py
│ │ └── etl_imperva.py
│ └── util
│ ├── __init__.py
│ ├── config.py
│ ├── daemon.py
│ ├── elastic
│ │ ├── __init__.py
│ │ └── impeva_index_config.py
│ └── imperva
│ ├── __init__.py
│ ├── kpe_config.py
│ └── query_config.py
├── scripts
│ └── etl_imperva
└── setup.py
And I have a script called "etl_imperva" in etllib/scripts. The code inside looks like this :
#!/usr/bin/python3
import sys
from etllib.etl.etl_imperva import ETL
# Run with python3 imperva_run.py start|run|stop|restart
ETL.startup(sys.argv)
If I install this package(etllib) and call this script, it works just fine. But when I need to test stuff, how can I tell python to use the modules that are on my working directory instead the ones are installed? Because each time I make a change on the modules, I need to reinstall the package and this is a little time consuming.
I also tried uninstalling the package for testing, but whe I run this script I get the following error :
Exception has occurred: ModuleNotFoundError
No module named 'etllib'
File "/home/jleonse/etl-lib/scripts/run_imperva", line 3, in <module>
from etllib.etl.etl_imperva import ETL
Is there a better way to do this?

Actually, it is not on the same level in the hierarchy.
from etllib.etl.etl_imperva import ETL
would work only if etllib was in the same directory or in a directory in your system PATH, but the etllib is in the parent directory, hence it can not find it.
so you can make it work if you change the project structure to be:
etl-lib
├── README.md
├── etllib
│ ├── __init__.py
│ ├── client
│ │ ├── __init__.py
│ │ ├── elastic.py
│ │ └── qradar.py
│ ├── etl
│ │ ├── __init__.py
│ │ └── etl_imperva.py
│ └── util
│ ├── __init__.py
│ ├── config.py
│ ├── daemon.py
│ ├── elastic
│ │ ├── __init__.py
│ │ └── impeva_index_config.py
│ └── imperva
│ ├── __init__.py
│ ├── kpe_config.py
│ └── query_config.py
├── etl_imperva
│
└── setup.py

subpackages can't be imported after package installed with pip

I have a project called build-your-own-computer with a structure that looks like this:
build-your-own-computer
├── computer
│   ├── arithmetic
│   ├── __init__.py
│   ├── logic
│   └── memory
├── README.md
├── setup.py
├── setup.py~
├── solutions
│   ├── arithmetic
│   │   ├── half_adder.py
│   │   ├── __init__.py
│   │   └── __init__.py~
│   ├── __init__.py
│   ├── __init__.py~
│   ├── logic
│   │   ├── _and.py
│   │   ├── __init__.py
│   │   ├── __init__.py~
│   │   ├── _not.py
│   │   ├── _or.py
│   │   └── xor.py
│   └── memory
│   └── __init__.py
└── tests
├── arithmetic
│   └── test_half_adder.py
├── logic
│   ├── test_and.py
│   ├── test_not.py
│   ├── test_or.py
│   └── test_xor.py
└── memory
My goal is to be able to install this project/package using pip and then to use it from anywhere on my system. I want to import the packages/modules like this:
from byoc.solutions.logic import _and
from byoc.computer.arithmetic import half_adder
As I understand it, build-your-own-computer itself can be considered a package, build-your-own-computer\computer a subpackage, and build-your-own-computer\computer\logic\ a sub-sub-package. All .py files besides init's and setup.py are modules. Is this correct, and does the import scheme above correspond with this project structure?
All of the __init__.py files are empty.
setup.py contains the following:
from setuptools import setup, find_packages
setup(
name='byoc',
packages=find_packages()
)
When I install this using pip and then try importing the submodules, I run into problems:
>>> from byoc.solutions.logic import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'byoc.solutions'
I can import byoc without errors.
I'm obviously doing something wrong, but what?

You can check here the suggested layout by setuptools. Also here is a good guide.
Here is my suggestion on your layout:
build-your-own-computer
├── README.md
├── setup.py
├── setup.cfg
├── src
│ └── byoc
│ ├── __init__.py
│ ├── computer
│ │ ├── arithmetic
│ │ ├── __init__.py
│ │ ├── logic
│ │ └── memory
│ └── solutions
│ ├── arithmetic
│ │ ├── half_adder.py
│ │ └── __init__.py
│ ├── __init__.py
│ ├── logic
│ │ ├── _and.py
│ │ ├── __init__.py
│ │ ├── __init__.py~
│ │ ├── _not.py
│ │ ├── _or.py
│ │ └── xor.py
│ └── memory
│ └── __init__.py
└── tests
├── arithmetic
│ └── test_half_adder.py
├── logic
│ ├── test_and.py
│ ├── test_not.py
│ ├── test_or.py
│ └── test_xor.py
└── memory
And then your setup.py should be:
from setuptools import setup, find_packages
setup(
name='byoc',
packages=find_packages('src/')
package_dir={'': 'src/'},
)
This way you should then be able to import it anywhere, e.g.:
from byoc.solutions import logic

how can I resolve a folders walking with python?

I'm trying to resolve this problem:
Have a tree structure like this:
── geny16
│   ├── gen26xgeny16
│   │   ├── sustrato270
│ │ │ ├── sustrato270_data01.dat
│ │ │ ├── sustrato270_data02.dat
│ │ │ └── sustrato270_data03.dat
│   │   ├── sustrato90
│ │ │ ├── sustrato90_data01.dat
│ │ │ ├── sustrato90_data02.dat
│ │ │ └── sustrato90_data03.dat
│   │   ├── tentata0
│ │ │ ├── tentata0_data01.dat
│ │ │ ├── tentata0_data02.dat
│ │ │ └── tentata0_data03.dat
│   │   └── tenteta90
│ │ │ ├── tentata90_data01.dat
│ │ │ ├── tentata90_data02.dat
│ │ │ └── tentata90_data03.dat
│   └── gen40xgeny16
│   ├── sustrato270
│   ├── sustrato90
│   ├── tenteta0
│   └── tenteta90
└── geny9
├── gen16xgeny9
│   ├── sustrato270
├── sustrato90
│   ├── tenteta0
│   └── tenteta90
├── gen26xgeny9
│   ├── sustrato270
│   ├── sustrato90
│   ├── tenteta0
│   └── tenteta90
└── gen40xgen9y
├── sustrato270
├── sustrato90
├── tenteta0
└── tenteta90
in each deeper folder there are some .dat files that I will manipulate and plot with matplotlib. When I use a python script inside the deeper folder, my work is done: read each data file, normalize a column, superspose and plot data; but as it's the same task for each deep folder, I would like to leave the script in the root folder and that goes through the subfolders to manipulate and plot data.
I can list the name of each element in the tree with this code:
rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir):
print('Directorio encontrado: %s' % subdirList)
for fname in fileList:
print('\t%s' % fname)
But it's only a list of names, how can I use this list to navigate and execute my script to manipulate and plot data in each deeper folders?
Thanks in advance for your comments and suggestions.
Gustavo.

You can use pathlib
from pathlib import Path
root_dir = '.' # __file__
d = Path(root_dir).glob('*/*/*/*.dat') #*==any folder/file
for i in d:
print(i.name, i.parent, i.parts[-2])
print(i.resolve())
print(i.stem)
# code to manipulate these files here
# e.g read, visualise and store data
See what you get. You can the do whatever you wish with the files

How to import a file from a subdirectory in python from a subpackage in another directory?

I have the following directory structure in my home projects folder.
|ALL-IN-ONE
|demo
|__init__.py
|__main__.py
|models
|grpc
|allinone_server.py
And I want to import from allinone_server.py a function defined in main.py called images_demo. I have tried
from demo.__main__ import images_demo
It is not working. How can I import it? The function I am trying to import is located inside main.py which is inside demo directory. I am trying to import it from the file allinone_server.py in grpc. I guess I have made my question clear now.
Here is the whole tree for the project
├── demo
│ ├── __init__.py
│ ├── __main__.py
│ └── __pycache__
│ ├── __init__.cpython-36.pyc
│ └── main.cpython-36.pyc
├── description
├── environment.yml
├── HEAD
├── hooks
│ ├── applypatch-msg.sample
│ ├── commit-msg.sample
│ ├── fsmonitor-watchman.sample
│ ├── post-update.sample
│ ├── pre-applypatch.sample
│ ├── pre-commit.sample
│ ├── prepare-commit-msg.sample
│ ├── pre-push.sample
│ ├── pre-rebase.sample
│ ├── pre-receive.sample
│ └── update.sample
├── imgs
│ └── 44.jpg
├── info
│ └── exclude
├── __init__.py
├── loggers
│ ├── __init__.py
│ └── __pycache__
│ └── __init__.cpython-36.pyc
├── models
│ ├── adience_large1.h5
│ ├── adience_small1.h5
│ ├── AgeModel.json
│ ├── detection_age_gender_large1.h5
│ ├── detection_age_gender_small1.h5
│ ├── detection_age_gender_smile_large1.h5
│ ├── detection_age_gender_smile_small1.h5
│ ├── detection_age_large1.h5
│ ├── detection_age_small1.h5
│ ├── detection_large1.h5
│ ├── detection_small1.h5
│ ├── grpc
│ │ ├── adele_2016.jpg
│ │ ├── allinone_client.py
│ │ ├── all_in_one_pb2_grpc.py
│ │ ├── all_in_one_pb2.py
│ │ ├── all_in_one.proto
│ │ ├── allinone_server.py
│ │ ├── benedict_cumberbatch_2014.png
│ │ ├── cat.png
│ │ ├── classroom_in_tanzania.jpg
│ │ ├── decoded1.py
│ │ ├── decoded.py
│ │ ├── elon_musk_2015.jpg
│ │ ├── laos.jpg
│ │ ├── model_face.jpg
│ │ ├── __pycache__
│ │ │ ├── all_in_one_pb2.cpython-36.pyc
│ │ │ ├── all_in_one_pb2_grpc.cpython-36.pyc
│ │ │ └── decoded.cpython-36.pyc
│ │ ├── sophia.jpg
│ │ ├── test
│ │ │ ├── __init__.py
│ │ │ ├── __pycache__
│ │ │ │ └── __init__.cpython-36.pyc
│ │ │ └── test_images
│ │ │ ├── adele_2016.jpg
│ │ │ ├── benedict_cumberbatch_2014.png
│ │ │ ├── classroom_in_tanzania.jpg
│ │ │ ├── elon_musk_2015.jpg
│ │ │ ├── __init__.py
│ │ │ ├── laos.jpg
│ │ │ ├── model_face.jpg
│ │ │ ├── sophia.jpg
│ │ │ ├── waaah.jpg
│ │ │ ├── woman.jpg
│ │ │ └── zebra_stripes.jpg
│ │ ├── waaah.jpg
│ │ ├── woman.jpg
│ │ └── zebra_stripes.jpg

So you've referred to main.py, but you also have __main__.py in your directory structure. I'll assume that your directory actually contains main.py instead of __main__.py.
To import from levels up in a package, start your import with a period.
To import just one function you would use from .main import images_demo
Now, let's start by saying main.py is in grpc/ along with allinone_server.py, then we'll move it to different directories and see how the import changes.
If it were in grpc/ from .main import images_demo
If it were in models/ from ..main import images_demo
If it were in __ALL-IN-ONE/ from ...main import images_demo
If it were in __demo/ from ...__demo.main import images_demo
Every extra period brings you up one level in the hierarchy, then you use the name of the next level down in the target path until you reach where you want to be.
Now let's suppose you wanted to import the whole of main.py.
If it were in grpc/ from . import main
If it were in models/ from .. import main
If it were in __ALL-IN-One/ from ... import main
If it were in __demo/ from ...__demo import main
Finally, the dot notation to move up a level only works if the file that uses it is in a package, so this will work fine if at the top level you start your program in a scope outside of this package then use from __ALL-IN-ONE.models.grpc import allinone_server
However, if you run allinone_server.py directly then it will fail to import anything above it as it isn't being imported as part of a package. Try that out, and let me know if that needs better explanation.
Good luck!

You can't import a function from another folder directly and for that you have to use this:
import sys
sys.path.insert(0, "../../demo/")
Another step is to rename __main__ to main.
here is the exact example that worked for me:
The tree:
.
├── demo
│   ├── __init__.py
│   ├── main.py
│  
└── models
└── grpc
└── allinone_server.py
main.py:
def images_demo():
print("hello there")
The calling file(allinone_server.py):
import sys
sys.path.insert(0, "../../demo/")
import main
main.images_demo()