We Keep Coding

Import subfolder file from Jupyter Notebook

With a folder structure like this:
~/
+-- some_module/
| +-- subfolder/
| | +-- submodule.py
| +-- helpers.py
| +-- api.py
+-- notebook.ipynb
This is the content of api.py:
from subfolder.submodule import submodule_fun
def print_something(string):
print(submodule_fun(string))
# ...
if __name__ == '__main__':
run()
And submodule.py:
from helpers import helper_fun
def submodule_fun(string):
stuff = helper_fun(string)
return stuff
This works perfectly fine when running python api.py from the command line. The submodule.py file is simply using some function from helpers.py and no drama needed.
However, I also want to be able to use submodule.py from notebook.ipynb which lives outside some_module. When I add the following lines in this notebook:
from some_module.subfolder import submodule
I get the following error:
~/some_module/subfolder/submodule.py in <module>
---> from helpers import helper_fun
ModuleNotFoundError: No module named 'helpers'
I tried packaging all of some_module into a proper module using setuptools but I'm running into weird errors, apparently because the module is not living directly in the repository's root.
Without resorting to packaging some_module, would there be a way to be able to access submodule.py from the notebook.ipynb in this scenario?

Try adding some_module to your PATH or PYTHONPATH environment variable, either through OS or programatically (shown below)
import sys
import os
abspath = r"c:\your\path\to\some_module"
sys.path.append(os.path.abspath(abspath ))
from some_module.subfolder import submodule
As an alternative to packaging, you can use poetry to manage your dependencies, virtual environment, and it will install your project into the environment without actually needing to produce a package.

Virtual environment for a package

I am working on a program and access a set of custom modules. I organized the modules in a subfolder as own package with an __init__.py. Moreover, on the main level of the directory I have created a virtual environment that holds my dependencies. The folder structure is as follows:
project
+-- main_program.py
+-- venv
| +-- cool_package.py
+---mypackage
| +-- module1.py
| +-- module2.py
| +-- __init__.py
The issue is, that module2.py depends on a package I installed in venv. Running module2.py from main_program.py gives an error "cool_package.py" not found.
How do I organize stuff like that so that I can accesses cool_package.py from main_program.py with all the other needed packaged. And make cool_package.py accessible for the custom package with module2.py as well?

I may have misunderstood what you mean by your virtual env but based on your folder and file layout I think you need to add an __init.py__ file to your venv folder to make it a package and then you should be able to import venv.cool_package.

thanks for all answers - it eventually worked by properly activating the environment before running the script. Something must have gone wrong the first time - all works now and the folder structure is correct.
Best
Moritz

VS Code Pylint unresolved import on not yet installed package

The structure of my package is the following:
package
|___package
| |_subpackage
| | |_ __init__.py
| | |_ module_Y.py
| |_ __init__.py
| |_ module_X.py
|_ main.py
|_ setup.py
My __init__.py files are all empty and module_Y.py has the line from package import module_X.
I have not yet installed the package since it's not even remotely close to be working, but I want Pylint to be able to understand that the import statement in module_Y.py is going to be correct. I know that this must be possible because cloning the repo of TF-Agents and opening it in VS code, pylint understand the references inside the files1 even if I have not yet installed the TF-agents repo.
I know that I could use relative imports like from .. import module_X, and I know that I could just disable these pylint warnings, but these two me are half solutions. The first is not as clean and clear as the statement from package import module_X and the second solution possibly doesn't tell me of something being actually wrong.
What am I missing?
1Take for example tf_agents/agents/dqn/dqn_agent.py which is able to resolve the imports to tf_agents.policies

According to your description, I reproduced this problem, the following is my solution and you could refer to it:
Way 1:
Please add the following code at the beginning of the file "module_Y.py", which adds the file path to the upper level directory "package":
import sys
sys.path.append("..")
Please set "cwd": "${fileDirname}", in "launch.json";
Click F5 to debug the code: (Since this warning does not affect the use of the code, we can close it after the code is available: use "python.analysis.disabled": [ "unresolved-import" ], in settings.json )
Way 2:
Or you could also set in "launch.json": (It adds the folder path of the current workspace to the system path.)
"env": {
"PYTHONPATH": "${workspaceFolder}"
},

Import error in VSCode despite setting the PYTHONPATH

I have an airflow environment locally deployed on WSL, and I am using VScode to debug and code.
My app folder structure is as follows:
~/workspaces
|--- .env
|---organization/gcp/datalake
|--- dags
|--- //My dags
|--- plugins
|--- __init.py__
|--- operators
|--- __init.py__
|--- facebook_operators.py
|--- hooks
|--- __init.py__
|--- facebook_hooks.py
I am having trouble understanding the behavior of VSCode regarding the imports.
I added the dags and plugins folders to the PYTHONPATH via .env file. My VSCode is opened directly on the workspaces directory.
The problem :
I get import errors, although I can successfully go to definition of the class I want to import.
Example : In my facebook_operators.py
from hooks.facebook_hooks import FacebookAdsHook
raises the following error :
No name 'facebook_hooks' in module 'hooks'
The contents of my .env file:
PROJECTDIR=~/workspaces/organization/gcp/datalake
PYTHONPATH=${PROJECTDIR}/plugins
PYTHONPATH=${PROJECTDIR}/dags:${PYTHONPATH}
Where did I go wrong? I'd like to understand and solve this error please.

Add the following statement at the beginning of the file "facebook_operators.py" to help VSCode find the file that needs to be imported:
import os,sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
(VSCode defaults to find the file from the parent folder of the currently opened file. We can use the above statement to add the file path that needs to be imported to the system path, and then VSCode can find it.)

Importing multiple python files in subfolders

I want to import a python file Loader within a subfolder sub1. This file that I want to import, imports another file detector within that same subfolder. However, Loader gives the following error:
ModuleNotFoundError: No module named 'detector'
I've tried using the exec command in Python and
import sub1.Loader
The folder structure looks like this:
Project
|
+-- File_for_loading_Loader.py
|
+-- sub1
|
+-- __init__.py
+-- Loader.py
+-- detector.py
Can anyone help?
Edit
I now use:
import subprocess
subprocess.call(["python", "Loader.py"], cwd="sub1")
which does the trick nicely. No need for relative imports etc.

Since you created sub1 as module you have to import files from it always like sub1.<module_name>.
So for you it should be from sub1.detector import detect_faces in your Loader.

You need to import loader in File_for_loading_Loader.py by using import sub1.Loader as abc.
And in Loader.py you need to import detector by using import sub1.detector as xyz.