We Keep Coding

Executing external python script function

I have two scripts in the same folder:
5057_Basic_Flow_Acquire.xyzpy
5006_Basic_Flow_Execute.xyzpy
I need to call function run() from 5057_Basic_Flow_Acquire file in the 5006_Basic_Flow_Execute file.
I tried two approaches:
1.
import time
import os
import sys
import types
dir_path = os.path.dirname(os.path.realpath(__file__))
#os.chdir(str(dir_path))
sys.path.append(str(dir_path))
sensor = __import__('5057_Basic_Flow_Acquire')
import time
import os
import sys
import types
import importlib
import importlib.util
dir_path = os.path.dirname(os.path.realpath(__file__))
spec = importlib.util.spec_from_file_location('run', dir_path+'\\5057_Basic_Flow_Acquire')
module = importlib.util.module_from_spec(spec)
Pycharm is reporting this type of errors:
case:
ModuleNotFoundError: No module named '5057_Basic_Flow_Acquire'
case
AttributeError: 'NoneType' object has no attribute 'loader'
So in both cases module was not found.
Does someone have any suggestion?

You should rename your files to something like :
5057_Basic_Flow_Acquire_xyz.py
5006_Basic_Flow_Execute_xyz.py
so that they are recognized as modules and can be imported.
Then, in your 5057_Basic_Flow_Acquire_xyz module, you can import the run function with the following line :
from 5006_Basic_Flow_Execute_xyz import run
Both files are in the same directory, so you should be able to import without changing your PATH environment variable as the current directory is automatically added at the top of the list.

Can't import module from different directory python

So below I have is my file structure:
/
api.py
app.py
code/
data_api/
weather_data/
get_data_dir/
get_data.py
master_call_dir/
master_call.py
My current problem is that I'm importing master_call.py from inside app.py I do this with the following code:
-- app.py
import sys
sys.path.append('./code/data_api/weather_data/master_call_dir')
from master_call import master_call_interface as master_call
Importing master_call itself works perfectly fine, the issue is that inside off master_call I import get_data.py. I do this with the following code:
-- master_call.py
import sys
sys.path.append("../get_data_dir")
from get_data import get_data_module
When printing out sys.path I can see ../get_data_dir inside of it but Python still isn't able to find the get_data.py file in /get_data_dir.
Does anyone know how to fix this?
(there is an __init__.py file in every directory, I removed it here for readability)

So I figured it out. The problem is while the directory of the master_call.py file is /code/data_api/weather_data/master_call the current working directory (CWD) is / since that is where app.py is located.
To fix it we simply fetch absolute file path in each python script with
current_path = os.path.dirname(os.path.realpath(__file__)).replace('\\', '/')
and pre-pend it to any file path we're appending with sys.path.append
So it'd look like this in practice:
import sys
import os
current_path = os.path.dirname(os.path.realpath(__file__)).replace('\\', '/')
sys.path.append(f'{current_path}./code/data_api/weather_data/master_call_dir')
from master_call import master_call_interface as master_call

Python ModuleNotFoundError - No module named 'pytorch_net'

I am trying to import a file from a folder named pytorch_net from a folder named AI_physicist into a script named models.py. I have tried to change the folder locations of the files, get an init.py file into the main AI_physicist folder, and change the sys.path.append command to get only the folder with the files inside of it. I have also looked at other posts and have tried their solutions but to no avail.
Here are the posts that I have found and have tried:
What does os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir)) mean? python
import python module using sys.path.append
file structure:
C:-----
Users----
trevo----
Desktop----
AI_physicist----
pytorch_net----
theory_learning----
Here is the error in its entirety:
No module named 'pytorch_net'
File "C:\Users\trevo\OneDrive\Desktop\AI_physicist\theory_learning\models.py", line 13, in <module>
from pytorch_net.net import MLP
File "C:\Users\trevo\OneDrive\Desktop\AI_physicist\theory_learning\theory_model.py", line 24, in <module>
from AI_physicist.theory_learning.models import Loss_Fun_Cumu, get_Lagrangian_loss
And lastly here is the code that I am using:
models.py:
# coding: utf-8
# In[ ]:
import numpy as np
import torch
import torch.nn as nn
from torch.autograd import grad
import sys, os
sys.path.append(os.path.join(os.path.dirname("__file__"), '..', '..'))
from AI_physicist.pytorch_net.net import MLP
from AI_physicist.pytorch_net.util import get_criterion, MAELoss, to_np_array, to_Variable, Loss_Fun
from AI_physicist.settings.global_param import PrecisionFloorLoss, Dt
from AI_physicist.theory_learning.util_theory import logplus
Any help on what I would be missing would be very appreciated, thank you!

Try to put __init__.py not init.py in that folder.
Alternatively:
sys.path.append(f'{os.path.dirname(__file__)}\\folder_name')
Please look on the difference between my and yours sys.append.
That should always work. Bare in mind these will work only if your script is in a subfolder of your main folder with main.py in.

File import issue in Python

Here's my folder structure:
src
->deployment_pipeline
->__init__.py, train_pipeline.py
src
->dags
->__init__.py,airflow_dag.py
src
->db_connector_mlflow
-> __init__.py, db_connector_mlflow.py
Now, I am trying to import a function start_final_train from train_pipeline.py(which is inside the folder deployment_pipeline) to airflow_dag.py and from db_connector_mlflow.py(which is inside the folder db_connector_mlflow) to airflow_dag.py
My import statement:
from deployment_pipeline import start_final_train
But I keep getting this error:
ModuleNotFoundError: No module named 'deployment_pipeline'

imports must be globally installed or be in the same directory or subdirectory thereof as your main file.
If you move your main file to the src folder and run everything from there it works out.
Your main file should import:
from dags.airflow_dag import <stuff you need from airflow_dag.py>
....
You should keep the same structure in airflow_dag.py to import your function (as if you were importing from src):
from deployment_pipeline.train_pipeline import start_final_train

Proper way to dynamically import a module with relative imports?

I need to dynamically import modules into my project from another package.
The structure is like:
project_folder/
project/
__init__.py
__main__.py
plugins/
__init__.py
plugin1/
__init__.py
...
plugin2/
__init__.py
...
I made this function to load a module:
import os
from importlib.util import spec_from_file_location, module_from_spec
def load_module(path, name=""):
""" loads a module by path """
try:
name = name if name != "" else path.split(os.sep)[-1] # take the module name by default
spec = spec_from_file_location(name, os.path.join(path, "__init__.py"))
plugin_module = module_from_spec(spec)
spec.loader.exec_module(plugin_module)
return plugin_module
except Exception as e:
print("failed to load module", path, "-->", e)
It works, unless the module uses relative imports:
failed to load module /path/to/plugins/plugin1 --> Parent module 'plugin1' not loaded, cannot perform relative import
What am I doing wrong?

I managed to solve my own issue after a LOT of googling. Turns out I needed to import using relative paths:
>>> from importlib import import_module
>>> config = import_module("plugins.config")
>>> config
<module 'plugins.config' from '/path/to/plugins/config/__init__.py'>
>>>

I had a similar problem not long ago. I added the path of the project folder to the sys.path using the module's absolute path like this:
import sys
import os
sys.path.append(os.path.dirname(os.path.realpath(__file__))+'/..')
This adds the project_folder to the sys.path thus allowing the import statement to find the plugin modules.