Here's my folder structure:
src
->deployment_pipeline
->__init__.py, train_pipeline.py
src
->dags
->__init__.py,airflow_dag.py
src
->db_connector_mlflow
-> __init__.py, db_connector_mlflow.py
Now, I am trying to import a function start_final_train from train_pipeline.py(which is inside the folder deployment_pipeline) to airflow_dag.py and from db_connector_mlflow.py(which is inside the folder db_connector_mlflow) to airflow_dag.py
My import statement:
from deployment_pipeline import start_final_train
But I keep getting this error:
ModuleNotFoundError: No module named 'deployment_pipeline'
imports must be globally installed or be in the same directory or subdirectory thereof as your main file.
If you move your main file to the src folder and run everything from there it works out.
Your main file should import:
from dags.airflow_dag import <stuff you need from airflow_dag.py>
....
You should keep the same structure in airflow_dag.py to import your function (as if you were importing from src):
from deployment_pipeline.train_pipeline import start_final_train
Related
I am brand new to working with python so this question might be basic. I am attempting to import five helper files into a primary script, the directory setup is as follows and both the script I'm calling from and the helper scripts are located within src here in this path-
/Users/myusername/Desktop/abd-datatable/src
I am currently importing as-
import helper1 as fdh
import helper2 as hdh
..
import helper5 as constants
The error I see is
File "/Users/myusername/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/22.77756/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
ModuleNotFoundError: No module named 'src'
I have also attempted the following for which the import was still unsuccessful:
import src.helper1 as fdh
..
import src.helper5 as constants
and
from src import src.helper1 as fdh
...
from src import helper5 as constants
and tried adding the following to the head of the script-
import sys
import os
module_path = os.path.abspath(os.getcwd())
if module_path not in sys.path:
sys.path.append(module_path)
Would be very grateful for any pointers on how to debug this!
Seems likely that it's a path issue.
In your case, if you want to import from src, you would need to add Users/myusername/Desktop/abd-datatable to the path.
os.getcwd() is getting the path you invoke the script from, so it would only work if you are invoking the script from Users/myusername/Desktop/abd-datatable.
So below I have is my file structure:
/
api.py
app.py
code/
data_api/
weather_data/
get_data_dir/
get_data.py
master_call_dir/
master_call.py
My current problem is that I'm importing master_call.py from inside app.py I do this with the following code:
-- app.py
import sys
sys.path.append('./code/data_api/weather_data/master_call_dir')
from master_call import master_call_interface as master_call
Importing master_call itself works perfectly fine, the issue is that inside off master_call I import get_data.py. I do this with the following code:
-- master_call.py
import sys
sys.path.append("../get_data_dir")
from get_data import get_data_module
When printing out sys.path I can see ../get_data_dir inside of it but Python still isn't able to find the get_data.py file in /get_data_dir.
Does anyone know how to fix this?
(there is an __init__.py file in every directory, I removed it here for readability)
So I figured it out. The problem is while the directory of the master_call.py file is /code/data_api/weather_data/master_call the current working directory (CWD) is / since that is where app.py is located.
To fix it we simply fetch absolute file path in each python script with
current_path = os.path.dirname(os.path.realpath(__file__)).replace('\\', '/')
and pre-pend it to any file path we're appending with sys.path.append
So it'd look like this in practice:
import sys
import os
current_path = os.path.dirname(os.path.realpath(__file__)).replace('\\', '/')
sys.path.append(f'{current_path}./code/data_api/weather_data/master_call_dir')
from master_call import master_call_interface as master_call
I have two directory on same level
1.Scripts->(Inside it we have app.py file)
2. Keymaker-->(Inside it we have keymaker.py file)
Now I want to import keymaker.py in app.py, So how Can I do it
Write these lines in app.py file.
import sys
sys.path.append('path/to/Keymaker/dir')
After that you can import anything from keymaker.py file using
from keymaker import *
Add empty __init__.py file to Keymaker directory and do from Keymaker import keymaker
I am trying to import a file from a folder named pytorch_net from a folder named AI_physicist into a script named models.py. I have tried to change the folder locations of the files, get an init.py file into the main AI_physicist folder, and change the sys.path.append command to get only the folder with the files inside of it. I have also looked at other posts and have tried their solutions but to no avail.
Here are the posts that I have found and have tried:
What does os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir)) mean? python
import python module using sys.path.append
file structure:
C:-----
Users----
trevo----
Desktop----
AI_physicist----
pytorch_net----
theory_learning----
Here is the error in its entirety:
No module named 'pytorch_net'
File "C:\Users\trevo\OneDrive\Desktop\AI_physicist\theory_learning\models.py", line 13, in <module>
from pytorch_net.net import MLP
File "C:\Users\trevo\OneDrive\Desktop\AI_physicist\theory_learning\theory_model.py", line 24, in <module>
from AI_physicist.theory_learning.models import Loss_Fun_Cumu, get_Lagrangian_loss
And lastly here is the code that I am using:
models.py:
# coding: utf-8
# In[ ]:
import numpy as np
import torch
import torch.nn as nn
from torch.autograd import grad
import sys, os
sys.path.append(os.path.join(os.path.dirname("__file__"), '..', '..'))
from AI_physicist.pytorch_net.net import MLP
from AI_physicist.pytorch_net.util import get_criterion, MAELoss, to_np_array, to_Variable, Loss_Fun
from AI_physicist.settings.global_param import PrecisionFloorLoss, Dt
from AI_physicist.theory_learning.util_theory import logplus
Any help on what I would be missing would be very appreciated, thank you!
Try to put __init__.py not init.py in that folder.
Alternatively:
sys.path.append(f'{os.path.dirname(__file__)}\\folder_name')
Please look on the difference between my and yours sys.append.
That should always work. Bare in mind these will work only if your script is in a subfolder of your main folder with main.py in.
I am trying to import python module from one directory to another directory but not able to do it.
Here is my folder structure
Parent_Folder
---------Module1
----__init__.py (empty file)
----mymodule1.py
---------Module2
----__init__.py (empty file)
----mymodule2.py
I am trying to import
mymodule1.py
into
mymodule2.py
Here is the command that I have used to import the python module:
import Module1.mymodule1
but getting an error like
ModuleNotFoundError: No module named 'Module1'
I do see there are option like system-path at runtime, PYTHONPATH but I don't want to use those options.
Do you all have any other recommended solutions?
You can insert this at the top of module1.py, or in the init.py in Module1 if you're importing module1.py as well:
import sys
sys.path.append("../Module2") #assuming linuxOS
import module2
This adds the Module2 directory to the system path and allows you to import files as if they were local to your current directory.
Other answers can be found in this link: beyond top level package error in relative import
I'll give you an example of importing modules
I created the module1.py file
print("Module : 1")
def module01():#function
print("module : 1.2")
class Module02:# class
def module021(self):# function
print('MODULE 1')
to import you need to indicate the file name
module2.py
from module1 import module01 #import the function
from module1 import Module02 #importing the class from module2.py
module01()
mod1 = Module02() #
mod1.module021()# imported class function