I am pretty new to python and have been working on a data validation program. I am trying to run my main.py file but I am getting the following error.
Traceback (most recent call last):
File "/Users/user/data_validation/main.py", line 5, in <module>
from src.compile_csv import combine_csv_files
File "/Users/user/data_validation/src/compile_csv.py", line 5, in <module>
from helpers.helper_methods import set_home_path
ModuleNotFoundError: No module named 'helpers'
I am using python version 3.94
This is my folder structure
data_validation
- src
- files
validation.csv
- folder_one
combined.csv
- folder_two
combined_two.csv
- helpers
helper_methods.py
compile_csv.py
mysql.py
split.py
main.py
Both compile_csv.py and split.py use methods from helpers.helper_methods.py
My main.py which is throwing the error when being run, looks like the following:
import os
import sys
import earthpy as et
from src.mysql import insert_data, output_non_matches
from src.compile_csv import combine_csv_files
from src.split import final_merged_file
from src.helpers.helper_methods import set_home_path, does_file_exist
home_path = et.io.HOME
file_path = home_path, "data_validation", "src", "files"
folder_name = sys.argv[1]
def configure_file_path():
master_aims_file = os.path.join(
file_path, "validation.csv")
output_csv = os.path.join(
file_path, "output.csv.csv")
gdpr_file_csv = set_home_path(folder_name + "_output.csv")
output_csv = does_file_exist(os.path.join(
file_path, folder_name + "_mismatch_output.csv"))
return output_csv, master_aims_file, gdpr_file_csv
output_csv, master_aims_file, gdpr_file_csv = configure_file_path()
if __name__ == "__main__":
print("🏁 Finding names which do not match name in master file")
if (folder_name == "pd") and (folder_name == "wu"):
final_merged_file()
else:
combine_csv_files()
insert_failures = insert_data(output_csv, master_aims_file)
output_failures = output_non_matches(output_csv, gdpr_file_csv)
if insert_failures or output_failures:
exit(
"⚠️ There were errors in finding non-matching data, read above for more info"
)
os.remove(os.path.join(home_path, "data_validation", "members_data.db"))
exit(
f"✅ mismatches found and have been outputted to {output_csv} in the {folder_name} folder")
From what I understand in python 3 we do not need to use __init__.py and you can use . for defining the path during import, So I am not entirely sure as to what I am doing wrong.
I am executing the file from /Users/user/data_validation and using the following command python main.py pd
There it is. The error is happening in your compile_csv.py file. I'm guessing in that file you have from helpers.helper_methods import blah. But you need to change it to
from .helpers.helper_methods import blah
OR
from src.helpers.helper_methods import blah
Reason being is that imports are relative to cwd not to the file where the code is running. So you need to add the import relative to /Users/user/data_validation.
you can try by setting PYTHONPATH variable. check more about it here
Related
I have a script that have multiple files and a config file with all the variables store in one Python file.
Folder structure:
Config file:
If I try to run the main file which calls the head function imported, an error pops up saying that the config cannot be imported.
Imports:
Your Functions folder has a __init__.py file. If your app executes from Main.py (ie if Main.py satisfies __name__ == "__main__") therefore wherever you are in your app you could import the config like this:
from Functions.Config import *
Edit:
However,from module import * is not recommended with local import. You could give an alias to the import and call name.variable instead of calling name directly.
Example:
def head():
# import packages
import Function.Config as conf
print(conf.more_200)
head()
>>> 50
Your syntax are wrong.
It is supposed to be from Config import * and not import .Config import *
I have a folder structure similar to this (my example has all the necessary bits):
web-scraper/
scraper.py
modules/
__init__.py
config.py
website_one_scraper.py
Where config.py just stores some global variables. It looks a bit like:
global var1
var1 = "This is a test!"
Within website_one_scraper.py it looks like this:
import config
def test_function():
# Do some web stuff...
return = len(config.var1)
if __name__ == "__main__":
print(test_function)
And scraper.py looks like this:
from module import website_one_scraper
print(website_one_scraper.test_function())
website_scraper_one.py works fine when run by itself, and thus the code under if __name__ == "__main__" is run. However, when I run scraper.py, I get the error:
ModuleNotFoundError: No module named 'config'
And this is the full error and traceback (albeit with different names, as I've changed some names for the example above):
Traceback (most recent call last):
File "c:\Users\User\Documents\Programming\Work\intrack-web-scraper\satellite_scraper.py", line 3, in
<module>
from modules import planet4589
File "c:\Users\User\Documents\Programming\Work\intrack-web-scraper\modules\planet4589.py", line 5, in
<module>
import config
ModuleNotFoundError: No module named 'config'
Also note that In scraper.py I've tried replacing from modules import website_one_scraper with import website_one_scraper, from .modules import website_one_scraper, and from . import website_one_scraper, but they all don't work.
What could the cause of my error be? Could it be something to do with how I'm importing everything?
(I'm using Python 3.9.1)
In your website_scraper_one.py, instead of import config.py try to use from . import config
Explanation:
. is the current package or the current folder
config is the module to import
I am trying to import some variables from a different python file resides in the same directory from a another python file.
I have two files in the same directory as below:
constantvariables.py
test.py
This is how constantvariables.py looks like
class CONST(object):
FOO = 1234
NAMESPACE = "default"
DEPLOYMENT_NAME = "deployment-test"
DOCKER_IMAGE_NAME = "banukajananathjayarathna/bitesizetroubleshooter:v1"
SERVICE_CLUSTER = "deployment-test-clusterip"
SERVICE_NODEPORT = "deployment-test-nodeport"
INGRESS_NAME = "deployment-test-ingress"
def __setattr__(self, *_):
pass
CONST = CONST()
and this is how my test.py looks like:
import os
from . import constantvariables
print(constantsvariables.NAMESPACE)
But I get this error:
Traceback (most recent call last):
File "test.py", line 7, in
from . import constantsvariables
ImportError: cannot import name 'constantsvariables'
can someone please help me?
Python version I am using python 2.7.5
Make constant file like that constant.py and put inside config folder for proper management.
FOO = 1234
NAMESPACE = "default"
DEPLOYMENT_NAME = "deployment-test"
DOCKER_IMAGE_NAME = "banukajananathjayarathna/bitesizetroubleshooter:v1"
SERVICE_CLUSTER = "deployment-test-clusterip"
SERVICE_NODEPORT = "deployment-test-nodeport"
INGRESS_NAME = "deployment-test-ingress"
Inside your base directory create main.py file and call the constant inside that.
import os
from config.constants import NAMESPACE, FOO
print(NAMESPACE)
If you want to keep your constant file as it is, you can write this:
import os
from constantvariables import CONST
print(CONST.NAMESPACE)
I have a basic parser app I'm building in Python. I monitors a folder and imports files when they are dropped there. I have a MongoDB that I'm trying to save the imports to. There's almost nothing to it. The problem happens when I try to include one of my class/mongo-document files. I'm sure it's a simple syntax issue I don't understand. I have all my requirements installed, and I'm running this in an virtual env. This is my first python app though, so it's likely something I'm not seeing.
My file structure is
application.py
requirements.txt
__init__.py
-services
parser.py
__init__.py
-models
hl7message.py
__init__.py
Here is application.py
from mongoengine import connect
import os, os.path, time
from services import parser
db = connect('testdb')
dr = 'C:\\Imports\\Processed'
def processimports():
while True:
files = os.listdir(dr)
print(str(len(files)) + ' files found')
for f in files:
msg = open(dr + '\\' + f).read().replace('\n', '\r')
parser.parse_message(msg)
print('waiting')
time.sleep(10)
processimports()
requirements.txt
mongoengine
hl7
parser.py
import hl7
from models import hl7message
def parse_message(message):
m = hl7.parse(str(message))
h = hl7message()
hl7message.py
from utilities import common
from application import db
import mongoengine
class Hl7message(db.Document):
message_type = db.StringField(db_field="m_typ")
created = db.IntField(db_field="cr")
message = db.StringField(db_field="m")
If I don't include the hl7message class in the parser.py it runs fine, but as soon as I include it I get the error, so I'm sure it has something to do with that file. The error message though isn't to helpful. I don't know if I've got myself into some kind of include loop or something.
Sorry, stack trace is below
Traceback (most recent call last):
File "C:/OneDrive/Dev/3/Importer/application.py", line 3, in <module>
from services import parser
File "C:\OneDrive\Dev\3\Importer\services\parser.py", line 2, in <module>
from models import hl7message
File "C:\OneDrive\Dev\3\Importer\models\hl7message.py", line 2, in <module>
from application import db
File "C:\OneDrive\Dev\3\Importer\application.py", line 23, in <module>
processimports()
File "C:\OneDrive\Dev\3\Importer\application.py", line 17, in processimports
parser.parse_message(msg)
AttributeError: module 'services.parser' has no attribute 'parse_message'
This is a circular import issue. Application.py imports parser, which imports h17 which imports h17message, which imports application which runs processimports before the whole code of the parser module has been run.
It seems to me that service modules should not import application. You could create a new module common.py containing the line db = connect('testdb') and import db from common both in application.py and in h17message.
In clean.py I have:
import datetime
import os
from flask_script import Manager
from sqlalchemy_utils import dependent_objects
from components import db, app
from modules.general.models import File
from modules.workflow import Workflow
manager = Manager(usage='Cleanup manager')
#manager.command
def run(dryrun=False):
for abandoned_workflow in Workflow.query.filter(Workflow.current_endpoint == "upload.upload_init"):
if abandoned_workflow.started + datetime.timedelta(hours=12) < datetime.datetime.utcnow():
print("Removing abandoned workflow {0} in project {1}".format(
abandoned_workflow.id, abandoned_workflow.project.name
))
if not dryrun:
db.session.delete(abandoned_workflow)
db.session.commit()
for file in File.query.all():
dependencies_number = dependent_objects(file).count()
print("File {0} at {1} has {2} dependencies".format(file.name, file.path, dependencies_number))
if not dependencies_number:
file_delete(file, dryrun)
if not dryrun:
db.session.delete(file)
db.session.commit()
# List all files in FILE_STORAGE directory and delete ones tat don't have records in DB
all_files_hash = list(zip(*db.session.query(File.hash).all()))
for file in os.listdir(app.config['FILE_STORAGE']):
if file.endswith('.dat'):
continue
if file not in all_files_hash:
file_delete(os.path.join(app.config['FILE_STORAGE'], file), dryrun)enter code here
I need start def run()
in console I write:
python clean.py
And I have outputs :
`Traceback (most recent call last):
File "cleanup_command.py", line 7, in <module>
from components import db, app
ImportError: No module named 'components'
clean.py is located in- C:\App\model\clean.py
components.py is located in - C:\components.py
Workflow.py is located in - C:\modules\workflow\Workflow.py
Please, tell me what could be the problem?
The problem is that modules for import are searched in certain locations: https://docs.python.org/2/tutorial/modules.html#the-module-search-path.
In your case you can put all source directory paths in PYTHONPATH var like:
PYTHONPATH=... python clean.py
But I guess it would be better to relocate your code files (i.e. put all the libs in one location)
To start run() when you call python clean.py, Add these lines at the end of the script.
if __name__ == '__main__':
r = run()
## 0-127 is a safe return range, and 1 is a standard default error
if r < 0 or r > 127: r = 1
sys.exit(r)
Also as Eugene Primako mentioned it is better to relocate your code files in one location.
from components import db, app
ImportError: No module named 'components'
This means its looking for script named components.py in a location where clean.py is placed. This is the reason why you have got import error.