We Keep Coding

Python ModuleNotFoundError when trying to run main.py

I am pretty new to python and have been working on a data validation program. I am trying to run my main.py file but I am getting the following error.
Traceback (most recent call last):
File "/Users/user/data_validation/main.py", line 5, in <module>
from src.compile_csv import combine_csv_files
File "/Users/user/data_validation/src/compile_csv.py", line 5, in <module>
from helpers.helper_methods import set_home_path
ModuleNotFoundError: No module named 'helpers'
I am using python version 3.94
This is my folder structure
data_validation
- src
- files
validation.csv
- folder_one
combined.csv
- folder_two
combined_two.csv
- helpers
helper_methods.py
compile_csv.py
mysql.py
split.py
main.py
Both compile_csv.py and split.py use methods from helpers.helper_methods.py
My main.py which is throwing the error when being run, looks like the following:
import os
import sys
import earthpy as et
from src.mysql import insert_data, output_non_matches
from src.compile_csv import combine_csv_files
from src.split import final_merged_file
from src.helpers.helper_methods import set_home_path, does_file_exist
home_path = et.io.HOME
file_path = home_path, "data_validation", "src", "files"
folder_name = sys.argv[1]
def configure_file_path():
master_aims_file = os.path.join(
file_path, "validation.csv")
output_csv = os.path.join(
file_path, "output.csv.csv")
gdpr_file_csv = set_home_path(folder_name + "_output.csv")
output_csv = does_file_exist(os.path.join(
file_path, folder_name + "_mismatch_output.csv"))
return output_csv, master_aims_file, gdpr_file_csv
output_csv, master_aims_file, gdpr_file_csv = configure_file_path()
if __name__ == "__main__":
print("🏁 Finding names which do not match name in master file")
if (folder_name == "pd") and (folder_name == "wu"):
final_merged_file()
else:
combine_csv_files()
insert_failures = insert_data(output_csv, master_aims_file)
output_failures = output_non_matches(output_csv, gdpr_file_csv)
if insert_failures or output_failures:
exit(
"⚠️ There were errors in finding non-matching data, read above for more info"
)
os.remove(os.path.join(home_path, "data_validation", "members_data.db"))
exit(
f"✅ mismatches found and have been outputted to {output_csv} in the {folder_name} folder")
From what I understand in python 3 we do not need to use __init__.py and you can use . for defining the path during import, So I am not entirely sure as to what I am doing wrong.
I am executing the file from /Users/user/data_validation and using the following command python main.py pd

There it is. The error is happening in your compile_csv.py file. I'm guessing in that file you have from helpers.helper_methods import blah. But you need to change it to
from .helpers.helper_methods import blah
OR
from src.helpers.helper_methods import blah
Reason being is that imports are relative to cwd not to the file where the code is running. So you need to add the import relative to /Users/user/data_validation.

you can try by setting PYTHONPATH variable. check more about it here

ImportError: cannot import name - Python

I am trying to import some variables from a different python file resides in the same directory from a another python file.
I have two files in the same directory as below:
constantvariables.py
test.py
This is how constantvariables.py looks like
class CONST(object):
FOO = 1234
NAMESPACE = "default"
DEPLOYMENT_NAME = "deployment-test"
DOCKER_IMAGE_NAME = "banukajananathjayarathna/bitesizetroubleshooter:v1"
SERVICE_CLUSTER = "deployment-test-clusterip"
SERVICE_NODEPORT = "deployment-test-nodeport"
INGRESS_NAME = "deployment-test-ingress"
def __setattr__(self, *_):
pass
CONST = CONST()
and this is how my test.py looks like:
import os
from . import constantvariables
print(constantsvariables.NAMESPACE)
But I get this error:
Traceback (most recent call last):
File "test.py", line 7, in
from . import constantsvariables
ImportError: cannot import name 'constantsvariables'
can someone please help me?
Python version I am using python 2.7.5

Make constant file like that constant.py and put inside config folder for proper management.
FOO = 1234
NAMESPACE = "default"
DEPLOYMENT_NAME = "deployment-test"
DOCKER_IMAGE_NAME = "banukajananathjayarathna/bitesizetroubleshooter:v1"
SERVICE_CLUSTER = "deployment-test-clusterip"
SERVICE_NODEPORT = "deployment-test-nodeport"
INGRESS_NAME = "deployment-test-ingress"
Inside your base directory create main.py file and call the constant inside that.
import os
from config.constants import NAMESPACE, FOO
print(NAMESPACE)

If you want to keep your constant file as it is, you can write this:
import os
from constantvariables import CONST
print(CONST.NAMESPACE)

Python program works in shell but in console

I'm writing a program that enables my Django application to consume a REST API. My problem is that Django rejects the program if I run it as a .py file, but accepts it via shell. The program is divided into two files: one that interacts with the API, and another that interacts with Django. The main folder structure is as follows:
/root folder
[rest of Django project]
-__init__.py
-test.py
-mid_side.py
-teamup_api/
-__init__.py
-teamup.py
where test.py is the file that starts the program, mid_side.py is the Django-side part, and teamup.py is the API-side part. Below is the specific code I'm having trouble with.
test.py
import django
from mid_side import dates
from datetime import datetime
start = datetime(2017,7,28)
end = datetime(2017,8,19)
test = dates(start,end)
print(test)
Where dates is a method from mid_side that accepts two datetime objects as parameters.
According to Django, the problem emerges in mid_side.py, as it imports models from an app in the main Django project. The specific line Django has issues with is from events.models import Service, where events is an app Django uses and Services is a model. The traceback is
Traceback (most recent call last):
File "test.py", line 2, in <module>
from mid_side import dates
File "/home/brian/Github/nydkc11/nydkcd11/mid_side.py", line 3, in <module>
from events.models import Service
File "/home/brian/Github/nydkc11/nydkcd11/events/models.py", line 1, in <module>
from embed_video.fields import EmbedVideoField
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/embed_video/fields.py", line 6, in <module>
from .backends import detect_backend, UnknownIdException, \
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/embed_video/backends.py", line 17, in <module>
from .settings import EMBED_VIDEO_BACKENDS, EMBED_VIDEO_TIMEOUT, \
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/embed_video/settings.py", line 7, in <module>
'embed_video.backends.SoundCloudBackend',
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
self._setup(name)
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/django/conf/__init__.py", line 39, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting EMBED_VIDEO_BACKENDS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
I know for a fact the traceback concerns an app my project uses, django-embed-video. What I don't understand is why Django is indicating it as a problem considering a) Service doesn't use fields from django-embed-video, b) I successfully deployed the application on my project earlier. My questions are thus as follows:
1) Why does this problem occur in the .py file and not in shell?
2) What is causing Django to output the traceback?
Below is relevant code from the program. Please notify me if additional clarifications are required. Thank you!
*I decided not to include teamup.py as this is more of a Django-side problem, but I will include it if requested.
mid_side.py
from dateutil.parser import parse
from events.models import Service
import datetime
from teamup_api.teamup import *
def dates(start,end):
event_list = day_query(start.year, start.month, start.day, end.year, end.month, end.day)['events']
service_objects = []
for event in event_list:
service_objects.append(Service(
title=event['title'],
school=none_parse(event['who']),
location=none_parse(event['location']),
start_time=parse(event['start_dt']),
end_time=parse(event['end_dt']),
all_day=event['all_day'],
description=none_parse(event['notes'])
))
return service_objects
models.py
class Service(models.Model):
title = models.CharField(max_length=100)
school = models.CharField(max_length=100)
location = models.CharField(max_length=100)
start_time = models.DateTimeField()
end_time = models.DateTimeField()
all_day = models.BooleanField()
description = models.TextField()
def __str__(self):
return self.title
UPDATE: I've been running 'shell' through manage.py shell, and 'console' via natively running a python file (e.g. python [file].py)

My problem was probably that I was using Django resources in the native python environment. As Chris indicates in the comments, creating a custom manage.py command instead of a .py start file should suffice.

Error trying to import between python files

I have a basic parser app I'm building in Python. I monitors a folder and imports files when they are dropped there. I have a MongoDB that I'm trying to save the imports to. There's almost nothing to it. The problem happens when I try to include one of my class/mongo-document files. I'm sure it's a simple syntax issue I don't understand. I have all my requirements installed, and I'm running this in an virtual env. This is my first python app though, so it's likely something I'm not seeing.
My file structure is
application.py
requirements.txt
__init__.py
-services
parser.py
__init__.py
-models
hl7message.py
__init__.py
Here is application.py
from mongoengine import connect
import os, os.path, time
from services import parser
db = connect('testdb')
dr = 'C:\\Imports\\Processed'
def processimports():
while True:
files = os.listdir(dr)
print(str(len(files)) + ' files found')
for f in files:
msg = open(dr + '\\' + f).read().replace('\n', '\r')
parser.parse_message(msg)
print('waiting')
time.sleep(10)
processimports()
requirements.txt
mongoengine
hl7
parser.py
import hl7
from models import hl7message
def parse_message(message):
m = hl7.parse(str(message))
h = hl7message()
hl7message.py
from utilities import common
from application import db
import mongoengine
class Hl7message(db.Document):
message_type = db.StringField(db_field="m_typ")
created = db.IntField(db_field="cr")
message = db.StringField(db_field="m")
If I don't include the hl7message class in the parser.py it runs fine, but as soon as I include it I get the error, so I'm sure it has something to do with that file. The error message though isn't to helpful. I don't know if I've got myself into some kind of include loop or something.
Sorry, stack trace is below
Traceback (most recent call last):
File "C:/OneDrive/Dev/3/Importer/application.py", line 3, in <module>
from services import parser
File "C:\OneDrive\Dev\3\Importer\services\parser.py", line 2, in <module>
from models import hl7message
File "C:\OneDrive\Dev\3\Importer\models\hl7message.py", line 2, in <module>
from application import db
File "C:\OneDrive\Dev\3\Importer\application.py", line 23, in <module>
processimports()
File "C:\OneDrive\Dev\3\Importer\application.py", line 17, in processimports
parser.parse_message(msg)
AttributeError: module 'services.parser' has no attribute 'parse_message'

This is a circular import issue. Application.py imports parser, which imports h17 which imports h17message, which imports application which runs processimports before the whole code of the parser module has been run.
It seems to me that service modules should not import application. You could create a new module common.py containing the line db = connect('testdb') and import db from common both in application.py and in h17message.

ImportError in python

In clean.py I have:
import datetime
import os
from flask_script import Manager
from sqlalchemy_utils import dependent_objects
from components import db, app
from modules.general.models import File
from modules.workflow import Workflow
manager = Manager(usage='Cleanup manager')
#manager.command
def run(dryrun=False):
for abandoned_workflow in Workflow.query.filter(Workflow.current_endpoint == "upload.upload_init"):
if abandoned_workflow.started + datetime.timedelta(hours=12) < datetime.datetime.utcnow():
print("Removing abandoned workflow {0} in project {1}".format(
abandoned_workflow.id, abandoned_workflow.project.name
))
if not dryrun:
db.session.delete(abandoned_workflow)
db.session.commit()
for file in File.query.all():
dependencies_number = dependent_objects(file).count()
print("File {0} at {1} has {2} dependencies".format(file.name, file.path, dependencies_number))
if not dependencies_number:
file_delete(file, dryrun)
if not dryrun:
db.session.delete(file)
db.session.commit()
# List all files in FILE_STORAGE directory and delete ones tat don't have records in DB
all_files_hash = list(zip(*db.session.query(File.hash).all()))
for file in os.listdir(app.config['FILE_STORAGE']):
if file.endswith('.dat'):
continue
if file not in all_files_hash:
file_delete(os.path.join(app.config['FILE_STORAGE'], file), dryrun)enter code here
I need start def run()
in console I write:
python clean.py
And I have outputs :
`Traceback (most recent call last):
File "cleanup_command.py", line 7, in <module>
from components import db, app
ImportError: No module named 'components'
clean.py is located in- C:\App\model\clean.py
components.py is located in - C:\components.py
Workflow.py is located in - C:\modules\workflow\Workflow.py
Please, tell me what could be the problem?

The problem is that modules for import are searched in certain locations: https://docs.python.org/2/tutorial/modules.html#the-module-search-path.
In your case you can put all source directory paths in PYTHONPATH var like:
PYTHONPATH=... python clean.py
But I guess it would be better to relocate your code files (i.e. put all the libs in one location)

To start run() when you call python clean.py, Add these lines at the end of the script.
if __name__ == '__main__':
r = run()
## 0-127 is a safe return range, and 1 is a standard default error
if r < 0 or r > 127: r = 1
sys.exit(r)
Also as Eugene Primako mentioned it is better to relocate your code files in one location.
from components import db, app
ImportError: No module named 'components'
This means its looking for script named components.py in a location where clean.py is placed. This is the reason why you have got import error.