We Keep Coding

Python ModuleNotFoundError when trying to run main.py

I am pretty new to python and have been working on a data validation program. I am trying to run my main.py file but I am getting the following error.
Traceback (most recent call last):
File "/Users/user/data_validation/main.py", line 5, in <module>
from src.compile_csv import combine_csv_files
File "/Users/user/data_validation/src/compile_csv.py", line 5, in <module>
from helpers.helper_methods import set_home_path
ModuleNotFoundError: No module named 'helpers'
I am using python version 3.94
This is my folder structure
data_validation
- src
- files
validation.csv
- folder_one
combined.csv
- folder_two
combined_two.csv
- helpers
helper_methods.py
compile_csv.py
mysql.py
split.py
main.py
Both compile_csv.py and split.py use methods from helpers.helper_methods.py
My main.py which is throwing the error when being run, looks like the following:
import os
import sys
import earthpy as et
from src.mysql import insert_data, output_non_matches
from src.compile_csv import combine_csv_files
from src.split import final_merged_file
from src.helpers.helper_methods import set_home_path, does_file_exist
home_path = et.io.HOME
file_path = home_path, "data_validation", "src", "files"
folder_name = sys.argv[1]
def configure_file_path():
master_aims_file = os.path.join(
file_path, "validation.csv")
output_csv = os.path.join(
file_path, "output.csv.csv")
gdpr_file_csv = set_home_path(folder_name + "_output.csv")
output_csv = does_file_exist(os.path.join(
file_path, folder_name + "_mismatch_output.csv"))
return output_csv, master_aims_file, gdpr_file_csv
output_csv, master_aims_file, gdpr_file_csv = configure_file_path()
if __name__ == "__main__":
print("🏁 Finding names which do not match name in master file")
if (folder_name == "pd") and (folder_name == "wu"):
final_merged_file()
else:
combine_csv_files()
insert_failures = insert_data(output_csv, master_aims_file)
output_failures = output_non_matches(output_csv, gdpr_file_csv)
if insert_failures or output_failures:
exit(
"⚠️ There were errors in finding non-matching data, read above for more info"
)
os.remove(os.path.join(home_path, "data_validation", "members_data.db"))
exit(
f"✅ mismatches found and have been outputted to {output_csv} in the {folder_name} folder")
From what I understand in python 3 we do not need to use __init__.py and you can use . for defining the path during import, So I am not entirely sure as to what I am doing wrong.
I am executing the file from /Users/user/data_validation and using the following command python main.py pd

There it is. The error is happening in your compile_csv.py file. I'm guessing in that file you have from helpers.helper_methods import blah. But you need to change it to
from .helpers.helper_methods import blah
OR
from src.helpers.helper_methods import blah
Reason being is that imports are relative to cwd not to the file where the code is running. So you need to add the import relative to /Users/user/data_validation.

you can try by setting PYTHONPATH variable. check more about it here

Python : Logging in module script to be called in another script

I have around 15 different python scripts for an application development, among of which 10 includes logging for debugging purposes. Now there is a main script say "hutextract.py" which is to run with a filename as an argument. Previously the log file name was fixed "test.log". Now I want to create the logging file with the same name as the input filename (except the extension). My "hutextract.py" code, where "randpage" and "mandate" are other python scripts :
from randpage import genrand
from mandatebase import formext
...# importing other scripts, functions
import logging
file_name=sys.argv[1]
def return_log_name():
return ".".join(file_name.split("/")[-1].split(".")[:-1]) + ".log"
log_file_name = return_log_name()
logging.basicConfig(filename=log_file_name, level=logging.DEBUG, format="%(asctime)s:%(levelname)s:%(message)s")
In randpage.py, mandatebase.py and other scripts, logging is also included there :
import logging
from hutextract import return_log_name
log_file_name = return_log_name()
logging.basicConfig(filename=log_file_name, level=logging.DEBUG, format="%(asctime)s:%(levelname)s:%(message)s")
This creates an error which is obvious that when we try to run hutextract.py with argument which calls the other scripts (and their functions) which in return again summons return_log_name function from hutextract.py for logging purpose :
Traceback (most recent call last):
File "hutextract.py", line 3, in <module>
from randpage import genrand
File "/home/src/randpage.py", line 3, in <module>
from mandate_final import sigext
File "/home/src/mandate_final.py", line 5, in <module>
from hutextract import return_log_name
File "/home/src/hutextract.py", line 3, in <module>
from randpage import genrand
ImportError: cannot import name 'genrand'
How to do the logging inside all the module scripts to save the log file with same name as the input filename given as an argument?

The error you have provided is because of circular import. You may see this circle in the traceback hutextract.py - randpage.py - mandate_final.py - hutextract.py
Now to the logging. You should use logging.basicConfig(...) only once across multiple scripts (in the starting script) because this line of code modifies so-called root logger of the logging module. This root logger is created when you first import logging module and it is located in the global scope. So root logger is always available - just use it like logging.debug('message') where and when you need.

Updating of a config file written in python

I'm writing a simulation software which should support reading parameters from a config file or from the command line. It is very important to be able to track what was the configuration of a simulation, I'm committing the config file to a local git repository at the start of simulation.
Now if I have parameters on the command line they have higher priority than the ones in the config file. But I also want to commit them. I guess I could save the python objects of a configured simulation, just before it is started. But it would be more elegant, if I could just update the config file with the command line parameters before committing it.
The reason I write the config file in python is that I have to define some python objects in it. I have something like
import SomeSimulationClass
SIMULATOR = SomeSimulationClass
in my config file and the SIMULATOR can then be swapped easily.
If I want to use something like configparser I can't have objects I believe.
Is there any easy way to update a python config file? All variable names in it are already defined, I just want to change the values. The only thing I can think of is parsing the file, comparing strings between the file and the command line parameters ...

You can write whatever you want into a file, and then later, Configparser can read
from it using values from your variables. Here is an example on how I used Configparser to read environment from config file.
import os
from ConfigParser import SafeConfigParser
conf_filename = os.getenv("CONFIG_FILE")
src_dir = os.getenv("CONFIG_DIR")
conf_file = os.path.join(src_dir,conf_filename)
parser = SafeConfigParser()
parser.read(conf_file)
section = env
server = parser.get(section, 'host')
db_port = parser.get(section, 'db_port')
ws_port = parser.get(section, 'ws_port')
and the config file itself:
[PROD]
host=xxx-yyy-15
db_port=1521
ws_port=8280
ora_server=xxx-xxx-xxx.com
sid=XXXXX
userid=xxxx
passwd=xxxx
[STAGE]
host=xxx-yyy-04
db_port=1521
ws_port=8280
ora_server=yyy-yyy-yyy.com
sid=YYYYYY
userid=yyyy
passwd=yyyy

I found a way to do what I want. Some small modifications were necessary to my python config module to allow it to be rewritten with the following script, but it works for my purposes:
with open('merged_config.py', 'w') as merged_config, \
open(base_config_module.__file__, 'r') as base_config:
for line in base_config:
if 'import' in line:
# copy imports from bas config
merged_config.write(line)
for item in dir(base_config_module):
if item.startswith("__"):
# ignore __variables like '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__' ...
continue
if item == 'SimulationSteps':
# ingoring my imports
continue
item_val = getattr(base_config_module, item)
# I had to overwrite the __repr__() method of Enums which I used. Everyting else worked fine.
merged_config.write('%s = %s\n' % (item, repr(item_val)))

How to give path for file passed through command line

I got answer for how to pass file as argument here How to access variables from file passed through command line
But I am asking separately since don't wanna mix two.
I am able to access variables in passed file as __import__(sys.argv[1]) and called python test.py config. But can I call config.py file by giving pythonpath? e/g/ python test.py ~/Desktop/config or PYTHONPATH='~/Desktop/' python test.py config? Because if I do this I get no module error.

You're trying to import a python module using the __import__ call. It only accepts the module name. If you need to add a directory to the PYTHONPATH, you can add it to sys.path and then import the module:
#File: ~/Projects/py/main.py
import sys
python_path = sys.argv[1]
module_name = sys.argv[2]
sys.path.insert(1, python_path)
print "Importing {} from {}".format(module_name, python_path)
__import__(module_name)
Now I created another file named masnun.py on ~/Desktop:
# File: ~/Desktop/masnun.py
print "Thanks for importing masnun.py"
Now I try to run main.py like this:
python main.py ~/Desktop masnun
So I am passing the Python path as argv[1] and the module name as argv[2]. Now it works. I get this output:

ConfigParser instance has no attribute '[extension]'

I am learning python and I am trying to do a simple task of reading information from a config file.
So using the Python Doc and this similar problem as a reference I created two files.
This is my config file config.ini (also tried config.cfg)
[DEFAULT]
OutDir = path_to_file/
[AUTH]
TestInt = 100
TestStr = blue
TestParse = blua
and this is my python file test.py
import ConfigParser
from ConfigParser import *
config = ConfigParser()
config.read(config.cfg)
for name in config.options('AUTH'):
print name
out = config.get('DEFAULT', 'OutDir')
print 'Output directory is ' + out
however when running the command python test.py I am erroring out and receiving this error
Traceback (most recent call last):
File "test.py", line 7, in <module>
config.read(config.cfg)
AttributeError: ConfigParser instance has no attribute 'cfg'
Note: I thought that meant it the extension couldn't be read so I created the .ini file and changed it in the code and I received the same error but it instead read ...has no attribute 'ini'
I am not sure what I am doing wrong since I am doing the exact same as the python doc and the solution someone used to fix this similar issue.

config.read takes a string as its argument. You forgot to quote the file name, and config was coincidentally the name of a Python object (the module) that potentially could have a cfg attribute. You'd get an entirely different error if you had written config.read(foobarbaz.ini).
The correct line is
config.read('config.cfg') # or 'config.ini', if that's the file name