I have around 15 different python scripts for an application development, among of which 10 includes logging for debugging purposes. Now there is a main script say "hutextract.py" which is to run with a filename as an argument. Previously the log file name was fixed "test.log". Now I want to create the logging file with the same name as the input filename (except the extension). My "hutextract.py" code, where "randpage" and "mandate" are other python scripts :
from randpage import genrand
from mandatebase import formext
...# importing other scripts, functions
import logging
file_name=sys.argv[1]
def return_log_name():
return ".".join(file_name.split("/")[-1].split(".")[:-1]) + ".log"
log_file_name = return_log_name()
logging.basicConfig(filename=log_file_name, level=logging.DEBUG, format="%(asctime)s:%(levelname)s:%(message)s")
In randpage.py, mandatebase.py and other scripts, logging is also included there :
import logging
from hutextract import return_log_name
log_file_name = return_log_name()
logging.basicConfig(filename=log_file_name, level=logging.DEBUG, format="%(asctime)s:%(levelname)s:%(message)s")
This creates an error which is obvious that when we try to run hutextract.py with argument which calls the other scripts (and their functions) which in return again summons return_log_name function from hutextract.py for logging purpose :
Traceback (most recent call last):
File "hutextract.py", line 3, in <module>
from randpage import genrand
File "/home/src/randpage.py", line 3, in <module>
from mandate_final import sigext
File "/home/src/mandate_final.py", line 5, in <module>
from hutextract import return_log_name
File "/home/src/hutextract.py", line 3, in <module>
from randpage import genrand
ImportError: cannot import name 'genrand'
How to do the logging inside all the module scripts to save the log file with same name as the input filename given as an argument?
The error you have provided is because of circular import. You may see this circle in the traceback hutextract.py - randpage.py - mandate_final.py - hutextract.py
Now to the logging. You should use logging.basicConfig(...) only once across multiple scripts (in the starting script) because this line of code modifies so-called root logger of the logging module. This root logger is created when you first import logging module and it is located in the global scope. So root logger is always available - just use it like logging.debug('message') where and when you need.
Related
I have a folder structure similar to this (my example has all the necessary bits):
web-scraper/
scraper.py
modules/
__init__.py
config.py
website_one_scraper.py
Where config.py just stores some global variables. It looks a bit like:
global var1
var1 = "This is a test!"
Within website_one_scraper.py it looks like this:
import config
def test_function():
# Do some web stuff...
return = len(config.var1)
if __name__ == "__main__":
print(test_function)
And scraper.py looks like this:
from module import website_one_scraper
print(website_one_scraper.test_function())
website_scraper_one.py works fine when run by itself, and thus the code under if __name__ == "__main__" is run. However, when I run scraper.py, I get the error:
ModuleNotFoundError: No module named 'config'
And this is the full error and traceback (albeit with different names, as I've changed some names for the example above):
Traceback (most recent call last):
File "c:\Users\User\Documents\Programming\Work\intrack-web-scraper\satellite_scraper.py", line 3, in
<module>
from modules import planet4589
File "c:\Users\User\Documents\Programming\Work\intrack-web-scraper\modules\planet4589.py", line 5, in
<module>
import config
ModuleNotFoundError: No module named 'config'
Also note that In scraper.py I've tried replacing from modules import website_one_scraper with import website_one_scraper, from .modules import website_one_scraper, and from . import website_one_scraper, but they all don't work.
What could the cause of my error be? Could it be something to do with how I'm importing everything?
(I'm using Python 3.9.1)
In your website_scraper_one.py, instead of import config.py try to use from . import config
Explanation:
. is the current package or the current folder
config is the module to import
I have a basic parser app I'm building in Python. I monitors a folder and imports files when they are dropped there. I have a MongoDB that I'm trying to save the imports to. There's almost nothing to it. The problem happens when I try to include one of my class/mongo-document files. I'm sure it's a simple syntax issue I don't understand. I have all my requirements installed, and I'm running this in an virtual env. This is my first python app though, so it's likely something I'm not seeing.
My file structure is
application.py
requirements.txt
__init__.py
-services
parser.py
__init__.py
-models
hl7message.py
__init__.py
Here is application.py
from mongoengine import connect
import os, os.path, time
from services import parser
db = connect('testdb')
dr = 'C:\\Imports\\Processed'
def processimports():
while True:
files = os.listdir(dr)
print(str(len(files)) + ' files found')
for f in files:
msg = open(dr + '\\' + f).read().replace('\n', '\r')
parser.parse_message(msg)
print('waiting')
time.sleep(10)
processimports()
requirements.txt
mongoengine
hl7
parser.py
import hl7
from models import hl7message
def parse_message(message):
m = hl7.parse(str(message))
h = hl7message()
hl7message.py
from utilities import common
from application import db
import mongoengine
class Hl7message(db.Document):
message_type = db.StringField(db_field="m_typ")
created = db.IntField(db_field="cr")
message = db.StringField(db_field="m")
If I don't include the hl7message class in the parser.py it runs fine, but as soon as I include it I get the error, so I'm sure it has something to do with that file. The error message though isn't to helpful. I don't know if I've got myself into some kind of include loop or something.
Sorry, stack trace is below
Traceback (most recent call last):
File "C:/OneDrive/Dev/3/Importer/application.py", line 3, in <module>
from services import parser
File "C:\OneDrive\Dev\3\Importer\services\parser.py", line 2, in <module>
from models import hl7message
File "C:\OneDrive\Dev\3\Importer\models\hl7message.py", line 2, in <module>
from application import db
File "C:\OneDrive\Dev\3\Importer\application.py", line 23, in <module>
processimports()
File "C:\OneDrive\Dev\3\Importer\application.py", line 17, in processimports
parser.parse_message(msg)
AttributeError: module 'services.parser' has no attribute 'parse_message'
This is a circular import issue. Application.py imports parser, which imports h17 which imports h17message, which imports application which runs processimports before the whole code of the parser module has been run.
It seems to me that service modules should not import application. You could create a new module common.py containing the line db = connect('testdb') and import db from common both in application.py and in h17message.
I am encountering an import error
Traceback (most recent call last):
File "C:\Users\bartis\Desktop\Python\TEC-KB\SlotMapper.pyw", line 9, in <module>
from SlotMapper import SlotMap
File "C:\Users\bartis\Desktop\Python\TEC-KB\SlotMapper.pyw", line 9, in <module>
from SlotMapper import SlotMap
ImportError: cannot import name 'SlotMap
This should be a straightforward issue, but I can’t seem to find the problem. If I place the SlotMapper.py file in the same directory as the GUI I am using the import of SlotMap occurs without error. If I move the file to a directory under the current working directory and add - sys.path.append(os.path.join(os.getcwd(), 'appLib')) I receive the error above. See import statements and modification of PYTHONPATH below. I know the PYTHONPATH has been modified after I checked it from the debugger. I also know since there are other files under appLib required for the GUI to operate. Finally, I have checked all of the imported files for a circular reference and find none… So stuck. Any suggestions welcome
import os
import sys
sys.path.append(os.path.join(os.getcwd(), 'appLib', 'KB-GUI'))
sys.path.append(os.path.join(os.getcwd(), 'appLib'))
from tkinter import *
from SlotMapper import SlotMap
from ShelfTypeSelection import ShelfTypeSelector
from PackTypeSelection import PackTypeSlotMappingSelector
from EntryWidgets import EntryBase, ShelfSlotEntry
The reason this is not working is because your file is named SlotMapper.pyw. The line
from SlotMapper import SlotMap
is trying to import SlotMap from your current file, hence the error. Try renaming your file to slotmapper_test.pyw or something like that, and everything should work as expected. You don't want your code files to have the same names as any modules you're trying to import, as the import mechanism will try to find the classes/functions there first, instead of searching your modules first.
Brief :
base.py:
import logging
from logging.handlers import TimedRotatingFileHandler
import os
slogFile = os.path.join(os.getcwd(), 'LOGS', 'App_Debug.log')
if True != os.path.isdir(os.path.join(os.getcwd(), 'LOGS')):
os.mkdir(os.path.join(os.getcwd(), 'LOGS'))
logging.basicConfig(filename=slogFile, level=logging.DEBUG)
logging.basicConfig(format='%(asctime)s %(message)s')
logger = logging.getLogger("myApp")
fmt = logging.Formatter(fmt='%(asctime)s %(message)s')
size=1024*1024*1 #1mb file size
logger = logging.getLogger("myApp")
fmt = logging.Formatter(fmt='%(asctime)s %(message)s')
hdlr = logging.handlers.RotatingFileHandler(filename = slogFile ,mode='w', maxBytes=size, backupCount=5, encoding=None, delay=0)
hdlr.setFormatter(fmt)
logger.addHandler(hdlr)</em>
app_main1.py:
import base
base.logger.debug('xxx')
app_main2.py:
import base
base.logger.debug('xxx')
NOTE : for my application i am using another module that also recording the logs into the same file .
I am getting this error:
Traceback (most recent call last):
File "C:\Python27\lib\logging\handlers.py", line 78, in emit
self.doRollover()
File "C:\Python27\lib\logging\handlers.py", line 141, in doRollover
os.rename(self.baseFilename, dfn)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process
Logged from file app_main1.py, line 59
Could you explain this to me?
I want do backup the log file when its reaches the max(1mb) size.
You probably have the log file open in some other program, which is why it can't be renamed. This could be one of your programs, or an anti-virus or full-text indexer operating on disk files.
I got the same error while developing a flask application. To solve the problem, I had to change the environmental variable from
"FLASK_DEBUG=1" to "FLASK_DEBUG=0". The reason being that turning debugging on leads to threading errors. I got the solution after reading this blog
My guess is that since you imported base.py twice, the RotatingFileHandler is setup twice, therefore it is accessed by two processes.
I had a similar problem today due to this reason. Details here.
I would suggest not importing base.py, but creating a child logger in app_main1.py and app_main2.py. You can refer to the documentation here.
I'm learning Python and I'm have some issues.
I got this error message:
Traceback (most recent call last):
File "main_console.py", line 8, in <module>
from util import Util
File "../utils/util.py", line 13, in <module>
class Util:
File "../utils/util.py", line 73, in Util
config.write(configfile)
NameError: name 'config' is not defined
Following is my code (this is inside a function):
config = ConfigParser.ConfigParser()
for index, list in enumerate(my_list):
config.add_section(str(index))
config.set(str(index), 'id', list.name)
config.set(str(index), 'host', list.host)
# Creating the folder
myFolder = "/etc/element/"
if not os.path.exists(myFolder):
os.makedirs(myFolder)
# Creating the file
filePath = "/etc/element/settings.cfg"
with open(filePath, 'wb') as configfile:
config.write(configfile)
Note: I'm using Sublime Text 3 as IDE. "myFolder" have the same problem a little time ago - if I type filePath = myFolder + '/settings.cfg' I got the same error of "not defined". Finally, I imported ConfigParser as following: import ConfigParser.
Is Python forgetting my variable name or I'm doing something wrong?
It is possible ConfigParser or some other import also defines a config variable that is being used by your config.write().
Sometimes it is safer to just import the functions you need, that way you know exactly what is defined in your file. It means you have to explicitly import everything that you use in other files, but it prevents any unknown duplication in your namespace.
You would do this like:
from ConfigParser import ConfigParser # instead of import ConfigParser
# Then
config = ConfigParser()
Secondly, config is a common variable - try renaming it to something like myConfig and see if it still happens.
configfile is the object that you want to use for write(). You are using a parser to write.
Thanks for replies, but I found the solution.
I go to Nano and edit the file with this error, so I see the indentation is absolutaly wrong. It was just a Sublime text 3 issue, now's solved.