We Keep Coding

Creating a class object to be used globally in Python

I wrote the module below that will standardize how my logfiles are written as well as easily changing whether events get printed/written to the logfile or not.
FILE: Logging.py
================
import os
import datetime
import io
class Logfile():
def __init__(self,name):
self.logFile = os.getcwd() + r'\.Log\\' + name + '_' + str(datetime.date.today().year) + ('00' + str(datetime.date.today().month))[-2:] + '.log'
self.printLog = False
self.debug = False
# Setup logFile and consolidated Folder
if not os.path.exists(os.path.dirname(self.logFile)):
os.mkdir(os.path.dirname(self.logFile))
#Check if logfile exists.
if not os.path.exists(self.logFile):
with open(self.logFile, 'w') as l:
pass
# Write LogFile Entry
def logEvent(self, eventText, debugOnly): # Function to add an event to the logfile
# If this is marked as debugging only AND debugging is off
if debugOnly == True and self.debug == False:
return
if self.printLog == True:
print(datetime.datetime.strftime(datetime.datetime.now(), '%m/%d/%Y, %I:%M:%S %p, ') + str(eventText))
with open(self.logFile, 'a') as l:
l.seek(0)
l.write(datetime.datetime.strftime(datetime.datetime.now(), '%m/%d/%Y, %I:%M:%S %p, ') + str(eventText) + '\n')
return
This is very handy but, I am having trouble understanding how to make this available to all of my classes. For example, If i import the following module, I am not sure how to use the logfile i created within my main script.
FILE: HelloWorld.py
===================
class HelloWorld():
def __init__(self):
log.logEvent('You have created a HelloWorld Object!', False)
Main Script Here:
import Logging
from HelloWorld import HelloWorld
log = logging.Logfile
hw = HelloWorld()
^^ Will fail because it does not know log is a thing. What is the proper way to handle these sort of situations?

I believe you're trying to do something like this. (and as a side note, you may want to look into using pythons default logging module)
FILE: HelloWorld.py
===================
# import LogFile
from .Logging import LogFile
# create new LogFile instance
log = LogFile(name='log name')
class HelloWorld():
def __init__(self):
# call logEvent method on your LogFile instance
log.logEvent('You have created a HelloWorld Object!', False)
FILE: Main.py
===================
# import HelloWorld
from .HellowWorld import HellowWorld
# create new HellowWorld instance
hw = HellowWorld()
Also, to create a module you will need to add an __init__.py file in that given directory.

This problem is easily solved by using the built-in "Logging" module. In an answer to the broader "how to use a thing(log) within all of my modules" question, I assume the answer to this can be found by reading through the code in the logging module and mimicking that.

How can I import one Jupyter notebook into another

Apparently it's possible to import one Jupyter notebook into another. The linked page has quite a bit of code to do it. Am I supposed to add that code to the importing notebook? The page isn't clear about it. It's supposed to be a general solution, so it doesn't make sense to add all that code to all notebooks that import other notebooks. Any help would be appreciated. Thanks.

Yes, you can add all of that code to a notebook if you want.
And yes, you shouldn't do so as a general solution.
A notebook is a complicated structure, as alluded to in the details of the text (I think it's JSON). It can contain python code, it can contain magics - cython, bash, latex and more - which would not be understood by the Python kernel. Essentially you have to replicate a portion of the functionality of the normal Python import process, as natively Python won't understand there is Python code inside an Ipython notebook.
However ... normally if you have a significant amount of Python code you would split it into modules, and then import the modules. These work as normal, because it is a normal Python import.
For example, once the code has been loaded to tell it how to understand what a notebook is, the actual import is only
import nbpackage.mynotebook
We can use the same technique with the module import code - find_notebook and NotebookLoader can be put into a helper module (e.g. helper.py), and all you would have to do is, from within your notebook, use from helper import NotebookFinder.
I suspect you'd still have to call sys.meta_path.append(NotebookFinder()) from inside your notebook along with the import.
Here is a specific example of how you can use the import capabilities to create an API drawn from a notebook:
Create a notebook. We'll call it scanner.ipynb:
import os, sys
def scanner(start):
for root, dirs,files in os.walk(start):
# remove any already processed file
if 'done' in dirs:
dirs.remove('done')
for names in files:
name, ext = os.path.splitext(names)
# only interested in media files
if ext == '.mp4' or ext == '.mkv':
print(name)
Create a regular python file called reuse.py. This is your general re-usable Ipython import module:
#! /usr/env/bin python
# *-* coding: utf-8 *-*
import io, os, sys, types
from IPython import get_ipython
from nbformat import read
from IPython.core.interactiveshell import InteractiveShell
def find_notebook(fullname, path=None):
"""find a notebook, given its fully qualified name and an optional path
This turns "foo.bar" into "foo/bar.ipynb"
and tries turning "Foo_Bar" into "Foo Bar" if Foo_Bar
does not exist.
"""
name = fullname.rsplit('.', 1)[-1]
if not path:
path = ['']
for d in path:
nb_path = os.path.join(d, name + ".ipynb")
if os.path.isfile(nb_path):
return nb_path
# let import Notebook_Name find "Notebook Name.ipynb"
nb_path = nb_path.replace("_", " ")
if os.path.isfile(nb_path):
return nb_path
class NotebookLoader(object):
"""Module Loader for Jupyter Notebooks"""
def __init__(self, path=None):
self.shell = InteractiveShell.instance()
self.path = path
def load_module(self, fullname):
"""import a notebook as a module"""
path = find_notebook(fullname, self.path)
print ("importing Jupyter notebook from %s" % path)
# load the notebook object
with io.open(path, 'r', encoding='utf-8') as f:
nb = read(f, 4)
# create the module and add it to sys.modules
# if name in sys.modules:
# return sys.modules[name]
mod = types.ModuleType(fullname)
mod.__file__ = path
mod.__loader__ = self
mod.__dict__['get_ipython'] = get_ipython
sys.modules[fullname] = mod
# extra work to ensure that magics that would affect the user_ns
# actually affect the notebook module's ns
save_user_ns = self.shell.user_ns
self.shell.user_ns = mod.__dict__
try:
for cell in nb.cells:
if cell.cell_type == 'code':
# transform the input to executable Python
code = self.shell.input_transformer_manager.transform_cell(cell.source)
# run the code in themodule
exec(code, mod.__dict__)
finally:
self.shell.user_ns = save_user_ns
return mod
class NotebookFinder(object):
"""Module finder that locates Jupyter Notebooks"""
def __init__(self):
self.loaders = {}
def find_module(self, fullname, path=None):
nb_path = find_notebook(fullname, path)
if not nb_path:
return
key = path
if path:
# lists aren't hashable
key = os.path.sep.join(path)
if key not in self.loaders:
self.loaders[key] = NotebookLoader(path)
return self.loaders[key]
Create your specific API file that connects the loader above with the notebook above. Call it scan_api.py:
# Note the python import here
import reuse, sys
# This is the Ipython hook
sys.meta_path.append(reuse.NotebookFinder())
import scanner
# And now we can drawn upon the code
dir_to_scan = "/username/location"
scanner.scanner(dir_to_scan)

Simple Solution with 2 lines of code
Use 'nbimporter' package in python for importing another notebook A (or its function) in notebook B. You can install 'nbimporter' by using the command - pip install nbimporter
Assume there are two Notebooks A.ipynb and B.ipynb. we are trying to import Notebook A (or its function) inside B Code sample below should solve the issue.
Inside Notebook B.ipynb
import nbimporter
import A # or do this --> from A import func1

how to preserve module path of a module executed as a script

I have a function called get_full_class_name(instance), which returns the full module-qualified class name of instance.
Example my_utils.py:
def get_full_class_name(instance):
return '.'.join([instance.__class__.__module__,
instance.__class__.__name__])
Unfortunately, this function fails when given a class that's defined in a currently running script.
Example my_module.py:
#! /usr/bin/env python
from my_utils import get_full_class_name
class MyClass(object):
pass
def main():
print get_full_class_name(MyClass())
if __name__ == '__main__':
main()
When I run the above script, instead of printing my_module.MyClass, it prints __main__.MyClass:
$ ./my_module.py
__main__.MyClass
I do get the desired behavior if I run the above main() from another script.
Example run_my_module.py:
#! /usr/bin/env python
from my_module import main
if __name__ == '__main__':
main()
Running the above script gets:
$ ./run_my_module.py
my_module.MyClass
Is there a way I could write the get_full_class_name() function such that it always returns my_module.MyClass regardless of whether my_module is being run as a script?

I propose handling the case __name__ == '__main__' using the techniques discussed in Find Path to File Being Run. This results in this new my_utils:
import sys
import os.path
def get_full_class_name(instance):
if instance.__class__.__module__ == '__main__':
return '.'.join([os.path.basename(sys.argv[0]),
instance.__class__.__name__])
else:
return '.'.join([instance.__class__.__module__,
instance.__class__.__name__])
This does not handle interactive sessions and other special cases (like reading from stdin). For this you may have to include techniques like discussed in detect python running interactively.

Following mkiever's answer, I ended up changing get_full_class_name() to what you see below.
If instance.__class__.__module__ is __main__, it doesn't use that as the module path. Instead, it uses the relative path from sys.argv[0] to the closest directory in sys.path.
One problem is that sys.path always includes the directory of sys.argv[0] itself, so this relative path ends up being just the filename part of sys.argv[0]. As a quick hack-around, the code below assumes that the sys.argv[0] directory is always the first element of sys.path, and disregards it. This seems unsafe, but safer options are too tedious for my personal code for now.
Any better solutions/suggestions would be greatly appreciated.
import os
import sys
from nose.tools import assert_equal, assert_not_equal
def get_full_class_name(instance):
'''
Returns the fully-qualified class name.
Handles the case where a class is declared in the currently-running script
(where instance.__class__.__module__ would be set to '__main__').
'''
def get_module_name(instance):
def get_path_relative_to_python_path(path):
path = os.path.abspath(path)
python_paths = [os.path.abspath(p) for p in sys.path]
assert_equal(python_paths[0],
os.path.split(os.path.abspath(sys.argv[0]))[0])
python_paths = python_paths[1:]
min_relpath_length = len(path)
result = None
for python_path in python_paths:
relpath = os.path.relpath(path, python_path)
if len(relpath) < min_relpath_length:
min_relpath_length = len(relpath)
result = os.path.join(os.path.split(python_path)[-1],
relpath)
if result is None:
raise ValueError("Path {} doesn't seem to be in the "
"PYTHONPATH.".format(path))
else:
return result
if instance.__class__.__module__ == '__main__':
script_path = os.path.abspath(sys.argv[0])
relative_path = get_path_relative_to_python_path(script_path)
relative_path = relative_path.split(os.sep)
assert_not_equal(relative_path[0], '')
assert_equal(os.path.splitext(relative_path[-1])[1], '.py')
return '.'.join(relative_path[1:-1])
else:
return instance.__class__.__module__
module_name = get_module_name(instance)
return '.'.join([module_name, instance.__class__.__name__])

Best way to dynamically load modules in a Python library

EDIT: Turns out the problem has to do with the path.
If I cd into the directory containing the library and run python __init__.py the imports all work fine. It's if I'm in a different directory and try to import the library itself (i.e. in the parent directory, and trying to import) that the failure occurs.
I don't see any way to literally specify a path for an import statement.
So, I'm wondering if the best way is just to add the directory in scriptDir to the sys.path? Is this the best way to do it? I feel like there should be a more elegant method, but...
I want to write a library that I will be able to extend very easily.
Here's some skeleton/pseudo code for what I want to do. In reality this code is a lot more complex, but it follows the basic premise - import each file, check it, and determine if we should use it; then allocate it into a list of module references. All of this would be contained in a single library folder.
I want the library, when imported, to dynamically import any file found in its directory starting with "plugin_". See the code:
init.py:
import os.path
scriptDir = os.path.dirname(__file__)
mods = []
thisMod = 0
for file in os.listdir(scriptDir):
if (file[0:7] == "plugin_" and file[-3:] == ".py"):
thisMod = __import__(".".join(file.split(".")[0:-1]))
print "debug: imported %s" % thisMod.modName
if (thisMod.enable == True):
mods.append(thisMod)
else:
print "debug: not loading %s because it's disabled." % thisMod.modName
def listMods():
"This function should print the names of all loaded modules."
for m in mods:
print "debug: module %s" % m.modName
def runMods():
"This function should execute the run method for ALL modules."
for m in mods:
c = m.ModuleClass()
c.run()
def selectMod(modNum):
"This function should let us select a single module for the runSelectedMod function."
thisMod = mods[modNum]
def runSelectedMod():
"This function should run the 'run' method for ONLY the previously selected module."
if (thisMod == 0):
raise ArgumentError("you didn't assign a module yet.")
c = thisMod.ModuleClass()
c.run()
plugin_test1.py
modName = "test module 1"
enable = True
class ModuleClass:
def run(self):
print "test module 1 is running"
plugin_math.py
modName = "math module"
enable = True
class ModuleClass:
def run(self):
print "mathematical result: %d" % (1+1)
plugin_bad.py
modName = "bad module"
enable = False
class ModuleClass:
def __init__(self):
print "x"[4] # throws IndexError, this code should not run.
def run(self):
print "divide by zero: %f" % (5 / 0) # this code should not run.
The problem I've already found is import won't work since I'm not importing whole libraries, but rather individual files. I'm guessing there is either an alternate syntax to import for this purpose? For example, import plugin_test and from plugin_math import ModuleClass work but my use of import isn't. I get an error:
thisMod = __import__(".".join(file.split(".")[0:-1]))
ImportError: No module named plugin_test1
Now, another question is: How will this end up working if I use py2exe, py2app, etc. to compile this into a compact library? If I recall, don't these apps compress all the local libraries into a site_packages.zip file?...
I'm still learning how to do this type of advanced coding in Python, so any advice is appreciated.
Thanks!

I was able to run it in Python 3. The only change not regarding syntax is the scriptDir:
import os.path
scriptDir = os.path.dirname(os.path.abspath(__file__))
mods = []
thisMod = 0
for file in os.listdir(scriptDir):
if (file[0:7] == "plugin_" and file[-3:] == ".py"):
thisMod = __import__(".".join(file.split(".")[0:-1]))
print ("debug: imported %s" % thisMod.modName)
if (thisMod.enable == True):
mods.append(thisMod)
else:
print ("debug: not loading %s because it's disabled." % thisMod.modName)
def listMods():
"This function should print the names of all loaded modules."
for m in mods:
print ("debug: module %s" % m.modName)
def runMods():
"This function should execute the run method for ALL modules."
for m in mods:
c = m.ModuleClass()
c.run()
def selectMod(modNum):
"This function should let us select a single module for the runSelectedMod function."
thisMod = mods[modNum]
def runSelectedMod():
"This function should run the 'run' method for ONLY the previously selected module."
if (thisMod == 0):
raise ArgumentError("you didn't assign a module yet.")
c = thisMod.ModuleClass()
c.run()
gives:
C:\Users\mm\Desktop\ING.SW\python>python so\dina.py
debug: imported bad module
debug: not loading bad module because it's disabled.
debug: imported math module
debug: imported test module 1
I guess you have a different problem, since the file name is actually read (as it's printed in the error message), but still...

i got this error: "ImportError: cannot import name python" How do I fix it?

File "G:\Python25\Lib\site-packages\PyAMF-0.6b2-py2.5-win32.egg\pyamf\util\__init__.py", line 15, in <module>
ImportError: cannot import name python
How do I fix it?
If you need any info to know how to fix this problem, I can explain, just ask.
Thanks
Code:
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext import webapp
from TottysGateway import TottysGateway
import logging
def main():
services_root = 'services'
#services = ['users.login']
#gateway = TottysGateway(services, services_root, logger=logging, debug=True)
#app = webapp.WSGIApplication([('/', gateway)], debug=True)
#run_wsgi_app(app)
if __name__ == "__main__":
main()
Code:
from pyamf.remoting.gateway.google import WebAppGateway
import logging
class TottysGateway(WebAppGateway):
def __init__(self, services_available, root_path, not_found_service, logger, debug):
# override the contructor and then call the super
self.services_available = services_available
self.root_path = root_path
self.not_found_service = not_found_service
WebAppGateway.__init__(self, {}, logger=logging, debug=True)
def getServiceRequest(self, request, target):
# override the original getServiceRequest method
try:
# try looking for the service in the services list
return WebAppGateway.getServiceRequest(self, request, target)
except:
pass
try:
# don't know what it does but is an error for now
service_func = self.router(target)
except:
if(target in self.services_available):
# only if is an available service import it's module
# so it doesn't access services that should be hidden
try:
module_path = self.root_path + '.' + target
paths = target.rsplit('.')
func_name = paths[len(paths) - 1]
import_as = '_'.join(paths) + '_' + func_name
import_string = "from "+module_path+" import "+func_name+' as service_func'
exec import_string
except:
service_func = False
if(not service_func):
# if is not found load the default not found service
module_path = self.rootPath + '.' + self.not_found_service
import_string = "from "+module_path+" import "+func_name+' as service_func'
# add the service loaded above
assign_string = "self.addService(service_func, target)"
exec assign_string
return WebAppGateway.getServiceRequest(self, request, target)

You need to post your full traceback. What you show here isn't all that useful. I ended up digging up line 15 of pyamf/util/init.py. The code you should have posted is
from pyamf import python
This should not fail unless your local environment is messed up.
Can you 'import pyamf.util' and 'import pyamf.python' in a interactive Python shell? What about if you start Python while in /tmp (on the assumption that you might have a file named 'pyamf.py' in the current directory. Which is a bad thing.)
= (older comment below) =
Fix your question. I can't even tell where line 15 of util/__init__.py is supposed to be. Since I can't figure that out, I can't answer your question. Instead, I'll point out ways to improve your question and code.
First, use the markup language correctly, so that all the code is in a code block. Make sure you've titled the code, so we know it's from util/__init__.py and not some random file.
In your error message, include the full traceback, and not the last two lines.
Stop using parens in things like "if(not service_func):" and use a space instead, so its " if not service_func:". This is discussed in PEP 8.
Read the Python documentation and learn how to use the language. Something like "func_name = paths[len(paths) - 1]" should be "func_name = paths[-1]"
Learn about the import function and don't use "exec" for this case. Nor do you need the "exec assign_string" -- just do the "self.addService(service_func, target)"