I have two files: 'my_main.py' and 'my_functions.py'. I wrote two functions in 'my_functions.py', 'first_function' and 'second_function' and imported this module into 'my_main.py'. However, I then added a third function into 'my_functions.py' called 'third_function':
def first_function():
print("hello1")
def second_function():
print("hello2")
def third_function():
print("hello3")
I found that I could not import 'third_function' into 'my_main.py', getting an error - AttributeError: module 'my_functions' has no attribute 'third_function'.
To understand this, in 'my_main.py' I then ran:
import my_functions
help(my_functions)
and had as my output:
Help on module my_functions:
NAME
my_functions
FUNCTIONS
first_function()
second_function()
For some reason 'my_functions.py' would not recognise any more functions after the initial two, meaning I couldn't then import 'third_function' into 'my_main.py'. How can I fix this?
Make sure you have saved your file my_functions.py before running my_main.py. Unless the file is saved, the update (addition of third_function) won't be recognized. Have been doing this for years, and still make this mistake constantly. Editing not enough. Need to edit and save.
You need to explicitly reload the module if you've made changes since the last import. There is a built-in function for this
You can reload your module as follows:
import importlib
importlib.reload(my_functions)
See this answer for a much more detailed explanation.
Related
I have a Jupyter Notebook - we'll call it main, and I'm running it piecemeal in Jupyter Lab 3.0.14 using a Python 3 environment.
Within this notebook, I'm trying to import custom functions from a module, we'll call this one module.py, which is in the same directory as main.
It looks something like this:
//main
import sys
import os
from module import foo, bar
//module.py
def foo():
return
def bar():
return
When I run the import statements in main, it throws an error
ImportError: cannot import name 'bar' from 'module' (/module.py)
If I run
from module import foo
or
from module import *
It's fine (although calling the functions will throw an error that the function is not defined). In fact, it will import foo even if there's no function named foo in module.py. It won't import any of my own functions by name. I feel like I must be missing something fundamental, here...
Any help would be greatly appreciated!!
Okay, this is my first time using Jupyter Lab, and this problem came from a fundamental misunderstanding of how Jupyter (and maybe other IDEs?) manages packages.
In short, as I understand it, re-running the code which imported functions from module.py did not actually re-import the contents. The only things it imported were what I asked for the first time, and any subsequent calls to import functions only referenced that first static image I pulled.
Therefore, if module.py had one function, foo, and I ran:
from module import foo
...then made some changes, such as editing the contents of foo and adding bar, running:
from module import foo, bar
...didn't actually do anything but reference that first copy of module.py I pulled. Because bar didn't exist the first time I ran the import function, it "couldn't see it", no matter how many times I re-ran the import function. I had to restart the kernel to fix it, basically.
I noticed this because when I changed the name of the script, it would import the functions that weren't previously working, but then edits to the functions, even once imported, weren't recognized.
This link provided the clue I needed to figure this out.
Hopefully this helps somebody!
Restart the kernel in the Jupiter notebook. I solved a similar import name issue by restarting the kernel.
Is there a way to force import x to always reload x in Python (i.e., as if I had called reload(x), or imp.reload(x) for Python 3)? Or in general, is there some way to force some code to be run every time I run import x? I'm OK with monkey patching or hackery.
I've tried moving the code into a separate module and deleting x from sys.modules in that separate file. I dabbled a bit with import hooks, but I didn't try too hard because according to the documentation, they are only called after the sys.modules cache is checked. I also tried monkeypatching sys.modules with a custom dict subclass, but whenever I do that, from module import submodule raises KeyError (I'm guessing sys.modules is not a real dictionary).
Basically, I'm trying to write a debugging tool (which is why some hackery is OK here). My goal is simply that import x is shorter to type than import x;x.y.
If you really want to change the semantics of the import statement, you will have to patch the interpreter. import checks whether the named module already is loaded and if so it does nothing more. You would have to change exactly that, and that is hard-wired in the interpreter.
Maybe you can live with patching the Python sources to use myImport('modulename') instead of import modulename? That would make it possible within Python itself.
Taking a lead from Alfe's answer, I got it to work like this. This goes at the module level.
def custom_logic():
# Put whatever you want to run when the module is imported here
# This version is run on the first import
custom_logic()
def __myimport__(name, *args, **kwargs):
if name == 'x': # Replace with the name of this module
# This version is run on all subsequent imports
custom_logic()
return __origimport__(name, *args, **kwargs)
# Will only be run on first import
__builtins__['__origimport__'] = __import__
__builtins__['__import__'] = __myimport__
We are monkeypatching __builtins__, which is why __origimport__ is defined when __myimport__ is run.
I understand how to actually link python files, however, i don't understand how to get variable's from these linked files. I've tried to grab them and I keep getting NameError.
How do I go about doing this? The reason i want to link files is to simply neaten up my script and not make it 10000000000 lines long. Also, in the imported python script, do i have to import everything again? Another question, do i use the self function when using another scripts functions?
ie;
Main Script:
import sys, os
import importedpyfile
Imported Py File
import sys, os
I understand how to actually link python files, however, i don't
understand how to get variable's from these linked files. I've tried
to grab them and I keep getting NameError.
How are you doing that? Post more code. For instance, the following works:
file1.py
#!/usr/bin/env python
from file2 import file2_func, file2_variable
file2_func()
print file2_variable
file2.py:
#!/usr/bin/env python
file2_variable = "I'm a variable!"
def file2_func():
print "Hello World!"
Also, in the imported python script, do i have to import everything
again?
Nope, modules should be imported when the python interpreter reads that file.
Another question, do i use the self function when using another
scripts functions?
Nope, that's usually to access class members. See python self explained.
There is also more than one way to import files. See the other answer for some explanations.
I think what you are trying to ask is how to get access to global vars from on .py file without having to deal with namespaces.
In your main script, replace the call to import importedpyfile to say this instead
from importedpyfile import *
Ideally, you keep the code the way you have it. But instead, just reference those global vars with the importedpyfile namespace.
e.g.
import importedpyfile
importedpyfile.MyFunction() # calls "MyFunction" that is defined in importedpyfile.py
Python modules are not "linked" in the sense of C/C++ linking libraries into an executable. A Python import operation creates a name that refers to the imported module; without this name there is no (direct) way to access another module.
I'm working on pypreprocessor which is a preprocessor that takes c-style directives and I've been able to make it work like a traditional preprocessor (it's self-consuming and executes postprocessed code on-the-fly) except that it breaks library imports.
The problem is: The preprocessor runs through the file, processes it, outputs to a temporary file, and exec() the temporary file. Libraries that are imported need to be handled a little different, because they aren't executed, but rather they are loaded and made accessible to the caller module.
What I need to be able to do is: Interrupt the import (since the preprocessor is being run in the middle of the import), load the postprocessed code as a tempModule, and replace the original import with the tempModule to trick the calling script with the import into believing that the tempModule is the original module.
I have searched everywhere and so far and have no solution.
This Stack Overflow question is the closest I've seen so far to providing an answer:
Override namespace in Python
Here's what I have.
# Remove the bytecode file created by the first import
os.remove(moduleName + '.pyc')
# Remove the first import
del sys.modules[moduleName]
# Import the postprocessed module
tmpModule = __import__(tmpModuleName)
# Set first module's reference to point to the preprocessed module
sys.modules[moduleName] = tmpModule
moduleName is the name of the original module, and tmpModuleName is the name of the postprocessed code file.
The strange part is this solution still runs completely normal as if the first module completed loaded normally; unless you remove the last line, then you get a module not found error.
Hopefully someone on Stack Overflow know a lot more about imports than I do, because this one has me stumped.
Note: I will only award a solution, or, if this is not possible in Python; the best, most detailed explanation of why this is not impossible.
Update: For anybody who is interested, here is the working code.
if imp.lock_held() is True:
del sys.modules[moduleName]
sys.modules[tmpModuleName] = __import__(tmpModuleName)
sys.modules[moduleName] = __import__(tmpModuleName)
The 'imp.lock_held' part detects whether the module is being loaded as a library. The following lines do the rest.
Does this answer your question? The second import does the trick.
Mod_1.py
def test_function():
print "Test Function -- Mod 1"
Mod_2.py
def test_function():
print "Test Function -- Mod 2"
Test.py
#!/usr/bin/python
import sys
import Mod_1
Mod_1.test_function()
del sys.modules['Mod_1']
sys.modules['Mod_1'] = __import__('Mod_2')
import Mod_1
Mod_1.test_function()
To define a different import behavior or to totally subvert the import process you will need to write import hooks. See PEP 302.
For example,
import sys
class MyImporter(object):
def find_module(self, module_name, package_path):
# Return a loader
return self
def load_module(self, module_name):
# Return a module
return self
sys.meta_path.append(MyImporter())
import now_you_can_import_any_name
print now_you_can_import_any_name
It outputs:
<__main__.MyImporter object at 0x009F85F0>
So basically it returns a new module (which can be any object), in this case itself. You may use it to alter the import behavior by returning processe_xxx on import of xxx.
IMO: Python doesn't need a preprocessor. Whatever you are accomplishing can be accomplished in Python itself due to it very dynamic nature, for example, taking the case of the debug example, what is wrong with having at top of file
debug = 1
and later
if debug:
print "wow"
?
In Python 2 there is the imputil module that seems to provide the functionality you are looking for, but has been removed in python 3. It's not very well documented but contains an example section that shows how you can replace the standard import functions.
For Python 3 there is the importlib module (introduced in Python 3.1) that contains functions and classes to modify the import functionality in all kinds of ways. It should be suitable to hook your preprocessor into the import system.
I've done what I shouldn't have done and written 4 modules (6 hours or so) without running any tests along the way.
I have a method inside of /mydir/__init__.py called get_hash(), and a class inside of /mydir/utils.py called SpamClass.
/mydir/utils.py imports get_hash() from /mydir/__init__.
/mydir/__init__.py imports SpamClass from /mydir/utils.py.
Both the class and the method work fine on their own but for some reason if I try to import /mydir/, I get an import error saying "Cannot import name get_hash" from /mydir/__init__.py.
The only stack trace is the line saying that __init__.py imported SpamClass. The next line is where the error occurs in in SpamClass when trying to import get_hash. Why is this?
This is a pretty easy problem to encounter. What's happening is this that the interpreter evaluates your __init__.py file line-by line. When you have the following code:
import mydir.utils
def get_hash(): return 1
The interpreter will suspend processing __init__.py at the point of import mydir.utils until it has fully executed 'mydir/utils.py' So when utils.py attempts to import get_hash(), it isn't defined because the interpreter hasn't gotten to it's definition yet.
To add to what the others have said, another good approach to avoiding circular import problems is to avoid from module import stuff.
If you just do standard import module at the top of each script, and write module.stuff in your functions, then by the time those functions run, the import will have finished and the module members will all be available.
You then also don't have to worry about situations where some modules can update/change one of their members (or have it monkey-patched by a naughty third party). If you'd imported from the module, you'd still have your old, out-of-date copy of the member.
Personally, I only use from-import for simple, dependency-free members that I'm likely to refer to a lot: in particular, symbolic constants.
In absence of more information, I would say you have a circular import that you aren't working around. The simplest, most obvious fix is to not put anything in mydir/__init__.py that you want to use from any module inside mydir. So, move your get_hash function to another module inside the mydir package, and import that module where you need it.