I have a package named jiva_tasks, which I'm trying to import via celery (using the CELERY_IMPORTS attribute of celeryconfig. The import statement that celery is using is this:
__import__(module, [], [], [''])
Oddly enough, when this syntax is used, the module gets imported twice, once as jiva_tasks and another time as jiva_tasks. (with a period at the end). Now, chances are good that celery should be passing in globals rather than an empty list, but this seems broken to me. It seems odd that even if given the wrong arguments, __import__ would import something that isn't a valid python module name.
I know that the way to fix this is to pass in globals, but I want to understand why I'm getting this result. Is this a bug, or is there something I don't understand about how __import__ is working?
Update: It also seems to be working fine if I use importlib.
Update 2: Here's the sys.meta_path and sys.import_path right before the __import__ line gets executed:
>>> sys.meta_path
[]
>>> sys.path_hooks
[<type 'zipimport.zipimporter'>]
It doesn't appear to me that there's anything out of the ordinary. However, I just now realized that the package I'm importing is installed using setuptools' develop command. Does that make a difference?
Creating an empty file "foo.py" and then creating a "bar.py" that says:
__import__('foo', [], [], [''])
import sys
print sorted(sys.modules)
prints out a list that only includes foo once, and not foo. or anything else with a trailing dot — so it's not simply the fact that celery is using __import__ that is causing the problem here. Can you figure out what additional step celery is taking that adds the second module? (Or does a print sorted(sys.modules) right before and after the __import__() call suggest that both modules appear at that moment — in which case we need to look at what import hooks are defined at the moment of the import?)
Related
I am trying to have a module that provides a framework of sorts to short declarative one-file scripts that I write. In the short scripts, I would like to define a number of variables and functions that are accessed or called back by the framework module.
I am stuck ... I have tried two approaches, and I am not crazy about either one. Is there a better way?
First approach:
Framework module:
import __main__
def test():
print __main__.var
print __main__.func()
Script:
import framework
var="variable"
def func():
print "function"
framework.test()
I like this approach and it works, but eclipse gives me an 'Undefined variable from import: var' error on any imported variable or function from main. Obviously, this is not correct as it works, but it clutters up eclipse with many false errors.
Second Approach:
Framework module:
def test(glob):
print glob['var']
print glob['func']()
Script:
import framework
var="variable"
def func():
print "function"
framework.test(globals())
This seems to work and does not give me errors, but I don't like the dictionary type notation used to call variables and functions ... especially for functions: func'funcName'.
Is there a better way to implement this that results in clean code? Can I pass in a module name (ie main) as an argument to a function? Can Eclipse errors on main for the first approach be turned off?
I went with the first approach and told Eclipse to ignore the Undefined Variable from Import error by pressing command+1 (for my Mac, for Windows it would be ctrl+1) on each line where this error occurred.
command+1 adds the following comment to the end of each line:
# #UndefinedVariable
Thanks to #ekhumoro for pointing this out in the comments.
I'm teaching myself Python (I have experience in other languages).
I found a way to import a "module". In PHP, this would just be named an include file. But I guess Python names it a module. I'm looking for a simple, best-practices approach. I can get fancy later. But right now, I'm trying to keep it simple while not developing bad habits. Here is what I did:
I created a blank file named __init__.py, which I stored in Documents (the folder on the Mac)
I created a file named myModuleFile.py, which I stored in Documents
In myModuleFile.py, I created a function:
def myFunction()
print("hello world")
I created another file: myMainFile.py, which I stored in Documents
In this file, I typed the following:
import myModuleFile.py
myModuleFile.myFunction()
This successfully printed out "hello world" to the console when I ran it on the terminal.
Is this a best-practices way to do this for my simple current workflow?
I'm not sure the dot notation means I'm onto something good or something bad. It throws an error if I try to use myFunction() instead of myModuleFile.myFunction(). I kind of think it would be good. If there were a second imported module, it would know to call myFunction() from myModuleFile rather than the other one. So the dot notation makes everybody know exactly which file you are trying to call the function from.
I think there is some advanced stuff using sys or some sort of exotic configuration stuff. But I'm hoping my simple little way of doing things is ok for now.
Thanks for any clarification on this.
For your import you don't need the ".py" extension
You can use:
import myModuleFile
myModuleFile.myFunction()
Or
from myModuleFile import myFunction
myFunction()
Last syntax is common if you import several functions or globals of your module.
Besides to use the "main" function, I'd put this on your module:
from myModuleFile import myFunction
if __name__ == '__main__':
myFunction()
Otherwise the main code could be executed in imports or other cases.
I'd use just one module for myModuleFile.py and myMainFile.py, using the previous pattern let you know if your module is called from command line or as import.
Lastly, I'd change the name of your files to avoid the CamelCase, that is, I'd replace myModuleFile.py by my_module.py. Python loves the lowercase ;-)
You only need to have init.py if you are creating a package (a package in a simple sense is a subdirectory which has one or more modules in it, but I think it may be more complex than you need right now).
If you have just one folder which has MyModule.py and MyMainFile.py - you don't need the init.py.
In MyMainFile.py you can write :
import myModuleFile
and then use
myModuleFile.MyFunction()
The reason for including the module name is that you may reuse the same function name in more than one module and you need a way of saying which module your program is using.
Module Aliases
If you want to you can do this :
import myModuleFile as MyM
and then use
MyM.MyFunction()
Here you have created MyM as an alias for myModuleFile, and created less typing.
Here Lies Dragons
You will sometimes see one other forms of IMport, which can be dangerous, especially for the beginner.
from myModuleFile import MyFunction
if you do this you can use :
MyFunction()
but this has a problem if you have used the same function name in MyMainFile, or in any other library you have used, as you now can't get to any other definition of the name MyFunction. This is often termed Contaminating the namespace - and should really be avoided unless you are absolutely certain it is safe.
there is a final form which I will show for completeness :
from myModuleFile import *
While you will now be able to access every function defined in myModuleFile without using myModuleFile in front of it, you have also now prevented your MyMainFile from using any function in any library which matches any name defined in myModuleFile.
Using this form is generally not considered to be a good idea.
I hope this helps.
Is there a way to force import x to always reload x in Python (i.e., as if I had called reload(x), or imp.reload(x) for Python 3)? Or in general, is there some way to force some code to be run every time I run import x? I'm OK with monkey patching or hackery.
I've tried moving the code into a separate module and deleting x from sys.modules in that separate file. I dabbled a bit with import hooks, but I didn't try too hard because according to the documentation, they are only called after the sys.modules cache is checked. I also tried monkeypatching sys.modules with a custom dict subclass, but whenever I do that, from module import submodule raises KeyError (I'm guessing sys.modules is not a real dictionary).
Basically, I'm trying to write a debugging tool (which is why some hackery is OK here). My goal is simply that import x is shorter to type than import x;x.y.
If you really want to change the semantics of the import statement, you will have to patch the interpreter. import checks whether the named module already is loaded and if so it does nothing more. You would have to change exactly that, and that is hard-wired in the interpreter.
Maybe you can live with patching the Python sources to use myImport('modulename') instead of import modulename? That would make it possible within Python itself.
Taking a lead from Alfe's answer, I got it to work like this. This goes at the module level.
def custom_logic():
# Put whatever you want to run when the module is imported here
# This version is run on the first import
custom_logic()
def __myimport__(name, *args, **kwargs):
if name == 'x': # Replace with the name of this module
# This version is run on all subsequent imports
custom_logic()
return __origimport__(name, *args, **kwargs)
# Will only be run on first import
__builtins__['__origimport__'] = __import__
__builtins__['__import__'] = __myimport__
We are monkeypatching __builtins__, which is why __origimport__ is defined when __myimport__ is run.
I've done what I shouldn't have done and written 4 modules (6 hours or so) without running any tests along the way.
I have a method inside of /mydir/__init__.py called get_hash(), and a class inside of /mydir/utils.py called SpamClass.
/mydir/utils.py imports get_hash() from /mydir/__init__.
/mydir/__init__.py imports SpamClass from /mydir/utils.py.
Both the class and the method work fine on their own but for some reason if I try to import /mydir/, I get an import error saying "Cannot import name get_hash" from /mydir/__init__.py.
The only stack trace is the line saying that __init__.py imported SpamClass. The next line is where the error occurs in in SpamClass when trying to import get_hash. Why is this?
This is a pretty easy problem to encounter. What's happening is this that the interpreter evaluates your __init__.py file line-by line. When you have the following code:
import mydir.utils
def get_hash(): return 1
The interpreter will suspend processing __init__.py at the point of import mydir.utils until it has fully executed 'mydir/utils.py' So when utils.py attempts to import get_hash(), it isn't defined because the interpreter hasn't gotten to it's definition yet.
To add to what the others have said, another good approach to avoiding circular import problems is to avoid from module import stuff.
If you just do standard import module at the top of each script, and write module.stuff in your functions, then by the time those functions run, the import will have finished and the module members will all be available.
You then also don't have to worry about situations where some modules can update/change one of their members (or have it monkey-patched by a naughty third party). If you'd imported from the module, you'd still have your old, out-of-date copy of the member.
Personally, I only use from-import for simple, dependency-free members that I'm likely to refer to a lot: in particular, symbolic constants.
In absence of more information, I would say you have a circular import that you aren't working around. The simplest, most obvious fix is to not put anything in mydir/__init__.py that you want to use from any module inside mydir. So, move your get_hash function to another module inside the mydir package, and import that module where you need it.
In the emacs Python shell (I'm running 2.* Python) I am importing a .py file I'm working with and testing the code. If I change the code however I'm not sure how to import it again.
From my reading so far it seems that
reload(modulename)
should work, but it doesn't seem to.
Perhaps just shutting down the python shell and restarting it would be enough, is there a command for that or you just do it manually?
edit: It looks like python-send-defun and python-send-buffer would be ideal, but changes don't seem to be propagating.
While reload() does work, it doesn't change references to classes, functions, and other objects, so it's easy to see an old version. A most consistent solution is to replace reload() with either exec (which means not using import in the first place) or restarting the interpreter entirely.
If you do want to continue to use reload, be very careful about how you reference things from that module, and always use the full name. E.g. import module and use module.name instead of from module import name. And, even being careful, you will still run into problems with old objects, which is one reason reload() isn't in 3.x.
It seems to work for me:
Make a file (in your PYTHONPATH) called test.py
def foo():
print('bar')
Then in the emacs python shell (or better yet, the ipython shell), type
>>> import test
>>> test.foo()
bar
Now modify test.py:
def foo():
print('baz')
>>> reload(test)
<module 'test' from '/home/unutbu/pybin/test.py'>
>>> test.foo()
baz
After looking at this issue for quite an extensive amount of time, I came to the conclusion that the best solution to implement is, either based on an initialisation file of your python interpreter (ipython for exemple), or using the python build-in module "imp" and its function "reload". For instance at the beginning of your code:
import my_module
import imp
imp.reload(my_module)
#your code
This solution came to me from this page: https://emacs.stackexchange.com/questions/13476/how-to-force-a-python-shell-to-re-import-modules-when-running-a-buffer