I'm hoping someone can provide some insight on some extra name bindings that Python3 is creating during an import. Here's the test case:
I created a test package called spam (original, I know). It contains 3 files as follows:
The contents of the files are as follows:
__init__.py:
from .foo import Foo
from .bar import Bar
foo.py:
def Foo():
pass
bar.py:
def Bar():
pass
Pretty simple stuff. When I import the spam package, I can see that it creates name bindings to the Foo() and Bar() functions in the spam namespace, which is expected. What isn't expected is that it also binds a name to the foo and bar modules in the spam namespace, as shown below.
What's even more interesting is that these extra name bindings to the module don't occur if I import the Foo() and Bar() functions in __main__, as shown below:
Reading through the documentation on the import statement (language ref and tutorial), I don't see anything that would cause this to be.
Can anyone shed some light on why, when importing a function from a module inside a package, it also binds a name to the module containing the function?
Yes - that is correct, and part of Python import mechanism.
When you import a module a lot of things happen, but we can focus on a few:
1) Python checks if the module is already loaded - to that means, it checks if it is qualifyed bane (name with dots) is under sys.modules
2) If not, it actually loads the module: that includes checking for pre-compiled cached bytecode files, parse, compile the .py file otherwise, etc...
3) It actually makes the name bindings as they are in the import command: that is "from .foo import Foo" creates a variable "Foo" in the current namespace that points to the "spam.foo.Foo" object.
Perceive that the module is always loaded as a whole - and associated in the sys.modules dictionary. Besides that, the import process makes all sub-modules to a package available in the module namespace visible in that package - that is what causes the names "foo" and "bar" do be visible in your Spam package.
You could, at the end of your __init__.py file delete the names "foo" and "bar" - but that will break the way expected import and usage of spam.foo works in funamental ways - basically: sys.modules["spam.foo"] will exist, butsys.modules["spam"].foo wont - meaning that after
one tries to do:
import spam.foo
spam.foo.Foo()
Python will yield a name error on "foo".
The import machinery will report it as existing (it is in sys.modules), so it does nothing. But "spam.foo" has been removed, so it can't be reached.
Related
As per my understanding, the difference between 'import' and 'from import' in Python is: 'import' imports the whole library, 'from import' imports a specific member or members of the library, and there should not be any behavioral difference.
As per this, I was expecting both the test1.py and test2.py to show the same result i.e. "Lucky". But it is different for test2.py. Can anyone explain why?
mymodule.py
message = "Happy"
def set_message(msg):
global message
message = msg
test1.py
import mymodule
mymodule.set_message("Lucky")
print(mymodule.message) #output is Lucky
test2.py
from mymodule import *
set_message("Lucky")
print(message) #output is Happy
'import' imports the whole library, 'from import' imports a specific member or members of the library, and there should not be any behavioral difference.
Both versions - assuming the module has not already been imported - cause the top-level code to execute and a module object to be created.
import ... means that the module object is assigned to the corresponding name in the local namespace. (The reason the . syntax works is that the global variables from when the module's code was running, become attributes of that modle object.)
from ... import * means that Python iterates over the attributes of the module object, and assigns each name into the current namespace, similarly.
For subsequent code, the module object itself is the global namespace in which its functions run.
Your test2.py calls a method from mymodule, causing it to do a global lookup of message. That lookup finds the message attribute of the module object, and replaces it. The message global variable in your own code is unchanged, because neither was that name reassigned, nor was the value modified (it was replaced). It is the same as if you do:
# Functions are the easiest way to get an object with mutable attributes
def namespace(): pass
namespace.a = 3
a = namespace.a # "import" the name
namespace.a = 4 # replace the value; `a` does not change
This is the contents of script_one.py:
x = "Hello World"
This is the contents of script_two.py:
from script_one import x
print(x)
Now, if I ran script_two.py the output would be:
>>> Hello World
What I need is a way to detect if x was imported.
This is what I imagine the source code of script_one.py would look like:
x = "Hello World"
if x.has_been_imported:
print("You've just imported \"x\"!")
Then if I ran script_two.py the output "should" be:
>>> Hello World
>>> You've just imported "x"!
What is this called, does this feature exist in Python 3 and how do you use it?
You can't. Effort expended on trying to detect this are a waste of time, I'm afraid.
Python imports consist of the following steps:
Check if the module is already loaded by looking at sys.modules.
If the module hasn't been loaded yet, load it. This creates a new module object that is added to sys.modules, containing all objects resulting from executing the top-level code.
Bind names in the importing namespace. How names are bound depends on the exact import variant chosen.
import module binds the name module to the sys.modules[module] object
import module as othername binds the name othername to the sys.modules[module] object
from module import attribute binds the name attribute to the sys.modules[module].attribute object
from module import attribute as othername binds the name othername to the sys.modules[module].attribute object
In this context it is important to realise that Python names are just references; all Python objects (including modules) live on a heap and stand or fall with the number of references to them. See this great article by Ned Batchelder on Python names if you need a primer on how this works.
Your question then can be interpreted in two ways:
You want to know the module has been imported. The moment code in the module is executed (like x = "Hello World"), it has been imported. All of it. Python doesn't load just x here, it's all or nothing.
You want to know if other code is using a specific name. You'd have to track what other references exist to the object. This is a mammoth task involving recursively checking the gc.get_referrers() object chain to see what other Python objects might now refer to x.
The latter goal is made the harder all the further in any of the following scenarios:
import script_one, then use script_one.x; references like these could be too short-lived for you to detect.
from script_one import x, then del x. Unless something else still references the same string object within the imported namespace, that reference is now gone and can't be detected anymore.
import sys; sys.modules['script_one'].x is a legitimate way of referencing the same string object, but does this count as an import?
import script_one, then list(vars(script_one).values()) would create a list of all objects defined in the module, but these references are indices in a list, not named. Does this count as an import?
Looks like it is impossible previously. But ever since python 3.7+ introduces __getattr__ on module level, looks like it is possible now. At least we can distinguish whether a variable is imported by from module import varable or import module; module.variable.
The idea is to detect the AST node in the previous frame, whether it is an Attribute:
script_one.py
def _variables():
# we have to define the variables
# so that it dosen't bypass __getattr__
return {'x': 'Hello world!'}
def __getattr__(name):
try:
out = _variables()[name]
except KeyError as kerr:
raise ImportError(kerr)
import ast, sys
from executing import Source
frame = sys._getframe(1)
node = Source.executing(frame).node
if node is None:
print('`x` is imported')
else:
print('`x` is accessed via `script_one.x`')
return out
script_two.py
from script_one import x
print(x)
# `x` is imported
# 'Hello world!'
import script_one
print(script_one.x)
# `x` is accessed via `script_one.x`
# 'Hello world!'
I have a module some_module.py which contains the following code:
def testf():
print(os.listdir())
Now, in a file named test.py, I have this code:
import os
from some_module import testf
testf()
But executing test.py gives me NameError: name 'os' is not defined. I've already imported os in test.py, and testf is in the namespace of test.py. So why does this error occur?
import is not the same as including the content of the file as if you had typed it directly in place of the import statement. You might think it works this way if you're coming from a C background, where the #include preprocessor directive does this, but Python is different.
The import statement in Python reads the content of the file being imported and evaluates it in its own separate context - so, in your example, the code in some_module.py has no access to or knowledge of anything that exists in test.py or any other file. It starts with a "blank slate", so to speak. If some_module.py's code wants to access the os module, you have to import it at the top of some_module.py.
When a module is imported in Python, it becomes an object. That is, when you write
import some_module
one of the first things Python does is to create a new object of type module to represent the module being imported. As the interpreter goes through the code in some_module.py, it assigns any variables, functions, classes, etc. that are defined in that file to be attributes of this new module object. So in your example, the module object will have one attribute, testf. When the code in the function testf wants to access the variable os, it looks in the function itself (local scope) and sees that os is not defined there, so it then looks at the attributes of the module object which testf belongs to (this is the "global" scope, although it's not truly global). In your example, it will not see os there, so you get an error. If you add
import os
to some_module.py, then that will create an attribute of the module under the name os, and your code will find what it needs to.
You may also be interested in some other answers I've written that may help you understand Python's import statement:
Why import when you need to use the full name?
Does Python import statement also import dependencies automatically?
The name testf is in the namespace of test. The contents of the testf function are still in some_module, and don't have access to anything in test.
If you have code that needs a module, you need to import that module in the same file where that code is. Importing a module only imports it into the one file where you import it. (Multiple imports of the same module, in different files, won't incur a meaningful performance penalty; the actual loading of the module only happens once, and later imports of the same module just get a reference to the already-imported module.)
Importing a module adds its name as an attribute of the current scope. Since different modules have independent scopes, any code in some_module cannot use names in __main__ (the executed script) without having imported it first.
I can make this code work, but I am still confused why it won't work the first way I tried.
I am practicing python because my thesis is going to be coded in it (doing some cool things with Arduino and PC interfaces). I'm trying to import a class from another file into my main program so that I can create objects. Both files are in the same directory. It's probably easier if you have a look at the code at this point.
#from ArduinoBot import *
#from ArduinoBot import ArduinoBot
import ArduinoBot
# Create ArduinoBot object
bot1 = ArduinoBot()
# Call toString inside bot1 object
bot1.toString()
input("Press enter to end.")
Here is the very basic ArduinoBot class
class ArduinoBot:
def toString(self):
print ("ArduinoBot toString")
Either of the first two commented out import statements will make this work, but not the last one, which to me seems the most intuitive and general. There's not a lot of code for stuff to go wrong here, it's a bit frustrating to be hitting these kind of finicky language specific quirks when I had heard some many good things about Python. Anyway I must be doing something wrong, but why doesn't the simple 'import ClassName' or 'import FileName' work?
Thank you for your help.
consider a file (example.py):
class foo(object):
pass
class bar(object):
pass
class example(object):
pass
Now in your main program, if you do:
import example
what should be imported from the file example.py? Just the class example? should the class foo come along too? The meaning would be too ambiguous if import module pulled the whole module's namespace directly into your current namespace.
The idea is that namespaces are wonderful. They let you know where the class/function/data came from. They also let you group related things together (or equivalently, they help you keep unrelated things separate!). A module sets up a namespace and you tell python exactly how you want to bring that namespace into the current context (namespace) by the way you use import.
from ... import * says -- bring everything in that module directly into my namespace.
from ... import ... as ... says, bring only the thing that I specify directly into my namespace, but give it a new name here.
Finally, import ... simply says bring that module into the current namespace, but keep it separate. This is the most common form in production code because of (at least) 2 reasons.
It prevents name clashes. You can have a local class named foo which won't conflict with the foo in example.py -- You get access to that via example.foo
It makes it easy to trace down which module a class came from for debugging.
consider:
from foo import *
from bar import *
a = AClass() #did this come from foo? bar? ... Hmmm...
In this case, to get access to the class example from example.py, you could also do:
import example
example_instance = example.example()
but you can also get foo:
foo_instance = example.foo()
The simple answer is that modules are things in Python. A module has its own status as a container for classes, functions, and other objects. When you do import ArduinoBot, you import the module. If you want things in that module -- classes, functions, etc. -- you have to explicitly say that you want them. You can either import them directly with from ArduinoBot import ..., or access them via the module with import ArduinoBot and then ArduinoBot.ArduinoBot.
Instead of working against this, you should leverage the container-ness of modules to allow you to group related stuff into a module. It may seem annoying when you only have one class in a file, but when you start putting multiple classes and functions in one file, you'll see that you don't actually want all that stuff being automatically imported when you do import module, because then everything from all modules would conflict with other things. The modules serve a useful function in separating different functionality.
For your example, the question you should ask yourself is: if the code is so simple and compact, why didn't you put it all in one file?
Import doesn't work quite the you think it does. It does work the way it is documented to work, so there's a very simple remedy for your problem, but nonetheless:
import ArduinoBot
This looks for a module (or package) on the import path, executes the module's code in a new namespace, and then binds the module object itself to the name ArduinoBot in the current namespace. This means a module global variable named ArduinoBot in the ArduinoBot module would now be accessible in the importing namespace as ArduinoBot.ArduinoBot.
from ArduinoBot import ArduinoBot
This loads and executes the module as above, but does not bind the module object to the name ArduinoBot. Instead, it looks for a module global variable ArduinoBot within the module, and binds whatever object that referred to the name ArduinoBot in the current namespace.
from ArduinoBot import *
Similarly to the above, this loads and executes a module without binding the module object to any name in the current namespace. It then looks for all module global variables, and binds them all to the same name in the current namespace.
This last form is very convenient for interactive work in the python shell, but generally considered bad style in actual development, because it's not clear what names it actually binds. Considering it imports everything global in the imported module, including any names that it imported at global scope, it very quickly becomes extremely difficult to know what names are in scope or where they came from if you use this style pervasively.
The module itself is an object. The last approach does in fact work, if you access your class as a member of the module. Either if the following will work, and either may be appropriate, depending on what else you need from the imported items:
from my_module import MyClass
foo = MyClass()
or
import my_module
foo = my_module.MyClass()
As mentioned in the comments, your module and class usually don't have the same name in python. That's more a Java thing, and can sometimes lead to a little confusion here.
As I understand it python has the following outermost namespaces:
Builtin - This namespace is global across the entire interpreter and all scripts running within an interpreter instance.
Globals - This namespace is global across a module, ie across a single file.
I am looking for a namespace in between these two, where I can share a few variables declared within the main script to modules called by it.
For example, script.py:
import Log from Log
import foo from foo
log = Log()
foo()
foo.py:
def foo():
log.Log('test') # I want this to refer to the callers log object
I want to be able to call script.py multiple times and in each case, expose the module level log object to the foo method.
Any ideas if this is possible?
It won't be too painful to pass down the log object, but I am working with a large chunk of code that has been ported from Javascript. I also understand that this places constraints on the caller of foo to expose its log object.
Thanks,
Paul
There is no namespace "between" builtins and globals -- but you can easily create your own namespaces and insert them with a name in sys.modules, so any other module can "import" them (ideally not using the from ... import syntax, which carries a load of problems, and definitely not using tghe import ... from syntax you've invented, which just gives a syntax error). For example, in script.py:
import sys
import types
sys.modules['yay'] = types.ModuleType('yay')
import Log
import foo
yay.log = Log.Log()
foo.foo()
and in foo.py
import yay
def foo():
yay.log.Log('test')
Do not fear qualified names -- they're goodness! Or as the last line of the Zen of Python (AKA import this) puts it:
Namespaces are one honking great idea -- let's do more of those!
You can make and use "more of those" most simply -- just qualify your names (situating them in the proper namespace they belong in!) rather than insisting on barenames where they're just not a good fit. There's a bazillion things that are quite easy with qualified names and anywhere between seriously problematic and well-nigh unfeasible for those who're stuck on barenames!-)
There is no such scope. You will need to either add to the builtins scope, or pass the relevant object.
Actually, I did figure out what I was looking for.
This hack is actually used PLY and that is where is stumbled across.
The library code can raise a runtime exception, which then gives access to the callers stack.