I am not a professional programmer, and I'm just starting to study python.
I just figured out that just because I could do:
>>> from xml.etree.ElementTree import Element
>>> var = Element("Something")
doesn't mean I can do:
>>> import xml
>>> var = xml.etree.ElementTree.Element("Something")
In fact, doing this:
>>> import xml
>>> dir(xml)
['_MINIMUM_XMLPLUS_VERSION', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
etree doesn't even appear as one of the methods provided by xml.
What is the relation between xml and xml.etree.ElementTree?
Why can I not see etree as one of xml's methods?
xml.etree is a child module to xml package. It only shows up in xml's namespace after you import xml.etree. A parent may import its child modules on its initialization (e.g. in 2.7, os imports os.path1) but it's not required to do so.
Conversely, when you import a module from a package directly, the package is automatically imported first.
Strangely, I couldn't find any phrasing in the docs stating this. But a test shows that this is exactly what happens:
$ cat test/__init__.py
print "package init"
import traceback
traceback.print_stack()
$ cat test/module.py
print "module init"
import traceback
traceback.print_stack()
$ python
<...>
>>> import test.module
package init
File "<stdin>", line 1, in <module>
File "test\__init__.py", line 3, in <module>
traceback.print_stack()
module init
File "<stdin>", line 1, in <module>
File "test\module.py", line 3, in <module>
traceback.print_stack()
1this is not documented though so don't rely on this
Related
I am working in Jupyter notebook. I created a simple module called conv.py for converting miles to km. When I try to import this module in a separate code (in the same directory) the import seems to go successfully but it doesn't recognize either of the functions I defined in the 'conv' module.
I have imported os and os.getcwd() provides the correct folder for conv.py...
code for conv.py
in_n_ft = 12
ft_n_mile = 5280
m_n_km = 1000
cm_n_in = 2.54
cm_n_m = 100
mm_n_m = 1000
def ft_to_km(feet):
return feet*in_n_ft*cm_n_in/cm_n_m/m_n_km
print(ft_to_km(5280))
def mil_to_km(mile):
return mile*ft_n_mile*in_n_ft*cm_n_in/cm_n_m/m_n_km
print(mil_to_km(3.2))
Code for new module
import conv
km = conv.mil_to_km(5)
Error provided
AttributeError Traceback (most recent call last)
<ipython-input-111-bfd778724ae2> in <module>
3 import conv
4
----> 5 km = conv.mil_to_km(5)
AttributeError: module 'conv' has no attribute 'mil_to_km'
When I type
dir(conv)
I get
['__builtins__',
'__cached__',
'__doc__',
'__file__',
'__loader__',
'__name__',
'__package__',
'__spec__']
What am I doing wrong?
EDIT
I have also tried
from conv import mil_to_km
when I do that I get a different error
cannot import name 'mil_to_km' from 'conv' (C:\Users\223023441\Documents\python\conv.py)
I have also queried the module using:
from inspect import getmembers, isfunction
import conv
print(getmembers(conv, isfunction))
from here I get:
['__builtins__',
'__cached__',
'__doc__',
'__file__',
'__loader__',
'__name__',
'__package__',
'__spec__']
I am also unable to access any of the variables within the conv.py file after import... Am I doing something wrong when I save the py file? Jupyter makes ipynb as the common file, when I 'save as' to conv.py, is this screwing it up?
You should import from the module.
Try this:
from conv import mil_to_km
km = mil_to_km(5)
The reason is that when you import the module in that way, you are executing it.
In the way I shown, you are just importing the needed functions.
So the ultimate issue was the way I was saving the .py file... I was using the 'save as' command in jupyter notebook and typing 'conv.py' for my file save... This was showing up in the directory as a .py file, but my main file wasn't recognizing it properly. Once I downloaded the file as a .py file, cut from my downloads folder and pasted into my working directory everything worked...
I have a following directory structure:
source
source_1.py
__init__.py
source1.py has class Source defined
source1.py
class Source(object):
pass
I am able to import using this
>>> from source.source1 import Source
>>> Source
<class 'source.source1.Source'>
However when trying to import using the below method it fails.
>>> from source import *
>>> source1.Source
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'source1' is not defined
Please let me know how can we use the 2nd import ?
For importing from a package (unlike importing from a module) you need to specify what * means. To do that, in __init__.py add a line like this:
__all__ = ["source1"]
See the Python documentation for Importing * From a Package.
I'm trying to do a dynamic import (using "__import__()") of a submodule in web2py and it doesn't seem to be finding the submodule.
Here's an example path:
web2py/app/modules/module.py <-- This works
web2py/app/modules/module_folder/submodule.py <-- This won't get spotted.
Right now as a workaround I'm using 'exec()' but I'd rather not do that.
Answers to questions:
"Do you have __init__.py in module_folder?"
Yep.
"What exactly is the line of code you write to import web2py/app/modules/module_folder/submodule.py?"
mapping_module = __import__('communication_templates.mappings.%s' % module_name)
"what is the error?"
<type 'exceptions.AttributeError'> 'module' object has no attribute 'mapping_dict'
Explanation: I'm trying to get a variable named 'mapping_dict' from the module once I've loaded it.
The problem here is that __import__ does not do the most intuitive thing (neither does import btw).
Here's what happens:
>>> import sys
>>> x = __import__('package1.package2.module')
>>> x
<module 'package1' from 'C:\\Temp\\package1\\__init__.py'>
Even though package1.package2.module was imported, the variable returned is actually package1.
So, to access something that is in package1.package2.module, one has to dig down again:
>>> x.package2.module.Class
<class 'package1.package2.module.Class'>
Is it then the same as importing just package1?
Not really:
>>> x = __import__('package1')
>>> x
<module 'package1' from 'C:\\Temp\\package1\\__init__.py'>
>>> x.package2.module.Class
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'package1' has no attribute 'package2'
Why?
Well, __import__ is the same as `import, that is, these are the same:
package1 = __import__('package1.package2.module')
# is the same as:
import package1.package2.module
And:
package1 = __import__('package1')
# is the same as:
import package1
In each of those cases, you get just one local variable package1.
In the case of importing package1.package2.module, package2 is also imported and stored in package2 attribute of package package1 etc.
Solution
To access package1.package2.module.Class, if package1.package2.module is in a string:
>>> s = 'package1.package2.module'
>>> module = __import__(s)
>>> for submodule_name in s.split('.')[1:]:
... module = getattr(module, submodule_name)
...
>>> module.Class
<class 'package1.package2.module.Class'>
I'm trying to use the importlib library to verify whether the nmap library is installed on the computer executing the script in Python 3.5.2
I'm trying to use importlib.util.find_spec("nmap") but receive the following error.
>>> import importlib
>>> importlib.util.find_spec("nmap")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
Can someone tell me where I'm going wrong?
EDIT
I was able to get the function to work using the following code.
#!/usr/bin/pythonw
import importlib
from importlib import util
#check to see if nmap module is installed
find_nmap = util.find_spec("nmap")
if find_nmap is None:
print("Error")
Try this:
from importlib import util
util.find_spec("nmap")
I intend to investigate, but honestly I don't know why one works and the other doesn't. Also, observe the following interactive session:
>>> import importlib
>>> importlib.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
>>> from importlib import util
>>> util
<module 'importlib.util' from '/usr/lib/python3.5/importlib/util.py'>
>>> importlib.util
<module 'importlib.util' from '/usr/lib/python3.5/importlib/util.py'>
So...yeah. I am sure this makes perfect sense to someone, but not to me. I will update once I figure it out.
Update:
Comparing this to something like:
>>> import datetime
>>> datetime
<module 'datetime' from '/usr/lib/python3.5/datetime.py'>
>>> datetime.datetime
<class 'datetime.datetime'>
I think the difference is that in this case the first datetime is a module and the second is a class, while in the importlib.util case both are modules. So perhaps module.module is not OK unless the code from both modules has been loaded, while module.class is OK, because the class code is loaded when the module is imported.
Update #2
Nope, it seems like in many cases module.module is fine. For example:
>>> import urllib
>>> urllib
<module 'urllib' from '/usr/lib/python3.5/urllib/__init__.py'>
>>> urllib.error
<module 'urllib.error' from '/usr/lib/python3.5/urllib/error.py'>
So perhaps it is something specific to importlib.
Update #3
As #kfb pointed out in the comments, it does seem to be related to importlib specifically. See the following comment from the __init__.py for importlib:
# Until bootstrapping is complete, DO NOT import any modules that attempt
# to import importlib._bootstrap (directly or indirectly). Since this
# partially initialised package would be present in sys.modules, those
# modules would get an uninitialised copy of the source version, instead
# of a fully initialised version (either the frozen one or the one
# initialised below if the frozen one is not available).
importlib/util.py does import importlib._bootstrap so I would assume that this is realted. If my understanding is correct, when you do import importlib the submodules will be initialized, but are not initialized for the importlib module object that you have imported. At this point, if you do dir(importlib) you will not see util. Interestingly, after you have tried to access importlib.util and gotten an AttributeError, util (along with the other submodules) gets loaded/initialized, and now you can access importlib.util!
>>> import importlib
>>> dir(importlib)
['_RELOADING', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__import__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_bootstrap', '_bootstrap_external', '_imp', '_r_long', '_w_long', 'find_loader', 'import_module', 'invalidate_caches', 'reload', 'sys', 'types', 'warnings']
>>> importlib.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
>>> importlib.util
<module 'importlib.util' from '/usr/lib/python3.5/importlib/util.py'>
>>> dir(importlib)
['_RELOADING', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__import__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_bootstrap', '_bootstrap_external', '_imp', '_r_long', '_w_long', 'abc', 'find_loader', 'import_module', 'invalidate_caches', 'machinery', 'reload', 'sys', 'types', 'util', 'warnings']
It states in the Python documentation that pickle is not secure and shouldn't parse untrusted user input. If you research this; almost all examples demonstrate this with a system() call via os.system.
Whats not clear to me, is how os.system is interpreted correctly without the os module being imported.
>>> import pickle
>>> pickle.loads("cos\nsystem\n(S'ls /'\ntR.") # This clearly works.
bin boot cgroup dev etc home lib lib64 lost+found media mnt opt proc root run sbin selinux srv sys tmp usr var
0
>>> dir() # no os module
['__builtins__', '__doc__', '__name__', '__package__', 'pickle']
>>> os.system('ls /')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'os' is not defined
>>>
Can someone explain?
The name of the module (os) is part of the opcode, and pickle automatically imports the module:
# pickle.py
def find_class(self, module, name):
# Subclasses may override this
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
return klass
Note the __import__(module) line.
The function is called when the GLOBAL 'os system' pickle bytecode instruction is executed.
This mechanism is necessary in order to be able to unpickle instances of classes whose modules haven't been explicitly imported into the caller's namespace.
For altogether too much information on writing malicious Pickles that go much further than the standard os.system() example, see this presentation and its accompanying paper.
If you use pickletools.dis to disassemble the pickle you can see how this is working:
import pickletools
print pickletools.dis("cos\nsystem\n(S'ls ~'\ntR.")
Output:
0: c GLOBAL 'os system'
11: ( MARK
12: S STRING 'ls ~'
20: t TUPLE (MARK at 11)
21: R REDUCE
22: . STOP
Pickle uses a simple stack-based virtual machine that records the instructions used to reconstruct the object. In other words the pickled instructions in your example are:
Push self.find_class(module_name, class_name) i.e. push os.system
Push the string 'ls ~'
Build tuple from topmost stack items
Apply callable to argtuple, both on stack. i.e. os.system(*('ls ~',))
Source
Importing a module only adds it to the local namespace, which is not necessarily the one you're in. Except when it doesn't:
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__']
>>> __import__('os')
<module 'os' from '/usr/lib64/python2.7/os.pyc'>
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__']