I am working in Jupyter notebook. I created a simple module called conv.py for converting miles to km. When I try to import this module in a separate code (in the same directory) the import seems to go successfully but it doesn't recognize either of the functions I defined in the 'conv' module.
I have imported os and os.getcwd() provides the correct folder for conv.py...
code for conv.py
in_n_ft = 12
ft_n_mile = 5280
m_n_km = 1000
cm_n_in = 2.54
cm_n_m = 100
mm_n_m = 1000
def ft_to_km(feet):
return feet*in_n_ft*cm_n_in/cm_n_m/m_n_km
print(ft_to_km(5280))
def mil_to_km(mile):
return mile*ft_n_mile*in_n_ft*cm_n_in/cm_n_m/m_n_km
print(mil_to_km(3.2))
Code for new module
import conv
km = conv.mil_to_km(5)
Error provided
AttributeError Traceback (most recent call last)
<ipython-input-111-bfd778724ae2> in <module>
3 import conv
4
----> 5 km = conv.mil_to_km(5)
AttributeError: module 'conv' has no attribute 'mil_to_km'
When I type
dir(conv)
I get
['__builtins__',
'__cached__',
'__doc__',
'__file__',
'__loader__',
'__name__',
'__package__',
'__spec__']
What am I doing wrong?
EDIT
I have also tried
from conv import mil_to_km
when I do that I get a different error
cannot import name 'mil_to_km' from 'conv' (C:\Users\223023441\Documents\python\conv.py)
I have also queried the module using:
from inspect import getmembers, isfunction
import conv
print(getmembers(conv, isfunction))
from here I get:
['__builtins__',
'__cached__',
'__doc__',
'__file__',
'__loader__',
'__name__',
'__package__',
'__spec__']
I am also unable to access any of the variables within the conv.py file after import... Am I doing something wrong when I save the py file? Jupyter makes ipynb as the common file, when I 'save as' to conv.py, is this screwing it up?
You should import from the module.
Try this:
from conv import mil_to_km
km = mil_to_km(5)
The reason is that when you import the module in that way, you are executing it.
In the way I shown, you are just importing the needed functions.
So the ultimate issue was the way I was saving the .py file... I was using the 'save as' command in jupyter notebook and typing 'conv.py' for my file save... This was showing up in the directory as a .py file, but my main file wasn't recognizing it properly. Once I downloaded the file as a .py file, cut from my downloads folder and pasted into my working directory everything worked...
Related
i want to implement sort of plugin architecture that dynamically loads modules and calls a function from them
for instance plugin code looks like (in file "foo_func.py")
foo_local = []
def foo_add(a: int, b: int) -> int:
c = a + b
foo_local.append(c)
return c
def foo_print():
print(foo_local)
i need to support two plugins with the same code but with different memory state, so i created directory structure like this:
<ROOT_PROJECT>
app.py
bar/
apple/
foo/
foo_func.py
__init__.py
orange/
foo/
foo_func.py
__init__.py
code in "apple" and "orange" folders is the same.
then in app file i try to load modules and invoke functions from them
import importlib
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
apple = importlib.import_module('bar.apple.foo')
orange = importlib.import_module('bar.orange.foo')
apple_foo = getattr(apple, 'foo_func')
orange_foo = getattr(orange, 'foo_func')
apple_foo_add_my = getattr(apple_foo, 'foo_add')
apple_foo_print_my = getattr(apple_foo, 'foo_print')
apple_foo_add_my(1, 2)
apple_foo_print_my()
and this works fine, but you see these import lines at the top
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
they are not used in code (even pycharm complains about it)
but if i try to comment code and run it - then failure
AttributeError: module 'bar.apple.foo' has no attribute 'foo_func'
why ?
I suppose normal plugins should deal only with "importlib.import_module" and "getattr" and it must be enough ?
what is wrong here ?
Let's switch completely to direct imports for this explanation, because:
import something.whatever as name
is the same as:
name = importlib.import_module("something.whatever")
So let's rewrite your apple code:
apple = importlib.import_module('bar.apple.foo')
apple_foo = getattr(apple, 'foo_func')
will become:
import bar.apple.foo as apple
apple_foo = apple.foo_func
Now, the first line loads bar.apple.foo as a module. In case of packages, this means importing package's __init__.py code. And treating it as a module itself.
And what's the code in the package's init? Usually nothing! That's why the name lookup fails.
However, when you do any import my_package.whatever, the package gets its insides checked and the name becomes visible. You're basically pre-loading the module for interpreter to look at.
Why is pycharm giving you not used suggestion? Because it's not used as a variable anywhere. You're only using a side-effect + pycharm doesn't analyze strings for imports or attributes.
Visual example, with a part of standard library:
>>> import xml
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> xml.etree
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'xml' has no attribute 'etree'
>>>
>>> import xml.etree
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'etree']
And another example, what happens if there are multiple modules in the package:
>>> import dateutil
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_version']
but:
>>> import dateutil.parser
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_common', '_version', 'parser', 'relativedelta', 'tz']
All sub-modules are now visible and usable with their qualified name.
tl;dr: import my_package == only my_package/__init__.py is looked at. import my_package.whatever == python now knows it's a package and registers its insides, all modules of my_package are visible and usable.
I'm currently trying to import Python files downloaded from external sources into my code. This has worked perfectly fine within the Python shell, but when using the exact same code within an IDLE file, Python is unable to find the file in the given directory, even though it does find the directory itself.
My aim is to import the Python Natural Language Toolkit, which requires the module 'six'. So I've imported first 'six', then NLTK into a shell.
I've then tried to repeat the same code within script mode.
INTERACTIVE MODE
>>> import os
>>> path = "C:/Users/henri/AppData/Local/Programs/Python/Python37-32/lib/site-packages/pip/_vendor/urllib3/packages"
>>> os.chdir(path)
>>> os.getcwd()
Result:
'C:\\Users\\henri\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages\\pip\\_vendor\\urllib3\\packages' <br/>
>>> import six
>>> dir(six)
Result:
['BytesIO', 'Iterator', 'MAXSIZE', 'Module_six_moves_urllib', 'Module_six_moves_urllib_error', 'Module_six_moves_urllib_parse', 'Module_six_moves_urllib_request', 'Module_six_moves_urllib_response', 'Module_six_moves_urllib_robotparser', 'MovedAttribute', 'MovedModule', 'PY2', 'PY3', 'PY34', 'StringIO', '_LazyDescr', '_LazyModule', '_MovedItems', '_SixMetaPathImporter', '__author__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_add_doc', '_assertCountEqual', '_assertRaisesRegex', '_assertRegex', '_func_closure', '_func_code', '_func_defaults', '_func_globals', '_import_module', '_importer', '_meth_func', '_meth_self', '_moved_attributes', '_urllib_error_moved_attributes', '_urllib_parse_moved_attributes', '_urllib_request_moved_attributes', '_urllib_response_moved_attributes', '_urllib_robotparser_moved_attributes', 'absolute_import', 'add_metaclass', 'add_move', 'advance_iterator', 'assertCountEqual', 'assertRaisesRegex', 'assertRegex', 'b', 'binary_type', 'byte2int', 'callable', 'class_types', 'create_bound_method', 'create_unbound_method', 'exec_', 'functools', 'get_function_closure', 'get_function_code', 'get_function_defaults', 'get_function_globals', 'get_method_function', 'get_method_self', 'get_unbound_function', 'indexbytes', 'int2byte', 'integer_types', 'io', 'iterbytes', 'iteritems', 'iterkeys', 'iterlists', 'itertools', 'itervalues', 'moves', 'next', 'operator', 'print_', 'python_2_unicode_compatible', 'raise_from', 'remove_move', 'reraise', 'string_types', 'sys', 'text_type', 'types', 'u', 'unichr', 'viewitems', 'viewkeys', 'viewvalues', 'with_metaclass', 'wraps']
SCRIPT MODE
import os
# importing six
path="C:/Users/henri/AppData/Local/Programs/Python/Python37-32/lib/site-packages/pip/_vendor/urllib3/packages"
os.chdir(path)
os.getcwd()
print(os.getcwd())
import six
dir(six)
And this is the resulting error message when trying to run this code:
RESTART: C:\Users\henri\AppData\Local\Programs\Python\Python37-32\importing_nltk_file_2.py
C:\Users\henri\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pip_vendor\urllib3\packages
Traceback (most recent call last):
File "C:\Users\henri\AppData\Local\Programs\Python\Python37-32\importing_nltk_file_2.py", line 13, in
import six
ModuleNotFoundError: No module named 'six'
So, does anyone have an idea why Python can't find 'six', although the directory is unmistakeably correct?
when you run a python script all import statements are run first and then other parts of your code are executed. So, import six is executed before os.chdir() and hence, the error
I'm trying to use the importlib library to verify whether the nmap library is installed on the computer executing the script in Python 3.5.2
I'm trying to use importlib.util.find_spec("nmap") but receive the following error.
>>> import importlib
>>> importlib.util.find_spec("nmap")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
Can someone tell me where I'm going wrong?
EDIT
I was able to get the function to work using the following code.
#!/usr/bin/pythonw
import importlib
from importlib import util
#check to see if nmap module is installed
find_nmap = util.find_spec("nmap")
if find_nmap is None:
print("Error")
Try this:
from importlib import util
util.find_spec("nmap")
I intend to investigate, but honestly I don't know why one works and the other doesn't. Also, observe the following interactive session:
>>> import importlib
>>> importlib.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
>>> from importlib import util
>>> util
<module 'importlib.util' from '/usr/lib/python3.5/importlib/util.py'>
>>> importlib.util
<module 'importlib.util' from '/usr/lib/python3.5/importlib/util.py'>
So...yeah. I am sure this makes perfect sense to someone, but not to me. I will update once I figure it out.
Update:
Comparing this to something like:
>>> import datetime
>>> datetime
<module 'datetime' from '/usr/lib/python3.5/datetime.py'>
>>> datetime.datetime
<class 'datetime.datetime'>
I think the difference is that in this case the first datetime is a module and the second is a class, while in the importlib.util case both are modules. So perhaps module.module is not OK unless the code from both modules has been loaded, while module.class is OK, because the class code is loaded when the module is imported.
Update #2
Nope, it seems like in many cases module.module is fine. For example:
>>> import urllib
>>> urllib
<module 'urllib' from '/usr/lib/python3.5/urllib/__init__.py'>
>>> urllib.error
<module 'urllib.error' from '/usr/lib/python3.5/urllib/error.py'>
So perhaps it is something specific to importlib.
Update #3
As #kfb pointed out in the comments, it does seem to be related to importlib specifically. See the following comment from the __init__.py for importlib:
# Until bootstrapping is complete, DO NOT import any modules that attempt
# to import importlib._bootstrap (directly or indirectly). Since this
# partially initialised package would be present in sys.modules, those
# modules would get an uninitialised copy of the source version, instead
# of a fully initialised version (either the frozen one or the one
# initialised below if the frozen one is not available).
importlib/util.py does import importlib._bootstrap so I would assume that this is realted. If my understanding is correct, when you do import importlib the submodules will be initialized, but are not initialized for the importlib module object that you have imported. At this point, if you do dir(importlib) you will not see util. Interestingly, after you have tried to access importlib.util and gotten an AttributeError, util (along with the other submodules) gets loaded/initialized, and now you can access importlib.util!
>>> import importlib
>>> dir(importlib)
['_RELOADING', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__import__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_bootstrap', '_bootstrap_external', '_imp', '_r_long', '_w_long', 'find_loader', 'import_module', 'invalidate_caches', 'reload', 'sys', 'types', 'warnings']
>>> importlib.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
>>> importlib.util
<module 'importlib.util' from '/usr/lib/python3.5/importlib/util.py'>
>>> dir(importlib)
['_RELOADING', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__import__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_bootstrap', '_bootstrap_external', '_imp', '_r_long', '_w_long', 'abc', 'find_loader', 'import_module', 'invalidate_caches', 'machinery', 'reload', 'sys', 'types', 'util', 'warnings']
I am not a professional programmer, and I'm just starting to study python.
I just figured out that just because I could do:
>>> from xml.etree.ElementTree import Element
>>> var = Element("Something")
doesn't mean I can do:
>>> import xml
>>> var = xml.etree.ElementTree.Element("Something")
In fact, doing this:
>>> import xml
>>> dir(xml)
['_MINIMUM_XMLPLUS_VERSION', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
etree doesn't even appear as one of the methods provided by xml.
What is the relation between xml and xml.etree.ElementTree?
Why can I not see etree as one of xml's methods?
xml.etree is a child module to xml package. It only shows up in xml's namespace after you import xml.etree. A parent may import its child modules on its initialization (e.g. in 2.7, os imports os.path1) but it's not required to do so.
Conversely, when you import a module from a package directly, the package is automatically imported first.
Strangely, I couldn't find any phrasing in the docs stating this. But a test shows that this is exactly what happens:
$ cat test/__init__.py
print "package init"
import traceback
traceback.print_stack()
$ cat test/module.py
print "module init"
import traceback
traceback.print_stack()
$ python
<...>
>>> import test.module
package init
File "<stdin>", line 1, in <module>
File "test\__init__.py", line 3, in <module>
traceback.print_stack()
module init
File "<stdin>", line 1, in <module>
File "test\module.py", line 3, in <module>
traceback.print_stack()
1this is not documented though so don't rely on this
It states in the Python documentation that pickle is not secure and shouldn't parse untrusted user input. If you research this; almost all examples demonstrate this with a system() call via os.system.
Whats not clear to me, is how os.system is interpreted correctly without the os module being imported.
>>> import pickle
>>> pickle.loads("cos\nsystem\n(S'ls /'\ntR.") # This clearly works.
bin boot cgroup dev etc home lib lib64 lost+found media mnt opt proc root run sbin selinux srv sys tmp usr var
0
>>> dir() # no os module
['__builtins__', '__doc__', '__name__', '__package__', 'pickle']
>>> os.system('ls /')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'os' is not defined
>>>
Can someone explain?
The name of the module (os) is part of the opcode, and pickle automatically imports the module:
# pickle.py
def find_class(self, module, name):
# Subclasses may override this
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
return klass
Note the __import__(module) line.
The function is called when the GLOBAL 'os system' pickle bytecode instruction is executed.
This mechanism is necessary in order to be able to unpickle instances of classes whose modules haven't been explicitly imported into the caller's namespace.
For altogether too much information on writing malicious Pickles that go much further than the standard os.system() example, see this presentation and its accompanying paper.
If you use pickletools.dis to disassemble the pickle you can see how this is working:
import pickletools
print pickletools.dis("cos\nsystem\n(S'ls ~'\ntR.")
Output:
0: c GLOBAL 'os system'
11: ( MARK
12: S STRING 'ls ~'
20: t TUPLE (MARK at 11)
21: R REDUCE
22: . STOP
Pickle uses a simple stack-based virtual machine that records the instructions used to reconstruct the object. In other words the pickled instructions in your example are:
Push self.find_class(module_name, class_name) i.e. push os.system
Push the string 'ls ~'
Build tuple from topmost stack items
Apply callable to argtuple, both on stack. i.e. os.system(*('ls ~',))
Source
Importing a module only adds it to the local namespace, which is not necessarily the one you're in. Except when it doesn't:
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__']
>>> __import__('os')
<module 'os' from '/usr/lib64/python2.7/os.pyc'>
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__']