How can you raise an exception when you import a module that is less or greater than a given value for its __version__?
There are a lot of different ways you could do it, but I feel like there must be some really simple way that eludes me at the moment. In this case the version number is of the format x.x.x
Python comes with this inbuilt as part of distutils. The module is called distutils.version and is able to compare several different version number formats.
from distutils.version import StrictVersion
print StrictVersion('1.2.2') > StrictVersion('1.2.1')
For way more information than you need, see the documentation:
>>> import distutils.version
>>> help(distutils.version)
If you are talking about modules installed with easy_install, this is what you need
import pkg_resources
pkg_resources.require("TurboGears>=1.0.5")
this will raise an error if the installed module is of a lower version
Traceback (most recent call last):
File "tempplg.py", line 2, in <module>
pkg_resources.require("TurboGears>=1.0.5")
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (TurboGears 1.0.4.4 (/usr/lib/python2.5/site-packages/TurboGears-1.0.4.4-py2.5.egg), Requirement.parse('TurboGears>=1.0.5'))
Like this?
assert tuple(map(int,module.__version__.split("."))) >= (1,2), "Module not version 1.2.x"
This is wordy, but works pretty well.
Also, look into pip, which provides more advanced functionality.
You should be using setuptools:
It allows you to lock the dependancies of an application, so even if multiple versions of an egg or package exist on a system only the right one will ever be used.
This is a better way of working: Rather than fail if the wrong version of a dependancy is present it is better to ensure that the right version is present.
Setuptools provides an installer which guarantees that everything required to run the application is present at install-time. It also gives you the means to select which of the many versions of a package which may be present on your PC is the one that gets loaded when you issue an import statement.
If you know the exact formatting of the version string a plain comparison will work:
>>> "1.2.2" > "1.2.1"
True
This will only work if each part of the version is in the single digits, though:
>>> "1.2.2" > "1.2.10" # Bug!
True
Related
I have an issue with airflow, I definitely (100%) sure that the pandas is installed.
When I use pandas in the same file like
if __name__ == '__main__':print(pd.DataFrame(data, columns))
it works well but in WebUI Airflow there is an error:
Broken DAG: [PWD/aflow_stu/dags/amp_dag.py]
Traceback (most recent call last):
File "PWD/aflow_stu/dags/amp_dag.py", line 4, in <module>
from pipelines import sql_engine, check_last_date, amp, amp_extract
File "PWD/aflow_stu/dags/pipelines.py", line 3, in <module>
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
Don’t pay attention to PWD, just don't wanna post my folders.
The structure of /dags folder below:
dags/amp_dag.py
dags/pipelines.py #<- pandas is here
dags/api_services.py #<- api_methods for extracting data
So, do you guys know how to fix it?
Maybe I need to use another structure, traceback of python doesn’t see this error, why airflow see.
First of all, where do you see this error? Is this:
a. when the tasks executes? Then the error would show in the logs. If this is the case, it's relevant also to know more about your setup and what kind of executor you use as #rozumir already asked.
b. when the DAG parses. In this case, the error would show up at the home page of the the Airflow WebUI.
I am going to assume that it is B.
In this case you do not have pandas available in the Airflow virtual environment. If you are sure it is, I would recommend checking once more with what python paths you are running each. You can use the sys.executable for this.
import sys
raise ValueError(sys.executable)
Place these two lines at the top of your DAG file and make sure they return the same path. Only then you are 100% sure you are using the same virtual environment. If you still see a discrepancy, then also check sys.path.
If both are equal, pandas will for sure be available to both or neither as well.
After I import functools I receive such message from interpreter:
Traceback (most recent call last):
File "C:/Users/Admin/Documents/Python/decorator.py", line 1, in
import functools
File "C:\Python3\lib\functools.py", line 22, in
from types import MappingProxyType
ImportError: cannot import name 'MappingProxyType'
import functools
def trace(func):
def inner(*args, **kwargs):
print(func.__name__, args, kwargs)
return func(*args, **kwargs)
functools.update_wrapper(inner, func)
return inner
#trace
def foo(x):
return x
foo(42)
Using PyCharm as IDE and CPython as interpreter
This is not a real answer, but a comment.
First: I can't verify the problem presented: The code within the question works perfectly in the standard python I am using.
Second: I stumbled upon this question because I had the same error message, but maybe for a different reason. I accidentally named a module "type". Strangely the python framework selected this new "type" module while resolving references from within imported system modules. Which indicates that python's algorithm of resolving references is a bit ... well ... of a bit of very simple design.
Nevertheless if readers get this error message please check first if you accidentally named something "type" somewhere in your code as a first approach to solve the problem before checking other reasons for that error message.
Update: This is a quite common phenomenon in Python. Be careful! Whenever you give your module a name that has already been used for any imported Python module (and you might not be aware of these names!) you will have the chance of getting strange errors more or less similar to the one described in the question.
The accepted solution forces you to rename your module, but:
There's nothing wrong with having a module named types in your package, actually you'll get this error if the interpreter is run with either the current directory or a PYTHONPATH directory is set to the directory containing the module.
If you avoid this situation, you won't have to rename your module.
If you use an IDE, check the run configuration to make sure PYTHONPATH or current directory isn't set to include your modules.
I am new to python and trying to get a feel for python fuse with this tutorial. I installed pythonfuse with pip. I installed os x fuse by downloading a dmg and installing on os x. When I run this line of code from fuse import FUSE, FuseOSError, Operations from the tutorial I see this:
akh2103$ python myfuse.py
Traceback (most recent call last):
File "myfuse.py", line 10, in <module>
from fuse import FUSE, FuseOSError, Operations
ImportError: cannot import name FUSE
It seems like it can't find the fuse package, can't find the python fuse package or can't find the FUSE, FuseOSError and Operations methods within the package. Which one is it? When I type import fuse where does Python go to look for the fuse package? I'm used to class paths in java: is there a python equivalent? I'm very new to python. How do I begin to debug this.
It looks in /Library/Python/<version>/site-packages.
You may be having multiple versions which may be the cause of the problem.
Find out where pip installed fuse.
You can use the PYTHONPATH environment variable to add additional folders.
The fuse module was found (otherwise you would see "No module named fuse"). The error you got means that "FUSE" symbol is not found in the module fuse.
My guess is there are several python bindings for FUSE and you are probably looking at a tutorial for a different module than the one you are loading. The other alternative is some drastic changes in the library between different versions.
If you want to see all the symbols exported by a module, use dir():
import fuse
dir(fuse)
Say this was your directory structure:
myapp/
firstpackage/
__init__.py
firstmodule.py
secondpackage/
__init__.py
secondmodule.py
__init__.py
myfirstapp.py
firstmodule.py
def first_function(data):
return data
def second_function(data):
return data
Now let's say we're working from :mod:`myfirstapp`.
If we wanted to access :func:`first_function`, we'd import like:
from myapp.firstpackage.firstmodule import first_function
print first_function('foo')
:mod:`__init__` in 'firstpackage' directory allows :mod:`firstmodule` to be accessed from outside of it's directory. The inclusion of :file:`__init__.py` within a directory makes that directory a Python package.
However, it's better practice to import the whole module like:
import myapp.firstpackage.firstmodule as firstmodule
print firstmodule.first_function('foo')
print firstmodule.second_function('bar')
Another method would be:
from myapp.firstpackage import firstmodule
print firstmodule.second_function('foo')
That way everything is accessible from within your module, and better for readability.
That being said, the :exc:`ImportError` you're receiving is because 'FUSE' does not exist in :mod:`fuse`, whether it's data, class or a function.
Open fuse.py and do a search for 'FUSE' and see if anything comes up. Look for:
def FUSE(): ...
class FUSE(..): ...
FUSE = ...
I know the whole package/module lesson was off topic from your question, but you said you were new, so I thought I'd elaborate :P
I'm dealing with a Python library that does not define the __version__ variable (sqlalchemy-migrate), and I want to have different behavior in my code based on what version of the library I have installed.
Is there a way to check at runtime what version of the library is installed (other than, say, checking the output of pip freeze)?
This being Python, the accepted way of doing this is generally to call something in the library that behaves differently depending on the version you have installed, something like:
import somelibrary
try:
somelibrary.this_only_exists_in_11()
SOME_LIBRARY_VERSION = 1.1
except AttributeError:
SOME_LIBRARY_VERSION = 1.0
A more elegant way might be to create wrapper functions.
def call_11_feature():
try:
somelibrary.this_only_exists_in_11()
except AttributeError:
somelibrary.some_convoluted_methods()
somelibrary.which_mimic()
somelibrary.the_11_feature()
pkg_resources may help, but you'll need to use the package name:
>>> import pkg_resources
>>> env = pkg_resources.Environment()
>>> env['sqlalchemy-migrate'][0].version
'0.6.2.dev'
If the library doesn't know its own version, then you are basically SOL. However, if one of the versions you want to support would raise an exception if the code went down the "wrong" path, you could use a try/except block.
Occasionally you can evaluate the path of the library and it will be in there somewhere... /usr/lib/python2.6/site-packages/XlsXcessive-0.1.6-py2.6.egg
Today I tried using pyPdf 1.12 in a script I was writing that targets Python 2.6. When running my script, and even importing pyPdf, I get complaints about deprecated functionality (md5->hashsum, sets). I'd like to contribute a patch to make this work cleanly in 2.6, but I imagine the author does not want to break compatibility for older versions (2.5 and earlier).
Searching Google and Stack Overflow have so far turned up nothing. I feel like I have seen try/except blocks around import statements before that accomplish something similar, but can't find any examples. Is there a generally accepted best practice for supporting multiple Python versions?
There are two ways to do this:
(1) Just like you described: Try something and work around the exception for old versions. For example, you could try to import the json module and import a userland implementation if this fails:
try:
import json
except ImportError:
import myutils.myjson as json
This is an example from Django (they use this technique often):
try:
reversed
except NameError:
from django.utils.itercompat import reversed # Python 2.3 fallback
If the iterator reversed is available, they use it. Otherwise, they import their own implementation from the utils package.
(2) Explicitely compare the version of the Python interpreter:
import sys
if sys.version_info < (2, 6, 0):
# Do stuff for old version...
else:
# Do 2.6+ stuff
sys.version_info is a tuple that can easily be compared with similar version tuples.
You can certainly do
try:
import v26
except ImportError:
import v25
Dive Into Python—Using Exceptions for Other Purposes
Multiple versions of Python are supported here. You can a) conditionally use the newer version, which takes a little work, or b) turn off the warnings, which should really be the default (and is on newer Pythons).