Trying to use pyarrow to access hdfs file and not able to get it working, below is the code, thank you very much in advance.
[rxie#cedgedev03 code]$ python
Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:42:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
import pyarrow
import os
os.environ["JAVA_HOME"]="/usr/java/jdk1.8.0_121"
from pyarrow import hdfs
fs = hdfs.connect()
Traceback (most recent call last): File "", line 1, in File
"/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/pyarrow/hdfs.py",
line 183, in connect extra_conf=extra_conf) File
"/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/pyarrow/hdfs.py",
line 37, in init self._connect(host, port, user, kerb_ticket, driver,
extra_conf) File "pyarrow/io-hdfs.pxi", line 89, in
pyarrow.lib.HadoopFileSystem._connect File "pyarrow/error.pxi", line
83, in pyarrow.lib.check_status pyarrow.lib.ArrowIOError: Unable to
load libhdfs
You might need to manually locate this file and specify it with the ARROW_LIBHDFS_DIR environmental variable.
Find the file using locate -l 1 libhdfs.so. In my case, the file is located under /opt/mapr/hadoop/hadoop-0.20.2/c++/Linux-amd64-64/lib.
Then, restart your Python REPL with the environment variable ARROW_LIBHDFS_DIR set to this path. In my case, my command looks like:
ARROW_LIBHDFS_DIR=/opt/mapr/hadoop/hadoop-0.20.2/c++/Linux-amd64-64/lib python
This should solve this particular problem.
(Inspired by https://gist.github.com/priancho/357022fbe63fae8b097a563e43dd885b)
Related
I saw someone was answering the same question here: Python for Naoqi (dynamic module not initialized properly), but I follow the answers in the page, still cannot get it working. Could someone advice how to proceed?
Using Python 2.7.10 doesn't help.
Under Anaconda Python 2.7.10:
Python 2.7.10 |Continuum Analytics, Inc.| (default, Oct 19 2015, 18:31:17)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import naoqi
Traceback (most recent call last):
File "/opt/anaconda3/envs/pepper_test2/lib/python2.7/site.py", line 557, in <module>
main()
File "/opt/anaconda3/envs/pepper_test2/lib/python2.7/site.py", line 539, in main
known_paths = addusersitepackages(known_paths)
File "/opt/anaconda3/envs/pepper_test2/lib/python2.7/site.py", line 275, in addusersitepackages
user_site = getusersitepackages()
File "/opt/anaconda3/envs/pepper_test2/lib/python2.7/site.py", line 250, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/opt/anaconda3/envs/pepper_test2/lib/python2.7/site.py", line 240, in getuserbase
USER_BASE = get_config_var('userbase')
File "/opt/anaconda3/envs/pepper_test2/lib/python2.7/sysconfig.py", line 520, in get_config_var
return get_config_vars().get(name)
File "/opt/anaconda3/envs/pepper_test2/lib/python2.7/sysconfig.py", line 453, in get_config_vars
import re
File "/opt/anaconda3/envs/pepper_test2/lib/python2.7/re.py", line 108, in <module>
import _locale
SystemError: dynamic module not initialized properly
Under Mac System Python 2.7.10:
Python 2.7.10 (default, Oct 6 2017, 22:29:07)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import naoqi
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/cchan/Documents/technical/python_sdk/pynaoqi-python2.7-2.5.5.5-mac64/lib/python2.7/site-packages/naoqi.py", line 9, in <module>
import qi
File "/Users/cchan/Documents/technical/python_sdk/pynaoqi-python2.7-2.5.5.5-mac64/lib/python2.7/site-packages/qi/__init__.py", line 88, in <module>
from _qi import Application as _Application
ImportError: dlopen(/Users/cchan/Documents/technical/python_sdk/pynaoqi-python2.7-2.5.5.5-mac64/lib/python2.7/site-packages/_qi.so, 2): Library not loaded: /Library/Frameworks/Python.framework/Versions/2.7/Python
Referenced from: /Users/cchan/Documents/technical/python_sdk/pynaoqi-python2.7-2.5.5.5-mac64/lib/python2.7/site-packages/_qi.so
Reason: image not found
I have also added the PYTHONPATH in my ~/.bash_profile:
export PYTHONPATH="$PYTHONPATH:/Users/cchan/Documents/technical/python_sdk/pynaoqi-python2.7-2.5.5.5-mac64/lib/python2.7/site-packages"
export DYLD_LIBRARY_PATH="$DYLD_LIBRARY_PATH:/Users/cchan/Documents/technical/python_sdk/pynaoqi-python2.7-2.5.5.5-mac64/lib"
export DYLD_FRAMEWORK_PATH="$DYLD_FRAMEWORK_PATH:/Users/cchan/Documents/technical/python_sdk/pynaoqi-python2.7-2.5.5.5-mac64"
Could someone advise what to do? Thanks in advance!
I found out the problem. The system Python 2.7.10 came with the Mac OS X doesn't work. I have to download the Python 2.7.10 from the Python website and install that in order to have it work: Python Release Website
This answer was original posted by Calvin in question see revision 2
I've a problem with importing the PdfPages function from the matplotlib package on my anaconda environment. I get the following error:
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from matplotlib.backends.backend_pdf import PdfPages
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/hpc/capn/mppi033h/.virtualenv/h5_to_histo_env/lib/python2.7/site-packages/matplotlib/backends/backend_pdf.py", line 32, in <module>
from matplotlib.backend_bases import (RendererBase, GraphicsContextBase,
File "/home/hpc/capn/mppi033h/.virtualenv/h5_to_histo_env/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 63, in <module>
import matplotlib.textpath as textpath
File "/home/hpc/capn/mppi033h/.virtualenv/h5_to_histo_env/lib/python2.7/site-packages/matplotlib/textpath.py", line 17, in <module>
import matplotlib.font_manager as font_manager
File "/home/hpc/capn/mppi033h/.virtualenv/h5_to_histo_env/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1439, in <module>
_rebuild()
File "/home/hpc/capn/mppi033h/.virtualenv/h5_to_histo_env/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1421, in _rebuild
with cbook.Locked(cachedir):
File "/home/hpc/capn/mppi033h/.virtualenv/h5_to_histo_env/lib/python2.7/site-packages/matplotlib/cbook.py", line 2738, in __enter__
files = glob.glob(self.pattern)
AttributeError: 'module' object has no attribute 'glob'
Importing matplotlib (import matplotlib) works fine though. I'm running python in a virtual environment created with Anaconda (/.../.virtualenv/h5_to_histo_env/)
According to conda-V my version is 'conda 4.3.16' and my matplotlib version is 2.0.2 ('np112py27_0').
My Anaconda environment is up to date as well, does anybody know why I get this error?
Update: Reinstalling the virtual anaconda environment and specifying the needed packages at the beginning with the conda create command (and not after the virtual env is already created) fixed it somehow!
I have recently upgraded the system to 16.04 Gnome. The most troubling thing that I am facing is that I cannot import a NAOqi library for my work. The python version of this library was pretty simple to set-up. One just has to untar the file and then enter a path variable called PYTHONPATH pointing to this library and it worked like a charm in 14.04. Now since upgrade I am facing:
Python 2.7.12 (default, Jul 1 2016, 15:12:24)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import naoqi
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/dell/nao_sdk/pynaoqi/naoqi.py", line 7, in <module>
import qi
File "/home/dell/nao_sdk/pynaoqi/qi/__init__.py", line 72, in <module>
from _qi import Application as _Application
ImportError: libqipython.so: cannot open shared object file: No such file or directory
If I add a path variable:
export LD_LIBRARY_PATH=:/home/dell/nao_sdk/pynaoqi/
The error changes to:
Python 2.7.12 (default, Jul 1 2016, 15:12:24)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import naoqi
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/dell/nao_sdk/pynaoqi/naoqi.py", line 7, in <module>
import qi
File "/home/dell/nao_sdk/pynaoqi/qi/__init__.py", line 72, in <module>
from _qi import Application as _Application
ImportError: libboost_regex.so.1.55.0: cannot open shared object file: No such file or directory
Please help me what should I do to get it working? I have also used python 2.6.9 but same error occurs with error below.
ImportError: libboost_python.so.1.55.0: cannot open shared object file: No such file or directory
Installing libboost1.55 did the trick. 16.04 comes with libboost1.58 but naoqi is not yet compatible with it. Manual installation of libboost1.55 solved the import error.
I have gphoto installed properly and I can use all the features with my camera, but for some reason
gphoto = ctypes.CDLL('libgphoto2.dylib')
Throws an error.
$ python
Python 2.7.5 (default, Aug 25 2013, 00:04:04)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> gphoto = ctypes.CDLL('libgphoto2.dylib')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(libgphoto2.dylib, 6): image not found
>>>
I installed libgphoto2 using macports, and like I said gphoto works fine:
$ gphoto2 --capture-image-and-download
New file is in location /capt0000.nef on the camera
Saving file as capt0000.nef
Deleting file /capt0000.nef on the camera
Deleting 'capt0000.nef' from folder '/'...
So I don't understand why gphoto2 can use libgphoto2 but ctypes can't. Any suggestions? I'm very new to this so it's probably a really basic, stupid mistake.
I have Redhawk 1.9 loaded on a 32-bit CentOS 5 virtual machine. I am trying to run the Redhawk HelloWorld component described here: http://redhawksdr.github.io/Documentation/mainch3.html. I am able to launch and start the component in the eclipse sandbox. I cannot run it from Python though. I get the following error.
Python 2.7.2 (default, Feb 27 2012, 16:40:29)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ossie.utils import sb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/redhawk/core/lib/python/ossie/utils/sb/__init__.py", line 115, in <module>
from domainless import *
File "/usr/local/redhawk/core/lib/python/ossie/utils/sb/domainless.py", line 102, in <module>
from omniORB import CORBA, any
ImportError: No module named omniORB
>>>
Any ideas why it will not work?
You'll probably have to set your PYTHONPATH manually to include the appropriate directory. It looks like Eclipse is doing that for you.
Try running your example as root. If you can successfully run as root, then you have a permission problem on your directories or files. How do you identify the files that have the incorrect permission ? I ran the following command,
strace -o test.out python -c "from ossie.utils import sb"
This command will write the output to test.out. Look for the string "denied", which will identify the file that has the incorrect file permission.