everyone,I am new to PP but got stuck in a problem when importing NumPy with PP.
Basically what I tried to do was submitting a function to the ppserver which depends on NumPy. I have imported it at the very beginning of the code using (import NumPy as nu) but when I ran the code, it gave me the error that cannot find the shared object multiarray.so.
The situation is exactly the same here: parallel python forum
the code is attached as below: (I am running on python 2.7.2 + pp 1.6.0 + numpy 1.5.1)
import numpy as nu
import pylab as pl
import pp
job_server = pp.Server(secret="123456")
print "Starting pp with", job_server.get_ncpus(), "workers"
aa = GrRib()
job = job_server.submit(aa.plotwavefunc, (band,k),(nu,pl,signal))
result = job()
the error looks like :
An error has occured during the function import
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/ppworker.py", line 86, in run
exec __fobj
File "<string>", line 127, in <module>
ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python intepreter from there.
An error has occured during the function import
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/ppworker.py", line 86, in run
exec __fobj
File "<string>", line 1, in <module>
File "/usr/share/pyshared/matplotlib/__init__.py", line 135, in <module>
from matplotlib.rcsetup import (defaultParams,
File "/usr/share/pyshared/matplotlib/rcsetup.py", line 19, in <module>
from matplotlib.colors import is_color_like
File "/usr/share/pyshared/matplotlib/colors.py", line 52, in <module>
import numpy as np
File "/usr/share/pyshared/numpy/__init__.py", line 136, in <module>
import add_newdocs
File "/usr/share/pyshared/numpy/add_newdocs.py", line 9, in <module>
from numpy.lib import add_newdoc
File "/usr/share/pyshared/numpy/lib/__init__.py", line 4, in <module>
from type_check import *
File "/usr/share/pyshared/numpy/lib/type_check.py", line 8, in <module>
import numpy.core.numeric as _nx
File "/usr/share/pyshared/numpy/core/__init__.py", line 5, in <module>
import multiarray
ImportError: No module named multiarray
Can anyone help me out? I understand it as I will have to change the directory PP is looking for pyshared objects.
I could be wrong, but I believe the way you submit the job is wrong. As seen on the documentation (http://www.parallelpython.com/content/view/15/30/#QUICKSMP), the third argument (not including "self") are dependent functions. Also, the modules have to be strings. I'm assuming the "band" and "k" are dependent functions since I do not see their declarations:
job = job_server.submit(aa.plotwavefunc, depfuncs = (band,k), modules = ("numpy","pylab","signal"))
Related
I have a folder structure like so
root/
A0/
A1/
B0/
B1/
Lets say I have a file called test.py. In it, I import numpy like so
import numpy as np
That is all that is contained in the file. This works without issue when it is located in root, subfolder A0,A1,B0 but raises an error in subfolder B1. Is error raised is as follows:
Traceback (most recent call last):
File "pose/utils/test.py", line 1, in <module>
import numpy as np
File "/home/adrian/.local/lib/python3.6/site-packages/numpy/__init__.py", line 187, in <module>
from .testing import Tester
File "/home/adrian/.local/lib/python3.6/site-packages/numpy/testing/__init__.py", line 10, in <module>
from unittest import TestCase
File "/usr/lib/python3.6/unittest/__init__.py", line 59, in <module>
from .case import (TestCase, FunctionTestCase, SkipTest, skip, skipIf,
File "/usr/lib/python3.6/unittest/case.py", line 6, in <module>
import logging
File "/home/adrian/Projects/lpr-pose-estimation/pose/utils/logging.py", line 3, in <module>
from pose.utils.utils import mkdir_if_missing
File "/home/adrian/Projects/lpr-pose-estimation/pose/utils/utils.py", line 4, in <module>
from scipy.misc import imresize
File "/home/adrian/.local/lib/python3.6/site-packages/scipy/misc/__init__.py", line 68, in <module>
from scipy.interpolate._pade import pade as _pade
File "/home/adrian/.local/lib/python3.6/site-packages/scipy/interpolate/__init__.py", line 175, in <module>
from .interpolate import *
File "/home/adrian/.local/lib/python3.6/site-packages/scipy/interpolate/interpolate.py", line 32, in <module>
from .interpnd import _ndim_coords_from_arrays
File "interpnd.pyx", line 1, in init scipy.interpolate.interpnd
File "/home/adrian/.local/lib/python3.6/site-packages/scipy/spatial/__init__.py", line 98, in <module>
from .kdtree import *
File "/home/adrian/.local/lib/python3.6/site-packages/scipy/spatial/kdtree.py", line 8, in <module>
import scipy.sparse
File "/home/adrian/.local/lib/python3.6/site-packages/scipy/sparse/__init__.py", line 230, in <module>
from .base import *
File "/home/adrian/.local/lib/python3.6/site-packages/scipy/sparse/base.py", line 9, in <module>
from scipy._lib._numpy_compat import broadcast_to
File "/home/adrian/.local/lib/python3.6/site-packages/scipy/_lib/_numpy_compat.py", line 16, in <module>
_assert_warns = np.testing.assert_warns
AttributeError: module 'numpy' has no attribute 'testing'
Importing as modules (e.g cv2) also leads to errors, but only wrt to numpy:
ImportError: numpy.core.multiarray failed to import
Traceback (most recent call last):
File "pose/utils/test.py", line 1, in <module>
import cv2 as cv
File "/home/adrian/.local/lib/python3.6/site-packages/cv2/__init__.py", line 3, in <module>
from .cv2 import *
ImportError: numpy.core.multiarray failed to import
What may be the cause of such an error?
If your start script is located in .../pose/utils then every absolute import looks for modules there, too. This directory contains a module named logging (like the one in the standard library).
During the initialization of the numpy package (executing its __init__.py) and before numpy.testing is available the usual chain of imports happens (as can be seen in the traceback) which leads to the wrong logging module which in turn leads to import of _numpy_compat which tries to access numpy.testing too early.
To avoid this circular import problem you can either rename your logging module or move the start script to another directory.
I upgraded scikit-learn to version 0.20.1 using anaconda3 after getting an error message concerning balanced_accuracy_score from metrics in scikit-learn. This is apparently not available in version 0.19.
After this upgrade i get the following error:
Traceback (most recent call last):
File "test4.py", line 4, in <module>
from sklearn import metrics
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\__init__.py", line 6
4, in <module>
from .base import clone
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\base.py", line 13, i
n <module>
from .utils.fixes import signature
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\utils\__init__.py",
line 14, in <module>
from . import _joblib
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\utils\_joblib.py", l
ine 22, in <module>
from ..externals import joblib
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\externals\joblib\__i
nit__.py", line 119, in <module>
from .parallel import Parallel
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\externals\joblib\par
allel.py", line 28, in <module>
from ._parallel_backends import (FallbackToBackend, MultiprocessingBackend,
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\externals\joblib\_pa
rallel_backends.py", line 22, in <module>
from .executor import get_memmapping_executor
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\externals\joblib\exe
cutor.py", line 14, in <module>
from .externals.loky.reusable_executor import get_reusable_executor
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\externals\joblib\ext
ernals\loky\__init__.py", line 13, in <module>
from .reusable_executor import get_reusable_executor
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\externals\joblib\ext
ernals\loky\reusable_executor.py", line 11, in <module>
from .process_executor import ProcessPoolExecutor, EXTRA_QUEUED_CALLS
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\externals\joblib\ext
ernals\loky\process_executor.py", line 82, in <module>
from .backend.queues import Queue, SimpleQueue, Full
File "C:\Users\Pieter\Anaconda3\lib\site-packages\sklearn\externals\joblib\ext
ernals\loky\backend\queues.py", line 21, in <module>
from multiprocessing.queues import Full
File "C:\Users\Pieter\Anaconda3\lib\multiprocessing\queues.py", line 20, in <m
odule>
from queue import Empty, Full
ImportError: cannot import name 'Empty' from 'queue' (C:\Users\Pieter\Python pro
gs\queue.py)
Are you running this program from "C:\Users\Pieter\Python progs"?
Looks like you have a file named queue.py in "C:\Users\Pieter\Python pro
gs\" which is shadowing the actual module queue in multiprocessing lib. And hence the error.
There are two possible reasons for this:
Maybe this line:
from queue import Empty, Full
has been added in new version, so earlier it did not error even when your custom made queue.py was present.
Maybe you have created your queue.py recently after updating the scikit-learn and the above line was always present. In this case, error should occur on both versions.
Rename your program from queue.py and auto-compiled queue.pyc
to something else and it should work.
I am getting an exception from scipy whenever I try to import the nltk package. The command and the error looks like this:
>>> import nltk
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/nltk/__init__.py", line 114, in <module>
from nltk.collocations import *
File "/usr/local/lib/python2.7/dist-packages/nltk/collocations.py", line 39, in <module>
from nltk.metrics import ContingencyMeasures, BigramAssocMeasures, TrigramAssocMeasures
File "/usr/local/lib/python2.7/dist-packages/nltk/metrics/__init__.py", line 16, in <module>
from nltk.metrics.scores import (accuracy, precision, recall, f_measure,
File "/usr/local/lib/python2.7/dist-packages/nltk/metrics/scores.py", line 16, in <module>
from scipy.stats.stats import betai
File "/usr/local/lib/python2.7/dist-packages/scipy/stats/__init__.py", line 344, in <module>
from .stats import *
File "/usr/local/lib/python2.7/dist-packages/scipy/stats/stats.py", line 176, in <module>
from . import distributions
File "/usr/local/lib/python2.7/dist-packages/scipy/stats/distributions.py", line 10, in <module>
from ._distn_infrastructure import (entropy, rv_discrete, rv_continuous,
File "/usr/local/lib/python2.7/dist-packages/scipy/stats/_distn_infrastructure.py", line 43, in <module>
from new import instancemethod
File "new.py", line 107
return copy.deepcopy(self)
^
IndentationError: expected an indented block
I executed this command in a terminal (ubuntu 16.04 LTS system). I did not find anything about this error and the indentation doesn't look wrong.
The SyntaxError is in your file new.py (I guess it's in your current working directory). Try to switch to a directory where no new.py file is present (or move the new.py file in another directory), then the internal scipy import should work without problems.
On the other hand you can just upgrade to scipy 0.19.x. They removed the import there and it should work without problems.
Note that it's always a bad idea to name python files like builtin-modules! That's a common source for exceptions or unexpected behaviour.
I am using eventlet module. When I try to monkey_patch the code exits with error:
'__init__ called too many times'
Here's the code that's causing problem:
from eventlet import *
patcher.monkey_patch(all=True)
import os, glob, pandas, logging
import numpy as np
import boto3
import datetime
The import works fine, but monkey patching is the step where I see error.
EDIT:
Here's the stack trace:
Traceback (most recent call last):
File "check.py", line 4, in <module>
import os, glob, pandas, logging
File "d:\Python\Anaconda\lib\site-packages\pandas\__init__.py", line 45, in <module>
from pandas.core.api import *
File "d:\Python\Anaconda\lib\site-packages\pandas\core\api.py", line 9, in <module>
from pandas.core.groupby import Grouper
File "d:\Python\Anaconda\lib\site-packages\pandas\core\groupby.py", line 15, in <module>
from pandas.core.frame import DataFrame
File "d:\Python\Anaconda\lib\site-packages\pandas\core\frame.py", line 32, in <module>
from pandas.core.generic import NDFrame, _shared_docs
File "d:\Python\Anaconda\lib\site-packages\pandas\core\generic.py", line 16, in <module>
from pandas.core.internals import BlockManager
File "d:\Python\Anaconda\lib\site-packages\pandas\core\internals.py", line 22, in <module>
from pandas.sparse.array import _maybe_to_sparse, SparseArray
File "d:\Python\Anaconda\lib\site-packages\pandas\sparse\array.py", line 19, in <module>
import pandas.core.ops as ops
File "d:\Python\Anaconda\lib\site-packages\pandas\core\ops.py", line 15, in <module>
import pandas.computation.expressions as expressions
File "d:\Python\Anaconda\lib\site-packages\pandas\computation\expressions.py", line 15, in <module>
import numexpr as ne
File "d:\Python\Anaconda\lib\site-packages\numexpr\__init__.py", line 40, in <module>
from numexpr.expressions import E
File "d:\Python\Anaconda\lib\site-packages\numexpr\expressions.py", line 81, in <module>
_context = Context({})
File "d:\Python\Anaconda\lib\site-packages\numexpr\expressions.py", line 67, in __init__
raise SystemError('__init__ called too many times')
SystemError: __init__ called too many times
Try to run it without all=True:
patcher.monkey_patch()
(you basically have called a class several times)
This is an issue with Windows machines in particular.. patcher.monkey_patch() is working perfectly fine for me in OSX, while the same code returns with -
SystemError: init called too many times for my colleagues using wondows machines ..
I am still searching for a solution for this issue.
So I am having a bit of trouble using scipy. I have been importing data from a .mat file (matlab variables) and up until today it has worked perfectly. I have no idea what changed or when because I have been developing a text - based data format so that I can avoid depending on Matlab for writing my data. Anyways, I am getting a bizzare import error when I import scipy.io
import scipy.io
myData = scipy.io.loadmat('some_data_file')
When I run this I get the following error:
>>> import scipy.io
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "C:\pyzo2014a\lib\site-packages\scipy\io\__init__.py", line 85, in <module>
from .matlab import loadmat, savemat, whosmat, byteordercodes
File "C:\pyzo2014a\lib\site-packages\scipy\io\matlab\__init__.py", line 13, in <module>
from .mio import loadmat, savemat, whosmat
File "C:\pyzo2014a\lib\site-packages\scipy\io\matlab\mio.py", line 13, in <module>
from .mio4 import MatFile4Reader, MatFile4Writer
File "C:\pyzo2014a\lib\site-packages\scipy\io\matlab\mio4.py", line 11, in <module>
import scipy.sparse
File "C:\pyzo2014a\lib\site-packages\scipy\sparse\__init__.py", line 217, in <module>
from .csgraph import cs_graph_components
File "C:\pyzo2014a\lib\site-packages\scipy\sparse\csgraph\__init__.py", line 148, in <module>
from ._shortest_path import shortest_path, floyd_warshall, dijkstra,\
ImportError: No module named 'scipy.sparse.csgraph._shortest_path'
I thought there might be something wrong with my python distribution so I reinstalled it, however the problem persists, and now the installer says it can't find '_shortest_path.py' when it is installing. When I navigate to .../Libs/site-packages/scipy/sparse/csgraph/ I find that indeed there is no module named _shortest_path. I don't understand how this error came about or how anything in my code would change it. Has anyone else come across this problem?
I am using Pyzo 2014 on Windows 7 x64.
Restore shortest_path.pyd in your virust chest
http://www.blendernation.com/2014/06/28/getting-a-virus-warning-with-blender-2-71-heres-why/