My situation:
need to upload solution .zip file to some server where my solution.py file gets executed. However, I need to use scipy method from package spatial, which is not available on server.
Simple attach whole scipy lib ? - No, I can't beacuse it won't fit in .zip file size regulations, however, if I upload only spatial package from scipy (not using any other) everything would fit (size).
Is it possible ?
Because when pasted simply spatial package into my project folder (files from github repo) it didn't work because of errors like 'Couldn't find foofile' mainly in import instructions.
Thanks, any help appreciated
To see one reason why this does not work, you can do from the (i)python commandline
from scipy import spatial
import inspect
inspect.getsourcelines(spatial)
Under the comment text, you can see several imports. If you also look at the first of those,
inspect.getsourcelines(spatial.kdtree)
there is another import line
import scipy.sparse
This is just the first example I found, there are surely many more dependencies and scipy.spatial won't work as is if you don't have access to them. Note also that numpy is imported, and if your server does not have scipy, it probably does not have numpy either.
My suggestion is to either copy/rewrite the scipy code you need (if this is allowed), write your code without scipy or ask for the server admin to either allow larger files or install scipy.
Related
I am trying to learn the linearmodels package for python.
I want to do this by practicing with the data sets, as can be seen here.
Example code:
import numpy as np
from linearmodels.iv import IV2SLS
from linearmodels.datasets import mroz
data = mroz.load()
But my code breaks when i run data = mroz.load()
error message:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\...\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\linearmodels\\datasets\\mroz\\mroz.csv.bz2'
I have pip version: 19.1.1
Conda can't find the package at all
and i have the latest version of linearmodels package: 4.13
The folder specified in the error message i can find, i.e. datasets\mroz but not the csv.bz2 file.
The same holds for every other data set i try to open.
Why am i not able to open the datasets?
let me know if you need additional information.
This is a bug in the package. If you download and unpack the source distribution you would find it lacks all *.csv.bz2.
I see two problems in the package. First, MANIFEST.in lists *.csv.bz. It must be *.csv.bz2 or *.csv.bz*.
Second, they tried to add the datasets in setup.py but also failed, not sure why. Perhaps the files must be declared as belonged to different subpackages, not to the main package.
Please report the bugs to the issue tracker.
I'm working on a lambda function that partly depends on numpy. I created a deployment package zip with a test script that imports numpy and then tries to subtract two numbers using np.subtract since it was having trouble finding the numpy functions.
import numpy as np
a = np.subtract(4,2)
print(a)
I'm working with a python3.5 virtualenv on a linux EC2 instance. To create the deployment zip, I installed numpy, placed my script into site-packages and zipped the contents of the folder as described here. I can create the lambda function from the zip file with no problem, but when I trigger it, it gives the error:
module initialization error: module 'numpy' has no attribute 'subtract'
It appears to import numpy, but it can't find any of the functions. I assume I packaged the libraries/script wrong, but I thought I was following the directions correctly. Any help would be appreciated!
I think you need to reference the module when you import it. A bit of code always helps.
import numpy
a=2
b=1
c=numpy.subtract(a,b)
print c
For future reference, a similar question was asked here, and I was able to adapt the solution for my needs. It was how I packaged the libraries.
I am using f2py to wrap my PETSc-based fortran analysis code for use in OpenMDAO (as suggested in this post). Rather than use f2py directly, I'm instead using it to generate the relevant .c, .pyc, etc. files and then linking them myself using mpif90.
In a simple python environment, I can import my .so and run the code without any problems:
>>> import module_name
>>> module_name.execute()
expected code output...
However, when trying to do the same thing in an OpenMDAO component, I get the following error:
At line 72 of file driver.F90
Internal Error: list_formatted_write(): Bad type
This happens even when running in serial and the error appears to the first place in the fortran code where I use write(*,*). What could be different about running under OpenMDAO that might cause this issue? Might it have something to do with the need to pass a comm object, as mentioned in the answer to my original question? I am not doing that at the moment as it was not clear to me from the the relevant OpenMDAO example how that should be done in my case.
When I try to find specific information about the error I'm getting, search results almost always point to the mpif90 or gfortran libraries and possibly needing to recompile or update the libraries. However, that doesn't explain to me why my analysis would work perfectly well in a simple python code but not in OpenMDAO.
UPDATE: Per some others' suggestions, I've tried a few more things. Firstly, I get the error regardless of if I'm running using mpiexec python <script> or merely python <script>. I do have the PETSc implementation set up, assuming that doesn't refer to anything beyond the if MPI block in this example.
In my standalone test, I am able to successfully import a handful of things, including
from mpi4py import MPI
from petsc4py import PETSc
from openmdao.core.system import System
from openmdao.core.component import Component
from openmdao.core.basic_impl import BasicImpl
from openmdao.core._checks import check_connections, _both_names
from openmdao.core.driver import Driver
from openmdao.core.mpi_wrap import MPI, under_mpirun, debug
from openmdao.components.indep_var_comp import IndepVarComp
from openmdao.solvers.ln_gauss_seidel import LinearGaussSeidel
from openmdao.units.units import get_conversion_tuple
from openmdao.util.string_util import get_common_ancestor, nearest_child, name_relative_to
from openmdao.util.options import OptionsDictionary
from openmdao.util.dict_util import _jac_to_flat_dict
Not too much rhyme or reason to what I tested, just went down a few random rabbit holes (more direction would be fantastic). Here are some of the things that do result in error if they are imported in the same script:
from openmdao.core.group import Group
from openmdao.core.parallel_group import ParallelGroup
from openmdao.core.parallel_fd_group import ParallelFDGroup
from openmdao.core.relevance import Relevance
from openmdao.solvers.scipy_gmres import ScipyGMRES
from openmdao.solvers.ln_direct import DirectSolver
So it doesn't seem that the MPI imports are a problem? However, not knowing the OpenMDAO code too well, I am having trouble seeing the common thread in the problematic imports.
UPDATE 2: I should add that I'm becoming particularly suspicious of the networkx package. If my script is simply
import networkx as nx
import module_name
module_name.execute()
then I get the error. If I import my module before networkz, however (i.e. switch lines 1 and 2 in the above block), I don't get the error. More strangely, if I also import PETSc:
from petsc4py import PETSc
import networkx as nx
import module_name
module_name.execute()
Then everything works...
UPDATE 3: I'm running OS X El Capitan 10.11.6. I genuinely don't remember how I installed the python2.7 (need to use this rather than 3.x at the moment) I was using. Installed years ago and was located in /usr/local/bin. However, I switched to an anaconda installation, re-installed networkx, and still get the same error.
I've discovered that if I compile the f2py-wrapped stuff using gfortran (I assume this is what you guys do, yes?) rather than mpif90, I don't get the errors. Unfortunately, this causes the PETSc stuff in my fortran code yield some strange errors, probably because those .f90/.F90 files, according to the PETSc compilation rules, are compiled by mpif90 even if I force the final compile to use gfortran.
UPDATE 4: I was finally able to solve the Internal Error: list_formatted_write() issue. By using mpif90 --showme I could see what flags mpif90 is using (since it's essentially just gfortran plus some flags). It turns omitting the flag -Wl,-flat_namespace got rid of those print-related errors.
Now I can import most things and run my code without a problem, with one important exception. If I have a petsc-based fortran module (pc_fort_mod), then also importing PETSc into the python environment, i.e.
from petsc4py import PETSc
import pc_fort_mod
pc_fort_mod.execute()
results in PETSc errors in the fortran analysis (invalid matrices, unsuccessful preallocation). This seems reasonable to me since both would appear to be attempting to use the same PETSc libraries. Any idea if there is a way to do this so that the pc_fort_mod PETSc and petsc4py PETSC don't clash? I guess a workaround may be to have two PETSc builds...
SOLVED: I'm told that the problem described in Update 4 ultimately should not be a problem--it should be possible to simultaneously use PETSc in python and fortran. I was ultimately able to resolve my error by using a self-compiled PETSc build rather than the Homebrew recipe.
I've never quite seen anything like this before and we've used network-X with compiled fortran wrapped in F2py, running under MPI many many times.
I suggest that you remove and re-install your network-x package.
Which python are you using and what os are you running on? We've had very good luck running the anaconda python installation. You have to be a bit careful when installing petsc though. Building from source and running the PETSc tests is the safest way.
I have a small python script that needs to do some simple stats on some lists of numbers - the sort of thing the statistics.py library seems suitable for.
From what I can make out of the info on python.org, it should be part of the standard library in python 3.4.1 (on Win 64 bit), but I'm struggling to access the functions.
import statistics just gives a 'no module named 'statistics' error
Looking through the python34\Lib directory doesn't seem to show a statistics folder, and creating one and saving the statistics.py file available from python.org doesn't seem to make any difference.
Googling for how to install python libraries gives lots of examples referencing setup.py files and command line sequences, but i don't see how they relate to the statistics.py file available.
I'm obviously missing something (probably obvious!) but being a newbie at this i can't work out what it is. Any pointers?
I had the same problem, and instead of:
import statistics
I wrote:
from Lib import statistics
Good Luck!
I'm having trouble importing numpy into my script because of some path-name issues.
I'm running my script with python 2.7, rather than the default 2.6 on my server because I need some of the updates in the Collections module. I cant seem to import numpy like:
from numpy.random import poisson
So I am trying to use the python2.7 specific links to numpy on my server, which are installed in:
/opt/lib/python2.7/numpy
But this period in the path is really making this difficult. I cannot change the path in anyway.
I've found a similar problem here, but frankly the code just doesn't make enough sense to me for me to feel safe using it (plus several commenters seemed to suggest it was a bad idea.). If someone has another suggestion, or if you can give me a good explanation of the code there I would appreciate it.
Try setting PYTHONPATH to point to /opt/lib/python2.7, after which import numpy et cetera should pull libraries from there.
$ PYTHONPATH=/opt/lib/python2.7 python27 my_script.py