I'm trying to set up a Windows slave for our Jenkins server, but some of the unit tests are failing due to a sqlite error.
We're using conda and have built a custom version of sqlite that includes rtree. Then our application is built using this custom sqlite. When we install our app on the windows server, the correct version of sqlite is installed:
$ conda list
sqlite 3.16.2 vc9_0 [vc9] file://[package_server]
Running the app from the command line it works as expected. When we run tests though,
$ nosetests test
...
======================================================================
ERROR: test_vectorise_1 (test.system_test_wallet.test_rasters.test_vectorise.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "farm\rasters\vectorise.py", line 189, in run_vectorise
proc(progressfn)
File "farm\process.py", line 1278, in __call__
for feature in lyr:
File "C:\bin\Miniconda2\envs\farm_system_test_windows\lib\site-packages\osgeo\
ogr.py", line 2876, in next
feature = self.GetNextFeature()
File "C:\bin\Miniconda2\envs\farm_system_test_windows\lib\site-packages\osgeo\
ogr.py", line 1460, in GetNextFeature
return _ogr.Layer_GetNextFeature(self, *args)
RuntimeError: sqlite3_exec(CREATE VIRTUAL TABLE "rtree_GB_TEST_RP100_VE_5m_27700
_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: no such module: rtree
I added some logging to show python executable path and sqlite3 path and it is the same every time, so it's not a virtualenv issue:
C:\bin\Miniconda2\envs\farm_system_test_windows\python.exe
C:\bin\Miniconda2\envs\farm_system_test_windows\lib\sqlite3\__init__.pyc
Printing the sqlite version though, shows some differences:
print('sql ver', sqlite3.sqlite_version_info)
Running the app from the command line, you get (3, 16, 2), which is the correct version. Running nosetests, you get (3, 8, 11) - an older version and presumably the reason for rtree not being found.
Weirdly, if you call a test file with
python path/to/test.py
and let unittest do its stuff, version (3, 16, 2) is initially imported, but then when the test runs, you get (3, 8, 11) and the test errors.
What is going on here? Why do I get a different version of sqlite depending on how I call the application?
I'm using python 2.7.14, conda 4.4.11 on Windows Server 2012 R2.
UPDATE:
I'm still getting this error on Jenkins and now on my dev machine too. It's not just a nose issue, as when I run the test using unittest, I get the error. The PYTHONPATH and sys.path are the same in both cases, and the call stack is identical, except with unittest, there are an extra two frames at the top.
File "C:\ProgramData\Anaconda2\envs\farm\lib\runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\ProgramData\Anaconda2\envs\farm\lib\runpy.py", line 72, in _run_code
exec code in run_globals
Interestingly, the rest of the call stack shows an absolute path for main.py, the entry point, and relative paths for everything else when using unittest. When calling main.py directly however, main.py has a relative path and all the other files have absolute paths.
Does anyone have any idea why we're seeing this difference? It's really frustrating having to skip a load of tests when we're running on windows.
By looking for a solution in sys.path you are looking at the wrong place. You need to see what os.environ['PATH'] is at the critical points here since it's the Windows Loader that has picked the 'wrong' DLL, where by wrong I mean the first one it found according to its well documented DLL lookup procedure.
Related
I am trying to work on a Django application that I downloaded from bitbucket which uses Python2-based code. Previously, Python 3.6 and Django 2.0 was on my system, so I downgraded to Python 2.7 and Django 1.11 to try to get it to work.
When I go to the project directory and type in "python manage.py runserver," I get a syntax error (this is a shortened version of it, hence the dots):
Unhandled exception in thread started by <function wrapper at 0x0000000006A0A358>
Traceback (most recent call last):
File "C:\Users\M\Anaconda3\lib\site-packages\django\utils\autoreload.py", line 227, in wrapper
fn(*args, **kwargs)
.
.
.
File "C:\Users\M\Anaconda3\lib\site-packages\django\contrib\admindocs\urls.py", line 2, in <module>
from django.contrib.admindocs import views
File "C:\Users\M\Anaconda3\lib\site-packages\django\contrib\admindocs\views.py", line 9, in <module>
from django.contrib.admindocs import utils
File "C:\Users\M\Anaconda3\lib\site-packages\django\contrib\admindocs\utils.py", line 12, in <module>
import docutils.core
File "C:\Users\M\Anaconda3\lib\site-packages\docutils\core.py", line 246
print('\n::: Runtime settings:', file=self._stderr)
^
SyntaxError: invalid syntax
If you have a solution to this problem, please let me know.
As said by #emi, you want to use virtualenv. It's a great tool that allows you to create an isolated virtual environment for each Python project. Each virtual environment can have its own version of every dependency.
On Windows, I recommend using it alongside virtualenvwrapper, witch is is just a very thin wrapper with some higher level commands.
It seems your Python installation is broken so uninstall every version of Python you have, reinstall Python 2 and follow along creating a virtualenv for your project (and for every new one you start). If you need you can install Python 3 in the future as well without messing with you previous envs.
OK, trying to track down this issue of running a Python package.
Running on Windows.
Only one version of Python is installed on the computer: 3.4.3
Package has been installed (package name: Willie) (details of installation are more convoluted than usual; can provide steps if necessary)
Package installed at: C:\Python34\Lib\site-packages\willie
Startup script is: C:\Python34\Scripts\willie.py
Error is one of the first lines of the script:
from willie.tools import stderr
If I run willie.py from the command line, I get this error:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 2218, in _find_and_load_unlocked
AttributeError: 'module' object has no attribute '__path__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python34\Scripts\willie.py", line 15, in <module>
from willie.tools import stderr
File "C:\Python34\Scripts\willie.py", line 15, in <module>
from willie.tools import stderr
ImportError: No module named 'willie.tools'; 'willie' is not a package
The __path__ attribute is supposed to be set automatically when the import function is used, but it doesn't exist, which leads to the program being unable to execute.
C:\Python34\Scripts; is in my Path environment variable.
PYTHONPATH has been set for everything from C:\Python34; to C:\Python34;C:\Python34\Lib;C:\Python34\Lib\site-packages;C:\Python34\Lib\site-packages;C:\Python34\Lib\site-packages\willie;C:\Python34\Lib\site-packages\willie\tools;.
In all cases with these directories, if I try to execute the above import command from the interactive Python prompt, it runs correctly. I can use the stderr function, and examine its __file__ and __path__ fields.
If I add C:\Python34\Scripts; to PYTHONPATH, however, I get the same error as using it from the command line: __path__ doesn't exist, 'willie' is not a package.
I do know that willie.py is being called recursively, somehow, because I can add a print() at the top of the file that runs twice.
Looking for help in figuring out how to make this run. Is almost certainly a configuration issue on my end, but I have no idea what to do to fix it.
First, you absolutely do not want c:\Python34\Scripts in your PYTHONPATH. Files under \Scripts are NOT meant to be importable.
Second, willie does some screwing around to make sure willie.py gets installed to c:\Python34\Scripts\willie (note there is no .py suffix). This is decidedly nonstandard. The only reason this even works is because on Unix machines, the first line of that file is a special "shebang" that tells the Unix program launcher to execute the file using Python. There is no equivalent feature on Windows - naming the file willie with no .py means you simply can't execute it.
So it looks like the module willie is simply not prepared to be installed on Windows systems. (#811 and #822 both refer to weirdness on Windows caused by the nonstandard package setup.)
A standard package setup would have willie.py inside the willie package as willie/main.py and, in the package's setup.py, register willie.main as an "Entry Point" named willie. This incantation would create c:\Python34\Scripts\willie.exe on Windows systems when the package is installed using pip.
I believe the reason for the recursive import is because the second entry in sys.path is the name of the script that is run from the command line. So, it's always finding C:\Python34\Scripts\willie.py when it looks for the module willie which is not the one it needs.
As a workaround, you could try renaming C:\Python34\Scripts\willie.py to C:\Python34\Scripts\run-willie.py
I'm running Pycharm 2.6.3 with python 2.7 and django 1.5.1. When I try to run django's manage.py task from Pycharm (Tools / Run manage.py task), syncdb, for instance, I get the following:
bash -cl "/usr/bin/python2.7 /home/paulo/bin/pycharm-2.6.3/helpers/pycharm/django_manage.py syncdb /home/paulo/Projetos/repo2/Paulo Brito/phl"
Traceback (most recent call last):
File "/home/paulo/bin/pycharm-2.6.3/helpers/pycharm/django_manage.py", line 21, in <module>
run_module(manage_file, None, '__main__', True)
File "/usr/lib/python2.7/runpy.py", line 170, in run_module
mod_name, loader, code, fname = _get_module_details(mod_name)
File "/usr/lib/python2.7/runpy.py", line 103, in _get_module_details
raise ImportError("No module named %s" % mod_name)
ImportError: No module named manage
Process finished with exit code 1
If I run the first line on the console passing project path between single quotes, it runs without problems, like this:
bash -cl "/usr/bin/python2.7 /home/paulo/bin/pycharm-2.6.3/helpers/pycharm/django_manage.py syncdb '/home/paulo/Projetos/repo2/Paulo Brito/phl'"
I tried to format the path like that in project settings / django support, but Pycharm won't recognize the path.
How can I work in PyCharm with paths with spaces?
Thanks.
EDIT 1
PyCharm dont recognize path with baskslash as valid path either.
it's known bug http://youtrack.jetbrains.com/issue/PY-8449
Fixed in PyCharm 2.7
In UNIX you can escape whitespaces with a backslash:
/home/paulo/Projetos/repo2/Paulo\ Brito/phl
I had the same problem on ubuntu 18.4 LTS pycharm 2019.2.6.
I was trying to create a basic Django new project in folder 'Learing Django api' (my folder name had spaces in it). when I tried to run the server I got a error saying settings module not defined.
After spending few hours I realized pycharm was creating a seperate folder named 'Learning' in same directory.
I renamed my folder 'Learing Django api' to 'Learing_Django_api' and it worked.
I like pycharm but this folder insensitivity is insane.
Is it possible that you are taking in the argument with a space in it into a variable, then passing that to something that is seeing two arguments instead of one? If so, "requote" it before passing it along. Will you hit a point where that can't be corrected? If so, maybe the upgrade makes sense.
Here's the setup:
Django (1.2) app on mod_wsgi that imports ctypes
Python 2.6.5
Apache 2.2.3
SELinux disabled
RedHat EL 5 64bit
some of the file system is mounted over nfs
Occasionally, when I restart apache I get an import error when it's trying to import ctypes. Every incoming request fails with a 500 error. If I restart apache usually everything just starts working again.
Here's the stack trace of the error:
Traceback (most recent call last):
File "/home/appfirst/django/django/core/handlers/base.py", line 80, in get_response
response = middleware_method(request)
-------------- A BUNCH OF DJANGO MIDDLEWARE STUFF HERE -------------
File "/home/appfirst/django/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/home/appfirst/backend/backend/streamer/views.py", line 6, in <module>
import appfirst.main.models as FEmodels
File "/home/appfirst/frontend/appfirst/main/models.py", line 27, in <module>
import numpy, math, mpmath
File "/usr/lib64/python2.6/site-packages/numpy/__init__.py", line 43, in <module>
import ctypeslib
File "/usr/lib64/python2.6/site-packages/numpy/ctypeslib.py", line 9, in <module>
import ctypes
File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module>
CFUNCTYPE(c_int)(lambda: None)
MemoryError
I thought it might be related to this bug, but I have SELinux turned off which I thought would mean this case could never occur:
http://bugs.python.org/issue5504
Any suggestions on how to reproduce it consistently and/or fix it? This is really stumping me!
I've run into this bug too. In my case it occurs when I exec a Python script from within a PHP script running under Apache on a 64-bit Linux system. [The Python code being run is the front-end to a pypy sandbox.] The same bit of code works fine on a 32-bit system and even works fine when the PHP script is executed directly from the command line. My "fix" has been simply to comment out that line "CFUNCTYPE(c_int)(lambda: None)" in ctypes/init.py. It's the last line of the file and is preceded by the following comment, showing that the programmer doesn't understand what's going on, either!
# XXX for whatever reasons, creating the first instance of a callback
# function is needed for the unittests on Win64 to succeed. This MAY
# be a compiler bug, since the problem occurs only when _ctypes is
# compiled with the MS SDK compiler. Or an uninitialized variable?
CFUNCTYPE(c_int)(lambda: None)
Clearly there's a deeper problem somewhere in cpython, but the fix works for me.
In case others happen to run into this issue, my fix was to upgrade from Python 3.7 to 3.8.
This offending line: https://github.com/python/cpython/blob/3.7/Lib/ctypes/__init__.py#L273
Was removed in 3.8: https://github.com/python/cpython/blob/3.8/Lib/ctypes/__init__.py#L261-L270
Hopefully, it can save someone the headache of trying to monkey patch ctype/__init__.py which was unsuccessful for me due to the way conda pack handles standard Python libraries.
Hope that helps 🤞
Consider turning SELinux off.
It should solve the problem.
I looked through the other posts and bug reports and couldn't figure out what's causing this. I'm using Jython 2.5.1, in a Java project in Eclipse (Ubuntu 8.10). It has been added to the project as a standalone .jar file (I just replaced the old Jython 2.1 jar with this one).
I'm running a script that uses the threading.py class. At some point the statement "import os" is evaluated from linecache.py and I get this error, which I can't seem to figure out how to fix:
'Execution failed. Traceback (most recent call last):
File "<string>", line 1, in <module>
File "../lib/python/threading.py", line 6, in <module>
import traceback
File "../lib/python/traceback.py", line 3, in <module>
import linecache
File "../lib/python/linecache.py", line 9, in <module>
import os
ImportError: No module named os'
What do you mean with "the jar that comes with the 2.5 download"? Did you extract the contents and use jython.jar or did you run the installer? If you just extracted and didn't run the installer your jython.jar will miss the whole LIB folder.
Can you check if jython.jar contains a LIB folder? (e.g. open jython.jar with 7z or WinZip).
Or try copying the LIB folder in the same folder where jython.jar resides.
Did you try setting these properties. Jython Registry. e.g. via -Dpython.home in the eclipse run configuration.
python.cachedir
python.path
python.home
How is the jar named? If similar to jython-complete.jar try renaming it to jython.jar
Something is wrong at a very deep level, but it's probably easy to fix. You are seeing an error that happens while trying to report some other error.
Probably you have your PYTHONPATH misconfigured. I don't know the details of Jython or Eclipse running Jython, but it looks like you have no standard library available to you.
If you are getting maven, using the dependency jython-standalone instead of jython may help (at least it did for me in a maven project with jython-standalone-2.5.3)