I noticed that the performance of mpmath, as oddly as it sounds, depends on whether sagemath is installed or not, regardless of whether the sage module is loaded in the current session. In particular, I experienced this for operations with multiple precision floats.
Example:
from mpmath import mp
import time
mp.prec = 650
t = time.time()
for i in range(1000000):
x_mpmath + y_mpmath
w = time.time()
print('plus:\t', (w-t), 'μs')
t = time.time()
for i in range(1000000):
x_mpmath * y_mpmath
w = time.time()
print('times:\t', (w-t), 'μs')
# If sagemath is installed:
# plus: 0.12919950485229492 μs
# times: 0.17601895332336426 μs
#
# If sagemath is *not* installed:
# plus: 0.6239776611328125 μs
# times: 0.6283771991729736 μs
While in both cases the module mpmath is the exact same
import mpmath
print(mpmath.__file__)
# /usr/lib/python3.9/site-packages/mpmath/__init__.py
I thought that mpmath's backend would depend on some sagemath dependency, and if that is missing it falls back to a less optimized one, but I cannot figure out what it is precisely. My goal is to be able to install only the required packages to speed up mpmath instead of installing all of sagemath.
Since this may very well be dependent on how things are packaged, you might need to have details on my system: I am using Arch Linux and all packages are updated to the most recent versions (sagemath 9.3, mpmath 1.2.1, python 3.9.5).
I found the explanation. In /usr/lib/python3.9/site-packages/mpmath/libmp/backend.py at line 82 there is
if 'MPMATH_NOSAGE' not in os.environ:
try:
import sage.all
import sage.libs.mpmath.utils as _sage_utils
sage = sage.all
sage_utils = _sage_utils
BACKEND = 'sage'
MPZ = sage.Integer
except:
pass
This loads all of sage if sagemath is installed and also sets it as a backend. This means that the following library is loaded next:
import sage.libs.mpmath.ext_libmp as ext_lib
From /usr/lib/python3.9/site-packages/mpmath/libmp/libmpf.py at line 1407. By looking at the __file__ of that module, one sees that it's a .so object, hence compiled, thus faster.
This also means that by exporting MPMATH_NOSAGE to any nonempty value will force the backend to be the default one (python or gmpy) and indeed I can confirm that the code I wrote in the question does run slower in this case, even with sagemath installed.
I am using numpy and my model involves intensive matrix-matrix multiplication.
To speed up, I use OpenBLAS multi-threaded library to parallelize the numpy.dot function.
My setting is as follows,
OS : CentOS 6.2 server #CPUs = 12, #MEM = 96GB
python version: Python2.7.6
numpy : numpy 1.8.0
OpenBLAS + IntelMKL
$ OMP_NUM_THREADS=8 python test_mul.py
code, of which I took from https://gist.github.com/osdf/
test_mul.py :
import numpy
import sys
import timeit
try:
import numpy.core._dotblas
print 'FAST BLAS'
except ImportError:
print 'slow blas'
print "version:", numpy.__version__
print "maxint:", sys.maxint
print
x = numpy.random.random((1000,1000))
setup = "import numpy; x = numpy.random.random((1000,1000))"
count = 5
t = timeit.Timer("numpy.dot(x, x.T)", setup=setup)
print "dot:", t.timeit(count)/count, "sec"
when I use OMP_NUM_THREADS=1 python test_mul.py, the result is
dot: 0.200172233582 sec
OMP_NUM_THREADS=2
dot: 0.103047609329 sec
OMP_NUM_THREADS=4
dot: 0.0533880233765 sec
things go well.
However, when I set OMP_NUM_THREADS=8.... the code starts to "occasionally works".
sometimes it works, sometimes it does not even run and and gives me core dumps.
when OMP_NUM_THREADS > 10. the code seems to break all the time..
I am wondering what is happening here ? Is there something like a MAXIMUM number threads that each process can use ? Can I raise that limit, given that I have 12 CPUs in my machine ?
Thanks
Firstly, I don't really understand what you mean by 'OpenBLAS + IntelMKL'. Both of those are BLAS libraries, and numpy should only link to one of them at runtime. You should probably check which of these two numpy is actually using. You can do this by calling:
$ ldd <path-to-site-packages>/numpy/core/_dotblas.so
Update: numpy/core/_dotblas.so was removed in numpy v1.10, but you can check the linkage of numpy/core/multiarray.so instead.
For example, I link against OpenBLAS:
...
libopenblas.so.0 => /opt/OpenBLAS/lib/libopenblas.so.0 (0x00007f788c934000)
...
If you are indeed linking against OpenBLAS, did you build it from source? If you did, you should see that in the Makefile.rule there is a commented option:
...
# You can define maximum number of threads. Basically it should be
# less than actual number of cores. If you don't specify one, it's
# automatically detected by the the script.
# NUM_THREADS = 24
...
By default OpenBLAS will try to set the maximum number of threads to use automatically, but you could try uncommenting and editing this line yourself if it is not detecting this correctly.
Also, bear in mind that you will probably see diminishing returns in terms of performance from using more threads. Unless your arrays are very large it is unlikely that using more than 6 threads will give much of a performance boost because of the increased overhead involved in thread creation and management.
Does anyone have any working implementations of the ISAAC cipher in python?
I tried to find this, but it seems no one ever did it.
Thanks.
I read some code examples on the ISAAC home page and the code appears straightforward to implement. If you need it in pure Python there are several examples in other languages to guide your porting effort.
Another way of using isaac() from Python is to build the C code as a shared library and access it via the ctypes module, which is standard in 2.5+, but can be found on PyPI for earlier versions. This should also perform much better than a direct port in pure Python.
Here's an example of building the C version of isaac() as a shared library and using it via ctypes. First you need to download rand.c, rand.h, and standard.h from the author's website, then build:
% gcc -shared -fPIC -o libisaac.so rand.c
Here is the Python code. Note that the sizes of the fields in the RandCtx are dependent on whether the code is built for 32- or 64-bit platform. I tested on 64-bit Ubuntu. For 32-bit you'd need to change all of the fields to use c_uint32:
from ctypes import *
class RandCtx(Structure):
RANDSIZL = 8
RANDSIZ = 1 << RANDSIZL
_fields_ = [
('randcnt', c_uint64),
('randrsl', c_uint64 * RANDSIZ),
('randmem', c_uint64 * RANDSIZ),
('randa', c_uint64),
('randb', c_uint64),
('randc', c_uint64)
]
ctx = RandCtx()
lib = cdll.LoadLibrary('./libisaac.so')
lib.randinit(byref(ctx), 0)
lib.isaac(byref(ctx))
for i in xrange(4):
print ctx.randrsl[i]
Output:
% python isaac.py
14012348966175605106
8193820543905647488
4194352129441799609
12121047914186473054
Does anyone know a way to get the amount of space available on a Windows (Samba) share via Python 2.6 with its standard library? (also running on Windows)
e.g.
>>> os.free_space("\\myshare\folder") # return free disk space, in bytes
1234567890
If PyWin32 is available:
free, total, totalfree = win32file.GetDiskFreeSpaceEx(r'\\server\share')
Where free is a amount of free space available to the current user, and totalfree is amount of free space total. Relevant documentation: PyWin32 docs, MSDN.
If PyWin32 is not guaranteed to be available, then for Python 2.5 and higher there is ctypes module in stdlib. Same function, using ctypes:
import sys
from ctypes import *
c_ulonglong_p = POINTER(c_ulonglong)
_GetDiskFreeSpace = windll.kernel32.GetDiskFreeSpaceExW
_GetDiskFreeSpace.argtypes = [c_wchar_p, c_ulonglong_p, c_ulonglong_p, c_ulonglong_p]
def GetDiskFreeSpace(path):
if not isinstance(path, unicode):
path = path.decode('mbcs') # this is windows only code
free, total, totalfree = c_ulonglong(0), c_ulonglong(0), c_ulonglong(0)
if not _GetDiskFreeSpace(path, pointer(free), pointer(total), pointer(totalfree)):
raise WindowsError
return free.value, total.value, totalfree.value
Could probably be done better but I'm not really familiar with ctypes.
The standard library has the os.statvfs() function, but unfortunately it's only available on Unix-like platforms.
In case there is some cygwin-python maybe it would work there?
Does anyone know how I would go about detected what bit version Windows is under Python. I need to know this as a way of using the right folder for Program Files.
Many thanks
I think the best solution to the problem has been posted by Mark Ribau.
The best answer to the question for Python 2.7 and newer is:
def is_os_64bit():
return platform.machine().endswith('64')
On windows the cross-platform-function platform.machine() internally uses the environmental variables used in Matthew Scoutens answer.
I found the following values:
WinXP-32: x86
Vista-32: x86
Win7-64: AMD64
Debian-32: i686
Debian-64: x86_64
For Python 2.6 and older:
def is_windows_64bit():
if 'PROCESSOR_ARCHITEW6432' in os.environ:
return True
return os.environ['PROCESSOR_ARCHITECTURE'].endswith('64')
To find the Python interpreter bit version I use:
def is_python_64bit():
return (struct.calcsize("P") == 8)
I guess you should look in os.environ['PROGRAMFILES'] for the program files folder.
platform module -- Access to underlying platform’s identifying data
>>> import platform
>>> platform.architecture()
('32bit', 'WindowsPE')
On 64-bit Windows, 32-bit Python returns:
('32bit', 'WindowsPE')
And that means that this answer, even though it has been accepted, is incorrect. Please see some of the answers below for options that may work for different situations.
Came here searching for properly detecting if running on 64bit windows, compiling all the above into something more concise.
Below you will find a function to test if running on 64bit windows, a function to get the 32bit Program Files folder, and a function to get the 64bit Program Files folder; all regardless of running 32bit or 64bit python. When running 32bit python, most things report as if 32bit when running on 64bit, even os.environ['PROGRAMFILES'].
import os
def Is64Windows():
return 'PROGRAMFILES(X86)' in os.environ
def GetProgramFiles32():
if Is64Windows():
return os.environ['PROGRAMFILES(X86)']
else:
return os.environ['PROGRAMFILES']
def GetProgramFiles64():
if Is64Windows():
return os.environ['PROGRAMW6432']
else:
return None
Note: Yes, this is a bit hackish. All other methods that "should just work", do not work when running 32bit Python on 64bit Windows (at least for the various 2.x and 3.x versions I have tried).
Edits:
2011-09-07 - Added a note about why only this hackish method works properly.
def os_platform():
true_platform = os.environ['PROCESSOR_ARCHITECTURE']
try:
true_platform = os.environ["PROCESSOR_ARCHITEW6432"]
except KeyError:
pass
#true_platform not assigned to if this does not exist
return true_platform
http://blogs.msdn.com/b/david.wang/archive/2006/03/26/howto-detect-process-bitness.aspx
Many of these proposed solutions, such as platform.architecture(), fail because their results depend on whether you are running 32-bit or 64-bit Python.
The only reliable method I have found is to check for the existence of os.environ['PROGRAMFILES(X86)'], which is unfortunately hackish.
You should be using environment variables to access this. The program files directory is stored in the environment variable PROGRAMFILES on x86 Windows, the 32-bit program files is directory is stored in the PROGRAMFILES(X86) environment variable, these can be accessed by using os.environ('PROGRAMFILES').
Use sys.getwindowsversion() or the existence of PROGRAMFILES(X86) (if 'PROGRAMFILES(X86)' in os.environ) to determine what version of Windows you are using.
Following this documentation, try this code:
is_64bits = sys.maxsize > 2**32
Im aware that in comments of the question this method was already used.
This is the method the .net framework uses:
import ctypes
def is64_bit_os():
""" Returns wethever system is a 64bit operating system"""
is64bit = ctypes.c_bool()
handle = ctypes.windll.kernel32.GetCurrentProcess() # should be -1, because the current process is currently defined as (HANDLE) -1
success = ctypes.windll.kernel32.IsWow64Process(handle, ctypes.byref(is64bit)) #should return 1
return (success and is64bit).value
print(is64_bit_os())
I just found another way to do this, which may be useful in some situations.
import subprocess
import os
def os_arch():
os_arch = '32-bit'
if os.name == 'nt':
output = subprocess.check_output(['wmic', 'os', 'get', 'OSArchitecture'])
os_arch = output.split()[1]
else:
output = subprocess.check_output(['uname', '-m'])
if 'x86_64' in output:
os_arch = '64-bit'
else:
os_arch = '32-bit'
return os_arch
print 'os_arch=%s' % os_arch()
I tested this code in the following environments:
Ubuntu 16.04 + Python 2.7.12
Mac OS Sierra + Python 2.7.11
Windows 7 Pro 32-bit + Python 2.7.5 (32-bit)
Windows 10 Home 64-bit + Python 2.7.13 (32-bit)
The subject lines asks about detecting 64 or 32bit OS, while the body talks about determining the location of ProgramFiles. The latter has a couple of workable answers here. I'd like to add another solution generalized to handle StartMenu, Desktop, etc. as well as ProgramFiles: How to get path of Start Menu's Programs directory?
When you need to find out things about windows system, it is usually somewhere in the registry, according to MS documentation, you should look at (http://support.microsoft.com/kb/556009) this key value:
HKLM\HARDWARE\DESCRIPTION\System\CentralProcessor\0
and if it is:
0x00000020 (32 in decimal)
It is a 32 bit machine.
64-bit versions of Windows use something called registry redirection and reflection keys. There is a compatibility layer called WoW64 which enables compatibility of 32-bit applications. Starting from Windows 7 and Windows Server 2008 R2 WoW64 registry keys are not longer reflected but shared. You can read about it here:
registry-reflection: msdn.microsoft.com/en-us/library/aa384235(v=vs.85).aspx
affected-keys: msdn.microsoft.com/en-us/library/aa384253(v=vs.85).aspx
wikipedia: en.wikipedia.org/wiki/WoW64
All you need to do is detect existence of those keys. You can use _winreg for that. Use try: and try opening key, example:
try:
aReg = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE,"SOFTWARE\\Wow6432Node\\Microsoft\\Windows\\CurrentVersion\\Run")
import _winreg
def get_registry_value(key, subkey, value):
key = getattr(_winreg, key)
handle = _winreg.OpenKey(key, subkey )
(value, type) = _winreg.QueryValueEx(handle, value)
return value
windowsbit=cputype = get_registry_value(
"HKEY_LOCAL_MACHINE",
"SYSTEM\\CurrentControlSet\Control\\Session Manager\\Environment",
"PROCESSOR_ARCHITECTURE")
print windowsbit
just run this code
if you are working on 64 bit windows machine this will print AMD64
or if you are working on 32 bit it will print AMD32
i hope this code can help to solve this problem fully
This works for me in the Python versions I use: 2.7 and 2.5.4
import win32com.client
import _winreg
shell = win32com.client.Dispatch('WScript.Shell')
proc_arch = shell.ExpandEnvironmentStrings(r'%PROCESSOR_ARCHITECTURE%').lower()
if proc_arch == 'x86':
print "32 bit"
elif proc_arch == 'amd64':
print "64 bit"
else:
raise Exception("Unhandled arch: %s" % proc_arch)
Just to update this old thread - it looks like the platform module reports the correct architecture now (at least, in Python 2.7.8):
c:\python27\python.exe -c "import platform; print platform.architecture(), platform.python_version()"
('32bit', 'WindowsPE') 2.7.6
c:\home\python278-x64\python.exe -c "import platform; print platform.architecture(), platform.python_version()"
('64bit', 'WindowsPE') 2.7.8
(sorry I don't have the rep to comment on the first answer which still claims to be wrong :)
import platform
platform.architecture()[0]
It will return '32bit' or '64bit' depending on system architecture.
The solution posted by Alexander Brüsch is the correct solution, but it has a bug that only reveals itself on python3.x. He neglected to cast the returned value from GetCurrentProcess() to a HANDLE type. Passing a simple integer as the first parameter of IsWow64Process() returns 0 (which is an error flag from win32api). Also, Alexander incorrectly handles the return statement (success has no .value attribute).
For those who stumble on this thread, here is the corrected code:
import ctypes
def is64_bit_os():
"""Returns True if running 32-bit code on 64-bit operating system"""
is64bit = ctypes.c_bool()
handle = ctypes.wintypes.HANDLE(ctypes.windll.kernel32.GetCurrentProcess())
success = ctypes.windll.kernel32.IsWow64Process(handle, ctypes.byref(is64bit))
return success and is64bit.value
print(is64_bit_os())
There is a function named machine in platform module. I installed both Python3.8 32-bit and 64-bit versions on the same 64-bit machine with 64-bit Windows 10 and here is what I found:
And it looks like platform.machine returns machine architecture without bothering what type of python is installed. so here is my
final compilation
import platform
def is_64bit():
return platform.machine().endswith('64')
Most of the answers here are incorrect :/
Here is a simple translation of the well known method used in CMD and this is how microsoft do it too.
import os
_os_bit=64
if os.environ.get('PROCESSOR_ARCHITECTURE').lower() == 'x86' and os.environ.get('PROCESSOR_ARCHITEW6432') is None: _os_bit=32
print(_os_bit)
but remember: Windows 10 on ARM includes an x86-on-ARM64 emulation, so the possible values for PROCESSOR_ARCHITECTURE are: AMD64 or IA64 or ARM64 or x86
A solution, putting together the options from the links below and using os module:
import os
#next save the response from the command prompt saved to a file
window = os.system('PowerShell.exe "gwmi win32_operatingsystem | select osarchitecture" > prompt.txt')
#next read the file
f = open('prompt.txt','r')
windowsos = f.readlines()
f.close()
print(windowsos[3][:-1])
https://datatofish.com/command-prompt-python/
https://www.radishlogic.com/windows/how-to-check-if-your-windows-10-is-64-bit-or-32-bit/
https://www.tutorialspoint.com/how-to-run-a-powershell-script-from-the-command-prompt
import struct
def is64Windows():
return struct.calcsize('P') * 8 == 64
There should be a directory under Windows 64bit, a Folder called \Windows\WinSxS64 for 64 bit, under Windows 32bit, it's WinSxS.