We Keep Coding

How to store the last displayed value from the console?

My python script passes changing inputs to a program called "Dymola", which in turn performs a simulation to generate outputs. Those outputs are stored as numpy arrays "out1.npy".
for i in range(0,100):
#code to initiate simulation
print(startValues, 'ParameterSet:', ParameterSet,'time:', stoptime)
np.save('out1.npy', output_data)
Unfortunately, Dymola crashes very often, which makes it necessary to rerun the loop from the time displayed in the console when it has crashed (e.g.: 50) and increase the number of the output file by 1. Otherwise the data from the first set would be overwritten.
for i in range(50,100):
#code to initiate simulation
print(startValues, 'ParameterSet:', ParameterSet,'time:', stoptime)
np.save('out2.npy', output_data)
Is there any way to read out the 'stoptime' value (e.g. 50) out of the console after Dymola has crashed?

I'm assuming dymola is a third-party entity that you cannot change.
One possibility is to use the subprocess module to start dymola and read its output from your program, either line-by-line as it runs, or all after the created process exits. You also have access to dymola's exit status.
If it's a Windows-y thing that doesn't do stream output but manipulates a window GUI-style, and if it doesn't generate a useful exit status code, your best bet might be to look at what files it has created while or after it has gone. sorted( glob.glob("somepath/*.out")) may be useful?

I assume you're using the dymola interface to simulate your model. If so, why don't you use the return value of the dymola.simulate() function and check for errors.
E.g.:
crash_counter = 1
from dymola.dymola_interface import DymolaInterface
dymola = DymolaInterface()
for i in range(0,100):
res = dymola.simulate("myModel")
if not res:
crash_counter += 1
print(startValues, 'ParameterSet:', ParameterSet,'time:', stoptime)
np.save('out%d.npy'%crash_counter, output_data)
As it is sometimes difficult to install the DymolaInterface on your machine, here is a useful link.
Taken from there:
The Dymola Python Interface comes in the form of a few modules at \Dymola 2018\Modelica\Library\python_interface. The modules are bundled within the dymola.egg file.
To install:
The recommended way to use the package is to append the \Dymola 2018\Modelica\Library\python_interface\dymola.egg file to your PYTHONPATH environment variable. You can do so from the Windows command line via set PYTHONPATH=%PYTHONPATH%;D:\Program Files (x86)\Dymola 2018\Modelica\Library\python_interface\dymola.egg.
If this does not work, append the following code before instantiating the interface:
import os
import sys
sys.path.insert(0, os.path.join('PATHTODYMOLA',
'Modelica',
'Library',
'python_interface',
'dymola.egg'))

How to debug PyQt5 appcrash on exit

I have been developing a PyQt5 software for a while now. I have managed to package my Python 3 PyQt5 software with py2exe fine in past with everything working perfectly.
However, now I have encountered an issue where the packaged exe-program will crash when user exits. More specifically I get APPCRASH with following details
Problem signature:
Problem Event Name: APPCRASH
Application Name: Sotilasmatrikkelit.exe
Application Version: 0.0.0.0
Application Timestamp: 54467a51
Fault Module Name: PyQt5.QtCore.pyd
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 549be77e
Exception Code: c0000005
Exception Offset: 0010c185
OS Version: 6.1.7601.2.1.0.256.4
Locale ID: 1035
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
I never get this crash when running the software from Pycharm during the development. Any idea of how to debug this or what could be the cause in general?
I suspect this might have something to do with memory management (that PyQt doesn't delete all the resources properly on exit and therefore segfaults) but does anyone have any good suggestions to figure out the exact problem without better debug information? Should I try to do some kind of cleanup on exit? Atm I start the software like this:
def start():
import sys
app = QApplication(sys.argv)
fixingtool = Mainwindow(app)
fixingtool.show()
sys.exit(app.exec_())
Additional investigation seems to suggest that I get Access Violation which is caused by C++ null-pointer. Sounds scary since I don't know too much of PyQt debugging. Anyway, I found an area on my businesslogic code which if removed will remove the problem. However, this code has nothing to do with PyQt and is just regular Python code and shouldn't differ in any way. Strangest part is that if I remove certain functions from the code, problem disappears even though functions are not called during runtime meaning that just importing the file with those functions cause the problem. Below is a code sample:
import re
from books.karelians.extraction.extractors.baseExtractor import BaseExtractor
from books.karelians.extraction.extractionExceptions import *
from books.karelians.extraction.extractors.dateExtractor import DateExtractor
from shared import textUtils
from books.karelians.extractionkeys import KEYS
from interface.valuewrapper import ValueWrapper
from shared import regexUtils
from books.karelians.extraction.extractors.professionextractor import ProfessionExtractor
class SpouseExtractor(BaseExtractor):
def extract(self, text, entry):
super(SpouseExtractor, self).extract(text)
self.entry = entry
self.PATTERN = r"Puol\.?(?P<spousedata>[A-ZÄ-Öa-zä-ö\s\.,\d-]*)(?=(Lapset|poika|tytär|asuinp))"
self.NAMEPATTERN = r"(?P<name>^[\w\s\.-]*)"
self.OPTIONS = (re.UNICODE | re.IGNORECASE) #TODO: TRY IGNORE CASE?
self.REQUIRES_MATCH_POSITION = False
self.SUBSTRING_WIDTH = 100
self.hasSpouse = False
self.spouseName = ""
self.profession = {KEYS["profession"] : ValueWrapper("")}
self.initVars(text)
self._findSpouse(text)
return self._constructReturnDict()
def initVars(self,text):
pass
def _findSpouse(self, text):
try:
self.foundSpouse = regexUtils.safeSearch(self.PATTERN, text, self.OPTIONS)
self.hasSpouse = True
self._findSpouseName(self.foundSpouse.group("spousedata"))
self._setFinalMatchPosition()
except regexUtils.RegexNoneMatchException:
pass
def _findSpouseName(self, text):
try:
name = regexUtils.safeSearch(self.NAMEPATTERN, text, self.OPTIONS)
self.spouseName = name.group("name").strip()
self._findProfession(text[name.end():])
except regexUtils.RegexNoneMatchException:
self.errorLogger.logError(SpouseNameException.eType, self.currentChild)
def _findProfession(self, text):
professionExt = ProfessionExtractor(self.entry, self.errorLogger, self.xmlDocument)
professionExt.setDependencyMatchPositionToZero()
self.profession = professionExt.extract(text, self.entry)
def _setFinalMatchPosition(self):
#Dirty fix for inaccuracy in positions which would screw the Location extraction
self.matchFinalPosition = self.foundSpouse.end() + self.matchStartPosition - 4
def _constructReturnDict(self):
print(self.profession)
return {KEYS["spouse"]: ValueWrapper({ KEYS["hasSpouse"]: ValueWrapper(self.hasSpouse),KEYS["spouseName"]: ValueWrapper(self.spouseName), KEYS["spouseProfession"]: ValueWrapper(self.profession[KEYS["profession"]].value) })}
Here if I remove or comment away all functions "initVars()" program exits properly. What gives?

This isn't really a solution to a problem itself but I'm leaving this here in case someone encounters a similar issue.
Today I decided to try cx_freeze instead of py2exe figuring that maybe the problem is with py2exe since the whole issue is not happening when running the application with normal python interpreter.
Turns out I was right and problem seemed to magically disappear after I packaged the app using cx_freeze instead of py2exe. I didn't do any changes to code. Someone more knowledgeable than me about how py2exe and cx_freeze work might be able to explain the difference. My wild guess is that for some reason the exiting from the Python interpreter is not handled perfectly in py2exe case somehow messing the end garbage cleanup. I have no idea if this is py2exe's or my fault by not configuring py2exe properly.
In any case I'm happy it works now since yesterday was a really frustrating day.

Set max number of threads at runtime on numpy/openblas

I'd like to know if it's possible to change at (Python) runtime the maximum number of threads used by OpenBLAS behind numpy?
I know it's possible to set it before running the interpreter through the environment variable OMP_NUM_THREADS, but I'd like to change it at runtime.
Typically, when using MKL instead of OpenBLAS, it is possible:
import mkl
mkl.set_num_threads(n)

You can do this by calling the openblas_set_num_threads function using ctypes. I often find myself wanting to do this, so I wrote a little context manager:
import contextlib
import ctypes
from ctypes.util import find_library
# Prioritize hand-compiled OpenBLAS library over version in /usr/lib/
# from Ubuntu repos
try_paths = ['/opt/OpenBLAS/lib/libopenblas.so',
'/lib/libopenblas.so',
'/usr/lib/libopenblas.so.0',
find_library('openblas')]
openblas_lib = None
for libpath in try_paths:
try:
openblas_lib = ctypes.cdll.LoadLibrary(libpath)
break
except OSError:
continue
if openblas_lib is None:
raise EnvironmentError('Could not locate an OpenBLAS shared library', 2)
def set_num_threads(n):
"""Set the current number of threads used by the OpenBLAS server."""
openblas_lib.openblas_set_num_threads(int(n))
# At the time of writing these symbols were very new:
# https://github.com/xianyi/OpenBLAS/commit/65a847c
try:
openblas_lib.openblas_get_num_threads()
def get_num_threads():
"""Get the current number of threads used by the OpenBLAS server."""
return openblas_lib.openblas_get_num_threads()
except AttributeError:
def get_num_threads():
"""Dummy function (symbol not present in %s), returns -1."""
return -1
pass
try:
openblas_lib.openblas_get_num_procs()
def get_num_procs():
"""Get the total number of physical processors"""
return openblas_lib.openblas_get_num_procs()
except AttributeError:
def get_num_procs():
"""Dummy function (symbol not present), returns -1."""
return -1
pass
#contextlib.contextmanager
def num_threads(n):
"""Temporarily changes the number of OpenBLAS threads.
Example usage:
print("Before: {}".format(get_num_threads()))
with num_threads(n):
print("In thread context: {}".format(get_num_threads()))
print("After: {}".format(get_num_threads()))
"""
old_n = get_num_threads()
set_num_threads(n)
try:
yield
finally:
set_num_threads(old_n)
You can use it like this:
with num_threads(8):
np.dot(x, y)
As mentioned in the comments, openblas_get_num_threads and openblas_get_num_procs were very new features at the time of writing, and might therefore not be available unless you compiled OpenBLAS from the latest version of the source code.

We recently developed threadpoolctl, a cross platform package to do control the number of threads used in calls to C-level thread-pools in python. It works similarly to the answer by #ali_m but detects automatically the libraries that needs to be limited by looping through all loaded libraries. It also comes with introspection APIs.
This package can be installed using pip install threadpoolctl and come with a context manager that allow you to control the number of threads used by packages such as numpy:
from threadpoolctl import threadpool_limits
import numpy as np
with threadpool_limits(limits=1, user_api='blas'):
# In this block, calls to blas implementation (like openblas or MKL)
# will be limited to use only one thread. They can thus be used jointly
# with thread-parallelism.
a = np.random.randn(1000, 1000)
a_squared = a # a
you can also have finer control on different threadpools (such as differenciating blas from openmp calls).
Note: this package is still in development and any feedback is welcomed.

Pyomo's SolverFactory cannot create Ipopt (OSX) - possibly related to COIN-OR

I'm trying to use Pyomo to find the optimal values of a Python model on OSX. I got the script from https://github.com/shoibalc/recem, and installed Pyomo and COIN-OR following the instructions to the extent that I could, changing a few things that were outdated or didn't seem to work on OSX.
The code that is causing problems is below.
import pyomo
from pyomo.opt.base import *
from pyomo.opt.parallel import SolverManagerFactory
from DICE2007 import createDICE2007
from DICEutils import DICE_results_writer
global start_time
start_time = time.time()
dice = createDICE2007()
dice.doc = 'OPTIMAL SCENARIO'
opt = SolverFactory('ipopt',solver_io='nl')
tee = False
options = """
halt_on_ampl_error=yes"""
solver_manager = SolverManagerFactory('serial')
print '[%8.2f] create model %s OPTIMAL SCENARIO\n' %(time.time()-start_time,dice.name)
instance = dice.create()
print '[%8.2f] created instance\n' %(time.time()-start_time)
results = solver_manager.solve(instance, opt=opt, tee=tee, options=options, suffixes=['dual','rc'])
This crashes on the last ("results") line, with the following error message:
The SolverFactory was unable to create the solver "ipopt"
and returned an UnknownSolver object. This error is raised at the point
where the UnknownSolver object was used as if it were valid (by calling
method "solve").
The original solver was created with the following parameters:
solver_io: nl
type: ipopt
_args: ()
options: {}
_options_str: []
I'm very new to all this, but thought that maybe Pyomo can't access the ipopt file it needs, which I think for me is located in the COIN-OR binaries I downloaded. I tried adding the relevant-seeming files to my PYTHONPATH and also importing them into the script, which didn't help. Any ideas what I should try next to either make this work, or to amend the script to something that would work?

A colleague of mine had the same issue and he managed to solve it by generating the solver object with the route to the IPOPT AMPL executable:
opt = SolverFactory('/route/to/ipopt',solver_io='nl')

system wide shortcut for Mac OS X

So I was asked to port some internal helper applications to Mac OS X 10.7.
Works all quite welll as the platform dependent code is minimal anyhow, but one application needs a system wide shortcut to function (i.e. RegisterHotkey functionality) and I can't find any documentation on how I'd do this on a Mac.
The program is using a PyQt gui with Python 3.2. and the corresponding code for windows is basically:
def register_hotkey(self):
hwnd = int(self.winId())
modifiers, key = self._get_hotkey()
user32.RegisterHotKey(hwnd, self._MESSAGE_ID, modifiers, key)
and then to receive the hotkey events:
def winEvent(self, msg):
if msg.message == w32.WM_HOTKEY:
self.handle_hotkey()
return True, id(msg)
return False, id(msg)
Note that I don't need a python variant, I can easily write a simple c extension - so C/objective-c solutions are welcome as well.

I recently coded up an extension to quodlibet capturing multimedia keys (since absorbed into quodlibet itself); for your setup the same process applies.
I used the Quartz CGEventTapCreate hook and event loop, and the Cocoa AppKit framework to decipher key codes to achieve this.
The following code registers a python callback which is passed global key presses, and starts the event loop:
import Quartz
from AppKit import NSKeyUp, NSSystemDefined, NSEvent
# Set up a tap, with type of tap, location, options and event mask
tap = Quartz.CGEventTapCreate(
Quartz.kCGSessionEventTap, # Session level is enough for our needs
Quartz.kCGHeadInsertEventTap, # Insert wherever, we do not filter
Quartz.kCGEventTapOptionListenOnly, # Listening is enough
Quartz.CGEventMaskBit(NSSystemDefined), # NSSystemDefined for media keys
keyboardTapCallback,
None
)
runLoopSource = Quartz.CFMachPortCreateRunLoopSource(None, tap, 0)
Quartz.CFRunLoopAddSource(
Quartz.CFRunLoopGetCurrent(),
runLoopSource,
Quartz.kCFRunLoopDefaultMode
)
# Enable the tap
Quartz.CGEventTapEnable(tap, True)
# and run! This won't return until we exit or are terminated.
Quartz.CFRunLoopRun()
I defined a tap for system defined keys only (media keys); you'll have to specify a different event mask (CGEventMaskBit with one or more Event Types); e.g. Quartz.CGEventMaskBit(Quartz.kCGEventKeyUp) for key up events.
The callback should have the following signature (it implements the CGEventTapCallBack method from the Quartz API:
def keyboardTapCallback(proxy, type_, event, refcon):
# Convert the Quartz CGEvent into something more useful
keyEvent = NSEvent.eventWithCGEvent_(event)
I converted the Quartz event into a NSEvent, because all the information I could find on Mac multimedia keys was referring to that class.
In principle you can achieve the same thing with the AppKit APIs too, but then your Python application is treated as a Mac Application (visible in the Dock with an icon and everything), while I wanted this to be kept in the background altogether.

Using the power of google, I found this snippet of code, which allows the registration of global hotkeys for Mac OS X
You'll need to add the Carbon framework, and probably a bridged cast for ARC when passing the Objective-C self pointer to the C function.
At a minimum, you'll also need to:
#import <Carbon/Carbon.h>
The keycodes can be seen on this page explaining the virtual key codes.

Why has nobody ever mentioned the hammerspoon, which supports custom global system shortcuts, which can invoke shell command or launch UI application like Safari, PHOTOSHOP.
The following is an example written by me demonstrating how to invoke shell function with global hotkeys.
https://gist.github.com/BigSully/0e59ab97f148bc167ea19dbd42ebef4b
Use hs.execute to execute shell command, either non-interactive or interactive.
hs.hotkey.bind({"cmd", "alt", "ctrl"}, "P", function()
local output = hs.execute("toggleProxy", true)
hs.alert.show(output)
end)
or
Use hs.application.launchOrFocus to launch application
hs.application.launchOrFocus("Safari")