We Keep Coding

Creating a Dask client results in endless loop of errors

When I run this code:
from dask.distributed import Client
client = Client(n_workers = 2, threads_per_worker = 2, memory_limit = '2GB', silence_logs='error')
client
I get endless loop of these errors:
tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: None, threads: 2>>
Traceback (most recent call last):
File "C:\Users\szlau\anaconda3\lib\site-packages\tornado\ioloop.py", line 907, in _run
return self.callback()
File "C:\Users\szlau\anaconda3\lib\site-packages\distributed\nanny.py", line 414, in memory_monitor
process = self.process.process
Note: I have also tried reinstalling the whole Anaconda environment and the error still persists.
I really have no idea what can be causing this or how to eliminate this.
Any suggestions are welcomed!

At a guess, you are running your code in a script. That means that when dask creates the cluster and associated worker processes, those will, in turn, also runt he script and also attempt to create new clusters and processes.
You should guard your client and cluster creation with
if __name__ == "__main__":
client = Client(...)

kochat in use RuntimeError: main thread is not in main loop

kochat is a Korean chatbot, and I ran into a problem while practicing it.
github imformation : https://github.com/hyunwoongko/kochat
It is environment setting
python3.8
pip install kochat
JPype is reinstalled ues JPype1-1.2.0-cp38-cp38-win_amd64.whl <-download
pytorch is same ver with cuda 11.1
start code
### from kochat.proc import DistanceClassifier ###
from kochat.data import Dataset
from kochat.proc import GensimEmbedder, DistanceClassifier
from kochat.model import intent, embed
from kochat.loss import CenterLoss
dataset = Dataset(ood=True)
emb = GensimEmbedder(model=embed.Word2Vec())
# 프로세서 생성
clf = DistanceClassifier(
model=intent.CNN(dataset.intent_dict),
loss=CenterLoss(dataset.intent_dict)
)
# 되도록이면 DistanceClassifier는 Margin 기반의 Loss 함수를 이용해주세요
# 현재는 CenterLoss, COCOLoss, Cosface, GausianMixture 등의
# 거리기반 Metric Learning 전용 Loss함수를 지원합니다.
# 모델 학습
clf.fit(dataset.load_intent(emb))
# 모델 추론 (인텐트 분류)
clf.predict(dataset.load_predict("오늘 서울 날씨 어떨까", emb))
error message
C:\projectkyc\kochat3.8\Scripts\python.exe "C:/projectkyc/kochat3.8/test3.8/!from kochat.proc import DistanceClassifier.py"`enter code here`
Exception ignored in: <function Image.__del__ at 0x000001E7E1566D30>
Traceback (most recent call last):
File "c:\users\user\appdata\local\programs\python\python38\lib\tkinter\__init__.py", line 4014, in __del__
Exception ignored in: <function Variable.__del__ at 0x000001E7E154B430>
Traceback (most recent call last):
File "c:\users\user\appdata\local\programs\python\python38\lib\tkinter\__init__.py", line 351, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x000001E7E1566D30>
Traceback (most recent call last):
File "c:\users\user\appdata\local\programs\python\python38\lib\tkinter\__init__.py", line 4014, in __del__
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (os_windows_x86.cpp:144), pid=11208, tid=0x00000000000037f4
# guarantee(result == EXCEPTION_CONTINUE_EXECUTION) failed: Unexpected result from topLevelExceptionFilter
#
# JRE version: Java(TM) SE Runtime Environment (8.0_281-b09) (build 1.8.0_281-b09)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.281-b09 mixed mode windows-amd64 compressed oops)
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\projectkyc\kochat3.8\test3.8\hs_err_pid11208.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Exception ignored in: <function Image.__del__ at 0x000001E7E1566D30>
Traceback (most recent call last):
File "c:\users\user\appdata\local\programs\python\python38\lib\tkinter\__init__.py", line 4014, in __del__
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Tcl_AsyncDelete: async handler deleted by the wrong thread
Process finished with exit code 1
If I haven't posted it or if I have the information I need, I will add it immediately upon confirmation if requested.

This is such an annoying issue and to me setting matplotlib.use('Agg') does not cut it. I dont want to comprise on this.
The only thing that seems to do it somewhat consistently is having to close the figure before creating a new plot via plt.close(). This is annoying and if i forget it does this.

this error
Matplotlib - Tcl_AsyncDelete: async handler deleted by the wrong thread?
same Question But did not know where to apply
The places to apply are kochat\utils\visualizer.py and kochat\proc\utils\visualizer.py
to be. In the
from matplotlib import pyplot as plt
part of these two places,
import matplotlib
matplotlib.use('Agg')
from matplotlib import pyplot as plt
The problem is solved.

JRC JJ1000 + dronekit >> ERROR:dronekit.mavlink:Exception in MAVLink input loop

I am trying to connect JRC JJ1000 drone using dronekit + python.
when executing the connect command:
dronekit.connect('com3', baud=115200, heartbeat_timeout=30)
I am getting the following error:
ERROR:dronekit.mavlink:Exception in MAVLink input loop
Traceback (most recent call last):
File "C:\Python37\lib\site-packages\dronekit\mavlink.py", line 211, in mavlink_thread_in
fn(self)
File "C:\Python37\lib\site-packages\dronekit\__init__.py", line 1371, in listener
self._heartbeat_error)
dronekit.APIException: No heartbeat in 5 seconds, aborting.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python37\lib\site-packages\dronekit\__init__.py", line 3166, in connect
vehicle.initialize(rate=rate, heartbeat_timeout=heartbeat_timeout)
File "C:\Python37\lib\site-packages\dronekit\__init__.py", line 2275, in initialize
raise APIException('Timeout in initializing connection.')
dronekit.APIException: Timeout in initializing connection.
I left no store unturned but no progress. I also tried both Python 2.7 and 3.7 with same result.

I have been getting the same error. I am using some custom code in a docker container to run simulations with dronekit and ArduPilot. The error is intermittent. So far it seems like the only way to get the error to stop is to:
Close all docker containers.
Open windows task manager and wait for vmmem to lower memory usage (5-10m).
Try again.
Maybe the problems are related somehow. To me it seems like the connection might be in use by a previous instance and it was not properly close. Since waiting for vmmem to free up resources appears to fix it. I would prefer a better solution if anyone finds one!
We are using python code like this to connect:
from dronekit import connect
...
# try to connect 5 times
while connected == False and fails < 5:
try:
vehicle = connect(connection_string, wait_ready=True)
except:
fails += 1
time.sleep(3)
print("Failed to connect to local mavlink sleeping for 3 seconds")
else:
connected = True
Where the connection_string is of the form:
"tcp:host:port"
Also, the documentation states "If the baud rate is not set correctly, connect may fail with a timeout error. It is best to set the baud rate explicitly." Are you sure that you have the correct baud rate?

Decoupled process from terminal still outputs Traceback to the terminal

While testing an application I made using a rest api I discovered this behaviour which I don't understand.
Lets start by reproducing a similar error as follows -
In file call.py -
Note that this file has code that manifests itself visually for example a GUI that runs forever. Here I am just showing you a representation and deliberately making it raise an Exception to show you the issue. Making a get request and then trying to parse the result as json will raise a JSONDecodeError.
import requests
from time import sleep
sleep(3)
uri = 'https://google.com'
r = requests.get(uri)
response_dict = r.json()
Since I want to run this as a daemon process, I decouple this process from the terminal that started it using the following trick -
In file start.py -
import subprocess
import sys
subprocess.Popen(["python3", "call.py"])
sys.exit(0)
And then I execute python3 start.py
It apparently decouples the process because if there are no exceptions the visual manifestation runs perfectly.
However in case of an exception I immediately see this output in the terminal, even though I got a new prompt after calling python3 start.py -
$ python3 start.py
$ Traceback (most recent call last):
File "call.py", line 7, in <module>
response_dict = r.json()
File "/home/walker/.local/lib/python3.6/site-packages/requests/models.py", line 896, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Now, I understand that all exceptions MUST be handled in the program itself. And I have done it after this strange issue, but what is not clear to me is why did this happen at all in the first place?
It doesn't happen if I quit the terminal and restart the terminal (the visual manifestation gets stuck in case of a Traceback, and no output on any terminal as expected)
Why is a decoupled process behaving this way?
NOTE: Decoupling is imperative to me. It is imperative that the GUI run as a background or daemon process and that the terminal that spawns it is freed from it.

by "decoupled", I assume you mean you want the stdout/stderr to go to /dev/null? assuming that's what you mean, that's not what you've told your code to do
from the docs:
stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object, and None.
With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent.
you therefore probably want to be doing:
from subprocess import Popen, DEVNULL
Popen(["python3", "call.py"], stdin=DEVNULL, stdout=DEVNULL, stderr=DEVNULL)
based on the OPs comment, I think they might be after a tool like GNU screen or tmux. terminal multiplexers, like these, allow you to create a virtual terminal that you can disconnect from and reconnect to at need. these answers see https://askubuntu.com/a/220880/106239 and https://askubuntu.com/a/8657/106239 have examples for tmux and screen respectively

Writing to file and running the files sometimes works, mostly only first one

This code produces WindowsError most times, rarely (like often first time it is run) not.
""" Running hexlified codes from codefiles module prepared previously """
import tempfile
import subprocess
import threading
import os
import codefiles
if __name__ == '__main__':
for ind, c in enumerate(codefiles.exes):
fn = tempfile.mktemp() + '.exe'
# for some reason hexlified code is sometimes odd length and one nibble
if len(c) & 1:
c += '0'
c = c.decode('hex')
with open(fn, 'wb') as f:
f.write(c)
threading.Thread(target=lambda:subprocess.Popen("", executable=fn)).start()
""" One time works, one time WindowsError 32
>>>
132096 c:\docume~1\admin\locals~1\temp\tmpkhhxxo.exe
991232 c:\docume~1\admin\locals~1\temp\tmp9ow6zz.exe
>>> ================================ RESTART ================================
>>>
132096 c:\docume~1\admin\locals~1\temp\tmp3hb0cf.exe
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Documents and Settings\Admin\My Documents\Google Drive\Python\Tools\runner.pyw", line 18, in <lambda>
threading.Thread(target=lambda:subprocess.Popen("", executable=fn)).start()
File "C:\Python27\lib\subprocess.py", line 710, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process
991232 c:\docume~1\admin\locals~1\temp\tmpnkfuon.exe
>>>
"""
Hexlification is done with this script, and it sometimes seems to produce odd number of nibbles which seems also odd.
# bootstrapper for running hexlified executables,
# hexlification can be done by running this file directly or calling function make
# running can be done by run function in runner module, which imports the code,
# runs them from temporary files
import os
modulename = 'codefiles.py'
def code_file_names(d='.', ext='.exe'):
return [n for n in os.listdir(d)
if n.lower().endswith(ext)]
def make():
codes = code_file_names(os.curdir)
with open(modulename, 'a') as f:
f.write('exes = (')
hex_codes = [open(n, 'rb').read().encode('hex') for n in codes]
assert all(len(c) & 1 == 0 for c in hex_codes)
print len(hex_codes),map(len, hex_codes)
hex_codes = [repr(c) for c in hex_codes]
f.write(',\n'.join(hex_codes))
f.write(')\n')
if __name__ == '__main__':
import make_exe
# prepare hexlified exes for exes in directory if codefiles not prepared
if modulename not in os.listdir('.'):
try:
os.remove(modulename)
except:
pass
make()
# prepare script for py2_exe to execute the scripts by run from runner
make_exe.py2_exe('runner.pyw', 'tools/dvd.ico')
After Mr. Martelli's suggestion, I got the error disappear, but still not expected result.
I pass in new version of code the creation of exe file if it exists (saved file names in new creation routine). It launches both codes (two hexlified codes) when creating the files, but multiple copies of first code afterwards.

tempfile.mktemp(), besides being deprecated since many years because of its security problems, guarantees unique names only within a single run of a program, since, per https://docs.python.org/2/library/tempfile.html#tempfile.mktemp , "The module uses a global variable that tell it how to construct a temporary name". So on the second run, as the very clear error message tells you, "The process cannot access the file because it is being used by another process" (specifically, the process started by the previous run, i.e you cannot re-write an .exe that's currently running in some process).
The fix is to make sure each run uses its own unique directory for temporary files. See mkdtemp at https://docs.python.org/2/library/tempfile.html#tempfile.mkdtemp . How to eventually clean up those temporary directories is a different issue since that can't be done as long as any process is running an .exe file within such a directory -- you'll probably need a separate "clean-up script" that does what it can for the purpose, run periodically (e.g in Unix I'd use cron) and a repository (e.g a small sqlite DB, or even just a file) to record which of the temporary directories created in previous runs still exist and need to be cleared up (catching the exceptions seen when they can't get cleaned up yet, so as to retry in the future).