We Keep Coding

Creating a Dask client results in endless loop of errors

When I run this code:
from dask.distributed import Client
client = Client(n_workers = 2, threads_per_worker = 2, memory_limit = '2GB', silence_logs='error')
client
I get endless loop of these errors:
tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: None, threads: 2>>
Traceback (most recent call last):
File "C:\Users\szlau\anaconda3\lib\site-packages\tornado\ioloop.py", line 907, in _run
return self.callback()
File "C:\Users\szlau\anaconda3\lib\site-packages\distributed\nanny.py", line 414, in memory_monitor
process = self.process.process
Note: I have also tried reinstalling the whole Anaconda environment and the error still persists.
I really have no idea what can be causing this or how to eliminate this.
Any suggestions are welcomed!

At a guess, you are running your code in a script. That means that when dask creates the cluster and associated worker processes, those will, in turn, also runt he script and also attempt to create new clusters and processes.
You should guard your client and cluster creation with
if __name__ == "__main__":
client = Client(...)

Why does the iMessage program throw a database error?

I'm a beginner and I've tried creating a iMessage program that sends a single word from a song's lyrics until it reaches the end. Unfortunately, I have encountered an error that I don't understand a single bit. I have tried researching about the bug, but I only see posts about actual SQL, and I haven't used it before.
It only sends one message and then throws errors like these:
Traceback (most recent call last):
File "/Users/levs/OneDrive/python/python-projects/14/messagebot.py", line 21, in <module>
sendMessage("+44 7871579435", getLyrics())
File "/Users/levs/OneDrive/python/python-projects/14/messagebot.py", line 18, in sendMessage
imessage.send(phone, message)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/py_imessage/imessage.py", line 16, in send
db_conn.open()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/py_imessage/db_conn.py", line 21, in open
db = sqlite3.connect(db_path, uri=True)
sqlite3.OperationalError: unable to open database file
And this is the code:
from time import sleep
from py_imessage import imessage
def getLyrics():
with open('lyrics.txt') as file:
return ' '.join([str(line.strip()) for line in file])
def getWords(lyrics):
return lyrics.split()
def sendMessage(phone, lyrics):
for message in getWords(lyrics):
imessage.send(phone, message)
sendMessage("+44 1122334455", getLyrics())
I would appreciate any kind of help! :)

I did a bit of research on the py_imessage package. It works by accessing the iMessages database on your Mac. The error message you are seeing,
sqlite3.OperationalError: unable to open database file
basically means that your Mac is not allowing the program to access that database. To fix this issue, you will need to boot your Mac into Recovery Mode and run:
crsutil disable
in the terminal. This can be done by holding Command-R when restarting your Mac. Once it boots into Recovery mode, go to Utilites > Terminal, and run csrutil disable there. Then you can type reboot and boot back into normal MacOS.

Perforce (P4) Python API complains about too many locks

I wrote an application that opens several subprocesses, which initiate connections individually to a Perforce server. After a while I get this error message in almost all of these child-processes:
Traceback (most recent call last):
File "/Users/peter/Desktop/test_app/main.py", line 76, in p4_execute
p4.run_login()
File "/usr/local/lib/python3.7/site-packages/P4.py", line 665, in run_login
return self.run("login", *args, **kargs)
File "/usr/local/lib/python3.7/site-packages/P4.py", line 611, in run
raise e
File "/usr/local/lib/python3.7/site-packages/P4.py", line 605, in run
result = P4API.P4Adapter.run(self, *flatArgs)
P4.P4Exception: [P4#run] Errors during command execution( "p4 login" )
[Error]: "Fatal client error; disconnecting!
Operation 'client-SetPassword' failed.
Too many trys to get lock /Users/peter/.p4tickets.lck."
Does anyone have any idea what could cause this? I open my connections properly and all double checked on all source locations that I disconnect from the server properly via disconnect.
Only deleting the .p4tickets.lck manually works until the error comes back after a few seconds

The relevant code is here:
https://swarm.workshop.perforce.com/projects/perforce_software-p4/files/2018-1/support/ticket.cc#200
https://swarm.workshop.perforce.com/projects/perforce_software-p4/files/2018-1/sys/filetmp.cc#147
I can't see that there's any code path where the ticket.lck file would fail to get cleaned up without throwing some other error.
Is there anything unusual about the home directory where the tickets file lives? Like, say, it's on a network filer with some latency and some kind of backup process? Or maybe one that doesn't properly enforce file locks between all these subprocesses you're spawning?
How often are your scripts running "p4 login" to refresh and re-write the ticket? Many times a second? If you change them to not do that (e.g. only login if there's not already a ticket) does the problem persist?

How to use pymtp

I have been trying to get my GalaxyS3 to play nice with Ubuntu12.04 to no avail. I've tried a bunch of different apps that dont work and now I'm trying to make one that does.
Now I am told that Samsung did some 'bad magic' with it's new devices and that I need to talk to it via the MTP protocol which Ubuntu cant do by default for whatever reason.
I like python so I'm trying this out using pymtp.
My current problem is that pymtp's connect function is failing:
import pymtp
oMTP = pymtp.MTP()
oMTP.connect()
This yields:
Device 0 (VID=04e8 and PID=6860) is a Samsung GT-P7310/P7510/N7000/I9100/Galaxy Tab 7.7/10.1/S2/Nexus/Note.
PTP_ERROR_IO: failed to open session, trying again after resetting USB interface
LIBMTP libusb: Attempt to reset device
LIBMTP PANIC: failed to open session on second attempt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/pymodules/python2.7/pymtp.py", line 443, in connect
raise NoDeviceConnected
pymtp.NoDeviceConnected

Python script freezes infinitely during a socket connection

I have a simple python script that updates that statuses of justin.tv streams in my database. It's a Django based web application. This script worked perfectly before I moved it to my production server, but now it has issues with timing out or freezing. I've solved the time out problem by adding try/except blocks and making the script retry, but I still can't figure out the freezing problem.
I know it freezes on the line streamOnline = manager.getStreamOnline(stream.name, LOG). That's the same point where the socket.timeout exception occurs. Some times however, it just locks up for ever. I just can't picture a scenario where python would freeze infinitely. Here is the code for the script that freezes. I'm linking website.networkmanagers below, as well as oauth and the justin.tv python library that I'm using.
import sys, os, socket
LOG = False
def updateStreamInfo():
# Set necessary paths
honstreams = os.path.realpath(os.path.dirname(__file__) + "../../../")
sys.path.append(honstreams)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
# Import necessary moduels
from website.models import Stream, StreamInfo
from website.networkmanagers import get_manager, \
NetworkManagerReturnedErrorException
# Get all streams
streams = Stream.objects.all()
try:
# Loop through them
for stream in streams:
skipstream = False
print 'Checking %s...' % stream.name,
# Get the appropriate network manager and
manager = get_manager(stream.network.name)
# Try to get stream status up to 3 times
for i in xrange(3):
try:
streamOnline = manager.getStreamOnline(stream.name, LOG)
break
except socket.error as e:
code, message = e
# Retry up to 3 times
print 'Error: %s. Retrying...'
# If this stream should be skipped
if(skipstream):
print 'Can\'t connect! Skipping %s' % stream.name
continue
# Skip if status has not changed
if streamOnline == stream.online:
print 'Skipping %s because the status has not changed' % \
stream.name
continue
# Save status
stream.online = streamOnline
stream.save()
print 'Set %s to %s' % (stream.name, streamOnline)
except NetworkManagerReturnedErrorException as e:
print 'Stopped the status update loop:', e
if(__name__ == "__main__"):
if(len(sys.argv) > 1 and sys.argv[1] == "log"):
LOG = True
if(LOG): print "Logging enabled"
updateStreamInfo()
networkmanagers.py
oauth.py
JtvClient.py
Example of the script freezing
foo#bar:/.../honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie... Skipping angrytestie because the status has not changed
Checking chustream... Skipping chustream because the status has not changed
Checking cilantrogamer... Skipping cilantrogamer because the status has not changed
| <- caret sits here blinking infinitely
Interesting update
Every time it freezes and I send a keyboard interrupt, it's on the same line in socket.py:
root#husta:/home/honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie... Skipping angrytestie because the status has not changed
Checking chustream... Skipping chustream because the status has not changed
^CChecking cilantrogamer...
Traceback (most recent call last):
File "website/scripts/updateStreamStatus.py", line 64, in <module>
updateStreamInfo()
File "website/scripts/updateStreamStatus.py", line 31, in updateStreamInfo
streamOnline = manager.getStreamOnline(stream.name, LOG)
File "/home/honstreams/honstreams/website/networkmanagers.py", line 47, in getStreamOnline
return self.getChannelLive(channelName, log)
File "/home/honstreams/honstreams/website/networkmanagers.py", line 65, in getChannelLive
response = client.get('/stream/list.json?channel=%s' % channelName)
File "/home/honstreams/honstreams/website/JtvClient.py", line 51, in get
return self._send_request(request, token)
File "/home/honstreams/honstreams/website/JtvClient.py", line 90, in _send_request
return conn.getresponse()
File "/usr/lib/python2.6/httplib.py", line 986, in getresponse
response.begin()
File "/usr/lib/python2.6/httplib.py", line 391, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.6/socket.py", line 397, in readline
data = recv(1)
KeyboardInterrupt
Any thoughts?

Have you tried using another application to open that connection? Given that it's an issue in production, perhaps you don't have some firewall issues.

Down in JtvClient.py it uses httplib to handle the connection. Have you tried changing this to use httplib2 instead?
Other than that stab in the dark, I would add a lot of logging statements to this code in order to track what actually happens and where it gets stuck. Then I would make sure that the point where it gets stuck can timeout on the socket (which usually involves either monkeypatching or forking the codebase) so that stuff fails instead of hanging.
You said:
I know it freezes on the line streamOnline = manager.getStreamOnline(stream.name, LOG). That's the same point where the socket.timeout exception occurs.
Wrong. It doesn't freeze on that line because that line is a function call which calls lots of other functions through several levels of other modules. So you do not yet know where the program freezes. Also, that line is NOT the point where the socket timeout occurs. The socket timeout will only occur on a low level socket operation like select or recv which is being called several times in the chain of activity triggered by getStreamOnline.
You need to trace your code in a debugger or add print statements to track down exactly where the hang occurs. It could possibly be an infinite loop in Python but is more likely to be a low-level call to an OS networking function. Until you find the source of the error, you can't do anything.
P.S. the keyboard interrupt is a reasonable clue that the problem is around line 90 in JtvClient.py, so put in some print statements and find out what happens. There may be a stupid loop in there that keeps calling getresponse, or you may be calling it with bad parameters or maybe the network server really is borked. Narrow it down to fewer possibilities.

It turns out this HTTP connection isn't passed a timeout in jtvClient.py
def _get_conn(self):
return httplib.HTTPConnection("%s:%d" % (self.host, self.port))
Changed the last line to
return httplib.HTTPConnection("%s:%d" % (self.host, self.port), timeout=10)
Which solved it