Python 3 Multiprocessing and Sockets - python

I have a client script that communicates with multiple servers. It appears from my testing and research that python's socket library is synchronous. No matter what I do, single socket variable or dynamic socket variables, the script will not send and receive data at the same time. Here's the basic version of my script using dynamic socket ids:
import socket
import logging
import os
import time
from time import time
import datetime
import multiprocessing
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logging.getLogger('requests').setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)
IP=[0,1,2,3,4,5]
IP[0] = '192.168.1.21'
IP[1] = '192.168.1.22'
IP[2] = '192.168.1.23'
IP[3] = '192.168.1.24'
IP[4] = '192.168.1.25'
IP[5] = '192.168.1.26'
PORT = 1292
def ClientWorker(ip, port, socketid):
name = multiprocessing.current_process().name
ts = time()
s = [0,1,2,3,4,5]
s[socketid] = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s[socketid].connect((ip, port))
s[socketid].send(b"Get status")
response = s[socketid].recv(50)
s[socketid].close()
print('Connection to {}:'.format(ip))
print('Process Took: {}'.format(time() - ts))
def main():
tsmain = time()
socketnum = 0
workers = []
for ipaddr in IP:
logger.info('Queueing {}'.format(ipaddr))
worker = multiprocessing.Process(target=ClientWorker, args=(ipaddr, PORT, socketnum))
workers.append(worker)
worker.start()
socketnum += 1
for w in workers:
w.join()
print('Main Took {}'.format(time() - tsmain))
if __name__ == '__main__':
main()
Multiprocessing output:
2016-07-06 10:32:17,178 - __main__ - INFO - Starting 192.168.1.21
2016-07-06 10:32:17,181 - __main__ - INFO - Starting 192.168.1.22
2016-07-06 10:32:17,181 - __main__ - INFO - Starting 192.168.1.23
2016-07-06 10:32:17,183 - __main__ - INFO - Starting 192.168.1.24
2016-07-06 10:32:17,184 - __main__ - INFO - Starting 192.168.1.25
2016-07-06 10:32:17,185 - __main__ - INFO - Starting 192.168.1.26
Connection to 192.168.1.22:
Process Took 0.36358118057250977
Connection to 192.168.1.25:
Process Took 0.37239646911621094
Connection to 192.168.1.21:
Process Took 0.3772423267364502
Connection to 192.168.1.26:
Process Took 0.4828777313232422
Connection to 192.168.1.24:
Process Took 0.6024985313415527
Connection to 192.168.1.23:
Process Took 0.7020003795623779
Main Took 0.708820104598999
It looks like all the processes are starting at pretty much the same time but each process takes longer to complete, which suggests only one socket can be open at once.
I've done research on twisted, asyncio & asyncore. I started going down the twisted road but I'm having trouble wrapping my head around that because the messages sent to each server will not be the same. The script will be designed to send different tasks to each server at the same time and wait for the results.
Edit
More detailed output:
(0, 0) = Process-1 Work started: 2016-07-06 15:42:02.227847
(0, 1) = Process-2 Work started: 2016-07-06 15:42:02.228606
(0, 2) = Process-3 Work started: 2016-07-06 15:42:02.229448
(0, 3) = Process-4 Work started: 2016-07-06 15:42:02.231957
(0, 4) = Process-5 Work started: 2016-07-06 15:42:02.232951
(0, 5) = Process-6 Work started: 2016-07-06 15:42:02.233697
(1, 0) = Process-1 Setting up network socket 0 # 2016-07-06 15:42:02.227894
(1, 1) = Process-2 Setting up network socket 1 # 2016-07-06 15:42:02.228655
(1, 2) = Process-3 Setting up network socket 2 # 2016-07-06 15:42:02.229498
(1, 3) = Process-4 Setting up network socket 3 # 2016-07-06 15:42:02.232006
(1, 4) = Process-5 Setting up network socket 4 # 2016-07-06 15:42:02.232994
(1, 5) = Process-6 Setting up network socket 5 # 2016-07-06 15:42:02.233742
(2, 0) = Process-1 Completed network socket setup 0 # 2016-07-06 15:42:02.228005 - took 0:00:00.000121
(2, 1) = Process-2 Completed network socket setup 1 # 2016-07-06 15:42:02.228781 - took 0:00:00.000137
(2, 2) = Process-3 Completed network socket setup 2 # 2016-07-06 15:42:02.229608 - took 0:00:00.000120
(2, 3) = Process-4 Completed network socket setup 3 # 2016-07-06 15:42:02.232109 - took 0:00:00.000112
(2, 4) = Process-5 Completed network socket setup 4 # 2016-07-06 15:42:02.233081 - took 0:00:00.000092
(2, 5) = Process-6 Completed network socket setup 5 # 2016-07-06 15:42:02.233854 - took 0:00:00.000122
(3, 0) = Process-1 Connecting to network socket # 2016-07-06 15:42:02.228046
(3, 1) = Process-2 Connecting to network socket # 2016-07-06 15:42:02.228823
(3, 2) = Process-3 Connecting to network socket # 2016-07-06 15:42:02.229647
(3, 3) = Process-4 Connecting to network socket # 2016-07-06 15:42:02.232145
(3, 4) = Process-5 Connecting to network socket # 2016-07-06 15:42:02.233109
(3, 5) = Process-6 Connecting to network socket # 2016-07-06 15:42:02.233893
(4, 0) = Process-1 Connected to network socket # 2016-07-06 15:42:02.232009 - took 0:00:00.003983
(4, 1) = Process-2 Connected to network socket # 2016-07-06 15:42:02.232624 - took 0:00:00.003818
(4, 2) = Process-3 Connected to network socket # 2016-07-06 15:42:02.233706 - took 0:00:00.004080
(4, 3) = Process-4 Connected to network socket # 2016-07-06 15:42:02.235946 - took 0:00:00.003827
(4, 4) = Process-5 Connected to network socket # 2016-07-06 15:42:02.236215 - took 0:00:00.003120
(4, 5) = Process-6 Connected to network socket # 2016-07-06 15:42:02.237408 - took 0:00:00.003530
(5, 0) = Process-1 Beginning to send data to network socket # 2016-07-06 15:42:02.228046
(5, 1) = Process-2 Beginning to send data to network socket # 2016-07-06 15:42:02.228823
(5, 2) = Process-3 Beginning to send data to network socket # 2016-07-06 15:42:02.229647
(5, 3) = Process-4 Beginning to send data to network socket # 2016-07-06 15:42:02.232145
(5, 4) = Process-5 Beginning to send data to network socket # 2016-07-06 15:42:02.233109
(5, 5) = Process-6 Beginning to send data to network socket # 2016-07-06 15:42:02.233893
(6, 0) = Process-1 Completed data send to network socket # 2016-07-06 15:42:02.232084 - took 0:00:00.000038
(6, 1) = Process-2 Completed data send to network socket # 2016-07-06 15:42:02.232699 - took 0:00:00.000039
(6, 2) = Process-3 Completed data send to network socket # 2016-07-06 15:42:02.233788 - took 0:00:00.000044
(6, 3) = Process-4 Completed data send to network socket # 2016-07-06 15:42:02.236024 - took 0:00:00.000035
(6, 4) = Process-5 Completed data send to network socket # 2016-07-06 15:42:02.236263 - took 0:00:00.000023
(6, 5) = Process-6 Completed data send to network socket # 2016-07-06 15:42:02.237456 - took 0:00:00.000022
(7, 0) = Process-1 Beginning to wait for data from network socket # 2016-07-06 15:42:02.232102
(7, 1) = Process-2 Beginning to wait for data from network socket # 2016-07-06 15:42:02.232717
(7, 2) = Process-3 Beginning to wait for data from network socket # 2016-07-06 15:42:02.233809
(7, 3) = Process-4 Beginning to wait for data from network socket # 2016-07-06 15:42:02.236041
(7, 4) = Process-5 Beginning to wait for data from network socket # 2016-07-06 15:42:02.236274
(7, 5) = Process-6 Beginning to wait for data from network socket # 2016-07-06 15:42:02.237467
(8, 0) = Process-1 Received data from network socket # 2016-07-06 15:42:02.273043 - took 0:00:00.040980
(8, 1) = Process-2 Received data from network socket # 2016-07-06 15:42:02.277420 - took 0:00:00.044757
(8, 2) = Process-3 Received data from network socket # 2016-07-06 15:42:02.267754 - took 0:00:00.033984
(8, 3) = Process-4 Received data from network socket # 2016-07-06 15:42:02.281372 - took 0:00:00.045383
(8, 4) = Process-5 Received data from network socket # 2016-07-06 15:42:02.278337 - took 0:00:00.042100
(8, 5) = Process-6 Received data from network socket # 2016-07-06 15:42:03.197792 - took 0:00:00.960364
(9, 0) = Process-1 Beginning to process response from server # 2016-07-06 15:42:02.273101
(9, 1) = Process-2 Beginning to process response from server # 2016-07-06 15:42:02.277494
(9, 2) = Process-3 Beginning to process response from server # 2016-07-06 15:42:02.267812
(9, 3) = Process-4 Beginning to process response from server # 2016-07-06 15:42:02.281444
(9, 4) = Process-5 Beginning to process response from server # 2016-07-06 15:42:02.278390
(9, 5) = Process-6 Beginning to process response from server # 2016-07-06 15:42:03.197861
(10, 0) = Process-1 Finished processing response from server # 2016-07-06 15:42:02.273140 - took 0:00:00.000048
(10, 1) = Process-2 Finished processing response from server # 2016-07-06 15:42:02.277529 - took 0:00:00.000043
(10, 2) = Process-3 Finished processing response from server # 2016-07-06 15:42:02.267847 - took 0:00:00.000045
(10, 3) = Process-4 Finished processing response from server # 2016-07-06 15:42:02.281475 - took 0:00:00.000038
(10, 4) = Process-5 Finished processing response from server # 2016-07-06 15:42:02.278418 - took 0:00:00.000035
(10, 5) = Process-6 Finished processing response from server # 2016-07-06 15:42:03.197889 - took 0:00:00.000035
(11, 0) = Process-1 Communicating with server to retreive additional information # 2016-07-06 15:42:02.273156
(11, 1) = Process-2 Communicating with server to retreive additional information # 2016-07-06 15:42:02.277544
(11, 2) = Process-3 Communicating with server to retreive additional information # 2016-07-06 15:42:02.267865
(11, 3) = Process-4 Communicating with server to retreive additional information # 2016-07-06 15:42:02.281488
(11, 4) = Process-5 Communicating with server to retreive additional information # 2016-07-06 15:42:02.278432
(11, 5) = Process-6 Communicating with server to retreive additional information # 2016-07-06 15:42:03.197903
(12, 0) = Process-1 Finished final retreival of data # 2016-07-06 15:42:02.848586 - took 0:00:00.575468
(12, 1) = Process-2 Finished final retreival of data # 2016-07-06 15:42:03.637331 - took 0:00:01.359825
(12, 2) = Process-3 Finished final retreival of data # 2016-07-06 15:42:02.765458 - took 0:00:00.497645
(12, 3) = Process-4 Finished final retreival of data # 2016-07-06 15:42:03.541048 - took 0:00:01.259597
(12, 4) = Process-5 Finished final retreival of data # 2016-07-06 15:42:02.818199 - took 0:00:00.539810
(12, 5) = Process-6 Finished final retreival of data # 2016-07-06 15:42:03.526207 - took 0:00:00.328342
(13, 0) = Process-1 Closing network socket 0 # 2016-07-06 15:42:02.848643
(13, 1) = Process-2 Closing network socket 1 # 2016-07-06 15:42:03.637387
(13, 2) = Process-3 Closing network socket 2 # 2016-07-06 15:42:02.765536
(13, 3) = Process-4 Closing network socket 3 # 2016-07-06 15:42:03.541101
(13, 4) = Process-5 Closing network socket 4 # 2016-07-06 15:42:02.818264
(13, 5) = Process-6 Closing network socket 5 # 2016-07-06 15:42:03.526262
(14, 0) = Process-1 Closed network socket 0 # 2016-07-06 15:42:02.848719 - took 0:00:00.000086
(14, 1) = Process-2 Closed network socket 1 # 2016-07-06 15:42:03.637463 - took 0:00:00.000086
(14, 2) = Process-3 Closed network socket 2 # 2016-07-06 15:42:02.765647 - took 0:00:00.000128
(14, 3) = Process-4 Closed network socket 3 # 2016-07-06 15:42:03.541168 - took 0:00:00.000075
(14, 4) = Process-5 Closed network socket 4 # 2016-07-06 15:42:02.818365 - took 0:00:00.000114
(14, 5) = Process-6 Closed network socket 5 # 2016-07-06 15:42:03.526333 - took 0:00:00.000080
(15, 0) = Process-1 Took 0:00:00.620923
(15, 1) = Process-2 Took 0:00:01.408908
(15, 2) = Process-3 Took 0:00:00.536262
(15, 3) = Process-4 Took 0:00:01.309261
(15, 4) = Process-5 Took 0:00:00.585469
(15, 5) = Process-6 Took 0:00:01.292729
(16, 0) = Process-1 Work finished: 2016-07-06 15:42:02.848763
(16, 1) = Process-2 Work finished: 2016-07-06 15:42:03.637487
(16, 2) = Process-3 Work finished: 2016-07-06 15:42:02.765685
(16, 3) = Process-4 Work finished: 2016-07-06 15:42:03.541191
(16, 4) = Process-5 Work finished: 2016-07-06 15:42:02.818397
(16, 5) = Process-6 Work finished: 2016-07-06 15:42:03.526357
Main Took 0:00:01.430665

I did indeed find out that my original code is good and that the discrepancies in timing were the result of the end machines being busy with other tasks or just slow to respond in general.

Related

Why the id of Queue in python mp.Manager is not the same?

**I print the id(Q) in calcul(), but the output is not same in different process, Why? **
Shouldn't the message queue be unique, so why would the output ids be different?
Code
from multiprocessing import Pool, Manager, freeze_support, Lock, Queue
class Model(object):
def __init__(self, id=0):
self.id = id
def act(self, s):
return len(str(s)) - len(str(self.id))
def init(lock1, lock2, lock3):
global model_lock_, text_lock_, rst_lock_
model_lock_ = lock1
text_lock_ = lock2
rst_lock_ = lock3
def calcul(s):
s, Q = s
request = Q.get(True)
rst = request.id
print((s, rst, os.getpid(), id(request), id(Q)))
time.sleep(1)
Q.put(request)
return (s, rst)
if __name__ == "__main__":
ptime = time.time()
freeze_support()
Q = Manager().Queue(5)
for i in range(5):
Q.put(Model(i))
params = [(i, Q) for i in range(100)]
p = Pool(processes=3)
rst = p.map(calcul, params)
Output:
(0, 0, 28836, 2421423514192, 2421423266352)
(9, 1, 26440, 1942850478672, 1942850230832)
(18, 2, 14888, 2433320739408, 2433320479280)
(1, 3, 28836, 2421423517504, 2421423266352)
(10, 0, 26440, 1942850481984, 1942850230832)
(19, 4, 14888, 2433320742720, 2433320479280)
(2, 1, 28836, 2421423514192, 2421423266352)
(20, 2, 14888, 2433320739408, 2433320479280)
(11, 3, 26440, 1942850478672, 1942850230832)
(3, 4, 28836, 2421423517504, 2421423266352)(21, 0, 14888, 2433320742720, 2433320479280)
(12, 1, 26440, 1942850481984, 1942850230832)
(22, 2, 14888, 2433320739408, 2433320479280)(4, 3, 28836, 2421423514192, 2421423266352)
(13, 4, 26440, 1942850478672, 1942850230832)
(5, 0, 28836, 2421423517504, 2421423266352)(14, 1, 26440, 1942850481984, 1942850230832)
(23, 3, 14888, 2433320742720, 2433320479280)
Can someone teach me:
Why the Queue id is not unique, or where I wrote wrong?

how to obtain SeDebugPrivilege in Python?

How does one obtain SeDebugPrivilege in Python? I believe both the Ctypes api and PyWin32 are both individually capable of doing it, i don't care which API is used. I found this broken code in the wild, it's probably pretty close
import win32api
import win32con
import win32security
#
def get_extra_privs():
# Try to give ourselves some extra privs (only works if we're admin):
# SeBackupPrivilege - so we can read anything
# SeDebugPrivilege - so we can find out about other processes (otherwise OpenProcess will fail for some)
# SeSecurityPrivilege - ??? what does this do?
# Problem: Vista+ support "Protected" processes, e.g. audiodg.exe. We can't see info about these.
# Interesting post on why Protected Process aren't really secure anyway: http://www.alex-ionescu.com/?p=34
th = win32security.OpenProcessToken(win32api.GetCurrentProcess(), win32con.TOKEN_ADJUST_PRIVILEGES | win32con.TOKEN_QUERY)
TokenPrivileges = 1
privs = win32security.GetTokenInformation(th, TokenPrivileges)
newprivs = []
for privtuple in privs:
if privtuple[0] == win32security.LookupPrivilegeValue(remote_server, "SeBackupPrivilege") or privtuple[0] == win32security.LookupPrivilegeValue(remote_server, "SeDebugPrivilege") or privtuple[0] == win32security.LookupPrivilegeValue(remote_server, "SeSecurityPrivilege"):
print ("Added privilege " + str(privtuple[0]))
# privtuple[1] = 2 # tuples are immutable. WHY?!
newprivs.append((privtuple[0], 2)) # SE_PRIVILEGE_ENABLED
else:
newprivs.append((privtuple[0], privtuple[1]))
# Adjust privs
privs = tuple(newprivs)
str(win32security.AdjustTokenPrivileges(th, False , privs))
but it crashes with
File "C:\cygwin64\home\hans\tibia\pybot\pybot.py", line 11, in init
processStuff.get_extra_privs()
File "C:\cygwin64\home\hans\tibia\pybot\processStuff.py", line 20, in get_extra_privs
if privtuple[0] == win32security.LookupPrivilegeValue(remote_server, "SeBackupPrivilege") or privtuple[0] == win32security.LookupPrivilegeValue(remote_server
, "SeDebugPrivilege") or privtuple[0] == win32security.LookupPrivilegeValue(remote_server, "SeSecurityPrivilege"):
TypeError: 'PySID' object is not subscriptable
Problems:
Wrong TokenPrivileges value: it should be 3, as per [MS.Docs]: TOKEN_INFORMATION_CLASS enumeration (winnt.h). Use the value exported by PyWin32
remote_server undefined
LookupPrivilegeValue being called 3 times for each existing privilege (minor optimization)
codee00.py:
#!/usr/bin/env python
import sys
import win32api as wapi
import win32con as wcon
import win32security as wsec
PRIV_NAMES = (
wsec.SE_BACKUP_NAME,
wsec.SE_DEBUG_NAME,
wsec.SE_SECURITY_NAME,
)
def enable_privs(remote_server=None, priv_names=PRIV_NAMES):
priv_ids = sorted(wsec.LookupPrivilegeValue(remote_server, e) for e in priv_names)
print("Privileges to be enabled IDs:", priv_ids)
tok = wsec.OpenProcessToken(wapi.GetCurrentProcess(), wcon.TOKEN_ADJUST_PRIVILEGES | wcon.TOKEN_QUERY)
proc_privs = wsec.GetTokenInformation(tok, wsec.TokenPrivileges)
print("Existing process privileges:", proc_privs)
new_proc_privs = []
need_change = False
for proc_priv in proc_privs:
if proc_priv[0] in priv_ids:
print("Checking privilege " + str(proc_priv[0]))
if proc_priv[1] != wcon.SE_PRIVILEGE_ENABLED:
need_change = True
new_proc_privs.append((proc_priv[0], wcon.SE_PRIVILEGE_ENABLED))
else:
new_proc_privs.append(proc_priv)
print("New process privileges:", new_proc_privs)
if need_change:
modif_privs = wsec.AdjustTokenPrivileges(tok, False, new_proc_privs)
res = wapi.GetLastError()
print("Changed privileges:", modif_privs) # Changed ones
if res != 0:
print("Error (partial) setting privileges:", res)
else:
print("Already set")
#wsec.GetTokenInformation(tok, wsec.TokenPrivileges) # To compare with proc_privs
wapi.CloseHandle(tok)
def main(*argv):
enable_privs()
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.")
sys.exit(rc)
Output:
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q072193556]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" code00.py
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32
Privileges to be added IDs: [17, 20, 8]
Existing process privileges: ((2, 0), (3, 0), (4, 0), (5, 0), (7, 0), (8, 2), (9, 0), (10, 0), (11, 0), (12, 0), (13, 0), (14, 0), (15, 0), (16, 0), (17, 0), (18, 0), (19, 0), (20, 0), (21, 0), (22, 0), (23, 3), (24, 0), (25, 0), (26, 0), (28, 0), (29, 3), (30, 3), (31, 0), (32, 0), (33, 0), (34, 0), (35, 0), (36, 0))
Checking privilege 8
Checking privilege 17
Checking privilege 20
New process privileges: [(2, 0), (3, 0), (4, 0), (5, 0), (7, 0), (8, 2), (9, 0), (10, 0), (11, 0), (12, 0), (13, 0), (14, 0), (15, 0), (16, 0), (17, 2), (18, 0), (19, 0), (20, 2), (21, 0), (22, 0), (23, 3), (24, 0), (25, 0), (26, 0), (28, 0), (29, 3), (30, 3), (31, 0), (32, 0), (33, 0), (34, 0), (35, 0), (36, 0)]
Changed privileges: ((17, 0), (20, 0))
Done
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q072193556]> :: Attempt running as non Administrator user
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q072193556]> runas /user:test "cmd /k e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe %CD%\code00.py"
Enter the password for test:
Attempting to start cmd /k e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe e:\Work\Dev\StackOverflow\q072193556\code00.py as user "CFATI-5510-0\test" ...
And the newly launched terminal:
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32
Privileges to be enabled IDs: [8, 17, 20]
Existing process privileges: ((19, 0), (23, 3), (25, 0), (33, 0), (34, 0))
New process privileges: [(19, 0), (23, 3), (25, 0), (33, 0), (34, 0)]
Already set
Done.
As for the final goal (adding extra new privileges - which is not clearly stated in the question):
It can't be done, which makes perfect sense, as one non privileged process being able to grant itself (administrative) privileges, would beat the purpose of the whole Windows security (privileges and rights) mechanism
Even if one would try adding new privilege entries to the existing list (which automatically implies that current code has design flaws), AdjustTokenPrivileges would (silently) fail with ERROR_NOT_ALL_ASSIGNED (1300). For more details, check [MS.Docs]: Changing Privileges in a Token which states (emphasis is mine):
AdjustTokenPrivileges cannot add or remove privileges from the token. It can only enable existing privileges that are currently disabled or disable existing privileges that are currently enabled.

MicroPython: Scheduling

I am very inexperienced with Python so bear with me. I am trying to set a schedule in a RaspiPico to turn a RaspPi 3B+ on at given times. The RaspPi 3B+ turns itself off at the 57th minute of every hour, the Pico is supposed to turn it on at the 24th minute of every hour. I've worked out the code so that when I manually run it through Thonny, it turns the RaspPi 3B+ on and doesnt give any error messages anymore, but when left on its own it doesn't appear to be following a schedule. To clarify there is also an RTC included in this Raspi Sandwich. Here is my code:
from machine import Pin, Timer, RTC, I2C
import time
import binascii
# I2C PINSs
I2C_PORT = 0
I2C_SDA = 20
I2C_SCL = 21
# Times when the B+ should be booted up
# Format is (hours, minutes)
# For now, keep it sorted; we could do this in code, but why not just do it by hand
SCHEDULE = [(0, 24, 0), (0, 56, 0), (1, 24, 0), (1, 56, 0), (2, 24, 0), (2, 56, 0), (3, 24, 0), (3, 56, 0), (4, 24, 0), (4, 56, 0), (5, 24, 0), (5, 56, 0), (6, 24, 0), (6, 56, 0), (7, 24, 0), (7, 56, 0), (8, 24, 0), (8, 56, 0), (9, 24, 0), (9, 56, 0)
, (10, 24, 0), (10, 56, 0), (11, 24, 0), (11, 56, 0), (12, 24, 0), (12, 56, 0), (13, 24, 0), (13, 56, 0), (14, 24, 0), (14, 56, 0), (15, 24, 0), (15, 56, 0), (16, 24, 0), (16, 56, 0), (17, 24, 0), (17, 56, 0), (18, 24, 0), (18, 56, 0), (19, 24, 0), (19, 56, 0),
(20, 24, 0), (20, 56, 0), (21, 24, 0), (21, 56, 0), (22, 24, 0), (22, 56, 0), (23, 24, 0), (23, 56, 0)]
# This function returns the next scheduled boot time
# after the hour, minute we provide
def grab_next_time(time_tuple):
current_hour, current_min, current_sec = time_tuple
# Go through all scheudled boot times from first to last
for h, m, s in SCHEDULE:
# If current hour is before hour, we found the next time
if current_hour < h:
return (h,m, s)
# If its the same hour
elif current_hour == h:
# If its before the minute, we found the next time
if current_min < m:
return (h,m,s)
# if its the same minute
elif current_min == m:
# If its before the second, we found the next time
if current_sec < s:
return (h,m,s)
# If the current hour is not the same or before, lets try the next
# We tried all times in the schedule,
# we're after, so lets start from the begining
return SCHEDULE[0]
# This function returns the time we need to wait in seconds
def wait_time(current_time, next_time):
next_hour, next_min, next_sec = next_time[0], next_time[1], next_time[2]
current_hour, current_min, current_sec = current_time[0], current_time[1], current_time[2]
wait_time = (next_sec - current_sec)
if (next_sec - current_sec) < 0:
wait_time += 60
next_min -= 1
wait_time += (next_min - current_min) * 60
if (next_min - current_min) < 0:
wait_time += (60 * 60)
next_hour -= 1
wait_time += (next_hour - current_hour) * 60 * 60
if (next_hour - current_hour) < 0:
wait_time += (60 * 60 * 24)
return wait_time
def start_local_rtc(date=(2017, 8, 23, 2, 12, 48, 0, 0)):
rtc = machine.RTC()
rtc.datetime(date)
return rtc
class ds3231(object):
NowTime = b'\x00\x45\x13\x02\x24\x05\x21'
w = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"];
address = 0x68
start_reg = 0x00
alarm1_reg = 0x07
control_reg = 0x0e
status_reg = 0x0f
def __init__(self,i2c_port,i2c_scl,i2c_sda):
self.bus = I2C(i2c_port,scl=Pin(i2c_scl),sda=Pin(i2c_sda))
def read_time(self):
t = self.bus.readfrom_mem(int(self.address),int(self.start_reg),7)
seconds = (t[0]&0x0F) + 10 * ((t[0]>>4)&0x07)
minutes = (t[1]&0x0F) + 10 * ((t[1]>>4)&0x07)
hour = (t[2]&0x0F) + 20 * ((t[2]>>4)&0x02) #hour ( 11 1111)
dow = t[3]&0x07 #week
day = (t[4]&0x0F) + 10 * ((t[4]>>4)&0x03) #day
month = (t[5]&0x1F) + 10 * ((t[5]>>4)&0x01) #month
year = 2000 + ((t[6]&0x0F) + 10 * ((t[6]>>4)&0x0F)) # year
#(year, month, day, weekday, hours, minutes, seconds, subseconds)
return (year, month, day, hour, minutes, seconds, 0, 0)
#print("20%x/%02x/%02x %02x:%02x:%02x %s" %(t[6],t[5],t[4],t[2],t[1],t[0],self.w[t[3]-1]))
# For a second, override the start pin
def start_pi():
boot = Pin(21, Pin.OUT)
boot.value(0)
time.sleep(0.1) # 100 ms should do
boot = Pin(21, Pin.IN) # Setting the pin to IN should make it float
if __name__ == '__main__':
# On startup grab the time from the RTC
hw_rtc = ds3231(I2C_PORT, I2C_SCL, I2C_SDA)
time_rtc = hw_rtc.read_time()
print(time_rtc)
local_rtc = start_local_rtc()
# This is our infinite loop
while True:
current_time = local_rtc.datetime()
current_time = (current_time[4], current_time[5], current_time[6])
print(current_time)
next_time = grab_next_time(current_time)
print("Next time: ", next_time)
wait_time = wait_time(current_time, next_time)
print("Wait time: ", wait_time)
# If within 30 seconds, start the pi
if wait_time < 30:
start_pi()
# If we need to wait for more than 30 seconds, keep waiting
else:
time.sleep(wait_time - 5)
The person I am working with mentioned that the Pico will "poke" the Pi and not stop poking until the specified time. hence why my schedule is listed as "(0,24,0), (0,56,0)". In my head this is supposed to start the "poking" at the 24th minute after midnight and stop at the 56th minute. As was stated before this obviously isn't what is going on. Some help would be appreciated.

Flexible jobshop problem with sequence dependent setup times

How do I add setup time depending on the sequence to the flexible job shop scheduling optimization problem. The code I have is as follows:
import collections
from ortools.sat.python import cp_model
class SolutionPrinter(cp_model.CpSolverSolutionCallback):
"""Print intermediate solutions."""
def __init__(self):
cp_model.CpSolverSolutionCallback.__init__(self)
self.__solution_count = 0
def on_solution_callback(self):
"""Called at each new solution."""
print('Solution %i, time = %f s, objective = %i' %
(self.__solution_count, self.WallTime(), self.ObjectiveValue()))
self.__solution_count += 1
def flexible_jobshop():
# Data part.
jobs = [ # task = (processing_time, machine_id)
[ # Job 0
[(3, 0), (1, 1), (5, 2)], # task 0 with 3 alternatives
[(2, 0), (4, 1), (6, 2)], # task 1 with 3 alternatives
[(2, 0), (3, 1), (1, 2)], # task 2 with 3 alternatives
],
[ # Job 1
[(2, 0), (3, 1), (4, 2)],
[(1, 0), (5, 1), (4, 2)],
],
[ # Job 2
[(2, 0), (1, 1), (4, 2)],
[(2, 0), (3, 1), (4, 2)],
[(3, 0), (1, 1), (5, 2)],
],
]
num_jobs = len(jobs)
all_jobs = range(num_jobs)
num_machines = 3
all_machines = range(num_machines)
# Model the flexible jobshop problem.
model = cp_model.CpModel()
horizon = 0
for job in jobs:
for task in job:
max_task_duration = 0
for alternative in task:
max_task_duration = max(max_task_duration, alternative[0])
horizon += max_task_duration
print('Horizon = %i' % horizon)
# Global storage of variables.
intervals_per_resources = collections.defaultdict(list)
starts = {} # indexed by (job_id, task_id).
presences = {} # indexed by (job_id, task_id, alt_id).
job_ends = []
# Scan the jobs and create the relevant variables and intervals.
for job_id in all_jobs:
job = jobs[job_id]
num_tasks = len(job)
previous_end = None
for task_id in range(num_tasks):
task = job[task_id]
min_duration = task[0][0]
max_duration = task[0][0]
num_alternatives = len(task)
all_alternatives = range(num_alternatives)
for alt_id in range(1, num_alternatives):
alt_duration = task[alt_id][0]
min_duration = min(min_duration, alt_duration)
max_duration = max(max_duration, alt_duration)
# Create main interval for the task.
suffix_name = '_j%i_t%i' % (job_id, task_id)
start = model.NewIntVar(0, horizon, 'start' + suffix_name)
duration = model.NewIntVar(min_duration, max_duration,
'duration' + suffix_name)
end = model.NewIntVar(0, horizon, 'end' + suffix_name)
interval = model.NewIntervalVar(start, duration, end,
'interval' + suffix_name)
# Store the start for the solution.
starts[(job_id, task_id)] = start
# Add precedence with previous task in the same job.
if previous_end is not None:
model.Add(start >= previous_end)
previous_end = end
# Create alternative intervals.
if num_alternatives > 1:
l_presences = []
for alt_id in all_alternatives:
alt_suffix = '_j%i_t%i_a%i' % (job_id, task_id, alt_id)
l_presence = model.NewBoolVar('presence' + alt_suffix)
l_start = model.NewIntVar(0, horizon, 'start' + alt_suffix)
l_duration = task[alt_id][0]
l_end = model.NewIntVar(0, horizon, 'end' + alt_suffix)
l_interval = model.NewOptionalIntervalVar(
l_start, l_duration, l_end, l_presence,
'interval' + alt_suffix)
l_presences.append(l_presence)
# Link the master variables with the local ones.
model.Add(start == l_start).OnlyEnforceIf(l_presence)
model.Add(duration == l_duration).OnlyEnforceIf(l_presence)
model.Add(end == l_end).OnlyEnforceIf(l_presence)
# Add the local interval to the right machine.
intervals_per_resources[task[alt_id][1]].append(l_interval)
# Store the presences for the solution.
presences[(job_id, task_id, alt_id)] = l_presence
# Select exactly one presence variable.
model.Add(sum(l_presences) == 1)
else:
intervals_per_resources[task[0][1]].append(interval)
presences[(job_id, task_id, 0)] = model.NewConstant(1)
job_ends.append(previous_end)
# Create machines constraints.
for machine_id in all_machines:
intervals = intervals_per_resources[machine_id]
if len(intervals) > 1:
model.AddNoOverlap(intervals)
# Makespan objective
makespan = model.NewIntVar(0, horizon, 'makespan')
model.AddMaxEquality(makespan, job_ends)
model.Minimize(makespan)
# Solve model.
solver = cp_model.CpSolver()
solution_printer = SolutionPrinter()
status = solver.Solve(model, solution_printer)
# Print final solution.
for job_id in all_jobs:
print('Job %i:' % job_id)
for task_id in range(len(jobs[job_id])):
start_value = solver.Value(starts[(job_id, task_id)])
machine = -1
duration = -1
selected = -1
for alt_id in range(len(jobs[job_id][task_id])):
if solver.Value(presences[(job_id, task_id, alt_id)]):
duration = jobs[job_id][task_id][alt_id][0]
machine = jobs[job_id][task_id][alt_id][1]
selected = alt_id
print(
' task_%i_%i starts at %i (alt %i, machine %i, duration %i)' %
(job_id, task_id, start_value, selected, machine, duration))
print('Solve status: %s' % solver.StatusName(status))
print('Optimal objective value: %i' % solver.ObjectiveValue())
flexible_jobshop()
This is implemented in the following example:
https://github.com/google/or-tools/blob/stable/examples/python/jobshop_ft06_distance_sat.py

What does "Killed: 9" error mean?

I am writing a program to find all the subsets of a given string. However it is resulting in an error called Killed: 9.
class Strings(object):
def __init__(self,string):
self.list1=[]
self.string=string
def appendToList(self,ch1):
self.list1.append(ch1)
for i in self.list1:
self.list1.append(i+ch1)
def subset(self):
for ch in self.string:
self.list1=self.appendToList(ch)
print self.list1
obj=Strings('hello')
obj.subset()
You can ask the program kill what that signal means:
$ kill -l | column
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR
31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3
38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8
43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7
58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2
63) SIGRTMAX-1 64) SIGRTMAX
So your program receives a SIGKILL (probably by the OOM killer):
SIGKILL
Default action: Abnormal termination of the process.
Description: Kill (cannot be caught or ignored).
C.f. Who "Killed" my process and why?
you enter an infinite loop when you append an element every time you iterate over existing elements (in appendToList) and no need to assign in subset method, check this out:
class Strings(object):
def __init__(self,string):
self.list1=[]
self.string=string
def appendToList(self,ch1):
self.list1.append(ch1)
# for i in self.list1: # no needed for loop is in subset
# self.list1.append(i+ch1)
def subset(self):
for ch in self.string:
self.appendToList(ch) # no need to assign to self.list1
print self.list1
obj=Strings('hello')
obj.subset()

Categories