We Keep Coding

How to frame a simple Factorio/Dyson Sphere factory problem in linear programming using `scipy.optimize`

I'm trying to frame a Linear Programming problem from the game Dyson Sphere (a Factorio clone) and then use scipy to optimize for it. The problem is as follows:
The problem is as follows: I have a copper mine which produces 150 copper per minute. I can build multiple copper plate factories each of which consume 60 copper per minute and produce 60 copper plates per minute. I want to determine how many factories I should build to maximize copper plate production.
After beginning the process of wrapping my head around the math I'm struggling to get a simplified example to work. I have the following:
from scipy.optimize import linprog
c = [-60]
A = [[-60]]
b = [150]
copper_plate_bounds = (0, None)
res = linprog(c, A_ub=A, b_ub=b, bounds=[copper_plate_bounds], method='highs')
print(res)
The code above will report message: The problem is unbounded. (HiGHS Status 10: model_status is Unbounded; primal_status is At upper bound) but I'm not too sure why. Obviously this is a silly matrix but you have Ax=b where here it is -60x=150. c should be negative here as I understand it because I'm trying to maximize the production of copper plates which I do by subtracting from the copper output (150).
Full output:
C:\Users\grant\Documents\code\math\venv\Scripts\python.exe C:\Users\grant\Documents\code\math\calcular_cobre.py
message: The problem is unbounded. (HiGHS Status 10: model_status is Unbounded; primal_status is At upper bound)
success: False
status: 3
fun: None
x: None
nit: 0
lower: residual: None
marginals: None
upper: residual: None
marginals: None
eqlin: residual: None
marginals: None
ineqlin: residual: None
marginals: None
Process finished with exit code 0
Since I'm new I'm not sure if I've misunderstood something mathematically or if I've improperly constructed the constraints of my program.

Python (for Revit Dynamo): unexpected token in while loop iteration

I'm a Python noobie trying to find my way around using it for Dynamo. I've had quite a bit of success using simple while loops/nested ifs to neaten my Dynamo scripts; however, I've been stumped by this recent error.
I'm attempting to pull in lists of data (flow rates from pipe fittings) and then output the maximum flow rate of each fitting by comparing the indices of each list (a cross fitting would have 4 flow rates in Revit, I'm comparing each pipe inlet/outlet flow rate and calculating the maximum for sizing purposes). For some reason, adding lists in the while loop and iterating the indices gives me the "unexpected token" error which I presume is related to "i += 1" according to online debuggers.
I've been using this while loop code format for a bit now and it has always worked for non-listed related iterations. Can anyone give me some guidance here?
Thank you in advance!
Error in Dynamo:
Warning: IronPythonEvaluator.EvaluateIronPythonScript
operation failed.
unexpected token 'i'
Code used:
import sys
import clr
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *
dataEnteringNode = IN
a = IN[0]
b = IN[1]
c = IN[2]
d = IN[3]
start = 0
end = 3
i = start
y=[]
while i < end:
y.append(max( (a[i], b[i], c[i] ))
i += 1
OUT = y

How to generate a time-ordered uid in Python?

Is this possible? I've heard Cassandra has something similar : https://datastax.github.io/python-driver/api/cassandra/util.html
I have been using a ISO timestamp concatenated with a uuid4, but that ended up way too large (58 characters) and probably overkill.
Keeping a sequential number doesn't work in my context (DynamoDB NoSQL)
Worth noticing that for my application it doesn't matter if items created in batch/same second are in a random order, as long as the uid don't collapse.
I have no specific restriction on maximum length, ideally I would like to see the different collision chance for different lengths, but it needs to be smaller than 58 (my original attempt)
This is to use with DynamoDB(NoSQL Database) as Sort-key

Why uuid.uuid1 is not sequential
uuid.uuid1(node=None, clock_seq=None) is effectively:
60 bits of timestamp (representing number of 100-ns intervals after 1582-10-15 00:00:00)
14 bits of "clock sequence"
48 bits of "Node info" (generated from network card's mac-address or from hostname or from RNG).
If you don't provide any arguments, then System function is called to generate uuid. In that case:
It's unclear if "clock sequence" is sequential or random.
It's unclear if it's safe to be used in multiple processes (can clock_seq be repeated in different processes or not?). In Python 3.7 this info is now available.
If you provide clock_seq or node, then "pure python implementation is used". IN this case even with "fixed value" for clock_seq:
timestamp part is guaranteed to be sequential for all the calls in current process even in threaded execution.
clock_seq part is randomly generated. But that is not critical annymore because timestamp is sequential and unique.
It's NOT safe for multiple processes (processes that call uuid1 with the same clock_seq, node might return conflicting values if called during the "same 100-ns time interval")
Solution that reuses uuid.uuid1
It's easy to see, that you can make uuid1 sequential by providing clock_seq or node arguments (to use python implementation).
import time
from uuid import uuid1, getnode
_my_clock_seq = getrandbits(14)
_my_node = getnode()
def sequential_uuid(node=None):
return uuid1(node=node, clock_seq=_my_clock_seq)
# .hex attribute of this value is 32-characters long string
def alt_sequential_uuid(clock_seq=None):
return uuid1(node=_my_node, clock_seq=clock_seq)
if __name__ == '__main__':
from itertools import count
old_n = uuid1() # "Native"
old_s = sequential_uuid() # Sequential
native_conflict_index = None
t_0 = time.time()
for x in count():
new_n = uuid1()
new_s = sequential_uuid()
if old_n > new_n and not native_conflict_index:
native_conflict_index = x
if old_s >= new_s:
print("OOops: non-sequential results for `sequential_uuid()`")
break
if (x >= 10*0x3fff and time.time() - t_0 > 30) or (native_conflict_index and x > 2*native_conflict_index):
print('No issues for `sequential_uuid()`')
break
old_n = new_n
old_s = new_s
print(f'Conflicts for `uuid.uuid1()`: {bool(native_conflict_index)}')
Multiple processes issues
BUT if you are running some parallel processes on the same machine, then:
node which defaults to uuid.get_node() will be the same for all the processes;
clock_seq has small chance to be the same for some processes (chance of 1/16384)
That might lead to conflicts! That is general concern for using
uuid.uuid1 in parallel processes on the same machine unless you have access to SafeUUID from Python3.7.
If you make sure to also set node to unique value for each parallel process that runs this code, then conflicts should not happen.
Even if you are using SafeUUID, and set unique node, it's still possible to have non-sequential (but unique) ids if they are generated in different processes.
If some lock-related overhead is acceptable, then you can store clock_seq in some external atomic storage (for example in "locked" file) and increment it with each call: this allows to have same value for node on all parallel processes and also will make id-s sequential. For cases when all parallel processes are subprocesses created using multiprocessing: clock_seq can be "shared" using multiprocessing.Value
As a result you always have to remember:
If you are running multiple processes on the same machine, then you must:
Ensure uniqueness of node. The problem for this solution: you can't be sure to have sequential ids from different processes generated during the same 100-ns interval. But this is very "light" operation executed once on process startup and achieved by: "adding" something to default node, e.g. int(time.time()*1e9) - 0x118494406d1cc000, or by adding some counter from machine-level atomic db.
Ensure "machine-level atomic clock_seq" and the same node for all processes on one machine. That way you'll have some overhead for "locking" clock_seq, but id-s are guaranteed to be sequential even if generated in different processes during the same 100-ns interval (unless you are calling uuid from several threads in the same process).
For processes on different machines:
either you have to use some "global counter service";
or it's not possible to have sequential ids generated on different machines during the same 100-ns interval.
Reducing size of the id
General approach to generate UUIDs is quite simple, so it's easy to implement something similar from scratch, and for example use less bits for node_info part:
import time
from random import getrandbits
_my_clock_seq = getrandbits(14)
_last_timestamp_part = 0
_used_clock_seq = 0
timestamp_multiplier = 1e7 # I'd recommend to use this value
# Next values are enough up to year 2116:
if timestamp_multiplier == 1e9:
time_bits = 62 # Up to year 2116, also reduces chances for non-sequential id-s generated in different processes
elif timestamp_multiplier == 1e8:
time_bits = 60 # up to year 2335
elif timestamp_multiplier == 1e7:
time_bits = 56 # Up to year 2198.
else:
raise ValueError('Please calculate and set time_bits')
time_mask = 2**time_bits - 1
seq_bits = 16
seq_mask = 2**seq_bits - 1
node_bits = 12
node_mask = 2**node_bits - 1
max_hex_len = len(hex(2**(node_bits+seq_bits+time_bits) - 1)) - 2 # 21
_default_node_number = getrandbits(node_bits) # or `uuid.getnode() & node_mask`
def sequential_uuid(node_number=None):
"""Return 21-characters long hex string that is sequential and unique for each call in current process.
Results from different processes may "overlap" but are guaranteed to
be unique if `node_number` is different in each process.
"""
global _my_clock_seq
global _last_timestamp_part
global _used_clock_seq
if node_number is None:
node_number = _default_node_number
if not 0 <= node_number <= node_mask:
raise ValueError("Node number out of range")
timestamp_part = int(time.time() * timestamp_multiplier) & time_mask
_my_clock_seq = (_my_clock_seq + 1) & seq_mask
if _last_timestamp_part >= timestamp_part:
timestamp_part = _last_timestamp_part
if _used_clock_seq == _my_clock_seq:
timestamp_part = (timestamp_part + 1) & time_mask
else:
_used_clock_seq = _my_clock_seq
_last_timestamp_part = timestamp_part
return hex(
(timestamp_part << (node_bits+seq_bits))
|
(_my_clock_seq << (node_bits))
|
node_number
)[2:]
Notes:
Maybe it's better to simply store integer value (not hex-string) in the database
If you are storing it as text/char, then its better to convert integer to base64-string instead of converting it to hex-string. That way it will be shorter (21 chars hex-string → 16 chars b64-encoded string):
from base64 import b64encode
total_bits = time_bits+seq_bits+node_bits
total_bytes = total_bits // 8 + 1 * bool(total_bits % 8)
def int_to_b64(int_value):
return b64encode(int_value.to_bytes(total_bytes, 'big'))
Collision chances
Single process: collisions not possible
Multiple processes with manually set unique clock_seq or unique node in each process: collisions not possible
Multiple processes with randomly set node (48-bits, "fixed" in time):
Chance to have the node collision in several processes:
in 2 processes out of 10000: ~0.000018%
in 2 processes out of 100000: 0.0018%
Chance to have single collision of the id per second in 2 processes with the "colliding" node:
for "timestamp" interval of 100-ns (default for uuid.uuid1 , and in my code when timestamp_multiplier == 1e7): proportional to 3.72e-19 * avg_call_frequency²
for "timestamp" interval of 10-ns (timestamp_multiplier == 1e8): proportional to 3.72e-21 * avg_call_frequency²

In the article you've linked too, the cassandra.util.uuid_from_time(time_arg, node=None, clock_seq=None)[source] seems to be exactly what you're looking for.
def uuid_from_time(time_arg, node=None, clock_seq=None):
"""
Converts a datetime or timestamp to a type 1 :class:`uuid.UUID`.
:param time_arg:
The time to use for the timestamp portion of the UUID.
This can either be a :class:`datetime` object or a timestamp
in seconds (as returned from :meth:`time.time()`).
:type datetime: :class:`datetime` or timestamp
:param node:
None integer for the UUID (up to 48 bits). If not specified, this
field is randomized.
:type node: long
:param clock_seq:
Clock sequence field for the UUID (up to 14 bits). If not specified,
a random sequence is generated.
:type clock_seq: int
:rtype: :class:`uuid.UUID`
"""
if hasattr(time_arg, 'utctimetuple'):
seconds = int(calendar.timegm(time_arg.utctimetuple()))
microseconds = (seconds * 1e6) + time_arg.time().microsecond
else:
microseconds = int(time_arg * 1e6)
# 0x01b21dd213814000 is the number of 100-ns intervals between the
# UUID epoch 1582-10-15 00:00:00 and the Unix epoch 1970-01-01 00:00:00.
intervals = int(microseconds * 10) + 0x01b21dd213814000
time_low = intervals & 0xffffffff
time_mid = (intervals >> 32) & 0xffff
time_hi_version = (intervals >> 48) & 0x0fff
if clock_seq is None:
clock_seq = random.getrandbits(14)
else:
if clock_seq > 0x3fff:
raise ValueError('clock_seq is out of range (need a 14-bit value)')
clock_seq_low = clock_seq & 0xff
clock_seq_hi_variant = 0x80 | ((clock_seq >> 8) & 0x3f)
if node is None:
node = random.getrandbits(48)
return uuid.UUID(fields=(time_low, time_mid, time_hi_version,
clock_seq_hi_variant, clock_seq_low, node), version=1)
There's nothing Cassandra specific to a Type 1 UUID...

You should be able to encode a timestamp precise to the second for a time range of 135 years in 32 bits. That will only take 8 characters to represent in hex. Added to the hex representation of the uuid (32 hex characters) that will amount to only 40 hex characters.
Encoding the time stamp that way requires that you pick a base year (e.g. 2000) and compute the number of days up to the current date (time stamp). Multiply this number of days by 86400, then add the seconds since midnight. This will give you values that are less than 2^32 until you reach year 2135.
Note that you have to keep leading zeroes in the hex encoded form of the timestamp prefix in order for alphanumeric sorting to preserve the chronology.
With a few bits more in the time stamp, you could increase the time range and/or the precision. With 8 more bits (two hex characters), you could go up to 270 years with a precision to the hundredth of a second.
Note that you don't have to model the fraction of seconds in a base 10 range. You will get optimal bit usage by breaking it down in 128ths instead of 100ths for the same number of characters. With the doubling of the year range, this still fits within 8 bits (2 hex characters)
The collision probability, within the time precision (i.e. per second or per 100th or 128th of a second) is driven by the range of the uuid so it will be 1 in 2^128 for the chosen precision. Increasing the precision of the time stamp has the most impact on reducing the collision chances. It is also the factor that has the lowest impact on total size of the key.
More efficient character encoding: 27 to 29 character keys
You could significantly reduce the size of the key by encoding it in base 64 instead of 16 which would give you 27 to 29 characters (depending on you choice of precision)
Note that, for the timestamp part, you need to use an encoding function that takes an integer as input and that preserves the collating sequence of digit characters.
For example:
def encode64(number, size):
chars = "+-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
result = list()
for _ in range(size):
result.append(chars[number%64])
number //= 64
return "".join(reversed(result))
a = encode64(1234567890,6) # '-7ZU9G'
b = encode64(9876543210,6) # '7Ag-Pe'
print(a < b) # True
u = encode64(int(uuid.uuid4()),22) # '1QA2LtMg30ztnugxaokVMk'
key = a+u # '-7ZU9G1QA2LtMg30ztnugxaokVMk' (28 characters)
You can save some more characters by combining the time stamp and uuid into a single number before encoding instead of concatenating the two encoded values.
The encode64() function needs one character every 6 bits.
So, for 135 years with precision to the second: (32+128)/6 = 26.7 --> 27 characters
instead of (32/6 = 5.3 --> 6) + (128/6 = 21.3 --> 22) ==> 28 characters
uid = uuid.uuid4()
timeStamp = daysSince2000 * 86400 + int(secondsSinceMidnight)
key = encode64( timeStamp<<128 | int(uid) ,27)
with a 270 year span and 128th of a second precision: (40+128)/6 = 28 characters
uid = uuid.uuid4()
timeStamp = daysSince2000 * 86400 + int(secondsSinceMidnight)
precision = 128
timeStamp = timeStamp * precision + int(factionOfSecond * precision)
key = encode64( timeStamp<<128 | int(uid) ,28)
With 29 characters you can raise precision to 1024th of a second and year range to 2160 years.
UUID masking: 17 to 19 characters keys
To be even more efficient, you could strip out the first 64 bits of the uuid (which is already a time stamp) and combine it with your own time stamp. This would give you keys with a length of 17 to 19 characters with practically no loss of collision avoidance (depending on your choice of precision).
mask = (1<<64)-1
key = encode64( timeStamp<<64 | (int(uid) & mask) ,19)
Integer/Numeric keys ?
As a final note, if your database supports very large integers or numeric fields (140 bits or more) as keys, you don't have to convert the combined number to a string. Just use it directly as the key. The numerical sequence of timeStamp<<128 | int(uid) will respect the chronology.

The uuid6 module (pip install uuid6) solves the problem. It aims at implementing the corresponding draft for a new uuid variant standard, see here.
Example code:
import uuid6
for i in range(0, 30):
u = uuid6.uuid7()
print(u)
time.sleep(0.1)
The package suggests to use uuid6.uuid7():
Implementations SHOULD utilize UUID version 7 over UUID version 1 and
6 if possible.
UUID version 7 features a time-ordered value field derived from the
widely implemented and well known Unix Epoch timestamp source, the
number of milliseconds seconds since midnight 1 Jan 1970 UTC, leap
seconds excluded. As well as improved entropy characteristics over
versions 1 or 6.

Not getting the output I need in my code and I can't figure out why. Python

The exercise I'm working on is supposed to display the following output:
Enter name of data file: gps.txt
---------------------------------------------------------------------------
Time Latitude Longitude Distance Pace
(hh:mm:ss) (deg) (deg) (miles) (min/miles)
---------------------------------------------------------------------------
10:57:32 38.898556 -77.037852 ***** *****
11:00:45 38.897147 -77.043934 0.341 9.421
11:03:28 38.896432 -77.052102 0.442 6.141
11:05:34 38.897411 -77.047212 0.272 7.730
11:08:58 38.898999 -77.040000 0.403 8.431
11:13:54 38.909032 -77.036894 0.713 6.915
11:19:24 38.920011 -77.029782 0.850 6.471
11:25:47 38.931111 -77.025432 0.802 7.957
---------------------------------------------------------------------------
Your fastest speed was 6 minutes and 9 seconds per mile.
When I run my program here is was gets outputted:
Enter name of data file: gps.txt
---------------------------------------------------------------------------
Time Latitude Longitude Distance Pace
(hh:mm:ss) (deg) (deg) (miles) (min/miles)
---------------------------------------------------------------------------
10:57:32 38.898556 -77.037852 ***** *****
11:00:45 38.897147 -77.043934 0.341 1935.282
11:03:28 38.896432 -77.052102 0.442 1500.225
11:05:34 38.897411 -77.047212 0.272 2449.924
11:08:58 38.898999 -77.040000 0.403 1658.896
11:13:54 38.909032 -77.036894 0.713 944.564
11:19:24 38.920011 -77.029782 0.850 799.326
11:25:47 38.931111 -77.025432 0.802 854.834
---------------------------------------------------------------------------
Your fastest speed was 854
As you can see, almost everything is correct except for pace and for the fastest speed down at the bottom. The file I am reading out of looks like this:
10:57:32 38.898556 -77.037852
11:00:45 38.897147 -77.043934
11:03:28 38.896432 -77.052102
11:05:34 38.897411 -77.047212
11:08:58 38.898999 -77.040000
11:13:54 38.909032 -77.036894
11:19:24 38.920011 -77.029782
11:25:47 38.931111 -77.025432

The main problem seems to be that you've set "this_pace" to "time elapsed", without bothering to incorporate the distance in your calculation.
In the future, add tracing print statements to trace both execution flow and the data computed. For instance, printing and checking each variable in your main calculation loop, would have shown the problem line immediately.
Also, consider incremental development: add a few lines or one loop at a time. Debug that before you add more.

I can also see that:
current_dist = float(current_time - previous_time)
time_elapsed = abs(((current_dist - previous_dist)/ 60) / distance)
this_pace = time_elapsed
Seems to me that the first two are backwards. The first calculates time elapsed. The second calculates pace; it should just calculate the distance, with the pace calculated on the this_pace line.

python passlib: what is the best value for "rounds"

from the passlib documentation
For most public facing services, you can generally have signin take upwards of 250ms - 400ms before users start getting annoyed.
so what is the best value for rounds in a login/registration if we consider that there is one call for the database for the login attempt, and it uses MongoDB with non-blocking call. (using Mongotor, and using the email as the _id, so it is by default indexed, the query is fast: 0.00299978256226 and of course tested with a database that has 3 records...)
import passlib.hash
import time
hashh = passlib.hash.pbkdf2_sha512
beg1 = time.time()
password = hashh.encrypt("test", salt_size = 32, rounds = 12000)
print time.time()- beg1 # returns 0.142999887466
beg2 = time.time()
hashh.verify("test", password) # returns 0.143000125885
print time.time()- beg2
now if i use half value:
password = hashh.encrypt("test", salt_size = 32, rounds = 4000) # returns 0.0720000267029
hashh.verify("test", password) # returns 0.0709998607635
am using Windows 7 64 bits on Dell XPS 15 i7 2.0 Ghz
NB: installed bcrypt, and of course, it's a real pain using it directly as its default values (rounds = 12):
hashh = passlib.hash.bcrypt
beg1 = time.time()
password = hashh.encrypt("test", rounds = 12) # returns 0.406000137329
print time.time()- beg1
beg2 = time.time()
hashh.verify("test", password) # returns 0.40499997139
print time.time()- beg2
half value:
password = hashh.encrypt("test", rounds = 12) # 0.00699996948242 wonderful?
hashh.verify("test", password) # 0.00600004196167
can you suggest me a good rounds value when using pbkdf2_sha512 that will be good for production?

(passlib developer here)
The amount of time pbkdf2_sha512 takes is linearly proportional to it's rounds parameter (elapsed_time = rounds * native_speed). Using the data for your system, native_speed = 12000 / .143 = 83916 iterations/second, which means you'll need around 83916 * .350 = 29575 rounds to get ~350ms delay.
Things are a little tricker for bcrypt, because the amount of time it takes is logarithmically proportional to it's rounds parameter (elapsed_time = (2 ** rounds) * native_speed). Using the data for your system, native_speed = (2 ** 12) / .405 = 10113 iterations/second, which means you'll need around log(10113 * .350, 2) = 11.79 rounds to get ~350 ms delay. But since BCrypt only accepts integer rounds parameters, so you'll need to pick rounds=11 (~200ms) or rounds=12 (~400ms).
All of this is something I'm hoping to fix in a future release of passlib. As a work in progress, passlib's mercurial repo currently contains a simple little script, choose_rounds.py, which takes care of choosing the correct rounds value for a given target time. You can download and run it directly as follows (it may take 20s or so to run):
$ python choose_rounds.py -h
usage: python choose_rounds.py <hash_name> [<target_in_milliseconds>]
$ python choose_rounds.py pbkdf2_sha512 350
hash............: pbkdf2_sha512
speed...........: 83916 iterations/second
target time.....: 350 ms
target rounds...: 29575
$ python choose_rounds.py bcrypt 350
hash............: bcrypt
speed...........: 10113 iterations/second
target time.....: 350 ms
target rounds...: 11 (200ms -- 150ms faster than requested)
target rounds...: 12 (400ms -- 50ms slower than requested)
(edit: added response regarding secure minimum rounds...)
Disclaimer: Determining a secure minimum is a surprisingly tricky question - there are a number of hard to quantify parameters, very little real world data, and some rigorously unhelpful theory. Lacking a good authority, I've been researching the topic myself; and for off-the-cuff calculations, I've boiled the raw data down to a short formula (below), which is generally what I use. Just be aware that behind it are a couple of pages of assumptions and rough estimates, making it more of a Fermi Estimation than an exact answer :|
My rule of thumb (mid 2012) for attacking PBKDF2-HMAC-SHA512 using GPUs is:
days * dollars = 2**(n-31) * rounds
days is the number of days before the attacker has a 50/50 chance of guessing the password.
dollars is the attackers' hardware budget (in $USD).
n is the average amount of entropy in your user's passwords (in bits).
To answer your script-kiddie question: if an average password has 32 bits of entropy, and the attacker has a $2000 system with a good GPU, then at 30000 rounds they will need 30 days (2**(32-31)*30000/2000) to have a 50/50 chance of cracking a given hash. I'd recommend playing around with the values until you arrive at a rounds/days tradeoff that you're comfortable with.
Some things to keep in mind:
The success rate of a dictionary attack isn't linear, it's more of "long tail" situation, so think of the 50/50 mark as more of a half-life.
That 31 is the key factor, as it encodes an estimation of the cost of attacking a specific algorithm using a specific technology level. The actual value, 2**-31, measures the "dollar-days per round" it will cost an attacker. For comparison, attacking PBKDF2-HMAC-SHA512 using an ASIC has a factor closer to 46 -- larger numbers mean more bang for the attacker's buck, and less security per round for you, though script kiddies generally won't have that kind of budget :)