I am trying to include a python "max" command inside a quicksum command using gurobi with python. There is obviously an error with doing so, under LinExpr limitations as it is not accepted.
shutdowncost = quicksum(quicksum(shutdown_cost[i] * max((v[hour -1, i] - v[hour, i]),0) for i in num_gen) for hour in hour_range)
V is a binary variable in the model, while the remainder are fixed variables. The issue is around shutdowncost being negative in the scenario where v[hour - 1, i] is 0, and v[hour, i] is 1.
Is there another command that can be used to replace the max command inside the quicksum?
Here is a paper that talks about startup and shutdown constraints: MIPFormulation. They use the notation:
u[t] 1 for online, 0 for offline, binary (state)
v[t] 1 for turned on that time period, binary (turn_on)
w[t] 1 for turned off that period, binary (turn_off)
These gurobi binary variables are defined with the constraints:
u[t] - u[t-1] == v[t] - w[t]
v[t] + w[t] <= 1
Then your shutdowncost can be defined:
shutdowncost = quicksum([shutdown_cost[i] * w[hour, i] for i in num_gen for hour in hour_range])
(No need for 2 quicksums!)
This shutdowncost can then be used in your objective function or another constraint. And it is easier to see what is happening.
Related
I am currently trying to obscure the contents of files or simply txt files without using any libraries. I know that this won't be very secure at all but what I basically want is a program that asks you what the password to "encrypt" it with is, then it asks what the files name is and then it finds that file and "encrypts" it. Then another program is used to "decrypt" it so it asks for the password and filename and then "decrypts" it. I don't care about actual security so if it can be easily opened it's fine I just need it so it doesn't just open if you click the file.
On top of that I don't want it to use ANY libraries so no pycrypto or anything like that.
I am on 64 bit windows.
I also am a complete beginner in tthe world of code and only know basic things such as how to get user input, print stuff, if loops and while loops.
Thanks in advance!
I don't know if this qualifies as an "external library" in your mind, but if you're on a linux machine you probably have the gpg command available to you. This is a reasonably* secure encryption protocol, which you could access from python - or directly from the command line, if you just want the files protected and you don't care about having it done through python.
Alternatively, you could bang together a trivial mechanism for obscuring a file's contents based on a known password. For example, you could "stretch" the password to the length of the file text (multiply the string by (1 + (text length / password length)) and then zip the two together. This gives you a bunch of tuples, which can by converted to their ordinal value (ord('f')=>102, for example) and xored together (ord('f')^ord('b')=>4) and converted back to chars (chr(4) => the unprintable '\x04'). The resulting chars are your cyphertext.
All of this is trivial to break, of course, but it's easy to implement, and decryption is trivial.
*intentional understatement :)
You can try using the password as a key to encrypt it. May be a logical operation on the file on a binary level such as or, and, or others will be able to encrypt it very simply -- but it won't be secure like you mentioned.
You can use XTEA (xTended Tiny Encryption Algorithm) by copying the python code (xTended Tiny Encryption Algorithm) code into your project, it is only 28 lines of python. It has been subjected to cryptanalysis and shown to be reasonably secure.
import struct
def crypt(key,data,iv='\00\00\00\00\00\00\00\00',n=32):
def keygen(key,iv,n):
while True:
iv = xtea_encrypt(key,iv,n)
for k in iv:
yield ord(k)
xor = [ chr(x^y) for (x,y) in zip(map(ord,data),keygen(key,iv,n)) ]
return "".join(xor)
def xtea_encrypt(key,block,n=32,endian="!"):
v0,v1 = struct.unpack(endian+"2L",block)
k = struct.unpack(endian+"4L",key)
sum,delta,mask = 0L,0x9e3779b9L,0xffffffffL
for round in range(n):
v0 = (v0 + (((v1<<4 ^ v1>>5) + v1) ^ (sum + k[sum & 3]))) & mask
sum = (sum + delta) & mask
v1 = (v1 + (((v0<<4 ^ v0>>5) + v0) ^ (sum + k[sum>>11 & 3]))) & mask
return struct.pack(endian+"2L",v0,v1)
def xtea_decrypt(key,block,n=32,endian="!"):
v0,v1 = struct.unpack(endian+"2L",block)
k = struct.unpack(endian+"4L",key)
delta,mask = 0x9e3779b9L,0xffffffffL
sum = (delta * n) & mask
for round in range(n):
v1 = (v1 - (((v0<<4 ^ v0>>5) + v0) ^ (sum + k[sum>>11 & 3]))) & mask
sum = (sum - delta) & mask
v0 = (v0 - (((v1<<4 ^ v1>>5) + v1) ^ (sum + k[sum & 3]))) & mask
return struct.pack(endian+"2L",v0,v1)
Attribution, code from: ActiveState Code ยป Recipes
in last equation i need to solve for q. Here is the problem from miranda feckler , I need to develop equivalent python code If my function is based on many variables and i need to solve non linear root finding problem for only one variable then how will i write-
when i write all the three variable, I get following error
TypeError: 'numpy.ndarray' object is not callable
and when i write only one of variables-
i get error-
TypeError: resid() missing 2 required positional arguments: 'p' and 'phi'
can anyone tell me my mistake and a better code for this.
broyden1(resid(co, p_node, q), co)
breaks because the term resid(co, p_node, q) gets evaluated (returning an array) before passing into the function.
broyden1(resid, co)
breaks because when broyden1 evaluates it calls resid(co) which is clearly not well defined. You want to be able to pass the initial guess as a single object (e.g. a tuple) in broyden1, so a simple solution is to just redefine resid to take in a tuple instead of three sepearate arguments, like so:
def resid(arg):
c,p,phi = arg
return p + (phi * c) * ((-1 / eta) * (p ** (eta + 1))) \
- alpha * (np.sqrt(np.abs(phi * c))) - (phi * c) ** 2
c1 = scipy.optimize.broyden1(resid, (co, p_node, q))
I am trying to assess the value of a column of a dataframe to determine the value of another column. I did this by using an if statement and .apply() function successfully. I.e.
if Col x < 0.3:
return y
elif Col x > 0.6:
return z
Etc. The problem is this takes quite a while to run with a lot of data. Instead I am trying to use the following logic to determine the new column value:
(x<0.3)*y + (x>0.6)*z
So Python evaluates TRUE/FALSE and applies the correct value. This seems to work much faster, the only thing is Python says:
"UserWarning: evaluating in Python space because the '*' operator is not supported by numexpr for the bool dtype, use '&' instead
unsupported[op_str]))"
Is this a problem? Should I be using "&"? I feel using "&" would be incorrect when multiplying.
Thank you!
From what I have read so far, the performance gap is issued by the parser backend chosen by pandas. There's the regular python parser as a backand and, additionally, a pandas parsing backend.
The docs say, that there is no performance gain if using plain old python over pandas here: Pandas eval Backends
However, you obviously hit a white spot in the pandas backend; i.e. you formed an expression that cannot be evaluated using pandas. The result is that pandas falls back to the original python parsing backend, as stated in the resulting UserWarning:
UserWarning: evaluating in Python space because the '*' operator is not supported by numexpr for the bool dtype, use '&' instead
unsupported[op_str]))
(More on this topic)
Timing evaluations
So, as we now know about different parsing backends, it's time to check a few options provided by pandas that are suitable for your desired dataframe operation (complete script below):
expr_a = '''(a < 0.3) * 1 + (a > 0.6) * 3 + (a >= 0.3) * (a <= 0.6) * 2'''
Evaluate the expression as a string using the pandas backend
Evaluate the same string using the python backend
Evaluate the expression string with external variable reference using pandas
Solve the problem using df.apply()
Solve the problem using df.applymap()
Direct submission of the expression (no string evaluation)
The results on my machine for a dataframe with 10,000,000 random float values in one column are:
(1) Eval (pd) 0.240498406269
(2) Eval (py) 0.197919774926
(3) Eval # (pd) 0.200814546686
(4) Apply 3.242620778595
(5) ApplyMap 6.542354086152
(6) Direct 0.140075372736
The major points explaining the performance differences are most likely the following:
Using a python function (as in apply() and applymap()) is (of course!) much slower than using functionality completely implemented in C
String evaluation is expensive (see (6) vs (2))
The overhead (1) has over (2) is probably the backend choice and fallback to also using the python backend, because pandas does not evaluate bool * int.
Nothing new, eh?
How to proceed
We basically just proved what our gut feeling was telling us before (namely: pandas chooses the right backend for a task).
As a consequence, I think it is totally okay to ignore the UserWarning, as long as you know the underlying hows and whys.
Thus: Keep going and have pandas use the fastest of all implementations, which is, as usual, the C functions.
The Test Script
from __future__ import print_function
import sys
import random
import pandas as pd
import numpy as np
from timeit import default_timer as timer
def conditional_column(val):
if val < 0.3:
return 1
elif val > 0.6:
return 3
return 2
if __name__ == '__main__':
nr = 10000000
df = pd.DataFrame({
'a': [random.random() for _ in range(nr)]
})
print(nr, 'rows')
expr_a = '''(a < 0.3) * 1 + (a > 0.6) * 3 + (a >= 0.3) * (a <= 0.6) * 2'''
expr_b = '''(#df.a < 0.3) * 1 + (#df.a > 0.6) * 3 + (#df.a >= 0.3) * (#df.a <= 0.6) * 2'''
fmt = '{:16s} {:.12f}'
# Evaluate the string expression using pandas parser
t0 = timer()
b = df.eval(expr_a, parser='pandas')
print(fmt.format('(1) Eval (pd)', timer() - t0))
# Evaluate the string expression using python parser
t0 = timer()
c = df.eval(expr_a, parser='python')
print(fmt.format('(2) Eval (py)', timer() - t0))
# Evaluate the string expression using pandas parser with external variable access (#)
t0 = timer()
d = df.eval(expr_b, parser='pandas')
print(fmt.format('(3) Eval # (pd)', timer() - t0))
# Use apply to map the if/else function to each row of the df
t0 = timer()
d = df['a'].apply(conditional_column)
print(fmt.format('(4) Apply', timer() - t0))
# Use element-wise apply (WARNING: requires a dataframe and walks ALL cols AND rows)
t0 = timer()
e = df.applymap(conditional_column)
print(fmt.format('(5) ApplyMap', timer() - t0))
# Directly access the pandas series objects returned by boolean expressions on columns
t0 = timer()
f = (df['a'] < 0.3) * 1 + (df['a'] > 0.6) * 3 + (df['a'] >= 0.3) * (df['a'] <= 0.6) * 2
print(fmt.format('(6) Direct', timer() - t0))
I have a matlab file (.m) and I want to run this file using python. I do not have matlab on my ubuntu system. Can I still run the matlab file?
isomonotone.m
% Checks vector v for monotonicity, and returns the direction (increasing,
% decreasing, constant, or none).
%
% Meaning of parameter tol:
% - if tol==0, check for non-increasing or non-decreasing sequence (default).
% - if tol>0, allow backward steps of size <= tol
% - if tol<0, require forward steps of size >= tol
%
% Inputs
% v: vector to check for monotonicity
% tol: see above
%
% Outputs
% b: a bitfield indicating monotonicity. Can be tested as follows:
% bitand(b,1)==true --> v is increasing (within tolerance)
% bitand(b,2)==true --> v is decreasing (within tolerance)
% bitand(b,3)==true --> v is both increasing and decreasing
% (i.e. v is constant, within tolerance).
% --------------------------------------------------------------------------
function b = ismonotone( v, tol )
if ( nargin < 2 )
tol = 0;
end
b = 0;
dv = diff(v);
if ( min(dv) >= -tol ) b = bitor( b, 1 ); end
if ( max(dv) <= tol ) b = bitor( b, 2 ); end
end
%!test assert(ismonotone(linspace(0,1,20)),1);
%!test assert(ismonotone(linspace(1,0,20)),2);
%!test assert(ismonotone(zeros(1,100)),3);
%!test
%! v=[0 -0.01 0 0 0.01 0.25 1];
%! assert(ismonotone(v,0.011),1);
%! assert(ismonotone(-v,0.011),2);
Can I run this file using python without having matlab on my ubuntu?
You can install Octave from the Ubuntu repository (or download and install the latest - that's more work)
Starting Octave in the directory where this file is, allows me to run the %! tests, thus:
octave:1> test ismonotone
PASSES 4 out of 4 tests
In fact the presence of those %! suggests that this file was originally written for Octave. Can someone confirm whether MATLAB can handle those doctest like lines?
edit - add interactive examples
octave:1> ismonotone(linspace(0,1,20))
ans = 1
octave:2> ismonotone(zeros(1,100))
ans = 3
Or from the linux shell
1424:~/myml$ octave -fq --eval 'ismonotone(linspace(0,1,20))'
ans = 1
For someone used to running Python scripts from the shell, Octave is more friendly than MATLAB. The startup overhead is much smaller, and the command line options are more familiar. In the interactive mode, doc opens a familiar UNIX info system.
Have you tried:
1. Small Matlab to Python compiler
2. LiberMate
3. Rewrite the code using SciPy module.
Hope this helps
Python cannot run Matlab programs natively. You would need to rewrite this in Python, likely using the SciPy libraries.
Afternoon everyone. I'm currently porting over an IDL code to python and it's been plain sailing up until this point so far. I'm stuck on this section of IDL code:
nsteps = 266
ind2 = ((lindgen(nsteps+1,nsteps+1)) mod (nsteps+1))
dk2 = (k2arr((ind2+1) < nsteps) - k2arr(ind2-1) > 0)) / 2.
My version of this includes a rewritten lindgen function as follows:
def pylindgen(shape):
nelem = numpy.prod(numpy.array(shape))
out = numpy.arange(nelem,dtype=int)
return numpy.reshape(out,shape)
... and the ported code where k2arr is an array of shape (267,):
ind2 = pylindgen((nsteps+1,nsteps+1)) % (nsteps+1)
dk2 = (k2arr[ (ind2+1) < nsteps ] - k2arr[ (ind2-1) > 0. ]) / 2.
Now, the problem is that my code makes ind2 an array where, by looking at the IDL code and the errors thrown in the python script, I'm sure it's meant to be a scalar. Am I missing some feature of these IDL functions?
Any thoughts would be greatly appreciated.
Cheers.
My knowledge of IDL is not what it used to be, I had to research a little. The operator ">" in IDL is not an equivalent of python (or other languages). It stablishes a maximum, anything above it will be set to that value. Same goes for "<", obviously, it sets a minimum.
dk2 = (k2arr((ind2+1) < nsteps) - k2arr(ind2-1) > 0))
where k2arr is 266 and ind2 is (266,266) is equivalent to saying:
- (ind2+1 < nsteps) take ind2+1 and, in any place that ind2+1
is greater than nsteps, replace by nsteps.
- (ind2-1 > 0) take ind2-1 and, in any place that ind2-1 is
less than zero, put zero instead.
The tricky part is now. k2arr (266,) is evaluated for each of the rows of (ind2+1) and (ind2-1), meaning that if (ind2+1 < nsteps) = [1,2,3,...,nsteps-1, nsteps, nsteps] the k2arr will be evaluated for exactly that 266 times, one on top of the other, with the result being (266,266) array.
And NOW I remember why I stopped programming in IDL!
The code for pylindgen works perfectly for me. Produces an array of (267,267), though. IF k2array is a (267,) array, you should be getting an error like:
ValueError: boolean index array should have 1 dimension
Is that your problem?
Cheers