DataFrame with array of Letters - python

data['Ln']
Out[46]:
0 [C, C, C, C, C, C, G, I, O, P, P, P, R, R, R, ...
1 [C, C, C, C, C, C, G, I, O, P, P, P, R, R, R, ...
2 [C, C, C, C, C, C, G, I, O, P, P, R, R, R, R, ...
3 [C, C, C, C, C, C, G, I, O, P, P, R, R, R, R, ...
4 [C, C, C, C, C, C, G, I, O, P, P, P, R, R, R, ...
...
43244 [G, I, O, P, P, P, R, R, R, R]
43245 [G, I, O, P, P, P, R, R, R, R]
43246 [G, I, O, P, P, R, R, R]
43247 [G, I, O, P, P, R, R, R]
43248 [G, I, O, P, R, R]
Name: Ln, Length: 43249, dtype: object
How can i structure a for loop to iterate over every row, and every letter either using sklearn.preprocessing.LebelEncoding or ord()?
For instance, I want every 'C' in every row to be the same number, as well as G, I, etc.

Create a dict then map it
alphabet_dict = {'C': 0, 'G': 1, }
data['Ln'].map(lambda x: [alphabet_dict.get(i) for i in x])
0 [0, 0, 0, 0, 0]
1 [1, 1, 1, 1, 1]

Related

how to post automatically generated passwords from websites

I have a problem with the password form, which when accessed from the browser, the password form fills itself after I press login on the webstie. And the value of the password changes every time a request is sent.
so i am confused to do auto login using python
maybe this is what my request looks like when i use the browser:
username=ExamplePassword123&password=20b4355f844d296868513bbc3daedc60&dst=&popup=true
it successfully logged into the website, but when I use python to post data, it doesn't work
import requests
sess= requests.session()
url = "http://example.com/loginform"
data={
'username':'ExamplePassword123',
'password':'',
'dst':'',
'popup':'true',
}
r= sess.post(url, data=data)
print (r.text)
the above code doesn't work. maybe because the password form is not filled in. I think if leave the password blank, it will automatically fill itself when it's sent. but apparently not.
is there a way to retrieve the value of the password using python and send the request so that I can login successfully?
ok, i found this md5.js. but i dont really understand about java script
/*
* A JavaScript implementation of the RSA Data Security, Inc. MD5 Message
* Digest Algorithm, as defined in RFC 1321.
* Version 1.1 Copyright (C) Paul Johnston 1999 - 2002.
* Code also contributed by Greg Holt
* See http://pajhome.org.uk/site/legal.html for details.
*/
/*
* Add integers, wrapping at 2^32. This uses 16-bit operations internally
* to work around bugs in some JS interpreters.
*/
function safe_add(x, y)
{
var lsw = (x & 0xFFFF) + (y & 0xFFFF)
var msw = (x >> 16) + (y >> 16) + (lsw >> 16)
return (msw << 16) | (lsw & 0xFFFF)
}
/*
* Bitwise rotate a 32-bit number to the left.
*/
function rol(num, cnt)
{
return (num << cnt) | (num >>> (32 - cnt))
}
/*
* These functions implement the four basic operations the algorithm uses.
*/
function cmn(q, a, b, x, s, t)
{
return safe_add(rol(safe_add(safe_add(a, q), safe_add(x, t)), s), b)
}
function ff(a, b, c, d, x, s, t)
{
return cmn((b & c) | ((~b) & d), a, b, x, s, t)
}
function gg(a, b, c, d, x, s, t)
{
return cmn((b & d) | (c & (~d)), a, b, x, s, t)
}
function hh(a, b, c, d, x, s, t)
{
return cmn(b ^ c ^ d, a, b, x, s, t)
}
function ii(a, b, c, d, x, s, t)
{
return cmn(c ^ (b | (~d)), a, b, x, s, t)
}
/*
* Calculate the MD5 of an array of little-endian words, producing an array
* of little-endian words.
*/
function coreMD5(x)
{
var a = 1732584193
var b = -271733879
var c = -1732584194
var d = 271733878
for(i = 0; i < x.length; i += 16)
{
var olda = a
var oldb = b
var oldc = c
var oldd = d
a = ff(a, b, c, d, x[i+ 0], 7 , -680876936)
d = ff(d, a, b, c, x[i+ 1], 12, -389564586)
c = ff(c, d, a, b, x[i+ 2], 17, 606105819)
b = ff(b, c, d, a, x[i+ 3], 22, -1044525330)
a = ff(a, b, c, d, x[i+ 4], 7 , -176418897)
d = ff(d, a, b, c, x[i+ 5], 12, 1200080426)
c = ff(c, d, a, b, x[i+ 6], 17, -1473231341)
b = ff(b, c, d, a, x[i+ 7], 22, -45705983)
a = ff(a, b, c, d, x[i+ 8], 7 , 1770035416)
d = ff(d, a, b, c, x[i+ 9], 12, -1958414417)
c = ff(c, d, a, b, x[i+10], 17, -42063)
b = ff(b, c, d, a, x[i+11], 22, -1990404162)
a = ff(a, b, c, d, x[i+12], 7 , 1804603682)
d = ff(d, a, b, c, x[i+13], 12, -40341101)
c = ff(c, d, a, b, x[i+14], 17, -1502002290)
b = ff(b, c, d, a, x[i+15], 22, 1236535329)
a = gg(a, b, c, d, x[i+ 1], 5 , -165796510)
d = gg(d, a, b, c, x[i+ 6], 9 , -1069501632)
c = gg(c, d, a, b, x[i+11], 14, 643717713)
b = gg(b, c, d, a, x[i+ 0], 20, -373897302)
a = gg(a, b, c, d, x[i+ 5], 5 , -701558691)
d = gg(d, a, b, c, x[i+10], 9 , 38016083)
c = gg(c, d, a, b, x[i+15], 14, -660478335)
b = gg(b, c, d, a, x[i+ 4], 20, -405537848)
a = gg(a, b, c, d, x[i+ 9], 5 , 568446438)
d = gg(d, a, b, c, x[i+14], 9 , -1019803690)
c = gg(c, d, a, b, x[i+ 3], 14, -187363961)
b = gg(b, c, d, a, x[i+ 8], 20, 1163531501)
a = gg(a, b, c, d, x[i+13], 5 , -1444681467)
d = gg(d, a, b, c, x[i+ 2], 9 , -51403784)
c = gg(c, d, a, b, x[i+ 7], 14, 1735328473)
b = gg(b, c, d, a, x[i+12], 20, -1926607734)
a = hh(a, b, c, d, x[i+ 5], 4 , -378558)
d = hh(d, a, b, c, x[i+ 8], 11, -2022574463)
c = hh(c, d, a, b, x[i+11], 16, 1839030562)
b = hh(b, c, d, a, x[i+14], 23, -35309556)
a = hh(a, b, c, d, x[i+ 1], 4 , -1530992060)
d = hh(d, a, b, c, x[i+ 4], 11, 1272893353)
c = hh(c, d, a, b, x[i+ 7], 16, -155497632)
b = hh(b, c, d, a, x[i+10], 23, -1094730640)
a = hh(a, b, c, d, x[i+13], 4 , 681279174)
d = hh(d, a, b, c, x[i+ 0], 11, -358537222)
c = hh(c, d, a, b, x[i+ 3], 16, -722521979)
b = hh(b, c, d, a, x[i+ 6], 23, 76029189)
a = hh(a, b, c, d, x[i+ 9], 4 , -640364487)
d = hh(d, a, b, c, x[i+12], 11, -421815835)
c = hh(c, d, a, b, x[i+15], 16, 530742520)
b = hh(b, c, d, a, x[i+ 2], 23, -995338651)
a = ii(a, b, c, d, x[i+ 0], 6 , -198630844)
d = ii(d, a, b, c, x[i+ 7], 10, 1126891415)
c = ii(c, d, a, b, x[i+14], 15, -1416354905)
b = ii(b, c, d, a, x[i+ 5], 21, -57434055)
a = ii(a, b, c, d, x[i+12], 6 , 1700485571)
d = ii(d, a, b, c, x[i+ 3], 10, -1894986606)
c = ii(c, d, a, b, x[i+10], 15, -1051523)
b = ii(b, c, d, a, x[i+ 1], 21, -2054922799)
a = ii(a, b, c, d, x[i+ 8], 6 , 1873313359)
d = ii(d, a, b, c, x[i+15], 10, -30611744)
c = ii(c, d, a, b, x[i+ 6], 15, -1560198380)
b = ii(b, c, d, a, x[i+13], 21, 1309151649)
a = ii(a, b, c, d, x[i+ 4], 6 , -145523070)
d = ii(d, a, b, c, x[i+11], 10, -1120210379)
c = ii(c, d, a, b, x[i+ 2], 15, 718787259)
b = ii(b, c, d, a, x[i+ 9], 21, -343485551)
a = safe_add(a, olda)
b = safe_add(b, oldb)
c = safe_add(c, oldc)
d = safe_add(d, oldd)
}
return [a, b, c, d]
}
/*
* Convert an array of little-endian words to a hex string.
*/
function binl2hex(binarray)
{
var hex_tab = "0123456789abcdef"
var str = ""
for(var i = 0; i < binarray.length * 4; i++)
{
str += hex_tab.charAt((binarray[i>>2] >> ((i%4)*8+4)) & 0xF) +
hex_tab.charAt((binarray[i>>2] >> ((i%4)*8)) & 0xF)
}
return str
}
/*
* Convert an array of little-endian words to a base64 encoded string.
*/
function binl2b64(binarray)
{
var tab = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
var str = ""
for(var i = 0; i < binarray.length * 32; i += 6)
{
str += tab.charAt(((binarray[i>>5] << (i%32)) & 0x3F) |
((binarray[i>>5+1] >> (32-i%32)) & 0x3F))
}
return str
}
/*
* Convert an 8-bit character string to a sequence of 16-word blocks, stored
* as an array, and append appropriate padding for MD4/5 calculation.
* If any of the characters are >255, the high byte is silently ignored.
*/
function str2binl(str)
{
var nblk = ((str.length + 8) >> 6) + 1 // number of 16-word blocks
var blks = new Array(nblk * 16)
for(var i = 0; i < nblk * 16; i++) blks[i] = 0
for(var i = 0; i < str.length; i++)
blks[i>>2] |= (str.charCodeAt(i) & 0xFF) << ((i%4) * 8)
blks[i>>2] |= 0x80 << ((i%4) * 8)
blks[nblk*16-2] = str.length * 8
return blks
}
/*
* Convert a wide-character string to a sequence of 16-word blocks, stored as
* an array, and append appropriate padding for MD4/5 calculation.
*/
function strw2binl(str)
{
var nblk = ((str.length + 4) >> 5) + 1 // number of 16-word blocks
var blks = new Array(nblk * 16)
for(var i = 0; i < nblk * 16; i++) blks[i] = 0
for(var i = 0; i < str.length; i++)
blks[i>>1] |= str.charCodeAt(i) << ((i%2) * 16)
blks[i>>1] |= 0x80 << ((i%2) * 16)
blks[nblk*16-2] = str.length * 16
return blks
}
/*
* External interface
*/
function hexMD5 (str) { return binl2hex(coreMD5( str2binl(str))) }
function hexMD5w(str) { return binl2hex(coreMD5(strw2binl(str))) }
function b64MD5 (str) { return binl2b64(coreMD5( str2binl(str))) }
function b64MD5w(str) { return binl2b64(coreMD5(strw2binl(str))) }
/* Backward compatibility */
function calcMD5(str) { return binl2hex(coreMD5( str2binl(str))) }
and also i found important stuff
<script type="text/javascript" src="md5.js"></script>
<script type="text/javascript">
function doLogin() {
document.sendin.username.value = document.login.username.value;
document.sendin.password.value = hexMD5('\065' + document.login.password.value +
'\057\373\261\270\106\356\071\007\121\106\031\115\166\071\222\233');
document.sendin.submit();
return false;
}
</script>
It's seems like the real password is somehow encoded before sending to the server. Considering the given example, a string consist of 32 hex characters, I guess it's encoded using the md5 algorithm.
But if the value changes everytime, it may contains some random factors. You need to dive into the js code to find out how exactly the password field is filled.

Is there a function in Tensorflow can do the following math?

I have two tensors, x and y, of shape [B, D]. I want to do something like the following code
B, D = x.shape
x = tf.expand_dims(x, 1) # [B, 1, D]
y = tf.expand_dims(y, -1) # [B, D, 1]
z = x * y # [B, D, D]
z = tf.reshape(z, (B, D**2))
Is there a function in Tensorflow that already does this?

how to extract the best population from different arrays?

I have a 8-10 sample arrays.
[W, T, I, Z, F]
[a, F, k, D, P]
[A, B, C, D, E]
[W, F, T, P, I]
[W, T, I, Z, F]
[H, c, B, V, C]
...
...
Each array has a unique value.
[0.4729,
0.7075,
0.8611,
0.8512,
0.4769,
0.3074
...
...]
I want to select best population.This population should include 5 different values. Any idea?
Note:If results are near to 1 , this means this population is better.
Thanks,

Index pytorch 4d tensor by values in 2d tensor

I have two pytorch tensors:
X with shape (A, B, C, D)
I with shape (A, B)
Values in I are integers in range [0, C).
What is the most efficient way to get tensor Y with shape (A, B, D), such that:
Y[i][j][k] = X[i][j][ I[i][j] ][k]
You probably want to use torch.gather for the indexing and expand to adjust I to the required size:
eI = I[..., None, None].expand(-1, -1, 1, X.size(3)) # make eI the same for the last dimension
Y = torch.gather(X, dim=2, index=eI).squeeze()
testing the code:
A = 3
B = 4
C = 5
D = 7
X = torch.rand(A, B, C, D)
I = torch.randint(0, C, (A, B), dtype=torch.long)
eI = I[..., None, None].expand(-1, -1, 1, X.size(3))
Y = torch.gather(X, dim=2, index=eI).squeeze()
# manually gather
refY = torch.empty(A, B, D)
for i in range(A):
for j in range(B):
refY[i, j, :] = X[i, j, I[i,j], :]
(refY == Y).all()
# Out[]: tensor(1, dtype=torch.uint8)

How to construct the following matrix elegantly in numpy?

Suppose I have a 5 dimensional matrix v and now I want a new matrix D fulfilling
D[a, b, n, m, d] = v[a, b, n, n, d]-v[a, b, m, m, d].
How do I elegantly do this in numpy?
How do you want to change the dimensionality? You can reshape it like this
import numpy as np
a, b, n, d = 2, 3, 4, 5
v = np.zeros((a, b, n, n, d))
D = v.reshape((a, b, n*n, d))
I found einsum can do this:
D = np.einsum('abiic->abic', v)[..., None, :] - np.einsum('abiic->abic', v)[:, :, None, ...]

Categories