I want to convert the string to bytes first, and then convert it to numpy array:
utf8 string -> bytes -> numpy.array
And then:
numpy.array -> bytes -> utf8 string
Here is the test:
import numpy as np
string = "any_string_in_utf8: {}".format(123456)
test = np.frombuffer(bytes(string, 'utf-8'))
print(test)
Here is the output:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_9055/3077694159.py in <cell line: 5>()
3 string = "any_string_in_utf8: {}".format(123456)
4
----> 5 test = np.frombuffer(bytes(string, 'utf-8'))
6 print(test)
ValueError: buffer size must be a multiple of element size
How to convert the string between numpy.array and bytes?
Solution :
Main Problem in your Code is that you haven't mentioned dtype. By default dtype was set as Float and we generally started Conversion from String that's why it was throwing ValueError: buffer size must be a multiple of element size.
But If we convert the same into unsigned int then it will work because it can't interpret Object. For more refer to the Code Snippet given below: -
# Import all the Important Modules
import numpy as np
# Initialize String
utf_8_string = "any_string_in_utf8: {}".format(123456)
# utf8 string -> bytes -> numpy.array
np_array = np.frombuffer(bytes(utf_8_string, 'utf-8'), dtype=np.uint8)
# Print 'np_array'
print("Numpy Array after Bytes Conversion : -")
print(np_array)
# numpy.array -> bytes -> utf8 string
result_str = np.ndarray.tobytes(np_array).decode("utf-8")
# Print Result for the Verification of 'Data Loss'
print("\nOriginal String After Conversion : - \n" + result_str)
To Know more about np.frombuffer(): - Click Here !
# Output of the Above Code: -
Numpy Array after Bytes Conversion : -
[ 97 110 121 95 115 116 114 105 110 103 95 105 110 95 117 116 102 56
58 32 49 50 51 52 53 54]
Original String After Conversion : -
any_string_in_utf8: 123456
I am trying to use pyamiibo from https://pypi.org/project/pyamiibo/ to read my amiibo bin dump file in python, and attempting to use amiiboapi.com to look up the details...
For Duck Hunt, pyamiibo's uid_hex returns "04 FC 30 82 03 49 80", but amiiboapi.com returns { "head": "07820000", "tail": "002f0002",}...
What should I do to link up the 2 outputs?
The UID is unique to each tag, the head and tail are in pages 21 and 22 respectively
I need an implementation of rand and rands from c++ in python to re-encrypt a bunch of files. But can't seem to get it right.
I have a exe that un-encrypts a file into text, I also have to source code, after editing the file I need to encrypt it using the same method.
Since I don't know how to write c++ code, I opted to write in python. Trying first to decrypt it to know the method is the same.
The following code is the c++ code that un-encrypted the files, where "buff" is the beginning of the encrypted block and "len" length of said block.
static const char KEY[] = "KF4QHcm2";
static const unsigned long KEY_LEN = sizeof(KEY) - 1;
void unobfuscate(unsigned char* buff, unsigned long len) {
srand((char)buff[len - 1]);
for (unsigned long i = 0; i < len - 2; i += 2) {
buff[i] ^= KEY[rand() % KEY_LEN];
}
}
From what I understand it takes the last character of the encrypted block as the seed, and from the beginning every 2 bytes it xors the value with an element of the KEY array, this index is determined by the remainder of a random number divided by the KEY length.
Searching around the web, I find that c++ uses a simple Linear Congruential Generator that shouldn't be used for encryption, but can't seem to make it work.
I found one example of the code and tried to implement the other but either don't seem to work.
#My try at implementing it
def rand():
global seed
seed = (((seed * 1103515245) + 12345) & 0x7FFFFFFF)
return seed
I also read that the rand function is between 0 and RAND_MAX, but can't find the value of RAND_MAX, if I found it maybe random.randrange() could be used.
It can also be the way I set the seed since it seems in c++ a char works but in python I'm setting it to the value of the character.
Here is what I observe when un-encrypting the file using the various methods. This is just the first 13 bytes, so if someone needs to check if it works it is possible to do so.
The block ends with the sequence: 4C 0A 54 C4 this means C4 is the seed
Example encrypted:
77 43 35 69 26 6B 0C 6E 3A 74 4B 33 71 wC5i&k.n:tK3q
Example un-encrypted using c++:
24 43 6C 69 63 6B 49 6E 69 74 0A 33 34 $ClickInit.34
Example un-encrypted using python example:
1A 43 7E 69 77 6B 38 6E 0E 74 1A 33 3A .C~iwk8n.t.3:
Example un-encrypted using python implementation:
3C 43 73 69 6E 6B 4A 6E 0E 74 1A 33 37 <CsinkJn.t.37
I can also have something wrong in my python script, so here is the file in case it has any errors:
import os
def srand(s):
global seed
seed = s
def rand():
global seed
#Example code
#seed = (seed * 214013 + 2531011) % 2**64
#return (seed >> 16)&0x7fff
#Implementation code
seed = (((seed * 1103515245) + 12345) & 0x7FFFFFFF)
return seed
KEY = ['K','F','4','Q','H','c','m','2']
KEY_LEN = len(KEY) - 1
for filename in os.listdir("."):
if filename.endswith(".dat"):
print(" Decoding " + filename)
#open file
file = open(filename, "rb")
#set file attributes
file_length = os.path.getsize(filename)
file_buffer = [0] * file_length
#copy contents of file to array
for i in range(file_length):
file_buffer[i] = int.from_bytes(file.read(1), 'big')
#close file
file.close()
print(" Random Seed: " + chr(file_buffer[-1]))
#set random generator seed
srand(file_buffer[-1])
#decrypt the file
for i in range(3600, file_length, 2):
file_buffer[i] ^= ord(KEY[rand() % KEY_LEN])
#print to check if output is un-encrypted
for i in range(3600, 3613, 1):
print(file_buffer[i])
print(chr(file_buffer[i]))
continue
else:
#Do not try to un-encrypt the python script
print("/!\ Can't decode " + filename)
continue
If anyone can help me figure this out I would be grateful, if possible I would love this to work in python but, from what I can gather, it seems like I will have to learn c++ to get it to work.
rand is not a cryptographic function.
rand's algorithm is not stable between systems compilers or anything else.
If you have no choice, your best bet is to use python-C/C++ interoperability techniques and actually run rand() and srand(). That will suck, but it will suck as much as the original code did.
This one is driving me crazy:
I defined these two functions to convert between bytes and int resp. bytes and bits:
def bytes2int(bytes) : return int(bytes.encode('hex'), 16)
def bytes2bits(bytes) : # returns the bits as a 8-zerofilled string
return ''.join('{0:08b}'.format(bytes2int(i)) for i in bytes)
This works as expected:
>>> bytes2bits('\x06')
'00000110'
Now I'm reading in a binary file (with a well defined data structure) and printing out some values, which works too. Here is an example:
The piece of code which reads the bytes from the file:
dataItems = f.read(dataSize)
for i in range(10) : // now only the first 10 items
dataItemBits = bytes2bits(dataItems[i*6:(i+1)*6]) // each item is 6 bytes long
dataType = dataItemBits[:3]
deviceID = dataItemBits[3:8]
# and here printing out the strings...
# ...
print(" => data type: %8s" % (dataType))
print(" => device ID: %8s" % (deviceID))
# ...
with this output:
-----------------------
Item #9: 011000010000000000111110100101011111111111111111
97 bits: 01100001
0 bits: 00000000
62 bits: 00111110
149 bits: 10010101
255 bits: 11111111
255 bits: 11111111
=> data type: 011 // first 3 bits
=> device ID: 00001 // next 5 bits
My problem is, that I'm unable to convert the 'bit-strings' to decimal numbers; if I try to print this
print int(deviceID, 2)
it gives me an the ValueError
ValueError: invalid literal for int() with base 2: ''
although deviceID is definitely a string (I'm using it before as string and printing it out) '00001', so it's not ''.
I also checked deviceID and dataType with __doc__ and they are strings.
This one works well in the console:
>>> int('000010', 2)
2
>>> int('000110', 2)
6
What is going on here?
UPDATE:
This is really weird: when I wrap it in a try/except block, it prints out the correct value.
try :
print int(pmtNumber, 2) // prints the correct value
except :
print "ERROR!" // no exception, so this is never printed
Any ideas?
I've got a string, a signature, and a public key, and I want to verify the signature on the string. The key looks like this:
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDfG4IuFO2h/LdDNmonwGNw5srW
nUEWzoBrPRF1NM8LqpOMD45FAPtZ1NmPtHGo0BAS1UsyJEGXx0NPJ8Gw1z+huLrl
XnAVX5B4ec6cJfKKmpL/l94WhP2v8F3OGWrnaEX1mLMoxe124Pcfamt0SPCGkeal
VvXw13PLINE/YptjkQIDAQAB
-----END PUBLIC KEY-----
I've been reading the pycrypto docs for a while, but I can't figure out how to make an RSAobj with this kind of key. If you know PHP, I'm trying to do the following:
openssl_verify($data, $signature, $public_key, OPENSSL_ALGO_SHA1);
Also, if I'm confused about any terminology, please let me know.
Use M2Crypto. Here's how to verify for RSA and any other algorithm supported by OpenSSL:
pem = """-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDfG4IuFO2h/LdDNmonwGNw5srW
nUEWzoBrPRF1NM8LqpOMD45FAPtZ1NmPtHGo0BAS1UsyJEGXx0NPJ8Gw1z+huLrl
XnAVX5B4ec6cJfKKmpL/l94WhP2v8F3OGWrnaEX1mLMoxe124Pcfamt0SPCGkeal
VvXw13PLINE/YptjkQIDAQAB
-----END PUBLIC KEY-----""" # your example key
from M2Crypto import BIO, RSA, EVP
bio = BIO.MemoryBuffer(pem)
rsa = RSA.load_pub_key_bio(bio)
pubkey = EVP.PKey()
pubkey.assign_rsa(rsa)
# if you need a different digest than the default 'sha1':
pubkey.reset_context(md='sha1')
pubkey.verify_init()
pubkey.verify_update('test message')
assert pubkey.verify_final(signature) == 1
The data between the markers is the base64 encoding of the ASN.1 DER-encoding of a PKCS#8 PublicKeyInfo containing an PKCS#1 RSAPublicKey.
That is a lot of standards, and you will be best served with using a crypto-library to decode it (such as M2Crypto as suggested by joeforker). Treat the following as some fun info about the format:
If you want to, you can decode it like this:
Base64-decode the string:
30819f300d06092a864886f70d010101050003818d0030818902818100df1b822e14eda1fcb74336
6a27c06370e6cad69d4116ce806b3d117534cf0baa938c0f8e4500fb59d4d98fb471a8d01012d54b
32244197c7434f27c1b0d73fa1b8bae55e70155f907879ce9c25f28a9a92ff97de1684fdaff05dce
196ae76845f598b328c5ed76e0f71f6a6b7448f08691e6a556f5f0d773cb20d13f629b6391020301
0001
This is the DER-encoding of:
0 30 159: SEQUENCE {
3 30 13: SEQUENCE {
5 06 9: OBJECT IDENTIFIER rsaEncryption (1 2 840 113549 1 1 1)
16 05 0: NULL
: }
18 03 141: BIT STRING 0 unused bits, encapsulates {
22 30 137: SEQUENCE {
25 02 129: INTEGER
: 00 DF 1B 82 2E 14 ED A1 FC B7 43 36 6A 27 C0 63
: 70 E6 CA D6 9D 41 16 CE 80 6B 3D 11 75 34 CF 0B
: AA 93 8C 0F 8E 45 00 FB 59 D4 D9 8F B4 71 A8 D0
: 10 12 D5 4B 32 24 41 97 C7 43 4F 27 C1 B0 D7 3F
: A1 B8 BA E5 5E 70 15 5F 90 78 79 CE 9C 25 F2 8A
: 9A 92 FF 97 DE 16 84 FD AF F0 5D CE 19 6A E7 68
: 45 F5 98 B3 28 C5 ED 76 E0 F7 1F 6A 6B 74 48 F0
: 86 91 E6 A5 56 F5 F0 D7 73 CB 20 D1 3F 62 9B 63
: 91
157 02 3: INTEGER 65537
: }
: }
: }
For a 1024 bit RSA key, you can treat "30819f300d06092a864886f70d010101050003818d00308189028181" as a constant header, followed by a 00-byte, followed by the 128 bytes of the RSA modulus. After that 95% of the time you will get 0203010001, which signifies a RSA public exponent of 0x10001 = 65537.
You can use those two values as n and e in a tuple to construct a RSAobj.
A public key contains both a modulus(very long number, can be 1024bit, 2058bit, 4096bit) and a public key exponent(much smaller number, usually equals one more than a two to some power). You need to find out how to split up that public key into the two components before you can do anything with it.
I don't know much about pycrypto but to verify a signature, take the hash of the string. Now we must decrypt the signature. Read up on modular exponentiation; the formula to decrypt a signature is message^public exponent % modulus. The last step is to check if the hash you made and the decrypted signature you got are the same.
I think ezPyCrypto might make this a little easier. The high-level methods of the key class includes these two methods which I hope will solve your problem:
verifyString - verify a string against a signature
importKey - import public key (and possibly private key too)
Rasmus points out in the comments that verifyString is hard-coded to use MD5, in which case ezPyCryto can't help Andrew unless he wades into its code. I defer to joeforker's answer: consider M2Crypto.
More on the DER decoding.
DER encoding always follows a TLV triplet format: (Tag, Length, Value)
Tag specifies the type (i.e. data structure) of the value
Length specifies the number of byte this value field occupies
Value is the actual value which could be another triplet
Tag basically tells how to interpret the bytes data in the Value field. ANS.1 does have a type system, e.g. 0x02 means integer, 0x30 means sequence (an ordered collection of one or more other type instances)
Length presentation has a special logic:
If the length < 127, the L field only uses one byte and coded as the
length number value directly
If the length > 127, then in the first byte of L field, the first bit
must be 1, and the rest 7 bits represents the number of following
bytes used to specifies the length of the Value field. Value, the
actually bytes of the value itself.
For example, say I want to encode a number of 256 bytes long, then it would be like this
02 82 01 00 1F 2F 3F 4F … DE AD BE EF
Tag, 0x02 means it's a number
Length, 0x82, bit presentation of it is 1000 0010, meaning the
following two bytes specifies the actually length of the value, which
his 0x0100 meaning the value field is 256 bytes long
Value, from 1F to EF, the actual 256 bytes.
Now looking at your example
30819f300d06092a864886f70d010101050003818d0030818902818100df1b822e14eda1fcb74336
6a27c06370e6cad69d4116ce806b3d117534cf0baa938c0f8e4500fb59d4d98fb471a8d01012d54b
32244197c7434f27c1b0d73fa1b8bae55e70155f907879ce9c25f28a9a92ff97de1684fdaff05dce
196ae76845f598b328c5ed76e0f71f6a6b7448f08691e6a556f5f0d773cb20d13f629b6391020301
0001
It interprets as just what Rasmus Faber put in his reply
Maybe this isn't the answer you're looking for, but if all you need is to turn the key into bits, it looks like it's Base64 encoded. Look at the codecs module (I think) in the standard Python library.
Using M2Crypto, the above answers does not work. Here is a tested example.
import base64
import hashlib
import M2Crypto as m2
# detach the signature from the message if it's required in it (useful for url encoded data)
message_without_sign = message.split("&SIGN=")[0]
# decode base64 the signature
binary_signature = base64.b64decode(signature)
# create a pubkey object with the public key stored in a separate file
pubkey = m2.RSA.load_pub_key(os.path.join(os.path.dirname(__file__), 'pubkey.pem'))
# verify the key
assert pubkey.check_key(), 'Key Verification Failed'
# digest the message
sha1_hash = hashlib.sha1(message_without_sign).digest()
# and verify the signature
assert pubkey.verify(data=sha1_hash, signature=binary_signature), 'Certificate Verification Failed'
And that's about it
I try the code given by joeforker but it does not work.
Here is my example code and it works fine.
from Crypto.Signature import PKCS1_v1_5
from Crypto.PublicKey import RSA
from Crypto.Hash import SHA
pem = """-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDfG4IuFO2h/LdDNmonwGNw5srW
nUEWzoBrPRF1NM8LqpOMD45FAPtZ1NmPtHGo0BAS1UsyJEGXx0NPJ8Gw1z+huLrl
XnAVX5B4ec6cJfKKmpL/l94WhP2v8F3OGWrnaEX1mLMoxe124Pcfamt0SPCGkeal
VvXw13PLINE/YptjkQIDAQAB
-----END PUBLIC KEY-----""" # your example key
key = RSA.importKey(pem)
h = SHA.new(self.populateSignStr(params))
verifier = PKCS1_v1_5.new(key)
if verifier.verify(h, signature):
print "verified"
else:
print "not verified"