We Keep Coding

How to convert the string between numpy.array and bytes

I want to convert the string to bytes first, and then convert it to numpy array:
utf8 string -> bytes -> numpy.array
And then:
numpy.array -> bytes -> utf8 string
Here is the test:
import numpy as np
string = "any_string_in_utf8: {}".format(123456)
test = np.frombuffer(bytes(string, 'utf-8'))
print(test)
Here is the output:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_9055/3077694159.py in <cell line: 5>()
3 string = "any_string_in_utf8: {}".format(123456)
4
----> 5 test = np.frombuffer(bytes(string, 'utf-8'))
6 print(test)
ValueError: buffer size must be a multiple of element size
How to convert the string between numpy.array and bytes?

Solution :
Main Problem in your Code is that you haven't mentioned dtype. By default dtype was set as Float and we generally started Conversion from String that's why it was throwing ValueError: buffer size must be a multiple of element size.
But If we convert the same into unsigned int then it will work because it can't interpret Object. For more refer to the Code Snippet given below: -
# Import all the Important Modules
import numpy as np
# Initialize String
utf_8_string = "any_string_in_utf8: {}".format(123456)
# utf8 string -> bytes -> numpy.array
np_array = np.frombuffer(bytes(utf_8_string, 'utf-8'), dtype=np.uint8)
# Print 'np_array'
print("Numpy Array after Bytes Conversion : -")
print(np_array)
# numpy.array -> bytes -> utf8 string
result_str = np.ndarray.tobytes(np_array).decode("utf-8")
# Print Result for the Verification of 'Data Loss'
print("\nOriginal String After Conversion : - \n" + result_str)
To Know more about np.frombuffer(): - Click Here !
# Output of the Above Code: -
Numpy Array after Bytes Conversion : -
[ 97 110 121 95 115 116 114 105 110 103 95 105 110 95 117 116 102 56
58 32 49 50 51 52 53 54]
Original String After Conversion : -
any_string_in_utf8: 123456

Can you help me understand packed binary data in python

I followed an example to convert 24bit audio to bytes.
For example:
struct.pack('<i', 4000000)
Gives:
b'\x00\t=\x00'
Can you help me understand the packed binary data
\x00\t=\x00
How do we interpret it?
Many thanks.

We can get the byte representation for 4000000 using this:
In [14]: x = 4000000
In [15]: print("%08X" % x)
003D0900
But on x86 and x64 machines, integers are stored LSB first, so the bytes in memory will be
00 09 3D 00
Those translate to the following characters:
In [46]: print(b"%c%c%c%c" % (0x00, 0x09, 0x3D, 0x00))
b'\x00\t=\x00'
So 0x09 is \t and 0x3D is =.
We can recreate the original value using ord, some bit shifts, and addition:
In [52]: (ord('=') << 16) + (ord('\t') << 8)
Out[52]: 4000000
In the comments, you asked "why it isn't just \t=". Hopefully this example answers that question:
In [8]: struct.unpack('<i', b'\t=')
---------------------------------------------------------------------------
error Traceback (most recent call last)
<ipython-input-8-c3b2a260fbdf> in <module>
----> 1 struct.unpack('<i', b'\t=')
error: unpack requires a buffer of 4 bytes
The '<i' tells pack and unpack that they are working with 4-byte little-endian integers, so we need to give them at least 4 bytes.
You also asked "why not \x00\t\x00=\x00". This time we promise 4 bytes but deliver 5:
In [10]: struct.unpack('<i', b'\x00\t\x00=\x00')
---------------------------------------------------------------------------
error Traceback (most recent call last)
<ipython-input-10-8f3448548a2a> in <module>
----> 1 struct.unpack('<i', b'\x00\t\x00=\x00')
error: unpack requires a buffer of 4 bytes
Maybe I confused you by leaving out the 0x00 bytes in the reassembly:
In [11]: (0 << 24) + (ord('=') << 16) + (ord('\t') << 8) + 0
Out[11]: 4000000

How to run a FOR LOOP over a csv file

I am trying to run the following code over a CSV file, but the code is showing an error that the input data type should be a str and not an int, but I have checked the data type and it is a float. I have tried every conversion from string, to float, to int, but nothing seems to work. please tell me what am I doing wrong.
print(stdized_data.X.dtypes)
for element in stdized_data:
if element != 0:
log(element + 1)
else:
log(element + 2)
###################################OUTPUT################################
float64
--------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-163-6e534ce31c6a> in <module>()
9 for element in stdized_data:
10 if element != 0:
---> 11 log(str(element) + 1)
12
13 else:
TypeError: must be str, not int
I have loaded the file using pd.read_csv function.

What you need to do is probably this -
log(str(element + 1))
What you are doing is -
log(str(element) + 1)
You have converted element to string but 1 is still an integer and you can't add string and integer

How to convert bin to float using 64bit IEE754, [duplicate]

I want to convert a binary number into a float number. Here's an example of a possibility:
>>> float(-0b1110)
gives me the correct output:
-14.0
Unfortunately, I am working with binary strings, i.e., I need something like float('-0b1110').
However, this doesn't work:
>>> float('-0b1110')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for float(): -0b1110
I tried to use binascii.a2b_qp(string[, header]) which converts a block of quoted-printable data back to binary and returns the binary data. But eventually, I get the same error:
>>> float(binascii.a2b_qp('-0b1110'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for float(): -0b1110
I understand the cases where the output number is an integer but what if I want to obtain the number 12.546? What would the function call for the binary string look like then?

In one of your comments you indicated that the binary number represents a float in 8 byte long IEEE 754 binary64 format. However that is inconsistent with the -0b1110 value you showed as an example, so I've ignored it and used my own which is in the proper format as example input data for testing the answer shown below.
Essentially what is done is first the binary string is converted into an integer value, then next into a string of raw bytes which is passed to struct.unpack() for final conversion to a floating point value. The bin_to_float() function shown below drives the process. Although not illustrated, binary input string arguments can be prefixed with '0b'.
from codecs import decode
import struct
def bin_to_float(b):
""" Convert binary string to a float. """
bf = int_to_bytes(int(b, 2), 8) # 8 bytes needed for IEEE 754 binary64.
return struct.unpack('>d', bf)[0]
def int_to_bytes(n, length): # Helper function
""" Int/long to byte string.
Python 3.2+ has a built-in int.to_bytes() method that could be used
instead, but the following works in earlier versions including 2.x.
"""
return decode('%%0%dx' % (length << 1) % n, 'hex')[-length:]
def float_to_bin(value): # For testing.
""" Convert float to 64-bit binary string. """
[d] = struct.unpack(">Q", struct.pack(">d", value))
return '{:064b}'.format(d)
if __name__ == '__main__':
for f in 0.0, 1.0, -14.0, 12.546, 3.141593:
print('Test value: %f' % f)
binary = float_to_bin(f)
print(' float_to_bin: %r' % binary)
floating_point = bin_to_float(binary) # Round trip.
print(' bin_to_float: %f\n' % floating_point)
Output:
Test value: 0.000000
float_to_bin: '0000000000000000000000000000000000000000000000000000000000000000'
bin_to_float: 0.000000
Test value: 1.000000
float_to_bin: '0011111111110000000000000000000000000000000000000000000000000000'
bin_to_float: 1.000000
Test value: -14.000000
float_to_bin: '1100000000101100000000000000000000000000000000000000000000000000'
bin_to_float: -14.000000
Test value: 12.546000
float_to_bin: '0100000000101001000101111000110101001111110111110011101101100100'
bin_to_float: 12.546000
Test value: 3.141593
float_to_bin: '0100000000001001001000011111101110000010110000101011110101111111'
bin_to_float: 3.141593

This works for me.
Tested with Python3.4:
def float_to_bin(num):
return bin(struct.unpack('!I', struct.pack('!f', num))[0])[2:].zfill(32)
def bin_to_float(binary):
return struct.unpack('!f',struct.pack('!I', int(binary, 2)))[0]
float_to_bin(bin_to_float(float_to_bin(123.123))) == float_to_bin(123.123)
>>> True

Another option is to do
from ast import literal_eval
float_str = "-0b101010101"
result = float(literal_eval(float_str))
Unlike the built-in "eval", literal_eval is safe to be run even on user inputs, as it can only parse Python literals - and will not execute expressions, which means it will not call functions as well.

float(int('-0b1110',0))
That works for me.
If you have a 64-bit string that represents a floating point number rather than an integer, you can do a three-step conversion - the first step turns the string into an integer, the second converts it into an 8-byte string, and the third re-interprets those bits as a float.
>>> import struct
>>> s = '0b0100000000101001000101111000110101001111110111110011101101100100'
>>> q = int(s, 0)
>>> b8 = struct.pack('Q', q)
>>> struct.unpack('d', b8)[0]
12.546
Of course you can combine all those steps into a single line.
>>> s2 = '0b1100000000101100000000000000000000000000000000000000000000000000'
>>> struct.unpack('d', struct.pack('Q', int(s2, 0)))[0]
-14.0

You could use eval('') and then cast it as a float if needed. Example:
>> eval('-0b1110')
-14
>> float(eval('-0b1110'))
-14.0

You can convert a binary number in string form to an int by setting the base to 2 in the built-in int([x[, base]]) function, however you need to get rid of the 0b first. You can then pass the result into float() to get your final result:
>>> s = '-0b1110'
>>> float(int(s.replace('0b', ''), 2))
-14.0
edit: Apparently getting rid of the 0b is only necessary on Python 2.5 and below, Mark's answer works fine for me on Python 2.6 but here is what I see on Python 2.5:
>>> int('-0b1110', 2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 2: '-0b1110'

def bin_to_float(binary):
di = binary.find('.')
if di == -1:
return int(binary,2)
else:
whole = binary[:di] or '0'
fraction = binary [di+1:] or '0'
return int(whole,2) + int(whole,2)/abs(int(whole,2)) * int(fraction,2) / 2** len(fraction)
samples = [
'-0b1110.010101011',
'-0b1001110',
'101',
'0b1110010.11010111011',
'1100.',
'-00011',
'+0b1100.0011',
]
for binary in samples:
print(binary,'=',bin_to_float(binary))
Returns
-0b1110.010101011 = -14.333984375
-0b1001110 = -78
101 = 5
0b1110010.11010111011 = 114.84130859375
1100. = 12.0
-00011 = -3
+0b1100.0011 = 12.1875

How to convert a binary (string) into a float value?

I want to convert a binary number into a float number. Here's an example of a possibility:
>>> float(-0b1110)
gives me the correct output:
-14.0
Unfortunately, I am working with binary strings, i.e., I need something like float('-0b1110').
However, this doesn't work:
>>> float('-0b1110')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for float(): -0b1110
I tried to use binascii.a2b_qp(string[, header]) which converts a block of quoted-printable data back to binary and returns the binary data. But eventually, I get the same error:
>>> float(binascii.a2b_qp('-0b1110'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for float(): -0b1110
I understand the cases where the output number is an integer but what if I want to obtain the number 12.546? What would the function call for the binary string look like then?

In one of your comments you indicated that the binary number represents a float in 8 byte long IEEE 754 binary64 format. However that is inconsistent with the -0b1110 value you showed as an example, so I've ignored it and used my own which is in the proper format as example input data for testing the answer shown below.
Essentially what is done is first the binary string is converted into an integer value, then next into a string of raw bytes which is passed to struct.unpack() for final conversion to a floating point value. The bin_to_float() function shown below drives the process. Although not illustrated, binary input string arguments can be prefixed with '0b'.
from codecs import decode
import struct
def bin_to_float(b):
""" Convert binary string to a float. """
bf = int_to_bytes(int(b, 2), 8) # 8 bytes needed for IEEE 754 binary64.
return struct.unpack('>d', bf)[0]
def int_to_bytes(n, length): # Helper function
""" Int/long to byte string.
Python 3.2+ has a built-in int.to_bytes() method that could be used
instead, but the following works in earlier versions including 2.x.
"""
return decode('%%0%dx' % (length << 1) % n, 'hex')[-length:]
def float_to_bin(value): # For testing.
""" Convert float to 64-bit binary string. """
[d] = struct.unpack(">Q", struct.pack(">d", value))
return '{:064b}'.format(d)
if __name__ == '__main__':
for f in 0.0, 1.0, -14.0, 12.546, 3.141593:
print('Test value: %f' % f)
binary = float_to_bin(f)
print(' float_to_bin: %r' % binary)
floating_point = bin_to_float(binary) # Round trip.
print(' bin_to_float: %f\n' % floating_point)
Output:
Test value: 0.000000
float_to_bin: '0000000000000000000000000000000000000000000000000000000000000000'
bin_to_float: 0.000000
Test value: 1.000000
float_to_bin: '0011111111110000000000000000000000000000000000000000000000000000'
bin_to_float: 1.000000
Test value: -14.000000
float_to_bin: '1100000000101100000000000000000000000000000000000000000000000000'
bin_to_float: -14.000000
Test value: 12.546000
float_to_bin: '0100000000101001000101111000110101001111110111110011101101100100'
bin_to_float: 12.546000
Test value: 3.141593
float_to_bin: '0100000000001001001000011111101110000010110000101011110101111111'
bin_to_float: 3.141593

This works for me.
Tested with Python3.4:
def float_to_bin(num):
return bin(struct.unpack('!I', struct.pack('!f', num))[0])[2:].zfill(32)
def bin_to_float(binary):
return struct.unpack('!f',struct.pack('!I', int(binary, 2)))[0]
float_to_bin(bin_to_float(float_to_bin(123.123))) == float_to_bin(123.123)
>>> True

Another option is to do
from ast import literal_eval
float_str = "-0b101010101"
result = float(literal_eval(float_str))
Unlike the built-in "eval", literal_eval is safe to be run even on user inputs, as it can only parse Python literals - and will not execute expressions, which means it will not call functions as well.

float(int('-0b1110',0))
That works for me.
If you have a 64-bit string that represents a floating point number rather than an integer, you can do a three-step conversion - the first step turns the string into an integer, the second converts it into an 8-byte string, and the third re-interprets those bits as a float.
>>> import struct
>>> s = '0b0100000000101001000101111000110101001111110111110011101101100100'
>>> q = int(s, 0)
>>> b8 = struct.pack('Q', q)
>>> struct.unpack('d', b8)[0]
12.546
Of course you can combine all those steps into a single line.
>>> s2 = '0b1100000000101100000000000000000000000000000000000000000000000000'
>>> struct.unpack('d', struct.pack('Q', int(s2, 0)))[0]
-14.0

You could use eval('') and then cast it as a float if needed. Example:
>> eval('-0b1110')
-14
>> float(eval('-0b1110'))
-14.0

You can convert a binary number in string form to an int by setting the base to 2 in the built-in int([x[, base]]) function, however you need to get rid of the 0b first. You can then pass the result into float() to get your final result:
>>> s = '-0b1110'
>>> float(int(s.replace('0b', ''), 2))
-14.0
edit: Apparently getting rid of the 0b is only necessary on Python 2.5 and below, Mark's answer works fine for me on Python 2.6 but here is what I see on Python 2.5:
>>> int('-0b1110', 2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 2: '-0b1110'

def bin_to_float(binary):
di = binary.find('.')
if di == -1:
return int(binary,2)
else:
whole = binary[:di] or '0'
fraction = binary [di+1:] or '0'
return int(whole,2) + int(whole,2)/abs(int(whole,2)) * int(fraction,2) / 2** len(fraction)
samples = [
'-0b1110.010101011',
'-0b1001110',
'101',
'0b1110010.11010111011',
'1100.',
'-00011',
'+0b1100.0011',
]
for binary in samples:
print(binary,'=',bin_to_float(binary))
Returns
-0b1110.010101011 = -14.333984375
-0b1001110 = -78
101 = 5
0b1110010.11010111011 = 114.84130859375
1100. = 12.0
-00011 = -3
+0b1100.0011 = 12.1875