We Keep Coding

How to decode a string representation of a bytes object?

I have a string which includes encoded bytes inside it:
str1 = "b'Output file \xeb\xac\xb8\xed\x95\xad\xeb\xb6\x84\xec\x84\x9d.xlsx Created'"
I want to decode it, but I can't since it has become a string. Therefore I want to ask whether there is any way I can convert it into
str2 = b'Output file \xeb\xac\xb8\xed\x95\xad\xeb\xb6\x84\xec\x84\x9d.xlsx Created'
Here str2 is a bytes object which I can decode easily using
str2.decode('utf-8')
to get the final result:
'Output file 문항분석.xlsx Created'

You could use ast.literal_eval:
>>> print(str1)
b'Output file \xeb\xac\xb8\xed\x95\xad\xeb\xb6\x84\xec\x84\x9d.xlsx Created'
>>> type(str1)
<class 'str'>
>>> from ast import literal_eval
>>> literal_eval(str1).decode('utf-8')
'Output file 문항분석.xlsx Created'

Based on the SyntaxError mentioned in your comments, you may be having a testing issue when attempting to print due to the fact that stdout is set to ascii in your console (and you may also find that your console does not support some of the characters you may be trying to print). You can try something like the following to set sys.stdout to utf-8 and see what your console will print (just using string slice and encode below to get bytes rather than the ast.literal_eval approach that has already been suggested):
import codecs
import sys
sys.stdout = codecs.getwriter('utf-8')(sys.stdout.buffer)
s = "b'Output file \xeb\xac\xb8\xed\x95\xad\xeb\xb6\x84\xec\x84\x9d.xlsx Created'"
b = s[2:-1].encode().decode('utf-8')

A simple way is to assume that all the characters of the initial strings are in the [0,256) range and map to the same Unicode value, which means that it is a Latin1 encoded string.
The conversion is then trivial:
str1[2:-1].encode('Latin1').decode('utf8')

Finally I have found an answer where i use a function to cast a string to bytes without encoding.Given string
str1 = "b'Output file \xeb\xac\xb8\xed\x95\xad\xeb\xb6\x84\xec\x84\x9d.xlsx Created'"
now i take only actual encoded text inside of it
str1[2:-1]
and pass this to the function which convert the string to bytes without encoding its values
import struct
def rawbytes(s):
"""Convert a string to raw bytes without encoding"""
outlist = []
for cp in s:
num = ord(cp)
if num < 255:
outlist.append(struct.pack('B', num))
elif num < 65535:
outlist.append(struct.pack('>H', num))
else:
b = (num & 0xFF0000) >> 16
H = num & 0xFFFF
outlist.append(struct.pack('>bH', b, H))
return b''.join(outlist)
So, calling the function would convert it to bytes which then is decoded
rawbytes(str1[2:-1]).decode('utf-8')
will give the correct output
'Output file 문항분석.xlsx Created'

Transform JSON String to Dictionary using shell without escape

I am calling a python script from the shell, with one input argument.
python main.py """{"key1":"value1", "key2":"value2"}"""
All keys and values are strings. Once in python, I would like to convert the JSON string to a dictionary, so I have acess to the values by using the keys.
I tried the following
import json
import sys
dict_in = json.loads(sys.argv[1])
But dict_in would end up a string like that {key1:value1, key2:value2}
So it seems like I need to find a way to pass the string with quotation marks from the shell to python. I can not use escape characters since the string is provided by a different program.
Is there an elegant way to solve this?

I've found a python 2 module which can handle such cases.
Suppose you have this string:
>>> str = '{foo: bar, id: 23}'
Then you can use yaml as follows:
>>> import yaml
>>> dict = yaml.load(str)
>>> dict
{'foo': 'bar', 'id': 23}
>>> dict['foo']
'bar'
Now you have what you needed.
More info (and also python 3 support and etc.) can be found here: https://pyyaml.org/wiki/PyYAMLDocumentation

Not sure if what you passing in is important but you can pass following and get desired output:
"{\"key1\":\"value1\", \"key2\":\"value2\"}"
or
'{"key1":"value1", "key2":"value2"}'
Here is the code and output:
$cat json_convert.py
import json
import sys
dict_in = json.loads(sys.argv[1])
print (dict_in)
$ python json_convert.py '{"key1":"value1", "key2":"value2"}'
{'key1': 'value1', 'key2': 'value2'}
Also what you are passing """{"key1":"value1", "key2":"value2"}""" translates to "" + "{" + key1 + ":" + value1 + ", " + + key2 + ":" + value2 + "}" + "" if you are asking bash, if you were calling the function with that as a argument from the python itself you would get the desired results.
So really goes down to what you are calling it from.
If you still like quotes go ahead and pass """{"'"key1"'":"'"value1"'", "'"key2"'":"'"value2"'"}""" to get desired result :)

use either:
$ your_other_program | python main.py
to send the output of the other program to python, or use base64.b64encode(json.dumps(blah)) and you'll get pretty code like
'eyJtQXV0b21hdGljVGVzdExpc3QiOiBbeyJtWSI6IDguMTE0MTA1LCAibU5hbWUiOiAiYWNjZWxlcmF0b3JFbnRpdHkiLCAibVRlc3RTdGF0dXMiOiB0cnVlLCAibVgiOiAzLjgwNDM1MTgsICJtWiI6IC0zLjM4OTU3MjF9LCB7Im1OYW1lIjogImJhcm9tZXRlckVudGl0eSIsICJtVmFsdWUiOiAwLCAibVRlc3RTdGF0dXMiOiBmYWxzZX1dLCAibUF1dG9tYXRpY1Rlc3RDb21wbGV0ZWQiOiB0cnVlfQ=='
to put in the command line, and then decode it back from base64 into JSON.
Or, even better, use:
$ your_other_program >output_file.tmp
$ python main.py < output_file.tmp
$ rm output_file.tmp

Ok so here is what is my test script:
print("original sys.argv output\n" + (sys.argv[1]))
string_temp=(yaml.load(sys.argv[1]))
print ("first transformation\n" +string_temp)
string_temp=string_temp.replace(":",": ")
dict_in=yaml.load(string_temp)
print("This is the dictionary")
print(dict_in)
This is what I type into the console
python test_script.py """{foo:bar, id:23}"""
And This is the output
original sys.argv output
"{foo:bar, id:23}"
first transformation
{foo:bar, id:23}
This is the dictionary
{'foo': 'bar', 'id': 23}
This only workds if I use tripple quotes ("""). If I use (") or (') to define the input string I get an error.
Alternatively one can remove the (") from the sys.argv[1]
print("original sys.argv output\n" + (sys.argv[1]))
string_temp=(sys.argv[1])[1:-1]
print ("first transformation\n" +string_temp)
string_temp=string_temp.replace(":",": ")
dict_in=yaml.load(string_temp)
print("This is the dictionary")
print(dict_in)

different outputs in unpack function in python

I am observing a different output in the unpack function of python when I accept the string input from the console and when I read the string input from a variable.
I read the string input from the variable, input:
>>> import struct
>>> input="\x0d\x00\x00\x00"
>>> print struct.unpack("I",input)[0]
13
I read the string input from the console:
>>> import sys
>>> import struct
>>> print struct.unpack("I",sys.stdin.read(4))[0]
\x0d\x00\x00\x00
1680898140
The input string is the same but the output is different. Does it interpret the input read from the console in a different way? How can I get the same input by reading the data from console?

"\x0d\x00\x00\x00" (from the first code) is different from r"\x0d\x00\x00\x00" (== "\\x0x\\x00\x00\x00") from the second code.
>>> struct.unpack("I", '\x0d\x00\x00\x00')[0]
13
>>> struct.unpack("I", r'\x0d\x00\x00\x00'[:4])[0]
1680898140
Try following:
>>> struct.unpack("I", sys.stdin.readline().decode('string-escape')[:4])[0]
\x0d\x00\x00\x00
13

seems like you are unpacking the wrong data...
>>> struct.unpack('I','\\x0d')[0]
1680898140
your call to sys.stdin.read(4) reads only 4 characters: \, x, 0 and d.
>>> import sys
>>> import struct
>>> value = raw_input().decode('string-escape')
\x0d\x00\x00\x00
>>> print struct.unpack("I", value)[0]
13

Str to hex in Python

I receive on my socket a 4 bytes value that I want to print as hex value. I am trying:
print "%08x" % (nonce)
However, I get an error message that string can't be converted to hex. Anyone an idea
how this could be quickly resolved?

Use the struct module to unpack the octets received from the network into an actual number. The %08x format will work on the number:
import struct
n, = struct.unpack('>I', nonce)
print "%08x" % n

You most likely have a string containing the bytes. However, to print them as a number you need well.. a number.
You can easily create the hex string you are looking for like this:
''.join('%02x' % ord(x) for x in nonce)
Demo:
>>> nonce = os.urandom(4)
>>> nonce
'X\x19e\x07'
>>> ''.join('%02x' % ord(x) for x in nonce)
'58196507'

Another option is:
from binascii import hexlify
>>> hexlify('a1n4')
'61316e34'

num = "9999"
print hex(int(num))
#0x270f

If your data can be converted to a string, you could use the str.encode() method:
>>> s = "XYZ"
>>> s.encode('hex')
'58595a'

python pack output in string format

I have done the following.
from struct import pack, unpack
t = 1234
tt = str(pack("<I", t))
printing tt gives \xf3\xe0\x01\x00. How do I get original value of t back from tt?
I tried using unpacking the repr(tt) but that does not work out. How do I go about doing this?

>>> t=1234
>>> tt=pack('<I', t)
>>> tt
'\xd2\x04\x00\x00'
>>> unpack('<I', tt)
(1234,)
>>> ttt, = unpack('<I', tt)
>>> ttt
1234

you are using the wrong package for serialization. the struct package is only useful for python code which interacts with C code.
for serialization into a string, you should use the pickle module.
import pickle
t = 1234
tt = pickle.dumps(t)
t = pickle.loads(tt)

unpack('<I', tt) will give you (1234,).
repr doesn't work since it adds quotes to the string:
>>> repr('foo')
'"foo"'