We Keep Coding

Python - How can I avoid b"\n" being written in a text file with "wb" causing a line break?

Here is my case:
# Small hashing script example
import hashlib
import os
def original_pass(Password):
salt = os.urandom(64)
hashed = hashlib.pbkdf2_hmac("sha512", Password.encode(), salt, 300000, dklen = 124)
with open("Hash.txt", "wb") as file:
file.write(salt + b"\n" + hashed)
file.close()
def check_pass(New_Pass):
with open("Hash.txt", "rb") as file:
f = file.readlines()
print (f)
check_hash = hashlib.pbkdf2_hmac("sha512", New_Pass.encode(), f[0].strip(b"\n"), 300000, dklen = 124)
file.close()
if check_hash == f[1]:
return True
else:
return False
original_pass("Password")
print (check_pass("Password"))
My problem is that occasionally the hash will contain characters such as \n. E.G. b"x004\n4\no5". This line gets split into b"x004\n", b"4\no5". This causes errors when I try to read things like the salt, since it may be split up into multiple pieces. Is there any way of avoiding it being read like that, or just stopping it from writing it that way?
To address the duplicate remark
I am dealing specifically with byte strings here and not regular strings. Both are a distinct data type, moreso in python 3 (the version I'm using) as seen here: What is the difference between a string and a byte string?. This distinction means that certain string methods such as .encode() don't work on byte strings. Thus there is a clear difference in the data types I'm dealing with and how they are manipulated, etc...

Based on #Blckknght comment, code using knowledge of fixed length salt:
import hashlib
import os
SALT_LEN = 64
def hash(password,salt):
return hashlib.pbkdf2_hmac('sha512',password.encode(),salt,300000,dklen=124)
def original_pass(password):
salt = os.urandom(SALT_LEN)
hashed = hash(password,salt)
with open('hash.bin','wb') as file:
file.write(salt + hashed)
def check_pass(password):
with open('hash.bin','rb') as file:
data = file.read()
salt,hashed = data[:SALT_LEN],data[SALT_LEN:]
check_hash = hash(password,salt)
return check_hash == hashed
original_pass('Password')
print(check_pass('Password'))

Read Null terminated string in python

I'm trying to read a null terminated string but i'm having issues when unpacking a char and putting it together with a string.
This is the code:
def readString(f):
str = ''
while True:
char = readChar(f)
str = str.join(char)
if (hex(ord(char))) == '0x0':
break
return str
def readChar(f):
char = unpack('c',f.read(1))[0]
return char
Now this is giving me this error:
TypeError: sequence item 0: expected str instance, int found
I'm also trying the following:
char = unpack('c',f.read(1)).decode("ascii")
But it throws me:
AttributeError: 'tuple' object has no attribute 'decode'
I don't even know how to read the chars and add it to the string, Is there any proper way to do this?

Here's a version that (ab)uses __iter__'s lesser-known "sentinel" argument:
with open('file.txt', 'rb') as f:
val = ''.join(iter(lambda: f.read(1).decode('ascii'), '\x00'))

How about:
myString = myNullTerminatedString.split("\x00")[0]
For example:
myNullTerminatedString = "hello world\x00\x00\x00\x00\x00\x00"
myString = myNullTerminatedString.split("\x00")[0]
print(myString) # "hello world"
This works by splitting the string on the null character. Since the string should terminate at the first null character, we simply grab the first item in the list after splitting. split will return a list of one item if the delimiter doesn't exist, so it still works even if there's no null terminator at all.
It also will work with byte strings:
myByteString = b'hello world\x00'
myStr = myByteString.split(b'\x00')[0].decode('ascii') # "hello world" as normal string
If you're reading from a file, you can do a relatively larger read - estimate how much you'll need to read to find your null string. This is a lot faster than reading byte-by-byte. For example:
resultingStr = ''
while True:
buf = f.read(512)
resultingStr += buf
if len(buf)==0: break
if (b"\x00" in resultingStr):
extraBytes = resultingStr.index(b"\x00")
resultingStr = resultingStr.split(b"\x00")[0]
break
# now "resultingStr" contains the string
f.seek(0 - extraBytes,1) # seek backwards by the number of bytes, now the pointer will be on the null byte in the file
# or f.seek(1 - extraBytes,1) to skip the null byte in the file

(edit version 2, added extra way at the end)
Maybe there are some libraries out there that can help you with this, but as I don't know about them lets attack the problem at hand with what we know.
In python 2 bytes and string are basically the same thing, that change in python 3 where string is what in py2 is unicode and bytes is its own separate type, which mean that you don't need to define a read char if you are in py2 as no extra work is required, so I don't think you need that unpack function for this particular case, with that in mind lets define the new readString
def readString(myfile):
chars = []
while True:
c = myfile.read(1)
if c == chr(0):
return "".join(chars)
chars.append(c)
just like with your code I read a character one at the time but I instead save them in a list, the reason is that string are immutable so doing str+=char result in unnecessary copies; and when I find the null character return the join string. And chr is the inverse of ord, it will give you the character given its ascii value. This will exclude the null character, if its needed just move the appending...
Now lets test it with your sample file
for instance lets try to read "Sword_Wea_Dummy" from it
with open("sword.blendscn","rb") as archi:
#lets simulate that some prior processing was made by
#moving the pointer of the file
archi.seek(6)
string=readString(archi)
print "string repr:", repr(string)
print "string:", string
print ""
#and the rest of the file is there waiting to be processed
print "rest of the file: ", repr(archi.read())
and this is the output
string repr: 'Sword_Wea_Dummy'
string: Sword_Wea_Dummy
rest of the file: '\xcd\xcc\xcc=p=\x8a4:\xa66\xbfJ\x15\xc6=\x00\x00\x00\x00\xeaQ8?\x9e\x8d\x874$-i\xb3\x00\x00\x00\x00\x9b\xc6\xaa2K\x15\xc6=;\xa66?\x00\x00\x00\x00\xb8\x88\xbf#\x0e\xf3\xb1#ITuB\x00\x00\x80?\xcd\xcc\xcc=\x00\x00\x00\x00\xcd\xccL>'
other tests
>>> with open("sword.blendscn","rb") as archi:
print readString(archi)
print readString(archi)
print readString(archi)
sword
Sword_Wea_Dummy
ÍÌÌ=p=Š4:¦6¿JÆ=
>>> with open("sword.blendscn","rb") as archi:
print repr(readString(archi))
print repr(readString(archi))
print repr(readString(archi))
'sword'
'Sword_Wea_Dummy'
'\xcd\xcc\xcc=p=\x8a4:\xa66\xbfJ\x15\xc6='
>>>
Now that I think about it, you mention that the data portion is of fixed size, if that is true for all files and the structure on all of them is as follow
[unknow size data][know size data]
then that is a pattern we can exploit, we only need to know the size of the file and we can get both part smoothly as follow
import os
def getDataPair(filename,knowSize):
size = os.path.getsize(filename)
with open(filename, "rb") as archi:
unknown = archi.read(size-knowSize)
know = archi.read()
return unknown, know
and by knowing the size of the data portion, its use is simple (which I get by playing with the prior example)
>>> strins_data, data = getDataPair("sword.blendscn", 80)
>>> string_data, data = getDataPair("sword.blendscn", 80)
>>> string_data
'sword\x00Sword_Wea_Dummy\x00'
>>> data
'\xcd\xcc\xcc=p=\x8a4:\xa66\xbfJ\x15\xc6=\x00\x00\x00\x00\xeaQ8?\x9e\x8d\x874$-i\xb3\x00\x00\x00\x00\x9b\xc6\xaa2K\x15\xc6=;\xa66?\x00\x00\x00\x00\xb8\x88\xbf#\x0e\xf3\xb1#ITuB\x00\x00\x80?\xcd\xcc\xcc=\x00\x00\x00\x00\xcd\xccL>'
>>> string_data.split(chr(0))
['sword', 'Sword_Wea_Dummy', '']
>>>
Now to get each string a simple split will suffice and you can pass the rest of the file contained in data to the appropriated function to be processed

Doing file I/O one character at a time is horribly slow.
Instead use readline0, now on pypi: https://pypi.org/project/readline0/ . Or something like it.
In 3.x, there's a "newline" argument to open, but it doesn't appear to be as flexible as readline0.

Here is my implementation:
import struct
def read_null_str(f):
r_str = ""
while 1:
back_offset = f.tell()
try:
r_char = struct.unpack("c", f.read(1))[0].decode("utf8")
except:
f.seek(back_offset)
temp_char = struct.unpack("<H", f.read(2))[0]
r_char = chr(temp_char)
if ord(r_char) == 0:
return r_str
else:
r_str += r_char

Binary to ASCII in Python

I'm trying to decode binary which are located in a .txt file, but I'm stuck. I don't see any possibilities this can go around.
def code(): testestest
ascii = {'01000001':'A', ...}
binary = {'A':'01000001', ...}
print (ascii, binary)
def encode():
pass
def decode(code,n):
f = open(code, mode='rb') # Open a file with filename <code>
while True:
chunk = f.read(n) # Read n characters at time from an open file
if chunk == '': # This is one way to check for the End Of File in Python
break
if chunk != '\n':
# Process it????
pass
How can I take the binary in the .txt file and output it as ASCII?

From your example, your input looks like a string of a binary formatted number.
If so, you don't need a dictionnary for that:
def byte_to_char(input):
return chr(int(input, base=2))
Using the data you gave in the comments, you have to split your binary string into bytes.
input ='01010100011010000110100101110011001000000110100101110011001000000110101001110101011100110111010000100000011000010010000001110100011001010111001101110100001000000011000100110000001110100011000100110000'
length = 8
input_l = [input[i:i+length] for i in range(0,len(input),length)]
And then, per byte, you convert it into a char:
input_c = [chr(int(c,base=2)) for c in input_l]
print ''.join(input_c)
Putting it all together:
def string_decode(input, length=8):
input_l = [input[i:i+length] for i in range(0,len(input),length)]
return ''.join([chr(int(c,base=2)) for c in input_l])
decode(input)
>'This is just a test 10:10'

python cant get valid conversion from hex to binary

I have a program that must be able to search a binary file as either hex or ascii. I have the ascii search/replace function working fine, but the previously working hex search/ replace is no longer working. Below is an example of what is happening: The LH is after the change and hex was transposed into the "ascii layer" the RH is the original.
Included code:
if a_chars>a_digits:alphex = 'a'
else: alphex = 'h'
print alphex
# 3b. if alpha
if alphex == 'a' :
data = open(tar,'rb').read(160); print('Opening tar file')
if len(old)>len(new):
old.ljust(len(old));pad = len(old)- len(new);new = new+(pad*' ')
if len(old)<len(new):
print 'NEW: data cannot exceed length of OLD:'
# 3c. if hex
if alphex == 'h' :
with open(tar,'rb') as f:
data = binascii.hexlify (f.read(100));print('Opening tar bfile')
if len(old)>len(new): print 'OLD>NEW hex size must be ='
if len(old)<len(new): print 'OLD<NEW hex size must be ='
old = old.lower();old = ''.join(old.split());print 'old: ', old
new = new.lower();new = ''.join(new.split());print 'new: ', new
##sub write new file
fo= open(tarname+time+'.'+tarext,'wb');print 'Creating new file...'
fo.write (data.replace(old,new));print 'Writing file...'

Previous answer given to me on the 3rd of Feb. chasing the trees around I got lost in the forest.
data = data.replace(old,new);data = binascii.unhexlify(data)
fo.write(data);print 'Writing bfile...'
I forgot to reassign data back to data (data = binascii.unhexlify(data)) to obtain the corrected output.

Read Wireshark dump file for packet times

My instructions were to read the wireshark.bin data file dumped from the Wireshark program and pick out the packet times. I have no idea how to skip the header and find the first time.
"""
reads the wireshark.bin data file dumped from the wireshark program
"""
from datetime import datetime
import struct
import datetime
#file = r"V:\workspace\Python3_Homework08\src\wireshark.bin"
file = open("wireshark.bin", "rb")
idList = [ ]
with open("wireshark.bin", "rb") as f:
while True:
bytes_read = file.read(struct.calcsize("=l"))
if not bytes_read:
break
else:
if len(bytes_read) > 3:
idList.append(struct.unpack("=l", bytes_read)[0])
o = struct.unpack("=l111", bytes_read)[0]
print( datetime.date.fromtimestamp(o))

Try reading the entire file at once, and then accessing it as a list:
data = open("wireshark.bin", "rb").read() # let Python automatically close file
magic = data[:4] # magic wireshark number (also reveals byte order)
gmt_correction = data[8:12] # GMT offset
data = data[24:] # actual packets
Now you can loop through data in (16?) byte size chunks, looking at the appropriate offset in each chunk for the timestamp.
The magic number is 0xa1b2c3d4, which takes four bytes, or two words. We can determine the order (big-endian or little-endian) by examining those first four bytes by using the struct module:
magic = struct.unpack('>L', data[0:4])[0] # before the data = data[24:] line above
if magic == 0xa1b2c3d4:
order = '>'
elif magic == 0xd4c3b2a1:
order = '<'
else:
raise NotWireSharkFile()
Now that we have the order (and know it's a wireshark file), we can loop through the packets:
field0, field1, field2, field3 = \
struct.unpack('%sllll' % order, data[:16])
payload = data[16 : 16 + field?]
data = data[16 + field?]
I left the names vague, since this is homework, but those field? names represent the information stored in the packet header which includes the timestamp and the length of the following packet data.
This code is incomplete, but hopefully will be enough to get you going.