I'm asking for your help because I've been stuck with the same problem for 3 days.
If I have :
Value1 = 0, Value2 = 3.10 and IPv6 = '2001::1'
I would like to pack all 3 values with this command: package = struct.pack(*format*, value1, value2, IPv6)
My problem is: I don't know what format characters in C type I can use to pack the IPv6 and keep its 16 bytes.
I know that I can use format = 'i f ?' with i for integer / f for float but I need to find with what to replace the '?' which is the format characters in C type for an IPv6 address to pack the three values.
Please, someone can help me?
It is not that straightforward to use struct for this, as you'd have to know the proper byte values of the IPv6 address before adding then to a struct.
'2001::1' is a textual representation that is nowhere close to giving you those values: you'd have to split the string on :, replace missing values with "0", then you 'd have a 4 16 bit number list to pack in the struct. and them, certainly there are corner cases and special syntax in the IPv6 string representation you'd have to account for.
Fortunately Python already handles that for you in the. ipaddress module of the stdlib.
Just import ipaddress, format the struct for the first part of your package and concatenate it with the "packed" attribute in the IPv6Address Python automatically genreates for you:
import struct
import ipaddress
Value1 = 0
Value2 = 3.10
IPv6 = '2001::1'
payload = struct.pack("if", (Value1, Value2)) + ipaddress.ip_address(IPv6).packed
However, I wonder if it will be productive to simply pack an int and a float along with an IP address in this way - whatever code will be reading this will be super coupled with the code you are writing to that.
If you are simply storing it to a file to be read back by a Python program under your control just use pickle instead. If you intend to send these values to a non Python program over a network, a schemaless textual way of conveying them, like JSON, might be much simpler.
If you really want to store these, and only these, in a compact way in order to save space, and there are tens of thousands of them, and they will be read back by the same program: try numpy arrays. They will take care of the compact binary representation for each object type and can be read and written to binary files, and numpy will take care of record offset for you.
The only use case I could see for that is if you have a program not under your control in a low level protocol that would be expecting exactly this record format. Since you are speculating about how to create the payload, and trying to convey "3.10" as a floating point value, this does not seem to be the case. Talking about that, "3.10" or other numbers might not round trip well as a nicely formed 2-decimal digit value with structs like this, due to how floating points are represented internally. I suggest you review your goals and needs there, and not overcomplicate things.
To unpack back, the easier thing is to use struct to recover just the numeric values, and pass the remaining 16 bytes back to the ip_address factory function - it automatically creates an IPv6 object, which string representation is the human friendly "2001::1".
I type the "roundtrip" in an interactive prompt:
In [30]: import struct, ipaddress
In [31]: x = ipaddress.ip_address('2001::1')
In [32]: v1 = 2;v2 = 3.10
In [33]: payload = struct.pack(">if",v1, v2) + x.packed
In [34]: print(payload)
b'\x00\x00\x00\x02#Fff \x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'
In [35]: v3, v4, nx = struct.unpack(">if", aa[:-16]) + (ipaddress.ip_address(aa[-16:]),)
In [36]: print(v3, v4, nx)
2 3.0999999046325684 2001::1
Two things to be noted: the added > prefix on the struct formatting string, to ensure byte order when encoding its contents,
and that the "3.10" value, as described above, does not roundtrip as a "nice two decimal places" value. Use round(number, 2) when representing the number on the server side, if needed.
Related
I'm making a script to get Valve's server information (players online, map, etc)
the packet I get when I request for information is this:
'\xff\xff\xff\xffI\x11Stargate Central CAP SBEP\x00sb_wuwgalaxy_fix\x00garrysmod\x00Spacebuild\x00\xa0\x0f\n\x0c\x00dw\x00\x0114.09.08\x00\xb1\x87i\x06\xb4g\x17.\x15#\x01gm:spacebuild3\x00\xa0\x0f\x00\x00\x00\x00\x00\x00'
This may help you to see what I'm trying to do https://developer.valvesoftware.com/wiki/Server_queries#A2S_INFO
The problem is, I don't know how to decode this properly, it's easy to get the string but I have no idea how to get other types like byte and short
for example '\xa0\x0f'
For now I'm doing multiple split but do you know if there is any better way of doing this?
Python has functions for encoding/decoding different data types into bytes. Take a look at the struct package, the functions struct.pack() and struct.unpack() are your friends there.
taken from https://docs.python.org/2/library/struct.html
>>> from struct import *
>>> pack('hhl', 1, 2, 3)
'\x00\x01\x00\x02\x00\x00\x00\x03'
>>> unpack('hhl', '\x00\x01\x00\x02\x00\x00\x00\x03')
(1, 2, 3)
The first argument of the unpack function defines the format of the data stored in the second argument. Now you need to translate the description given by valve to a format string. If you wanted to unpack 2 bytes and a short from a data string (that would have a length of 4 bytes, of course), you could do something like this:
(first_byte, second_byte, the_short) = unpack("cc!h", data)
You'll have to take care yourself, to get the correct part of the data string (and I don't know if those numbers are signed or not, be sure to take care of that, too).
The strings you'll have to do differently (they are null-terminated here, so start were you know a string starts and read to the first "\0" byte).
pack() work's the other way around and stores data in a byte string. Take a look at the examples on the python doc and play around with it a bit to get a feel for it (when a tuple is returned/needed, e.g.).
struct supports you in getting the right byte order, which most of the time is network byte order and different from your system. That is of course only necessary for multi byte integers (like short) - so a format string of `"!h" should unpack a short correctly.
so far this is what i found:
from random import randint
randint(100000000000000000000, 999999999999999999999)
the output is:
922106555361958347898L
but i do not want that L there..
i can only use this as an int if there is no "L" there at the end of it.
UPDATE
would it be a better idea to generate two small numbers and then combine them if the goal is to simply have a random number that is 30 digits long ?
The reason there's an L there is because this is too large to fit into an int,* so it's a long. See Numeric Types — int, long, float, complex in the docs for more details.
So, why do you get that L, and how do you get rid of it?
Python has two different ways to turn a value into a string representation:
repr either returns the canonical source-code representation, or something like <__main__.Eggs at 0x10fbd8908>).
str returns a human-friendly representation.
For example, for strings, str('abc') is abc, while repr('abc') is 'abc'.
And for longs, str(1L) is 1, while repr(1L) is 1L.
When you just type an expression at the interactive prompt, it uses repr. But when you use the print command, it uses str. So, if you want to see the value without the L, just print it:
print randint(100000000000000000000, 999999999999999999999)
If you want to, e.g., save the string in a variable or write it to a file, you have to call str explicitly.
But if you just want to use it as a number, you don't have to worry about this at all; it's a number, and int and long values can be intermixed freely (as of Python 2.3 or so).
And if you're trying to store it in a MySQL database, whichever MySQL interface you use won't care whether you're giving it int values or long, as long as they fit into the column type.**
Or you could upgrade to Python 3.x, where there is no separate long type anymore (all integers are int, no matter how big) and no L suffix.
* The exact cutoff isn't documented anywhere, but at least for CPython, it's whatever fits into a C long on your platform. So, on most 64-bit platforms, the max value is (1<<63)-1; on the other 64-bit platforms, and all 32-bit platforms, it's (1<<31)-1. You can see for yourself on your platform by printing sys.maxint. At any rate, your number takes 70 bits, so unless someone ports Python 2.x to a platform with 128-bit C longs, it won't fit.
** Note that your values are too big to fit into even a MySQL BIGINT, so you're going to be using either DECIMAL or NUMERIC. Depending on which interface you're using, and how you've set things up, you may have to convert to and from strings manually. But you can do that with the str and int functions, without worrying about which values fit into the int type and which don't.)
If you're on the interactive prompt, explicitly print the value. The repr of the value has an L, but the str of the value doesn't.
>>> 922106555361958347898
922106555361958347898L
>>> print 922106555361958347898
922106555361958347898
The output in the REPL has an L suffixed; if you print the value, it is not displayed.
>>> from random import randint
>>> print randint(100000000000000000000, 999999999999999999999)
106315199286113607384
>>>
I usually perform things like this in C++, but I'm using python to write a quick script and I've run into a wall.
If I have a binary list (or whatever python stores the result of an "fread" in). I can access the individual bytes in it with: buffer[0], buffer[1], etc.
I need to change the bytes [8-11] to hold a new 32-bit file-size (read: there's already a filesize there, I need to update it). In C++ I would just get a pointer to the location and cast it to store the integer, but with python I suddenly realized I have no idea how to do something like this.
How can I update 4 bytes in my buffer at a specific location to hold the value of an integer in python?
EDIT
I'm going to add more because I can't seem to figure it out from the solutions (though I can see they're on the right track).
First of all, I'm on python 2.4 (and can't upgrade, big corporation servers) - so that apparently limits my options. Sorry for not mentioning that earlier, I wasn't aware it had so many less features.
Secondly, let's make this ultra-simple.
Lets say I have a binary file named 'myfile.binary' with the five-byte contents '4C53535353' in hex - this equates to the ascii representations for letters "L and 4xS" being alone in the file.
If I do:
f = open('myfile.binary', 'rb')
contents = f.read(5)
contents should (from Sven Marnach's answer) hold a five-byte immutable string.
Using Python 2.4 facilities only, how could I change the 4 S's held in 'contents' to an arbitrary integer value? I.e. give me a line of code that can make byte indices contents [1-4] contain the 32-bit integer 'myint' with value 12345678910.
What you need is this function:
struct.pack_into(fmt, buffer, offset, v1, v2, ...)
It's documented at http://docs.python.org/library/struct.html near the top.
Example code:
import struct
import ctypes
data=ctypes.create_string_buffer(10)
struct.pack_into(">i", data, 5, 0x12345678)
print list(data)
Similar posting: Python: How to pack different types of data into a string buffer using struct.pack_into
EDIT: Added a Python 2.4 compatible example:
import struct
f=open('myfile.binary', 'rb')
contents=f.read(5)
data=list(contents)
data[0:4]=struct.pack(">i", 0x12345678)
print data
Have a look at Struct module. You need pack function.
EDIT:
The code:
import struct
s = "LSSSS" # your string
s = s[0] + struct.pack('<I', 1234567891) # note "shorter" constant than in your example
print s
Output:
L╙☻ЦI
struct.pack should be available in Python2.4.
Your number "12345678910" cannot be packed into 4 bytes, I shortened it a bit.
The result of file.read() is a string in Python, and it is immutable. Depending on the context of the task you are trying to accomplish, there are different solutions to the problem.
One is using the array module: Read the file directly as an array of 32-bit integers. You can modify this array and write it back to the file.
with open("filename") as f:
f.seek(0, 2)
size = f.tell()
f.seek(0)
data = array.array("i")
assert data.itemsize == 4
data.fromfile(f, size // 4)
data[2] = new_value
# use data.tofile(g) to write the data back to a new file g
You could install the numpy module, which is often used for scientific computing.
read_data = numpy.fromfile(file=id, dtype=numpy.uint32)
Then access the data at the desired location and make your changes.
The following is just a demonstration for you to understand what really happens when the four bytes are converted into an integer.
Suppose you have a number: 15213
Decimal: 15213
Binary: 0011 1011 0110 1101
Hex: 3 B 6 D
On little-endian systems (i.e x86 machines), this number can be represented using a length-4 bytearray as: b"\x6d\x3b\x00\x00" or b"m;\x00\x00" when you print it on the screen, to convert the four bytes into an integer, we simply do a bit of base conversion, which in this case, is:
sum(n*(256**i) for i,n in enumerate(b"\x6d\x3b\x00\x00"))
This gives you the result: 15213
serial.write() method in pyserial seems to only send string data. I have arrays like [0xc0,0x04,0x00] and want to be able to send/receive them via the serial port? Are there any separate methods for raw I/O?
I think I might need to change the arrays to ['\xc0','\x04','\x00'], still, null character might pose a problem.
An alternative method, without using the array module:
def a2s(arr):
""" Array of integer byte values --> binary string
"""
return ''.join(chr(b) for b in arr)
You need to convert your data to a string
"\xc0\x04\x00"
Null characters are not a problem in Python -- strings are not null-terminated the zero byte behaves just like another byte "\x00".
One way to do this:
>>> import array
>>> array.array('B', [0xc0, 0x04, 0x00]).tostring()
'\xc0\x04\x00'
I faced a similar (but arguably worse) issue, having to send control bits through a UART from a python script to test an embedded device. My data definition was "field1: 8 bits , field2: 3 bits, field3 7 bits", etc. It turns out you can build a robust and clean interface for this using the BitArray library. Here's a snippet (minus the serial set-up)
from bitstring import BitArray
cmdbuf = BitArray(length = 50) # 50 byte BitArray
cmdbuf.overwrite('0xAA', 0) # Init the marker byte at the head
Here's where it gets flexible. The command below replaces the 4 bits at
bit position 23 with the 4 bits passed. Note that it took a binary
bit value, given in string form. I can set/clear any bits at any location
in the buffer this way, without having to worry about stepping on
values in adjacent bytes or bits.
cmdbuf.overwrite('0b0110', 23)
# To send on the (previously opened) serial port
ser.write( cmdbuf )
Similar to this question, I am trying to read in an ID3v2 tag header and am having trouble figuring out how to get individual bytes in python.
I first read all ten bytes into a string. I then want to parse out the individual pieces of information.
I can grab the two version number chars in the string, but then I have no idea how to take those two chars and get an integer out of them.
The struct package seems to be what I want, but I can't get it to work.
Here is my code so-far (I am very new to python btw...so take it easy on me):
def __init__(self, ten_byte_string):
self.whole_string = ten_byte_string
self.file_identifier = self.whole_string[:3]
self.major_version = struct.pack('x', self.whole_string[3:4]) #this
self.minor_version = struct.pack('x', self.whole_string[4:5]) # and this
self.flags = self.whole_string[5:6]
self.len = self.whole_string[6:10]
Printing out any value except is obviously crap because they are not formatted correctly.
If you have a string, with 2 bytes that you wish to interpret as a 16 bit integer, you can do so by:
>>> s = '\0\x02'
>>> struct.unpack('>H', s)
(2,)
Note that the > is for big-endian (the largest part of the integer comes first). This is the format id3 tags use.
For other sizes of integer, you use different format codes. eg. "i" for a signed 32 bit integer. See help(struct) for details.
You can also unpack several elements at once. eg for 2 unsigned shorts, followed by a signed 32 bit value:
>>> a,b,c = struct.unpack('>HHi', some_string)
Going by your code, you are looking for (in order):
a 3 char string
2 single byte values (major and minor version)
a 1 byte flags variable
a 32 bit length quantity
The format string for this would be:
ident, major, minor, flags, len = struct.unpack('>3sBBBI', ten_byte_string)
Why write your own? (Assuming you haven't checked out these other options.) There's a couple options out there for reading in ID3 tag info from MP3s in Python. Check out my answer over at this question.
I am trying to read in an ID3v2 tag header
FWIW, there's already a module for this.
I was going to recommend the struct package but then you said you had tried it. Try this:
self.major_version = struct.unpack('H', self.whole_string[3:5])
The pack() function convers Python data types to bits, and the unpack() function converts bits to Python data types.