I want to design a game server in python. The game will mostly just be passing small packets filled with ints, strings, and bytes stuffed into one message. As I'm using a different language to write the game, a normal packet would be sent like so:
Writebyte(buffer, 5); // Delimit type of message
Writestring(buffer, "Hello");
Sendmessage(buffer, socket);
As you can see, it writes the bytes to the buffer, and sends the buffer. Is there any way to read something like this in python? I am aware of the struct module, and I've used it to pack things, but I've never used it to actually read something with mixed types stuck into one message. Thanks for the help.
I would recommend using Google Protocol Buffers. Protocol Buffers gives you a multi-language, fast, extensible message serialization framework. You can easily add fields later, parse messages in most popular languages, and embed message types within other message types. It will save you a lot of time as compared to coding your own serialization framework.
Check out http://twistedmatrix.com/ and http://construct.wikispaces.com/
Related
I have an application in PyQT (UDP client) that send some parameters over UDP/IP to an application on raspberry (UDP server).
This Qt application has several fields like PID parameters, speed of the motor, sensors presets and so on.
Actually, the UDP client sends a string by getting the values from each field in the QT application and appending the data into the string with separator character (','), always in the same sequence. For instance, "142.0, 10.0, 2.0, negative, positive".
The UDP server receives this message, splits the message and moves each item of the list to the respective variable.
It works, but it is not smart, all parameters are sent even when one the parameter is not changed.
Whats should be the smart way to send only specific parameters, not depending of the right sequence? or only the changed ones?
Maybe some encapsulate protocol over the UDP message?
If you really want to keep things simple and change the existing code the least, you can include empty values for parameters with values that didn't change. E.g. if you have four parameters, then assuming they all changed you'd send 142,10,2,negative,positive, but if only the first two changed you'd send 142,10,,. But, such ad-hoc schemes should be IMHO discouraged.
You could use json with very short member strings. E.g.{"a":142,"b":10}. You'd have to keep a human-readable mapping between the short string keys and their meaning separate from the data. Since the strings can be any Unicode character, you have quite a way to go before you ran out of single characters to use. Also, Python natively supports json.
If you don't care much about the length of the packet, then you don't even need short member strings: make your packets-self documenting by using meaningful strings, such as {"velocity":142,"acceleration":10}.
I'm building up a nice application in python from the bottom up that allows encrypted communication between peers. This application able users to dynamically install new plugins and therefore new protocols.
How is used '\0' among common socket operations ? Is it a good separator or should I use something else ?
I would like to be able to manage my own socket code which prevents me from using libs that abstract those bytes constructions.
By the way I'm using Python 3 so all data sent or received is encoded. I'm using utf-8 by default.
The NUL-Byte (b'\0') has been and still is commonly used in binary protocols as separator or when transferring numbers (e.g.: 1 as a 32 bit integer is b'\x01\x00\x00\x00'). Its usage can therefor be considered completely safe with socket implementations on all platforms.
When encoding and decoding strings in Python 3 however, I'd recommend you insert those NUL-Bytes after encoding your string to bytes and stripping them (on the receiver side) before decoding your strings to Unicode.
I want to parse application layer protocols from network trace using Google protocol buffer and replay the trace (I am using python). I need suggestions to automatically generate protocol message description (in .proto file) from a network trace.
So you want to reconstruct what .proto messages were being passed over the application-layer protocol?
This isn't as easy as it sounds. First, .proto messages can't be sent raw over the wire, as the receiver needs to know how long they are. They need to be encapsulated somehow, maybe in an HTTP POST or with a raw 4-byte size prepended. I don't know what it would be for your application, but you'll need to deal with that.
Second, you can't reconstruct the full .proto from the messages alone. You only get tag numbers and types, not names. In addition, you will lose information about submessages - submessages and plain strings are encoded identically (you could probably tell which is which by eyeballing them, but I don't think you could do it automatically). You also will never know about optional items that never got sent. But you could parse the buffer without the proto and get some reasonable data (ints, repeated strings, and such).
Third, you need to reconstruct the application byte stream from the pcap log. I'm not sure how to do that, but I suspect there are tools that would do that for you.
I've been expanding on python version of Armstrong's classical example of interfaces. Everything works fine while I communicate bytes. But, I'd like to communicate long integers and floats. Mabye even (oh, no) strings. Here is my code:
http://pastebin.com/epxgDmvu
http://pastebin.com/rD7CWRkz
First of all, all that I know how to send are bytes. Can erlang send anything else to it's inteface? Or do I have to convert float to list of bytes, send it to python and then assemble it back to float in python?
And, the ohter way around: if I pack with 'f' format, erlang recognises that as a list of bytes. Is there a way to persuade erlang to take those bytes as one single float? If not, where can I find out how to convert that erlang list to erlang float?
In case erlang can communicate only bytes, how do you propose that I send a list of integers? If I convert integers to lists of bytes, then I can't send them in one message since reciever won't know where one integer ends and other begins, right? Should I then send integers one by one?
Yes, I am looking into ErlPort and py-interface and some other, but I'd like to start from fundamentals.
Regards,
dijxtra
At a low (from programming point of view) level you always send/receive only a bunch of bytes through different external interfaces (pipes, network, files...) independently from a programming language you're using. For example when you work with pipes (which you got calling open_port/2) you send/receive data as a stream of bytes.
To send any useful information through a stream you need to divide it to chunks which usually named messages. Message format define how messages can be extracted from a stream of bytes. For example using Type-Length-Value (TLV) encoding you can send values of different length marked with a type.
In ErlPort it all works as following:
When you call open_port/2 you must add option {packet, N} so all binary messages sent through this port are preceded by their length, sent in N bytes (Length-Value encoding).
Before to send a term to the port you must use term_to_binary/1 function which encode Erlang term to a bunch of bytes using Erlang external term format.
On Python side when you instantiate erlproto.Port class you must pass the same packet value as you pass to open_port/2 (packet=1 by default).
Port.read method first read the length of the message from the stream and then the message body.
The message body decoded by the erlterms.decode function.
When you send a reply using Port.write method it's first encoded to Erlang external term format by erlterms.encode function.
Then the encoded chunk prepended by its length sent back to Erlang.
On Erlang side binary_to_term/1 is used to decode the data.
In Twisted when implementing the dataReceived method, there doesn't seem to be any examples which refer to packets being fragmented. In every other language this is something you manually implement, so I was just wondering if this is done for you in twisted already or what? If so, do I need to prefix my packets with a length header? Or do I have to do this manually? If so, what way would that be?
In the dataReceived method you get back the data as a string of indeterminate length meaning that it may be a whole message in your protocol or it may only be part of the message that some 'client' sent to you. You will have to inspect the data to see if it comprises a whole message in your protocol.
I'm currently using Twisted on one of my projects to implement a protocol and decided to use the struct module to pack/unpack my data. The protocol I am implementing has a fixed header size so I don't construct any messages until I've read at least HEADER_SIZE amount of bytes. The total message size is declared in this header data portion.
I guess you don't really need to define a message length as part of your protocol but it helps. If you didn't define one you would have to have a special delimiter that determines when a message begins/ends. Sort of how the FIX protocol uses the SOH byte to delimit fields. Though it does have a required field that tells you how long a message is (just not how many fields are in a message).
When dealing with TCP, you should really forget all notion of 'packets'. TCP is a stream protocol - you stream data in and data streams out the other side. Once the data is sent, it is allowed to arrive in as many or as few blocks as it wants, as long as the data all arrives in the right order. You'll have to manually do the delimitation as with other languages, with a length field, or a message type field, or a special delimiter character, etc.
You can also use a LineReceiver protocol