We Keep Coding

Why are the UUIDs from this AWS network socket backwards?

When you use the AWS API to run a command on a remote docker container (ECS), the AWS API gives you back a websocket to read the output of your command from. When using the aws command line utility (which also uses the AWS API), reading the websocket stream is handled by session-manager-plugin.
session-manager-plugin is written in GoLang, and I've been trying to rewrite parts of it in Python. I don't speak GoLang, but I fumbled my way though adding some code to session-manager-plugin to output the raw binary data it is sending, and receiving when the binary is being used.
Essentially, the output of the command you ran is split up into messages, each one with headers, and a payload. One of the headers for each message is a messageID, which is a UUID. Each message needs to be acknowledged by telling the server that you're received the message with that UUID.
The issue I'm having is that when analyzing the raw binary data, I can see that a message that was received with UUID b'\x85\xc3\x12P\n\x08\xaf)\xfd\xba\x1b8\x1asMd' is being acknowledged by session-manager-plugin with a packet that says this:
b'{"AcknowledgedMessageType":"output_stream_data","AcknowledgedMessageId":"fdba1b38-1a73-4d64-85c3-12500a08af29","AcknowledgedMessageSequenceNumber":4,"IsSequentialMessage":true}'
To figure out what UUID b'\x85\xc3\x12P\n\x08\xaf)\xfd\xba\x1b8\x1asMd' is in Python, I do this:
import uuid
print(str(uuid.UUID(bytes=b'\x85\xc3\x12P\n\x08\xaf)\xfd\xba\x1b8\x1asMd')))
# 85c31250-0a08-af29-fdba-1b381a734d64
At first glance, the UUID of the message that was received, and the UUID of the message being acknowledged do not match, but if you look closely, you'll see that the UUID of the original message that was received is reversed from the UUID being acknowledged. Sort of. In the 16 byte UUID, the first 8 bytes come after the last 8 bytes.
85c31250-0a08-af29-fdba-1b381a734d64
fdba1b38-1a73-4d64-85c3-12500a08af29
Is there any reason this would be happening? Am I decoding b'\x85\xc3\x12P\n\x08\xaf)\xfd\xba\x1b8\x1asMd' wrong?
Note: As you can see from above, the UUID in the Acknowledgement packet is inside of JSON. If I was decoding it wrong, the whole thing would be gibberish.
Also note that this is just an analysis of a perfectly working session-manager-plugin communication stream. One way or another, this actually works. I'm just trying to figure out how so I can re-create it.

Looking at the source code for session-manager-plugin, it would appear it reads the first eight bytes as the least significant bytes, then reads the next eight bytes as the most significant bytes, then appends it in the order MSB, LSB. Seems to me like that would produce the behavior you're seeing.
// getUuid gets the 128bit uuid from an array of bytes starting from the offset.
func getUuid(log log.T, byteArray []byte, offset int) (result uuid.UUID, err error) {
byteArrayLength := len(byteArray)
if offset > byteArrayLength-1 || offset+16-1 > byteArrayLength-1 || offset < 0 {
log.Error("getUuid failed: Offset is invalid.")
return nil, errors.New("Offset is outside the byte array.")
}
leastSignificantLong, err := getLong(log, byteArray, offset)
if err != nil {
log.Error("getUuid failed: failed to get uuid LSBs Long value.")
return nil, errors.New("Failed to get uuid LSBs long value.")
}
leastSignificantBytes, err := longToBytes(log, leastSignificantLong)
if err != nil {
log.Error("getUuid failed: failed to get uuid LSBs bytes value.")
return nil, errors.New("Failed to get uuid LSBs bytes value.")
}
mostSignificantLong, err := getLong(log, byteArray, offset+8)
if err != nil {
log.Error("getUuid failed: failed to get uuid MSBs Long value.")
return nil, errors.New("Failed to get uuid MSBs long value.")
}
mostSignificantBytes, err := longToBytes(log, mostSignificantLong)
if err != nil {
log.Error("getUuid failed: failed to get uuid MSBs bytes value.")
return nil, errors.New("Failed to get uuid MSBs bytes value.")
}
uuidBytes := append(mostSignificantBytes, leastSignificantBytes...)
return uuid.New(uuidBytes), nil
}
Source Code

How to view c-message from azure service queue in Python

I am using Azure service bus to view message contents of a messages sent from an service bus service.
I am using the message.get_message() command and it returns a c message as specified in the documentation
https://learn.microsoft.com/en-us/python/api/uamqp/uamqp.message.message?view=azure-python
Is there a way to view the contents of the c message?
<uamqp.c_uamqp.cMessage object at 0x0000000003180F50>

What do you mean about the contents of the c message?
I think the content of the c message you received is the date to send in the message. So according to the offical API reference Message class, you should use message.get_data() to get the body of the message to view its content, as the figure below.
Or just only to view it by print(...) like the sample code azure-uamqp-python/samples/sample_uamqp_receive_simple.py.
message = uamqp.receive_message(uri, auth=plain_auth)
print("Received: {}".format(message))
If your real intention is to parse the uamqp.c_uamqp.cMessage data structure of the massage to view the internal contents, you can refer to the SO thread Parsing C Structs in Python with the source codes message.h & message.c of Azure/azure-uamqp-c and the Python wrapper code message.pyx to try to parse it.
After I reviewed them above, the core data structure of uamqp.c_uamqp.cMessage is as below.
typedef struct MESSAGE_INSTANCE_TAG
{
BODY_AMQP_DATA* body_amqp_data_items;
size_t body_amqp_data_count;
AMQP_VALUE* body_amqp_sequence_items;
size_t body_amqp_sequence_count;
AMQP_VALUE body_amqp_value;
HEADER_HANDLE header;
delivery_annotations delivery_annotations;
message_annotations message_annotations;
PROPERTIES_HANDLE properties;
application_properties application_properties;
annotations footer;
uint32_t message_format;
} MESSAGE_INSTANCE;
Hope it helps.

Google Speech API from client micro stream : data format

I'm trying to adapt this Java codeLabs to Python / SocketIO (Flask Socket IO on server side).
My socket is working, I'm able to pass both the sample rate (and to retrieve on server side), and the audio data.
The problem is that the format of my audio data is not correct. I send it this way:
// Create a node that sends raw bytes across the websocket
var scriptNode = context.createScriptProcessor(4096, 1, 1);
// Need the maximum value for 16-bit signed samples, to convert from float.
const MAX_INT = Math.pow(2, 16 - 1) - 1;
scriptNode.addEventListener('audioprocess', function(e) {
var floatSamples = e.inputBuffer.getChannelData(0);
// The samples are floats in range [-1, 1]. Convert to 16-bit signed
// integer.
socket.emit('audiodata', Int16Array.from(floatSamples.map(function(n) {
return n * MAX_INT;
})));
This is the code from the codeLabs but I use socket.emit instead of socket.send.
The python code to handle and send it to Speech API is this one:
credentials = service_account.Credentials.from_service_account_file(CONFIG[u"service_account"])
client = speech.SpeechClient(credentials=credentials)
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=44100,
language_code=u"fr"
)
streaming_config = types.StreamingRecognitionConfig(
config=config,
interim_results=True,
single_utterance=False)
#socketio.on(u'audiodata')
def handle_audio_data_msg(audio_data_msg):
print u'Received raw audio.'
print audio_data_msg
for response in client.streaming_recognize(streaming_config, [audio_data_msg]):
print response
emit(u'audiodatareceived', {u"AudioData": u"Acquired"})
I got TypeError: descriptor 'SerializeToString' requires a 'google.protobuf.pyext._message.CMessage' object but received a 'dict'
so I tried, on front side, to pass Int16Array.from(...).buffer but I get TypeError: descriptor 'SerializeToString' requires a 'google.protobuf.pyext._message.CMessage' object but received a 'str'
So I'm not sure how I should pass the data... Any help appreciated!
[edit]
I re-wrote my handler cause I think I was not using the right type, it is now like that:
#socketio.on(u'audiodata')
def handle_audio_data_msg(audio_data_msg):
print u'Received raw audio.'
request = types.StreamingRecognizeRequest(audio_content=audio_data_msg)
response = client.streaming_recognize(streaming_config, [request])
for item in response:
print u"error", item.get(u"error")
print u"results", item.get(u"results")
print u"resultIndex", item.get(u"resultIndex")
print u"endpointerType", item.get(u"endpointerType")
emit(u'audiodatareceived', {u"AudioData": u"Acquired"})
I don't get any error but I get no response (empty list) and so nothing printed...
[edit2]: I realized that the type of my input was never binary data but str. Could the problem come from here ? I try to instantiate my Flask Socket IO app as follow, but still the type is str.
socketio = SocketIO(app, binary=True)

Why does ZeroMQ req-rep give me an empty response between C++ and Python?

I'm trying to bridge between a Python client and a C++ based server using ZeroMQ but I'm struggling to get the correct response returned from the server back to the client.
The Python client looks like this:
import zmq
import time
# Socket to talk to server
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect ("tcp://127.0.0.1:5563")
requestId = 0
while True:
request = "Message %d"%requestId
print ("Sending '%s'.. "%request)
socket.send_string(request)
response = socket.recv()
print ("Response:",response)
requestId += 1
time.sleep(1)
The C++ server looks like this:
zmq::context_t* context = new zmq::context_t();
zmq::socket_t* publisher = new zmq::socket_t(*context, ZMQ_REP);
unsigned int port = 5563;
std::ostringstream bindDest;
bindDest << "tcp://*:" << port;
publisher->bind(bindDest.str());
zmq::message_t request(0);
bool notInterrupted = true;
while (notInterrupted)
{
// Wait for a new request to act upon.
publisher->recv(&request, 0);
// Turn the incoming message into a string
char* requestDataBuffer = static_cast<char*>(request.data());
std::string requestStr(requestDataBuffer, request.size());
printf("Got request: %s\n", requestStr.c_str());
// Call the delegate to get a response we can pass back.
std::string responseString = requestStr + " response";
printf("Responding with: %s\n", responseString.c_str());
// Turn the response string into a message.
zmq::message_t responseMsg(responseString.size());
char* responseDataBuffer = static_cast<char*>(responseMsg.data());
const int OffsetToCopyFrom = 0;
responseString.copy(responseDataBuffer, responseString.size(), OffsetToCopyFrom);
// Pass back our response.
publisher->send(&responseMsg, 0);
}
When I run this, the client reports:
Sending 'Message 0'..
The server reports:
Got request: Message 0
Responding with: Message 0 response
But the python receives a blank message:
Response: b''
If I replace the C++ version, with a Python implementation as follows it works as expected:
import zmq
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind ("tcp://127.0.0.1:5563")
replyId = 0
while True:
request = socket.recv_string()
print("Received: "+request)
response = request+" response %d"%replyId
print ("Sending response: "+response)
socket.send_string(response)
replyId += 1
What am I doing wrong in the C++ version?
Update - Checked versions...
Reading around other people's problems, one thing people have suggested is that different versions sometimes cause issues. I double-checked and Python is using v4.1.6 whereas C++ is on v4.0.4.
I'm using the C++ pre-built libraries from here: http://zeromq.org/distro:microsoft-windows so I guess that could be the cause? I made a similar setup to publish from C++ to Python and that worked fine but perhaps something in the Req-Rep area has changed.
Update2 - It doesn't appear to be the versions...
After a lot of hacking around I finally managed to get v4.1.6 to build for the C++ version of the server and I still get the same empty message.

You need to send the buffer, not the pointer.
publisher->send(responseMsg, 0);
did the trick when I tested your code.
Hope this helps,
Hannu

Golang - Sending binary encoded data via TLS Socket

I'm new to Golang and this kind of more lowlevel stuff, so maybe i'm just thinking in the wrong direction.
My project is a small golang monitoring client which sends some ping-message (current date) over a encrypted TLS connection (this is just for learning purpose).
The python server and client are working flawless for now.
The (python) server & client are packing the data like this:
[...]
def _send_msg(self, msg):
msg = struct.pack('>I', len(msg)) + bytes(msg, 'UTF-8')
self.client.send(msg)
def _recv_msg(self):
raw_msglen = self.client.recv(4)
if not raw_msglen:
return ""
msglen = struct.unpack('>I', raw_msglen)[0]
return self._recvall(msglen)
[...]
On the Go-Side I pack the data like this:
type packet struct {
a uint32
b []byte
}
func pack(d string) []byte {
buf := bytes.Buffer{}
u := len([]byte(d))
p := packet{
a: uint32(u),
b: []byte(d),
}
err := binary.Write(&buf, binary.BigEndian, p.a) // struct.unpack('>I' [...] in python
if err != nil {
fmt.Println(err)
}
buf.Write(p.b) // Append bytes to buffer
return buf.Bytes()
}
reader := bufio.NewReader(tlsConn) // Socket reader
[..]
// Writing binary data
tlsConn.Write(pack("login test:test")) // Works so far
// Reading response (from python server)
var p uint32 // packet size
binary.Read(reader, binary.BigEndian, &p) // Read packet length (4 bytes uint32)
buf1 := make([]byte, int(p))
reader.Read(buf1)
fmt.Println(string(buf1)) // Print response - this also works
// Send some ping
// This code part also get's passed, but the python server doesn't
// recive any message
c.Write(pack("ping 0000-00-00 00:00:00"))
binary.Read(reader, binary.BigEndian, &p)
buf1 = make([]byte, int(p))
fmt.Println(string(buf1)) // Here i don't get an response
Here is the golang client-side test code which I'm using: http://pastebin.com/dr1mJZ9Y
and the corresponding terminal output: http://pastebin.com/mrctJPs5
The strange thing is, that the login message (including the server response) all get sent (and received) correctly but the second time when i try to send a ping like:
tlsConn.Write(pack("ping 0000-00-00 00:00:00"))
The message seems not to reach the server and no response message gets send back to the client.
Is there an error with the binary encoding of the packet-data and the length prefixing?
I know the code looks a bit messy and not very go-idiomatic, sorry for that.
Thank you again in advance for your help!
Software dependencies / environment:
Python 3.4
Golang 1.5.3 (amd64 Linux)
No 3rd-Party libs