I'm new to Golang and this kind of more lowlevel stuff, so maybe i'm just thinking in the wrong direction.
My project is a small golang monitoring client which sends some ping-message (current date) over a encrypted TLS connection (this is just for learning purpose).
The python server and client are working flawless for now.
The (python) server & client are packing the data like this:
[...]
def _send_msg(self, msg):
msg = struct.pack('>I', len(msg)) + bytes(msg, 'UTF-8')
self.client.send(msg)
def _recv_msg(self):
raw_msglen = self.client.recv(4)
if not raw_msglen:
return ""
msglen = struct.unpack('>I', raw_msglen)[0]
return self._recvall(msglen)
[...]
On the Go-Side I pack the data like this:
type packet struct {
a uint32
b []byte
}
func pack(d string) []byte {
buf := bytes.Buffer{}
u := len([]byte(d))
p := packet{
a: uint32(u),
b: []byte(d),
}
err := binary.Write(&buf, binary.BigEndian, p.a) // struct.unpack('>I' [...] in python
if err != nil {
fmt.Println(err)
}
buf.Write(p.b) // Append bytes to buffer
return buf.Bytes()
}
reader := bufio.NewReader(tlsConn) // Socket reader
[..]
// Writing binary data
tlsConn.Write(pack("login test:test")) // Works so far
// Reading response (from python server)
var p uint32 // packet size
binary.Read(reader, binary.BigEndian, &p) // Read packet length (4 bytes uint32)
buf1 := make([]byte, int(p))
reader.Read(buf1)
fmt.Println(string(buf1)) // Print response - this also works
// Send some ping
// This code part also get's passed, but the python server doesn't
// recive any message
c.Write(pack("ping 0000-00-00 00:00:00"))
binary.Read(reader, binary.BigEndian, &p)
buf1 = make([]byte, int(p))
fmt.Println(string(buf1)) // Here i don't get an response
Here is the golang client-side test code which I'm using: http://pastebin.com/dr1mJZ9Y
and the corresponding terminal output: http://pastebin.com/mrctJPs5
The strange thing is, that the login message (including the server response) all get sent (and received) correctly but the second time when i try to send a ping like:
tlsConn.Write(pack("ping 0000-00-00 00:00:00"))
The message seems not to reach the server and no response message gets send back to the client.
Is there an error with the binary encoding of the packet-data and the length prefixing?
I know the code looks a bit messy and not very go-idiomatic, sorry for that.
Thank you again in advance for your help!
Software dependencies / environment:
Python 3.4
Golang 1.5.3 (amd64 Linux)
No 3rd-Party libs
Related
For some reason, I want to transfer file (not in stream) with grpc in python. The protobuf and python code are as follows
protobuf
syntax = "proto3";
import "google/protobuf/empty.proto";
package xxxx;
service gRPCComServeFunc {
rpc sendToClient(FileRequest) returns (google.protobuf.Empty) {}
}
message FileRequest {
FileInfo info = 1;
bytes chunk = 2;
}
message FileInfo {
int32 sender = 1;
int32 state = 2;
string timestamp = 3;
}
python
def get_file_content(path_file):
with open(path_file, 'rb') as f:
content = f.read()
return content
def pack_file_request(state, chunk):
request = gRPC_comm_manager_pb2.FileRequest(
info=gRPC_comm_manager_pb2.FileInfo(
sender=0,
state=state,
timestamp=""),
chunk=chunk
)
return request
...
chunk = get_file_content(path_file)
request = pack_file_request(state, chunk)
stub.sendToClient(request)
First, the server reports error as follows
...
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Received RST_STREAM with error code 8"
debug_error_string = "UNKNOWN:Error received from peer 0.0.0.0:50078 {created_time:"2023-01-04T22:02:11.242643217+08:00", grpc_status:1, grpc_message:"Received RST_STREAM with error code 8"}"
It looks like the message is sent successfully, but refused by the remote client at 0.0.0.0:50078.
So I guess it's caused by the file chunk, and this time I send the message without chunk as follows
def pack_file_request(state, chunk):
request = gRPC_comm_manager_pb2.FileRequest(
info=gRPC_comm_manager_pb2.FileInfo(
sender=0,
state=state,
timestamp=""),
)
return request
And it works. Therefore I guess I cannot directly read the file and pass it to grpc services in python.
So how should I read the file and pass it to the parameter chunk in grpc service sendToClient?
I have been trying to establish a bidirectional communication between a C# local server and a Python local server. What I wish to achieve:
Send a string from C# to Python
Compute the string in Python and send back the result to C#
Do this as the Python instance is continuously running as I require a time-expensive package import
My C# code for the pipe creation:
PipeSecurity pSecure = new PipeSecurity();
pSecure.SetAccessRule(new PipeAccessRule("Everyone", PipeAccessRights.FullControl, System.Security.AccessControl.AccessControlType.Allow));
using (NamedPipeServerStream pipeClient = new NamedPipeServerStream("NLP", PipeDirection.InOut)) {
pipeClient.SetAccessControl(pSecure);
Console.WriteLine("Attempting to connect to pipe...");
pipeClient.WaitForConnection();
Console.WriteLine("Is pipe connected? " + pipeClient.IsConnected);
Console.WriteLine("Connected to pipe");
using (StreamReader sr = new StreamReader(pipeClient)) {
using (StreamWriter sw = new StreamWriter(pipeClient)) {
// initialize python server
sw.WriteLine("true; nothing; false");
foreach (AlignedVariation variation in _nonConformancePatterns.Keys) {
foreach (NonConformancePattern pattern in _nonConformancePatterns[variation]) {
// create argument to be parsed by Python
string pythonParameter = "false; ";
foreach (string activityName in pattern.EventsSkippedNames()) {
pythonParameter += activityName + "|";
}
pythonParameter += "; false";
sw.WriteLine(pythonParameter);
pattern.SetLingvisticTag(sr.ReadLine()!);
}
}
// Close connection to Python server
sw.WriteLine("false; nothing; true");
}
}
}
My Python code for the pipe interpretation:
class PipeServer():
def __init__(self, pipeName):
self.pipe = win32pipe.CreateNamedPipe(r'\\.\pipe\NLP', win32pipe.PIPE_ACCESS_DUPLEX,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_READMODE_MESSAGE |
win32pipe.PIPE_WAIT, 1, 65536, 65536, 0, None)
#Carefull, this blocks until a connection is established
def connect(self):
win32pipe.ConnectNamedPipe(self.pipe, None)
#Message without tailing '\n'
def write(self, message):
win32file.WriteFile(self.pipe, message.encode()+b'\n')
def read(self):
return win32file.ReadFile(self.pipe)
def close(self):
win32file.CloseHandle(self.pipe)
I could not find a better solution that does not imply the creation of a file in Python for when reading from a pipe. Furthermore, I have read on the documentation of the pywin32 package but it mentions when reading from a pipe the only solution is to use a PyHandle class to handle the reading but I have seen solutions doing this only from reading a file.
How can I change the Python code to be able to access the pipeline? At the moment whenever I want to connect the Python instance to the pipe created in C# I get the error that permission is denied.
Also, is the code for sending and receiving messages correct in C#?
Please review below context and help resolve the issue
I define below message in device.proto
message DeviceConfiguration {
message Resolution {
uint32 width = 1;
uint32 height = 2;
}
string device_name = 1;
string brand_name = 2;
Resolution resolution = 3;
}
Then, compile this message into 2 languages: Nodejs (device_pb.js) and Python(device_pb2.py)
1. From Nodejs: send above message to Kafka
const {DeviceConfiguration} = require("./device_pb")
const resolution = new DeviceConfiguration.Resolution();
resolution.setWidth(1280);
resolution.setHeight(960);
const deviceConfiguration = new DeviceConfiguration();
deviceConfiguration.setDeviceName("S6");
deviceConfiguration.setBrandName("samsung");
deviceConfiguration.setResolution(resolution);
let binaryEvent = deviceConfiguration.serializeBinary();
Finally, use Kafka-node to send binaryEvent value to a Kafka topic
2. From Python component: consume the message via kafka
Here is binary value received in python component
b'10,8,111,98,106,101,99,116,73,100,26,21,10,2,83,54,18,7,115,97,109,115,117,110,103,26,6,8,128,10,16,192,7'
Then, I use below code to deserialize the message but it throws a message "Error parsing message"
from device_pb2 import DeviceConfiguration
device = DeviceConfiguration()
device.ParseFromString(message)
=> As I see that the ParseFromString doesn't work for above binary value and throws "Error parsing message". It only works if the bytes value is created by SerializeToString python code.
Note that: I'm able to deserialize the binary by using deserializeBinary (make it public) in Nodejs but there is no similar function in device_pb2.py
So, is there any way for me to deserialize the message in Python code?
Many Thanks.
I'm trying to adapt this Java codeLabs to Python / SocketIO (Flask Socket IO on server side).
My socket is working, I'm able to pass both the sample rate (and to retrieve on server side), and the audio data.
The problem is that the format of my audio data is not correct. I send it this way:
// Create a node that sends raw bytes across the websocket
var scriptNode = context.createScriptProcessor(4096, 1, 1);
// Need the maximum value for 16-bit signed samples, to convert from float.
const MAX_INT = Math.pow(2, 16 - 1) - 1;
scriptNode.addEventListener('audioprocess', function(e) {
var floatSamples = e.inputBuffer.getChannelData(0);
// The samples are floats in range [-1, 1]. Convert to 16-bit signed
// integer.
socket.emit('audiodata', Int16Array.from(floatSamples.map(function(n) {
return n * MAX_INT;
})));
This is the code from the codeLabs but I use socket.emit instead of socket.send.
The python code to handle and send it to Speech API is this one:
credentials = service_account.Credentials.from_service_account_file(CONFIG[u"service_account"])
client = speech.SpeechClient(credentials=credentials)
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=44100,
language_code=u"fr"
)
streaming_config = types.StreamingRecognitionConfig(
config=config,
interim_results=True,
single_utterance=False)
#socketio.on(u'audiodata')
def handle_audio_data_msg(audio_data_msg):
print u'Received raw audio.'
print audio_data_msg
for response in client.streaming_recognize(streaming_config, [audio_data_msg]):
print response
emit(u'audiodatareceived', {u"AudioData": u"Acquired"})
I got TypeError: descriptor 'SerializeToString' requires a 'google.protobuf.pyext._message.CMessage' object but received a 'dict'
so I tried, on front side, to pass Int16Array.from(...).buffer but I get TypeError: descriptor 'SerializeToString' requires a 'google.protobuf.pyext._message.CMessage' object but received a 'str'
So I'm not sure how I should pass the data... Any help appreciated!
[edit]
I re-wrote my handler cause I think I was not using the right type, it is now like that:
#socketio.on(u'audiodata')
def handle_audio_data_msg(audio_data_msg):
print u'Received raw audio.'
request = types.StreamingRecognizeRequest(audio_content=audio_data_msg)
response = client.streaming_recognize(streaming_config, [request])
for item in response:
print u"error", item.get(u"error")
print u"results", item.get(u"results")
print u"resultIndex", item.get(u"resultIndex")
print u"endpointerType", item.get(u"endpointerType")
emit(u'audiodatareceived', {u"AudioData": u"Acquired"})
I don't get any error but I get no response (empty list) and so nothing printed...
[edit2]: I realized that the type of my input was never binary data but str. Could the problem come from here ? I try to instantiate my Flask Socket IO app as follow, but still the type is str.
socketio = SocketIO(app, binary=True)
I'm trying to bridge between a Python client and a C++ based server using ZeroMQ but I'm struggling to get the correct response returned from the server back to the client.
The Python client looks like this:
import zmq
import time
# Socket to talk to server
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect ("tcp://127.0.0.1:5563")
requestId = 0
while True:
request = "Message %d"%requestId
print ("Sending '%s'.. "%request)
socket.send_string(request)
response = socket.recv()
print ("Response:",response)
requestId += 1
time.sleep(1)
The C++ server looks like this:
zmq::context_t* context = new zmq::context_t();
zmq::socket_t* publisher = new zmq::socket_t(*context, ZMQ_REP);
unsigned int port = 5563;
std::ostringstream bindDest;
bindDest << "tcp://*:" << port;
publisher->bind(bindDest.str());
zmq::message_t request(0);
bool notInterrupted = true;
while (notInterrupted)
{
// Wait for a new request to act upon.
publisher->recv(&request, 0);
// Turn the incoming message into a string
char* requestDataBuffer = static_cast<char*>(request.data());
std::string requestStr(requestDataBuffer, request.size());
printf("Got request: %s\n", requestStr.c_str());
// Call the delegate to get a response we can pass back.
std::string responseString = requestStr + " response";
printf("Responding with: %s\n", responseString.c_str());
// Turn the response string into a message.
zmq::message_t responseMsg(responseString.size());
char* responseDataBuffer = static_cast<char*>(responseMsg.data());
const int OffsetToCopyFrom = 0;
responseString.copy(responseDataBuffer, responseString.size(), OffsetToCopyFrom);
// Pass back our response.
publisher->send(&responseMsg, 0);
}
When I run this, the client reports:
Sending 'Message 0'..
The server reports:
Got request: Message 0
Responding with: Message 0 response
But the python receives a blank message:
Response: b''
If I replace the C++ version, with a Python implementation as follows it works as expected:
import zmq
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind ("tcp://127.0.0.1:5563")
replyId = 0
while True:
request = socket.recv_string()
print("Received: "+request)
response = request+" response %d"%replyId
print ("Sending response: "+response)
socket.send_string(response)
replyId += 1
What am I doing wrong in the C++ version?
Update - Checked versions...
Reading around other people's problems, one thing people have suggested is that different versions sometimes cause issues. I double-checked and Python is using v4.1.6 whereas C++ is on v4.0.4.
I'm using the C++ pre-built libraries from here: http://zeromq.org/distro:microsoft-windows so I guess that could be the cause? I made a similar setup to publish from C++ to Python and that worked fine but perhaps something in the Req-Rep area has changed.
Update2 - It doesn't appear to be the versions...
After a lot of hacking around I finally managed to get v4.1.6 to build for the C++ version of the server and I still get the same empty message.
You need to send the buffer, not the pointer.
publisher->send(responseMsg, 0);
did the trick when I tested your code.
Hope this helps,
Hannu