We Keep Coding

How do I send an Image rotate request to gprc server with python?

I am new to grpc. I am having trouble understanding it. Anyway, I am working based on a protobuf file called: image.proto
Below are the contents of my image.proto file:
syntax = "proto3";
option java_multiple_files = true;
message NLImage {
bool color = 1;
bytes data = 2;
int32 width = 3;
int32 height = 4;
}
message NLImageRotateRequest {
enum Rotation {
NONE = 0;
NINETY_DEG = 1;
ONE_EIGHTY_DEG = 2;
TWO_SEVENTY_DEG = 3;
}
Rotation rotation = 1;
NLImage image = 2;
}
service NLImageService {
rpc RotateImage(NLImageRotateRequest) returns (NLImage);
rpc MeanFilter(NLImage) returns (NLImage);
}
I was able to create a server.py file, and client.py. I also generated the image_pb2.py from the image.proto file and I generated the image_pb2_grpc.py.
Now, I am stuck into sending an image rotation request from the client to the server, and getting back an appropriate response.
Here is what I have so far tried in my client.py
import grpc
# import the generated files
import image_pb2
import image_pb2_grpc
#.......
#.......SEVERAL LINES OF CODE LATER
#.......
print('Input image path is', inputfile)
print('Output path is', outputfile)
print('Mean is', mean)
print("Rotate is", rotate)
channel = grpc.insecure_channel('localhost:5000')
stub = image_pb2_grpc.NLImageServiceStub(channel)
# SEND ROTATION REQUEST IF rotate
img_rotate_request = image_pb2.NLImageRotateRequest()
stub.RotateImage(img_rotate_request, inputfile)
I am not sure how to send that RotateImage request properly.
Below is my server.py:
import sys, getopt
from google.protobuf.descriptor import EnumDescriptor
import grpc
from concurrent import futures
import time
import numpy as np
# importing grpc generated classes
import image_pb2
import image_pb2_grpc
class NLImageServiceServicer(image_pb2_grpc.NLImageServiceServicer):
def RotateImage(self, request, context):
print("Request to rotate image received")
return image_pb2._NLIMAGE()
def MeanFilter(self, request, context):
print("Request to filter received")
return image_pb2.NLImage()
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
image_pb2_grpc.add_NLImageServiceServicer_to_server(NLImageServiceServicer, server)
server.add_insecure_port( host + ":" + str(port))
server.start()
print("Server started ...on " + host + ":" + port)
server.wait_for_termination()

Eyeballing your code, it appears OK; I've not run it.
You're correctly calling the method: stub.RotateImage but you're not correctly creating the message type (NLImageRotateRequest) for it. It will be an instance of the class generated for you by protoc, probably image_pb2.NLImageRotateRequest and this contains rotation and image properties. The image is itself an instance of a class image_pb2.Image etc.
Please see this link for using protobuf with Python because each language is distinct and Python has some quirks in how you create messages.
At some point, you'll need to read in inputfile as bytes in order to populate the data property of image.
If you've not, perhaps run through the Python HelloWorld example on grpc.io. It's basic but it will get you started with gRPC. Then review the link I included above and the protobuf sections on Python.

Create Shared Access Signature in Python - Azure Api Rest Management

sorry for my english.
I'm interacting with the Azure admin rest API and I want to programmatically create a SAS token. In Azure documentation is explained for C # (I attach the code below) but I need to implement it in Python (I'm new) and I can't get the data encoding and signing process correctly, even though I've searched a lot of information and tested in some different ways. Could someone help me to "translate" this code?. Thank you very much.
c#
using System;
using System.Text;
using System.Globalization;
using System.Security.Cryptography;
public class Program
{
public static void Main()
{
var id = "account-name";
var key = "account-key";
var expiry = DateTime.UtcNow.AddDays(10);
using (var encoder = new HMACSHA512(Encoding.UTF8.GetBytes(key)))
{
var dataToSign = id + "\n" + expiry.ToString("O", CultureInfo.InvariantCulture);
var hash = encoder.ComputeHash(Encoding.UTF8.GetBytes(dataToSign));
var signature = Convert.ToBase64String(hash);
var encodedToken = string.Format("SharedAccessSignature uid={0}&ex={1:o}&sn={2}", id, expiry, signature);
Console.WriteLine(encodedToken);
}
}
}

Based on #GaJsu 's solution
Following is my solution:
import base64
import hmac
import hashlib
from datetime import datetime
from dateutil.relativedelta import relativedelta
identifier = 'the "Identifier" value in "management API" tab'
end_date = datetime.now() + relativedelta(months=+6)
expiry = f'{end_date.isoformat()}0Z'
key_azure = 'the key in "management API" tab, primary/secondary key'
string_to_sign = f'{identifier}\n{expiry}'
signature = (
base64.b64encode(
hmac.new(
bytearray(key_azure, "utf-8"),
bytearray(string_to_sign,"utf-8") ,
hashlib.sha512).digest()
)
).decode("utf-8")
auth_sas = f"SharedAccessSignature uid={identifier}&ex={expiry}&sn={signature}"

Finally I can create the SAS token. Here is the code for the signature:
string_to_sign = '{}{}{}'.format(id_azure,'\n',expiry)
signature = (base64.b64encode(hmac.new(bytearray(key_azure, "utf-8") , bytearray(string_to_sign,"utf-8") , hashlib.sha512).digest())).decode("utf-8").replace("\n", "")

POST request remake from Java to python gets null

I am totally new to python flask and encountered a problem when writing some code using the requests and flask modules.
I am working on a project using the web API offered by the Panther platform. The project provided an example using Apache Java.
The source code is as below (see more for details).
public class TestProject {
public static void main(String args[]) throws Exception {
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
HttpPost httppost = new HttpPost("http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?");
StringBody organism = new StringBody("Homo sapiens", ContentType.TEXT_PLAIN);
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
StringBody enrichmentType = new StringBody("process", ContentType.TEXT_PLAIN);
StringBody testType = new StringBody("FISHER", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("FDR", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("BONFERRONI", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("NONE", ContentType.TEXT_PLAIN);
StringBody type = new StringBody("enrichment", ContentType.TEXT_PLAIN);
HttpEntity reqEntity = MultipartEntityBuilder.create()
.addPart("organism", organism)
.addPart("geneList", fileData)
.addPart("enrichmentType", enrichmentType)
.addPart("test_type", testType)
.addPart("type", type)
//.addPart("correction", cor)
.build();
httppost.setEntity(reqEntity);
CloseableHttpResponse response = httpclient.execute(httppost);
try {
//System.out.println("----------------------------------------");
//System.out.println(response.getStatusLine());
HttpEntity resEntity = response.getEntity();
if (resEntity != null) {
System.out.println(IOUtils.toString(resEntity.getContent(), StandardCharsets.UTF_8));
}
EntityUtils.consume(resEntity);
} finally {
response.close();
}
} finally {
httpclient.close();
}
}
}
The part I am mostly interested in is .addPart("organism", organism) and all the other code with similar structures. They will help pass the parameters from a third-party website to the web API offered by Panther.
I remade the JAVA code into python3 using requests. The code is as follows:
uploadTemp = {'file':open('./app/static/data_temp/temp.txt','rb')}
url="http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
params = {"organism":organism,"geneList":pantherName,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
# or params = {"organism":organism,"geneList":uploadTemp,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
Pantherpost= requests.post(url, params = params)
print(Pantherpost.text)
I am expecting an XML object from the web API including some basic biological information. However, the result I got was null (or \n\n\rnull\n when I print Pantherpost.content)
It seems that the parameters I have got from my own web were not correctly sent to the web API.
In addition to this getting null problem, as a beginner, I am also not quite sure about whether the "geneList" part should be receiving a plain-text object or a file. The manual says it is expecting a file, however, it may have been reformatted into plain-text by this command
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
Anyway, I did try both interpretations: pantherName is a list with name correctly formatted in plain-text and uploadTemp is a .txt file generated for the project. There must be some extra bugs in my code since it returned null in both cases.
Can someone please help out? Thank you very much.

I've found the following issues with your python code:
One. If you want to POST a file using requests, you should use keyword files=.
Two. Keys in files object should match respective parameters of the request (you're using file instead).
Three. You put your parameters in the wrong place of the request by writing params=params.
Function annotation from requests source code:
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:Request.
In example Java code StringBody is used to create parameters, which implies that parameters should be placed inside the body of HTTP request, not query string. So you should use data= keyword instead. If you use params=, output will be null.
SO article on difference between data and params keywords in requests.
So I've spent some time reading thier manual and made a test script:
import requests
url = "http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
filepath = "C:\\data\\YOUR_DATA.txt" # change to your file location
# all required parameters according to manual, except geneList which is a file (see below)
params = { # using defaults from manual
"type": "enrichment",
"organism": "Homo sapiens",
"enrichmentType": "process",
"test_type": "FISHER",
"correction": "FDR",
}
# note that the key here is the name of paramter: geneList
files = {'geneList': open(filepath, 'rb')}
# it outputs null, when 'params=params' is used
r = requests.post(url, data=params, files=files)
print(r.status_code)
print(r.text)
Output:
200
Id Name GeneId raw P-value FDR

Keeping track of multiple clients on grpc server

I am trying to create a grpc python server that can keep track of all clients connected.
I am referencing a talk/demo that Ray Tsang did where he kept a collection of StreamObservers and just iterated through them to send to all the clients. Here is a video of that for reference.
Now my question is how do you get a StreamObserver in python? I only see self, request and context as being available to me in the definition.
This is my first python project so there might be something obvious I am missing here.
Here is my proto, its basically the sample proto
syntax = "proto3";
package hellostreamingworld;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
// Sends another greeting
rpc SayHelloAgain (HelloRequest) returns (stream HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}

If I have understand you need to create a class ClientInterceptor that extends grpc.UnaryUnaryClientInterceptor(https://grpc.io/grpc/python/grpc.html?highlight=unaryunaryclientinterceptor#grpc.UnaryUnaryClientInterceptor)
and then assign it with the intercept_channel method in this way
self.channel = grpc.insecure_channel(address, options)
self.channel = grpc.intercept_channel(
self.channel,
ClientInterceptor()
)
You can use the intercept_unary_unary method for receive infos about the various clients.
If you want to have infos server side extends the ServerInterceptor(https://grpc.io/grpc/python/grpc.html?highlight=serverinterceptor#grpc.ServerInterceptor) and assign it on the server init
self.server = grpc.server(futures.ThreadPoolExecutor(max_workers=1000),
options=(('grpc.max_send_message_length', 1000 * 1024 * 1024,),
('grpc.max_receive_message_length', 1000 * 1024 * 1024,),
),
interceptors=(MyServerInterceptor())
)
And the use the intercept_service method for receive infos.

What you probably need is a bidirectional stream service. That is both the client and server are continuously sending data over. In this case gRPC + python will keep track of your client!
Check out this example code and corresponding explanation.

Why does ZeroMQ req-rep give me an empty response between C++ and Python?

I'm trying to bridge between a Python client and a C++ based server using ZeroMQ but I'm struggling to get the correct response returned from the server back to the client.
The Python client looks like this:
import zmq
import time
# Socket to talk to server
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect ("tcp://127.0.0.1:5563")
requestId = 0
while True:
request = "Message %d"%requestId
print ("Sending '%s'.. "%request)
socket.send_string(request)
response = socket.recv()
print ("Response:",response)
requestId += 1
time.sleep(1)
The C++ server looks like this:
zmq::context_t* context = new zmq::context_t();
zmq::socket_t* publisher = new zmq::socket_t(*context, ZMQ_REP);
unsigned int port = 5563;
std::ostringstream bindDest;
bindDest << "tcp://*:" << port;
publisher->bind(bindDest.str());
zmq::message_t request(0);
bool notInterrupted = true;
while (notInterrupted)
{
// Wait for a new request to act upon.
publisher->recv(&request, 0);
// Turn the incoming message into a string
char* requestDataBuffer = static_cast<char*>(request.data());
std::string requestStr(requestDataBuffer, request.size());
printf("Got request: %s\n", requestStr.c_str());
// Call the delegate to get a response we can pass back.
std::string responseString = requestStr + " response";
printf("Responding with: %s\n", responseString.c_str());
// Turn the response string into a message.
zmq::message_t responseMsg(responseString.size());
char* responseDataBuffer = static_cast<char*>(responseMsg.data());
const int OffsetToCopyFrom = 0;
responseString.copy(responseDataBuffer, responseString.size(), OffsetToCopyFrom);
// Pass back our response.
publisher->send(&responseMsg, 0);
}
When I run this, the client reports:
Sending 'Message 0'..
The server reports:
Got request: Message 0
Responding with: Message 0 response
But the python receives a blank message:
Response: b''
If I replace the C++ version, with a Python implementation as follows it works as expected:
import zmq
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind ("tcp://127.0.0.1:5563")
replyId = 0
while True:
request = socket.recv_string()
print("Received: "+request)
response = request+" response %d"%replyId
print ("Sending response: "+response)
socket.send_string(response)
replyId += 1
What am I doing wrong in the C++ version?
Update - Checked versions...
Reading around other people's problems, one thing people have suggested is that different versions sometimes cause issues. I double-checked and Python is using v4.1.6 whereas C++ is on v4.0.4.
I'm using the C++ pre-built libraries from here: http://zeromq.org/distro:microsoft-windows so I guess that could be the cause? I made a similar setup to publish from C++ to Python and that worked fine but perhaps something in the Req-Rep area has changed.
Update2 - It doesn't appear to be the versions...
After a lot of hacking around I finally managed to get v4.1.6 to build for the C++ version of the server and I still get the same empty message.

You need to send the buffer, not the pointer.
publisher->send(responseMsg, 0);
did the trick when I tested your code.
Hope this helps,
Hannu