I'm writing an http server in python3.3, just to learn how to do this sort of thing. In my function that parses a request, I want to use fcntl.ioctl to get the number of bytes that I can read in the socket, and I only do this when I see a kevent in the result of checking a kqueue that says there is stuff to read on the socket. But whenever I try to call fcntl.ioctl, I get OSError: [Errno 14] Bad address. What am I doing wrong? Also, this seems to be happening on the first call. Here is the relevant code:
def client_thread(kq, client_socket, methods):
while True:
events = kq.control([], 2, POLLTIME) #we pass an empty list of changes, because we don't have any changes to make to the events we are interested in.
#we want a list that is at most two long. We listen for POLLTIME seconds.
for event in events:
if event != KILL_KEV: #there are only two events in our kqueue
handle_client(client_socket, methods)
else: #KILL_SOCK has a connection
break
client_socket.close()
client_socket.shutdown()
def handle_client(client_socket, methods):
request = parse_request(client_socket) #parse the request data in the client socket
handlers = methods[request["request"]["method"]] #retrieve the appropriate list of handlers from the methods dict
for path_match_pred, handler_func in handlers:
if path_match_pred(path): #if the path matches whatever path predicate you've created...
break
response = handler_func(request) #... then call the appropriate handler function to handle the request
send_response(client_socket, response) #and finally, send the response.
def parse_request(client_socket):
"""Returns the request data, parsed into a dictionary like this:
{
"request": {
"method": method,
"path": path,
"version": HTTP version
},
"headers": header dictionary,
"body": body data as a string
}
This should only be called if the client socket is ready for reading!
"""
client_fd = client_socket.fileno() #get the file descriptor for the socket
bytes_in_socket = 0
fcntl.ioctl(client_fd, termios.FIONREAD, bytes_in_socket) #count the bytes in it
#^^^^^^^^^THIS IS WHERE IT BREAKS
print(bytes_in_socket, "bytes in socket")
msg = bytearray() #make empty byte array
while bytes_in_socket:
msg.extend(client_socket.recv(bytes_in_socket)) #read the bytes we counted earlier
fcntl.ioctl(client_fd, termios.FIONREAD, bytes_in_socket) #check for more bytes
print(bytes_in_socket, "bytes left to read")
Note that in fcntl.ioctl's documentation, they mention the acceptable types for arg (the argument to the ioctl operation). In some cases, (like this), you need to pass a buffer, or an object that supports its interface. In particular, when you want to receive a value back. You're just passing an integer.
Related
When I run my gRPC client and it attempts to stream a request to the server I get this error: "TypeError: has type list_iterator, but expected one of: bytes, unicode"
Do I need to encode the text I'm sending in some way? Error message makes some sense, as I am definitely passing in an iterator. I assumed from the gRPC documentation that this is what was needed. (https://grpc.io/docs/tutorials/basic/python.html#request-streaming-rpc)Anyway, sending a list or string yields a similar error.
At the moment I am sending a small test list of strings to the server in the request, but I plan to stream requests with very large amounts of text in the future.
Here's some of my client code.
def gen_tweet_space(text):
for tweet in text:
yield tweet
def run():
channel = grpc.insecure_channel('localhost:50050')
stub = ProseAndBabel_pb2_grpc.ProseAndBabelStub(channel)
while True:
iterator = iter(block_of_text)
response = stub.UserMarkov(ProseAndBabel_pb2.UserTweets(tweets=iterator))
Here's relevant server code:
def UserMarkov(self, request_iterator, context):
return ProseAndBabel_pb2.Babel(prose=markov.get_sentence(request_iterator.tweets))
Here's the proto where the rpc and messages are defined:
service ProseAndBabel {
rpc GetHaiku (BabelRequest) returns (Babel) {}
rpc GetBabel (BabelRequest) returns (Babel) {}
rpc UserMarkov (stream UserTweets) returns (UserBabel) {}
}
message BabelRequest{
string ask = 1;
}
message Babel{
string prose = 1;
}
message UserTweets{
string tweets = 1;
}
message UserBabel{
string prose = 1;
}
I've been successful getting the non-streaming rpc to work, but having trouble finding walkthroughs for request side streaming for python applications so I'm sure I'm missing something here. Any guidance/direction appreciated!
You need to pass the iterator of requests to the gRPC client stub, not to the protobuf constructor. The current code tries to instantiate a UserTweets protobuf with an iterator rather than an individual string, resulting in the type error.
response = stub.UserMarkov(ProseAndBabel_pb2.UserTweets(tweets=iterator))
You'll instead need to have your iterator to return instances of ProseAndBabel_pb2.UserTweets, each of which wraps one of the request strings you would like to send, and pass the iterator itself to the stub. Something like:
iterator = iter([ProseAndBabel_pb2.UserTweets(tweets=x) for x in block_of_text])
response = stub.UserMarkov(iterator)
I'm following this Route_Guide sample.
The sample in question fires off and reads messages without replying to a specific message. The latter is what i'm trying to achieve.
Here's what i have so far:
import grpc
...
channel = grpc.insecure_channel(conn_str)
try:
grpc.channel_ready_future(channel).result(timeout=5)
except grpc.FutureTimeoutError:
sys.exit('Error connecting to server')
else:
stub = MyService_pb2_grpc.MyServiceStub(channel)
print('Connected to gRPC server.')
this_is_just_read_maybe(stub)
def this_is_just_read_maybe(stub):
responses = stub.MyEventStream(stream())
for response in responses:
print(f'Received message: {response}')
if response.something:
# okay, now what? how do i send a message here?
def stream():
yield my_start_stream_msg
# this is fine, i receive this server-side
# but i can't check for incoming messages here
I don't seem to have a read() or write() on the stub, everything seems to be implemented with iterators.
How do i send a message from this_is_just_read_maybe(stub)?
Is that even the right approach?
My Proto is a bidirectional stream:
service MyService {
rpc MyEventStream (stream StreamingMessage) returns (stream StreamingMessage) {}
}
What you're trying to do is perfectly possible and will probably involve writing your own request iterator object that can be given responses as they arrive rather than using a simple generator as your request iterator. Perhaps something like
class MySmarterRequestIterator(object):
def __init__(self):
self._lock = threading.Lock()
self._responses_so_far = []
def __iter__(self):
return self
def _next(self):
# some logic that depends upon what responses have been seen
# before returning the next request message
return <your message value>
def __next__(self): # Python 3
return self._next()
def next(self): # Python 2
return self._next()
def add_response(self, response):
with self._lock:
self._responses.append(response)
that you then use like
my_smarter_request_iterator = MySmarterRequestIterator()
responses = stub.MyEventStream(my_smarter_request_iterator)
for response in responses:
my_smarter_request_iterator.add_response(response)
. There will probably be locking and blocking in your _next implementation to handle the situation of gRPC Python asking your object for the next request that it wants to send and your responding (in effect) "wait, hold on, I don't know what request I want to send until after I've seen how the next response turned out".
Instead of writing a custom iterator, you can also use a blocking queue to implement send and receive like behaviour for client stub:
import queue
...
send_queue = queue.SimpleQueue() # or Queue if using Python before 3.7
my_event_stream = stub.MyEventStream(iter(send_queue.get, None))
# send
send_queue.push(StreamingMessage())
# receive
response = next(my_event_stream) # type: StreamingMessage
This makes use of the sentinel form of iter, which converts a regular function into an iterator that stops when it reaches a sentinel value (in this case None).
I have a multi-room speaker system from Denon called Heos which I want to control by using python script. To communicate with the multi-room system I have to telnet to port 1255 on the device and send commands like this:
heos://player/set_play_state?pid=player_id&state=play_state
The response back is in json:
{
"heos": {
"command": " player/set_play_state ",
"result": "success",
"message": "pid='player_id'&state='play_state'"
}
}
I have successfully used python telnet lib to send simple commands like this:
command = "heos://player/set_play_state?pid=player_id&state=play_state"
telnet.write(command.encode('ASCII') + b'\r\n')
But what is the best way to get the response back in a usable format? Loop with telnet.read_until? I want to result and message lines back to a clean variable.
This method with using telnet to communicate with api feels a bit dirty. Is it possible to use something else, for example socket?
Thanks in advance
The API/CLI is documented here: http://rn.dmglobal.com/euheos/HEOS_CLI_ProtocolSpecification.pdf
While it may be possible to use loop_until() here, it would depend on exactly how the response JSON is formatted, and it would probably be unwise to rely on it.
If the remote device closes the connection after sending the response, the easy way would be a simple
response = json.loads(telnet.read_all().decode())
If it remains open for more commands, then you'll instead need to keep receiving until you have a complete JSON object. Here's a possibility that just keeps trying to parse the JSON until it succeeds:
response = ''
while True:
response += telnet.read_some().decode()
try:
response = json.loads(response)
break
except ValueError:
pass
Either way, your result and message are response['heos']['result'] and response['heos']['message'].
FWIW, here is my GitHub repo (inspired by this repo) for controlling a HEOS speaker with Python. It uses a similar approach as the accepted result, but additionally waits if the HEOS player is busy.
def telnet_request(self, command, wait = True):
"""Execute a `command` and return the response(s)."""
command = self.heosurl + command
logging.debug("telnet request {}".format(command))
self.telnet.write(command.encode('ASCII') + b'\r\n')
response = b''
while True:
response += self.telnet.read_some()
try:
response = json.loads(response)
if not wait:
logging.debug("I accept the first response: {}".format(response))
break
# sometimes, I get a response with the message "under
# process". I might want to wait here
message = response.get("heos", {}).get("message", "")
if "command under process" not in message:
logging.debug("I assume this is the final response: {}".format(response))
break
logging.debug("Wait for the final response")
response = b'' # forget this message
except ValueError:
# response is not a complete JSON object
pass
except TypeError:
# response is not a complete JSON object
pass
if response.get("result") == "fail":
logging.warn(response)
return None
return response
This shouldn't be that complicated, but it seems that both the Ruby and Python Telnet libs have awkward APIs. Can anyone show me how to write a command to a Telnet host and then read the response into a string for some processing?
In my case "SEND" with a newline retrieves some temperature data on a device.
With Python I tried:
tn.write(b"SEND" + b"\r")
str = tn.read_eager()
which returns nothing.
In Ruby I tried:
tn.puts("SEND")
which should return something as well, the only thing I've gotten to work is:
tn.cmd("SEND") { |c| print c }
which you can't do much with c.
Am I missing something here? I was expecting something like the Socket library in Ruby with some code like:
s = TCPSocket.new 'localhost', 2000
while line = s.gets # Read lines from socket
puts line # and print them
end
I found out that if you don't supply a block to the cmd method, it will give you back the response (assuming the telnet is not asking you for anything else). You can send the commands all at once (but get all of the responses bundled together) or do multiple calls, but you would have to do nested block callbacks (I was not able to do it otherwise).
require 'net/telnet'
class Client
# Fetch weather forecast for NYC.
#
# #return [String]
def response
fetch_all_in_one_response
# fetch_multiple_responses
ensure
disconnect
end
private
# Do all the commands at once and return everything on one go.
#
# #return [String]
def fetch_all_in_one_response
client.cmd("\nNYC\nX\n")
end
# Do multiple calls to retrieve the final forecast.
#
# #return [String]
def fetch_multiple_responses
client.cmd("\r") do
client.cmd("NYC\r") do
client.cmd("X\r") do |forecast|
return forecast
end
end
end
end
# Connect to remote server.
#
# #return [Net::Telnet]
def client
#client ||= Net::Telnet.new(
'Host' => 'rainmaker.wunderground.com',
'Timeout' => false,
'Output_log' => File.open('output.log', 'w')
)
end
# Close connection to the remote server.
def disconnect
client.close
end
end
forecast = Client.new.response
puts forecast
I am using python sockets to receive web style and soap requests. The code I have is
import socket
svrsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname();
svrsocket.bind((host,8091))
svrsocket.listen(1)
clientSocket, clientAddress = svrsocket.accept()
message = clientSocket.recv(4096)
Some of the soap requests I receive, however, are huge. 650k huge, and this could become several Mb. Instead of the single recv I tried
message = ''
while True:
data = clientSocket.recv(4096)
if len(data) == 0:
break;
message = message + data
but I never receive a 0 byte data chunk with firefox or safari, although the python socket how to says I should.
What can I do to get round this?
Unfortunately you can't solve this on the TCP level - HTTP defines its own connection management, see RFC 2616. This basically means you need to parse the stream (at least the headers) to figure out when a connection could be closed.
See related questions here - https://stackoverflow.com/search?q=http+connection
Hiya
Firstly I want to reinforce what the previous answer said
Unfortunately you can't solve this on the TCP level
Which is true, you can't. However you can implement an http parser on top of your tcp sockets. And that's what I want to explore here.
Let's get started
Problem and Desired Outcome
Right now we are struggling to find the end to a datastream. We expected our stream to end with a fixed ending but now we know that HTTP does not define any message suffix
And yet, we move forward.
There is one question we can now ask, "Can we ever know the length of the message in advance?" and the answer to that is YES! Sometimes...
You see HTTP/1.1 defines a header called Content-Length and as you'd expect it has exactly what we want, the content length; but there is something else in the shadows: Transfer-Encoding: chunked. unless you really want to learn about it, we'll stay away from it for now.
Solution
Here is a solution. You're not gonna know what some of these functions are at first, but if you stick with me, I'll explain. Alright... Take a deep breath.
Assuming conn is a socket connection to the desired HTTP server
...
rawheaders = recvheaders(conn,end=CRLF)
headers = dict_headers(io.StringIO(rawheaders))
l_content = headers['Content-Length']
#okay. we've got content length by magic
buffersize = 4096
while True:
if l_content <= 0: break
data = clientSocket.recv(buffersize)
message += data
l_content -= len(data)
...
As you can see, we enter the loop already knowing the Content-Length as l_content
While we iterate we keep track of the remaining content by subtracting the length of clientSocket.recv(buff) from l_content.
When we've read at least as much data as l_content, we are done
if l_content <= 0: break
Frustration
Note: For some these next bits I'm gonna give psuedo code because the code can be a bit dense
So now you're asking, what is rawheaders = recvheaders(conn), what is headers = dict_headers(io.StringIO(rawheaders)),
and HOW did we get headers['Content-Length']?!
For starters, recvheaders. The HTTP/1.1 spec doesn't define a message suffix, but it does define something useful: a suffix for the http headers! And that suffix is CRLF aka \r\n.That means we know when we've recieved the headers when we read CRLF. So we can write a function like
def recvheaders(sock):
rawheaders = ''
until we read crlf:
rawheaders = sock.recv()
return rawheaders
Next, parsing the headers.
def dict_header(ioheaders:io.StringIO):
"""
parses an http response into the status-line and headers
"""
#here I expect ioheaders to be io.StringIO
#the status line is always the first line
status = ioheaders.readline().strip()
headers = {}
for line in ioheaders:
item = line.strip()
if not item:
break
//headers look like this
//'Header-Name' : 'Value'
item = item.split(':', 1)
if len(item) == 2:
key, value = item
headers[key] = value
return status, headers
Here we read the status line then we continue to iterate over every remaining line
and build [key,value] pairs from Header: Value with
item = line.strip()
item = item.split(':', 1)
# We do split(':',1) to avoid cases like
# 'Header' : 'foo:bar' -> ['Header','foo','bar']
# when we want ---------> ['Header','foo:bar']
then we take that list and add it to the headers dict
#unpacking
#key = item[0], value = item[1]
key, value = item
header[key] = value
BAM, we've created a map of headers
From there headers['Content-Length'] falls right out.
So,
This structure will work as long as you can guarantee that you will always recieve Content-Length
If you've made it this far WOW, thanks for taking the time and I hope this helped you out!
TLDR; if you want to know the length of an http message with sockets, write an http parser