I'm not sure if I'm doing this right, but I would like to be able to send messages to my server running ZMQ from normal TCP connections. The server is running Python ZMQ on port 5555 using a TCP transport. I would like to be able to send messages to it using different clients (Python, Java, PHP) that use conventional TCP. This is what I have so far:
SERVER
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
while True:
message = socket.recv()
print message
socket.send('{"name":"someone"}')
CLIENT
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 5555))
s.send('Hello, World!')
data = s.recv(1024)
print data
Printing data on the client does not give me the message I am expecting. I get this: �. I tried doing bytes(data).decode('utf8') thinking what I'm getting was an array of bytes, but I get the following error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
I am just wondering: Is this possible at all? Or am I doing something wrong? Also, is it recommended?
My reason for not using ZMQ on the clients is that I want to reduce the number of dependencies (ZeroMQ being one).
Thank you for your help.
ZeroMQ is a protocol that sits on top of TCP (well, technically, ZMTP is a protocol; 0MQ is the library that implements the ZMTP protocol, and ZeroMQ is the Python implementation of 0MQ…). All the signaling it does, and all the framing (dividing the stream of bytes into separate messages), is done by sending bytes over the socket. A client that doesn't know how to do ZeroMQ signaling and framing is just going to see a bunch of garbage.
What you're trying to do is exactly like trying to write a web client that just reads off a socket without knowing anything about HTTP. You're going to get a bunch of framing stuff that you don't know what to do with. The only difference is that in the case of HTTP, the framing is sometimes (but not always—you can have MIME envelopes, gzip transport-encoding, chunked transport, …) just a bunch of human-readable ASCII that comes before the data, while with ZMTP it's never human-readable…
If you want to send data over a plain TCP socket, you have to do that by creating a plain TCP socket on the server side and calling its send (or sendall, etc.) method. If you want to send data over a ZMTP channel, you have to do that by parsing ZMTP, or using a library that does it for you (like ZeroMQ), on the client side.
One more thing to keep in mind is that, unlike ZMTP, TCP is not a message-oriented protocol; all TCP sends is a stream of bytes. From the receiving side, there's no way to know when one send ends and the next one begins. So, for almost anything but a "send a request, get a response, hang up" protocol, you need to write some kind of framing of your own. This can be as simple as "messages are strings that have no newlines in them, and each message is separated by a newline" (in which case you can just use socket.makefile), but often the message format has to be more complicated, or you have to send "commands" rather than just data, etc.
Since tdelaney's answer has been deleted (which I think means it's invisible to anyone under 10K rep), and had a useful suggestion, I'll repeat it here, with credit as due: You can (using the ZeroMQ library) write a piece of middleware that talks ZMTP to the server but your own simple TCP-based protocol to the clients. ZeroMQ has been specifically designed to make this reasonably easy; as tdelaney put it, it's "kinda like Lego, you build a robust communication infrastructure by building different communicating parts. Not all of the parts need to be zeromq."
Related
I send mouse coordinates from python server to python client via socket. Mouse coordinates are send every time when mouse movement event is catch on the server which means quite often (dozen or so per second).
Problem is when I use python server and python client on different hosts. Then only part of messages are delivered to the client.
e.g. 3 first messages are delivered, 4 messages aren't delivered, 4 messages are delivered etc...
Everything is fine when server and client are on the same host (localhost).
Everything is fine when server and client are on different hosts but instead of python client I use standard windows Telnet client to read messages from the server.
I noticed that when I use time.sleep(0.4) break between each message that is send then all messages are delivered. Problem is I need that information in real time not with such delay. Is it possible to achieve that in Python using sockets?
Below python client code that I use:
import pickle
import socket
import sys
host = '192.168.1.222'
port = 8888
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
print "Faile. Error:" + str(msg[0]), "Error message : " + msg[1]
sys.exit()
mySocket = socket.socket()
mySocket.connect((host,port))
while 1:
data = mySocket.recv(1024)
if not data: break
load_data = pickle.loads(data)
print 'parametr x: ' + str(load_data[0])
print 'parametr y : ' + str(load_data[1])
mySocket.close()
You are using TCP (SOCK_STREAM) which is a reliable protocol which (contrary to UDP) does not loose any messages, even if the recipient is not reading the data fast enough. Instead TCP will reduce the sending speed.
This means that the problem must be somewhere in your application code.
One possibility is that the problem is in your sender, i.e. that you use socket.send and do not check if all the bytes you've intended to send are really got send. But this check needs to be done since socket.send might only send part of the data if the socket buffer of the OS is full which can happen if the client does not read the data fast enough.
Another possibility is that your socket.recv call receives more data than your pickle.loads needs and that the rest of the data gets discarded (not sure if pickle.loads will throw an exception if too much data are provided). Note that TCP is not a message but a stream protocol so it might be that you have more that socket.recv will return a buffer which contains more than one pickled object but you only read the first. The chance that this will happen on a network is higher than on localhost because by default the TCP layer will try to concatenate multiple send buffers into a single TCP packet for better use of the connection (i.e. less overhead). And the chance is high that these will then be received within the same recv call. By putting a sleep(0.4) on the sender side you've effectively switch off this optimization of TCP, see NAGLE algorithm for details.
Thus the correct way to implement what you want would be:
Make sure that all data are delivered at the server, i.e. check the return of socket.send.
Make sure that you unpack all messages you receive. To do this you probable need to add some message layer on top of the TCP stream to find out where the message boundary is.
I have JSON messages (JSON hash table) represented as strings coming in through a websocket. Each read from the socket may return a string that does not end on a message boundary. What's the easiest way to parse the JSON messages in Python? How do I find where in the string a message terminates without writing a parser (or brace/paren matcher) myself?
Do other languages provide tools to make this easier?
Each read from the socket may return a string that does not end on a message boundary.
This is incorrect - Websockets is a message oriented protocol as opposed to the traditional stream oriented TCP socket protocol where you needed to worry about and handle message chunking.
It is built on top of TCP, however it automatically handles piecing the individual fragments into one complete message before delivering it at the application layer (your code).
So websockets are message-oriented like UDP without the maximum length constraints but with TCP’s delivery guarantees and congestion control. It turns out that TCP’s stream orientation isn’t all that useful (think about how many protocols build some sort of “message” concept on top of TCP). In fact SCTP (RFC 4960) provides many of the same benefits of messages-on-top-of-TCP but removes the TCP part to reduce the overhead. Unfortunately, SCTP is yet to gain widespread adoption.
Also from the official RFC:
layers a framing mechanism on top of TCP to get back to the IP packet mechanism that TCP is built on, but without length limits
I am sending some request from the server to my client but I have some problem.
When I'm sending messages to the client, if I send many messages, I'll receive all with socket.recv()
Is there a way to get the messages one by one ?
Thanks
You need to use some kind of protocol over otherwise bare sockets.
See python twisted or use something like nanomsg or ZeroMQ if you want a simple drop-in replacement which is message-oriented.
It is not transport-agnostic though, meaning they will only work if they are used on both ends.
No. TCP is a byte stream. There are no messages larger than one byte.
I assume that you are using TCP. TCP is a streaming protocol, not a datagram protocol. This means that the data are not a series of messages, but instead a single data stream without any message boundaries. If you need something like this either switch the protocol (UDP is datagram, but has other problems) or make your own protocol on top of TCP which knows about messages.
Typical message based protocols on top of TCP either use some message delimiter (often newline) or prefix each message with its size.
I may be going about this the wrong way but that's why I'm asking the question.
I have a source of serial data that is connected to a SOC then streams the serial data up to a socket on my server over UDP. The baud rate of the raw data is 57600, I'm trying to use Python to receive and parse the data. I tested that I'm receiving the data successfully on the port via the script below (found here: https://wiki.python.org/moin/UdpCommunication)
import socket
UDP_IP = "MY IP"
UDP_PORT = My PORT
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print "received message:", data
Since I'm not reading the data with the .serial lib in Python or setting the baud rate to read at it comes all garbled, as would be expected. My end goal is to be able to receive and parse the data for server side processing and also have another client connect to the raw data stream piped back out from the server (proxy) which is why I'm not processing the data directly from the serial port on the device.
So my question is, how can I have Python treat the socket as a serial port that I can set a baud rate on and #import serial and .read from? I can't seem to find any examples online which makes me think I'm overlooking something simple or am trying to do something stupid.
sadface
You can't treat a socket as a serial line. A socket can only send and receive data (data stream for TCP, packets for UDP). If you would need a facility to control the serial line on the SOC you would need to build an appropriate control protocol over the socket, i.e. either use another socket for control like FTP does or use in-band control and distinguish between controlling and data like HTTP does. And of course both sides of the connection have to understand this protocol.
Build on facts
A first thing to start with is to summarise facts -- begining from the very SystemOnChip (SOC) all the way up ...:
an originator serial-bitstream parameters ::= 57600 Bd, X-<dataBIT>-s, Y-<stopBIT>, Z-<parityBIT>,
a mediator receiving process de-framing <data> from a raw-bitstream
a mediator upStream sending to server-process integration needs specification ( FSA-model of a multi-party hand-shaking, fault-detection, failure-state-resilience, logging, ... )
T.B.C. ...
Design as per a valid requirement list
A few things work as a just bunch of SLOC one-liners. Design carefully against the validated Requirement List as a principle. It saves both your side and the cooperating Team(s).
Test on models
Worth a time to test on-the-fly, during the efforts to progress from simple parts to more complex, multiparty scenarios.
Integrate on smarter frameworks
Definitely a waste of time to reinvent wheel. Using a smart framework for server-side integration will unlock a lot of your time and energy on your ProblemDOMAIN-specific tasks, rather than to waste both the former and the latter for writing your just-another-socket-framework ( destined in majority of cases to failure ... )
Try ZeroMQ or a nanomsg Scale-able Formal Communication Patterns Framework for smart-services to send de-framed data from your serial-bitStream source and you are almost there.
I'm implementing client-server communication using UDP that's used for FTP. First off, you don't need to tell me that UDP is unreliable, I know. My approach is: client asks for a file, server blasts the client with udp packets with sequence numbers, then says "what'd you miss?", resending those. On a local network, packet loss is < 1%. I'm pretty new to socket programming, so I'm not familiar with all the socket options (of which most examples found on google are for tcp).
My problem is why my client's receiving of this data.
PACKET_SIZE = 9216
mysocket.sendto('GO!', server_addr)
while True:
resp = mysocket.recv(PACKET_SIZE)
worker_thread.enqeue_packet(resp)
But by the time it gets back up to .recv(), it's missed a few udp packets (that I've confirmed are being sent using wireshark). I can fix this by making the server send slightly slower (actually, including logging statements is enough of a delay to make everything function).
How can i make sure that socket.recv doesn't miss anything in the time it takes to process a packet? I've tried pushing the data out to a separate thread that pushes it into a queue, but it's still not enough.
Any ideas? select, recv_into, setblocking?
While you already know, that UDP is not reliable, you maybe missed the other advantages of TCP. Relevant for you is that TCP has flow control and automatically scales down if the receiver is unable to cope with the senders speed (e.g. packet loss). So for normal connections TCP should be preferred for data transfer. For high latency connections (satellite link) it behaves too bad in the default configuration, so that some people design there custom transfer protocols (mostly with UDP), while others just tune the existing TCP stack.
I don't know why you use UDP, but if you want to continue to use it you should add some kind of back channel to the sender to inform it from current packet loss, so that it can scale down. Maybe you should have a look at RTCP, which accompanies RTP (used for VoIP etc).