I'm designing a system like this: a Python process (let's call it "server") accepts inputs from another process ("client", written in Objective-C) on the same machine and returns outputs to the client.
What's a good architecture for this system? I mean, what's a good protocol for server/client communication? I think making the server an HTTP service is overkill because the client always lives on the same machine.
I would argue it's not the HTTP that adds a lot of overhead, but the TCP 3-way connection handshake.
Having said that, a lot of systems use TCP for inter-process communication, so if you want to use HTTP, it's only a very small extra load on top.
Of course with HTTP, you are creating a new connection with each request, however this is not so bad - you should be able to make each HTTP call within 1 or 2ms.
With HTTP comes a lot of nice properties like not having to maintain a persistent TCP connection, a ton of great libraries to easily make/receive your requests, and the request/response model seems to suit your system needs.
Related
Key points:
I need to send roughly ~100 float numbers every 1-30 seconds from one machine to another.
The first machine is catching those values through sensors connected to it.
The second machine is listening for them, passing them to an http server (nginx), a telegram bot and another program sending emails with alerts.
How would you do this and why?
Please be accurate. It's the first time I work with sockets and with python, but I'm confident I can do this. Just give me crucial details, lighten me up!
Some small portion (a few rows) of the core would be appreciated if you think it's a delicate part, but the main goal of my question is to see the big picture.
Main thing here is to decide on a connection design and to choose protocol. I.e. will you have a persistent connection to your server or connect each time when new data is ready to it.
Then will you use HTTP POST or Web Sockets or ordinary sockets. Will you rely exclusively on nginx or your data catcher will be another serving service.
This would be a most secure way, if other people will be connecting to nginx to view sites etc.
Write or use another server to run on another port. For example, another nginx process just for that. Then use SSL (i.e. HTTPS) with basic authentication to prevent anyone else from abusing the connection.
Then on client side, make a packet every x seconds of all data (pickle.dumps() or json or something), then connect to your port with your credentials and pass the packet.
Python script may wait for it there.
Or you write a socket server from scratch in Python (not extra hard) to wait for your packets.
The caveat here is that you have to implement your protocol and security. But you gain some other benefits. Much more easier to maintain persistent connection if you desire or need to. I don't think it is necessary though and it can become bulky to code break recovery.
No, just wait on some port for a connection. Client must clearly identify itself (else you instantly drop the connection), it must prove that it talks your protocol and then send the data.
Use SSL sockets to do it so that you don't have to implement encryption yourself to preserve authentication data. You may even rely only upon in advance built keys for security and then pass only data.
Do not worry about the speed. Sockets are handled by OS and if you are on Unix-like system you may connect as many times you want in as little time interval you need. Nothing short of DoS attack won't inpact it much.
If on Windows, better use some finished server because Windows sometimes do not release a socket on time so you will be forced to wait or do some hackery to avoid this unfortunate behaviour (non blocking sockets and reuse addr and then some flo control will be needed).
As far as your data is small you don't have to worry much about the server protocol. I would use HTTPS myself, but I would write myown light-weight server in Python or modify and run one of examples from internet. That's me though.
The simplest thing that could possibly work would be to take your N floats, convert them to a binary message using struct.pack(), and then send them via a UDP socket to the target machine (if it's on a single LAN you could even use UDP multicast, then multiple receivers could get the data if needed). You can safely send a maximum of 60 to 170 double-precision floats in a single UDP datagram (depending on your network).
This requires no application protocol, is easily debugged at the network level using Wireshark, is efficient, and makes it trivial to implement other publishers or subscribers in any language.
I have an app that consists of a local "server" and a GUI client. The server is written in Python, while the GUI is meant to be changable, and is written in Flex 4. The client queries the local server for information and displays it accordingly. Both applications are meant to be run on the same computer, and will only communicate locally. At the moment, the client and the Python server communicates via a basic socket, the client writes a request to the socket, and the socket returns some data.
However, since I'm writing a desktop app, I thought it might be easier to maintain and to polish a system that uses standard streams instead of a socket. The server would listen for raw_input() continuously, and output according to anything written to stdin, and the client, in this case, would use AIR's NativeProcess class to read and write to stdout and stdin, rather than using sockets.
The client and server are separate processes, but are meant to be started at more or less the same time. I don't really have need for complex networking, just local cross-language communication.
What are the pros and cons of each approach? What would I gain or lose from using sockets, vs what I would gain or lose from using standard streams to communicate? Which one is more efficient? Which one is easier to maintain?
On UNIX-like platforms, using stdin/stdout is a socket, so there's no difference. That is, if you launch a process with its stdout redirected, that'll typically be done with socketpair, so making your own UNIX-domain socket to communicate is unnecessary. The Python classes for handling stdin/stdout won't give you access to the full flexibility of the underlying socket though, so you'll have to set it up yourself I think if you want to do a half-close, for example (Python can't offer that cross-platform because the Windows sys::stdin can't be a socket, for example, nor is it always on UNIX.)
The nice thing about a local TCP connection is that it's cross-platform, and gives predictable semantics everywhere. If you want to be able to close your output and still read from the input, for example, it's much simpler to do this sort of thing with sockets which are the same everywhere. Even without that, for simplicity's sake, using a TCP socket is always a reasonable way to work around the wacky Windows mess of named pipes, although Python shields you from that reasonably well.
As the title says, I would like to send data using an existing tcp connection. Said connection has already been established by a 3rd party program. I haven't been able to find much information about this, and it's safe to say I don't know how this will work at all.
The operating system is Windows. My preferred programming language is python - I'd prefer not to use 3rd party python modules, but I will if they make my life easier.
Just to clarify, in case you aren't sure what I want to do: I want to send data as if it were sent by a different program; pretty much like WPE pro's send function does.
Update:
Technically, couldn't I manually design the TCP packet and then tell the network device (or operating system) to send that packet? Wouldn't that be exactly the same thing an injected socket would do?
Edit: Wikipedia says the receiving host acknowledges packets it receives, which makes this a bit more difficult. But if can drop that acknowledge-packet before the 3rd party program receives it, then this should work. Right?
Scapy/Pcapy are pretty powerful tools for monitoring and injecting packets into a live network interface. I've used them for several projects. These tools are ideal for stimulus/response low-level network protocols (ie DHCP, DNS, etc) and anything non-stateful sent over simple UDP.
Unfortunately, the TCP layer is very complicated and stateful. So injecting something that makes sense into the stream will be more difficult. Moreover, Scapy/Pcapy do not currently have support for tcp.
A TCP session is not intended to be a many-to-one connection. Its a point-to-point stateful protocol which keeps track of packets that have been sent versus those that have been received by the other end. I don't believe you can inject yourself into an already-established session. Your best bet, as was pointed out previously, is to create a proxy and act as a man-in-the-middle interloper. Still not a trivial thing but doable.
I am working out how to build a python app to do image processing. A client (not a web browser) sends an image and some text data to the server and the server's response is based on the received image.
One method is to use a web server + WSGI module and have clients make a HTTP POST request (using multipart/form-data). The http server then 'works out' the uploaded image and other data that the program can use.
Another method is to create a protocol that only sends the needed data and is handled within the application. The application would be doing everything (listening on the port, etc).
Is one of these a stand-out 'best' way (if yes, which one?), or is it more up to preference (or is there another way which is better)?
I believe it's more up to your needs, the size of the images, and your general knowledge of network programming.
In terms of simplicity, posting an image to the webserver using WSGI would be fairly simple, and you wouldn't have to worry about handling connections, sockets, error handling due to busy network ports, etc.
Another argument in favor of this approach is that you can easily reuse this "feature" if you already have it working on a webserver, say, by including a browser client. It might not be one of your needs now, but the door is left open.
This would be my choice.
Also, in Python you have a huge plethora of web frameworks to choose from, from the likes of Django, which is probably a huge overkill for your needs, to something alot simpler, like http://flask.pocoo.org/ which might just suit your needs and is really simple to set up.
In my opinion HTTP is an ideal protocol for sending files or large data, and its very common use, easy to suit any situation. If you use a self-created protocol, you may find it hard to transform when you get other client needs, like a web API.
Maybe the discussions about HTTP's lack of instantaneity and agility make you hesitate about choosing HTTP, but that mostly something about instant messaging and server push, there are better protocols. But when it comes to stability and flexiblity, HTTP is always a good choice.
I have written a little streaming mp3 server in python. So far all it does is accept a ServerSocket connection, and begin streaming all mp3 data in its queue to the request using socket.send(). I have implemented this to chunk in stream icy metadata, so the name of the playing song shows up in the client.
I would like to add playlist management to the server, so that I can manipulate the playlist of the running server. I have a vague idea that xmlrpclib would be suited to doing this, but I'm confused about two things:
Whether it's possible/advisable to integrate ICY and XMLRPC on a single server and a single port.
How to share state between the stream thread and the playlist, and manipulation thereof via xmlrpc.
Your initial attempt might be easier if you use two separate ports, each with its own server running in a separate thread. However, managing synchronization between the threads might be an annoying task in the long run.
ICY and HTTP are very similar, and if you've already implemented ICY on top of SocketServer, you could probably extend BaseHTTPServer.BaseHTTPRequestHandler to respond to both ICY and HTTP requests on the same port. Take a look at the standard library code for the BaseHTTPRequestHandler.parse_request() method, and think about how to override it in a subclass for a split personality.
Also, when you want to handle multiple concurrent requests using these classes, take a look at the SocketServer mixin classes.