I have an app that consists of a local "server" and a GUI client. The server is written in Python, while the GUI is meant to be changable, and is written in Flex 4. The client queries the local server for information and displays it accordingly. Both applications are meant to be run on the same computer, and will only communicate locally. At the moment, the client and the Python server communicates via a basic socket, the client writes a request to the socket, and the socket returns some data.
However, since I'm writing a desktop app, I thought it might be easier to maintain and to polish a system that uses standard streams instead of a socket. The server would listen for raw_input() continuously, and output according to anything written to stdin, and the client, in this case, would use AIR's NativeProcess class to read and write to stdout and stdin, rather than using sockets.
The client and server are separate processes, but are meant to be started at more or less the same time. I don't really have need for complex networking, just local cross-language communication.
What are the pros and cons of each approach? What would I gain or lose from using sockets, vs what I would gain or lose from using standard streams to communicate? Which one is more efficient? Which one is easier to maintain?
On UNIX-like platforms, using stdin/stdout is a socket, so there's no difference. That is, if you launch a process with its stdout redirected, that'll typically be done with socketpair, so making your own UNIX-domain socket to communicate is unnecessary. The Python classes for handling stdin/stdout won't give you access to the full flexibility of the underlying socket though, so you'll have to set it up yourself I think if you want to do a half-close, for example (Python can't offer that cross-platform because the Windows sys::stdin can't be a socket, for example, nor is it always on UNIX.)
The nice thing about a local TCP connection is that it's cross-platform, and gives predictable semantics everywhere. If you want to be able to close your output and still read from the input, for example, it's much simpler to do this sort of thing with sockets which are the same everywhere. Even without that, for simplicity's sake, using a TCP socket is always a reasonable way to work around the wacky Windows mess of named pipes, although Python shields you from that reasonably well.
Related
I have a large Pandas DataFrame in Python that I would like to access in a Julia program (as a Julia DataFrames.DataFrame object). As I would like to avoid writing to disk for each file send from Python to Julia, it seems as though storing the DataFrame in an Apache Arrow/Feather file in a buffer and sending that via TCP from python to Julia is ideal.
I have tried extensively but cannot figure out how to
Write Apache Arrow/Feather files to memory (not storage)
Send them over TCP from python
Access them from the TCP port in Julia
Thanks for your help.
Hmmm, good question. I'm not sure using a TCP socket is necessarily the easiest, since you need one end to be the "server" socket and the other to be the client. So typically the TCP flow is: 1) server binds and listens to a port, 2) server calls to "accept" a new connection, 3) client calls "connect" on the port to initialize connection, 4) once server accepts, the connection is established, then server/client can write data to each other over connected socket.
I've had success doing something similar to what you've described by using mmapped files, though maybe you have a hard requirement to not touch disk at all. This works nicely though because both the python and Julia processes just "share" the mmapped file.
Another approach you could check out is what I setup to do "round trip" testing in the Arrow.jl Julia package: https://github.com/apache/arrow-julia/blob/main/test/pyarrow_roundtrip.jl. It's setup to use PyCall.jl from Julia to share the bytes between python and Julia.
Hope that helps!
Key points:
I need to send roughly ~100 float numbers every 1-30 seconds from one machine to another.
The first machine is catching those values through sensors connected to it.
The second machine is listening for them, passing them to an http server (nginx), a telegram bot and another program sending emails with alerts.
How would you do this and why?
Please be accurate. It's the first time I work with sockets and with python, but I'm confident I can do this. Just give me crucial details, lighten me up!
Some small portion (a few rows) of the core would be appreciated if you think it's a delicate part, but the main goal of my question is to see the big picture.
Main thing here is to decide on a connection design and to choose protocol. I.e. will you have a persistent connection to your server or connect each time when new data is ready to it.
Then will you use HTTP POST or Web Sockets or ordinary sockets. Will you rely exclusively on nginx or your data catcher will be another serving service.
This would be a most secure way, if other people will be connecting to nginx to view sites etc.
Write or use another server to run on another port. For example, another nginx process just for that. Then use SSL (i.e. HTTPS) with basic authentication to prevent anyone else from abusing the connection.
Then on client side, make a packet every x seconds of all data (pickle.dumps() or json or something), then connect to your port with your credentials and pass the packet.
Python script may wait for it there.
Or you write a socket server from scratch in Python (not extra hard) to wait for your packets.
The caveat here is that you have to implement your protocol and security. But you gain some other benefits. Much more easier to maintain persistent connection if you desire or need to. I don't think it is necessary though and it can become bulky to code break recovery.
No, just wait on some port for a connection. Client must clearly identify itself (else you instantly drop the connection), it must prove that it talks your protocol and then send the data.
Use SSL sockets to do it so that you don't have to implement encryption yourself to preserve authentication data. You may even rely only upon in advance built keys for security and then pass only data.
Do not worry about the speed. Sockets are handled by OS and if you are on Unix-like system you may connect as many times you want in as little time interval you need. Nothing short of DoS attack won't inpact it much.
If on Windows, better use some finished server because Windows sometimes do not release a socket on time so you will be forced to wait or do some hackery to avoid this unfortunate behaviour (non blocking sockets and reuse addr and then some flo control will be needed).
As far as your data is small you don't have to worry much about the server protocol. I would use HTTPS myself, but I would write myown light-weight server in Python or modify and run one of examples from internet. That's me though.
The simplest thing that could possibly work would be to take your N floats, convert them to a binary message using struct.pack(), and then send them via a UDP socket to the target machine (if it's on a single LAN you could even use UDP multicast, then multiple receivers could get the data if needed). You can safely send a maximum of 60 to 170 double-precision floats in a single UDP datagram (depending on your network).
This requires no application protocol, is easily debugged at the network level using Wireshark, is efficient, and makes it trivial to implement other publishers or subscribers in any language.
I'm designing a system like this: a Python process (let's call it "server") accepts inputs from another process ("client", written in Objective-C) on the same machine and returns outputs to the client.
What's a good architecture for this system? I mean, what's a good protocol for server/client communication? I think making the server an HTTP service is overkill because the client always lives on the same machine.
I would argue it's not the HTTP that adds a lot of overhead, but the TCP 3-way connection handshake.
Having said that, a lot of systems use TCP for inter-process communication, so if you want to use HTTP, it's only a very small extra load on top.
Of course with HTTP, you are creating a new connection with each request, however this is not so bad - you should be able to make each HTTP call within 1 or 2ms.
With HTTP comes a lot of nice properties like not having to maintain a persistent TCP connection, a ton of great libraries to easily make/receive your requests, and the request/response model seems to suit your system needs.
I need to have a tcp socket client connected to a server to send data and receive.
But this socket must be always on and i cannot open another socket.
I have always some data to send over the time and then later process the answer to the data sent previously.
If i could open many sockets, i think it was more easy. But in my case i have to send everything on the same socket asynchronously.
So the question is, what do you recommend to use within the Python ecosystem? (twisted, tornado, etc)
Should i consider node.js or another option?
I highly recommend Twisted for this:
It comes with out-of-the-box support for many TCP protocols.
It is easy to maintain a single connection, there is a ReconnectingClientFactory that will deal with disconnections and use exponential backoff, and LoopingCall makes it easy to implement a heartbeat.
Stateful protocols are also easy to implement and intermingle with complex business logic.
It's fun.
I have a service that is exactly like the one you mention (single login, stays on all the time, processes data). It's been on for months working like a champ.
Twisted is possibly hard to get your head around, but the tutorials here are a great start. Knowing Twisted will get you far in the long run!
"i have to send everything on the same socket asynchronously"
Add your data to a queue, have a separate thread taking items out of the queue and sending via socket.send()
As the title says, I would like to send data using an existing tcp connection. Said connection has already been established by a 3rd party program. I haven't been able to find much information about this, and it's safe to say I don't know how this will work at all.
The operating system is Windows. My preferred programming language is python - I'd prefer not to use 3rd party python modules, but I will if they make my life easier.
Just to clarify, in case you aren't sure what I want to do: I want to send data as if it were sent by a different program; pretty much like WPE pro's send function does.
Update:
Technically, couldn't I manually design the TCP packet and then tell the network device (or operating system) to send that packet? Wouldn't that be exactly the same thing an injected socket would do?
Edit: Wikipedia says the receiving host acknowledges packets it receives, which makes this a bit more difficult. But if can drop that acknowledge-packet before the 3rd party program receives it, then this should work. Right?
Scapy/Pcapy are pretty powerful tools for monitoring and injecting packets into a live network interface. I've used them for several projects. These tools are ideal for stimulus/response low-level network protocols (ie DHCP, DNS, etc) and anything non-stateful sent over simple UDP.
Unfortunately, the TCP layer is very complicated and stateful. So injecting something that makes sense into the stream will be more difficult. Moreover, Scapy/Pcapy do not currently have support for tcp.
A TCP session is not intended to be a many-to-one connection. Its a point-to-point stateful protocol which keeps track of packets that have been sent versus those that have been received by the other end. I don't believe you can inject yourself into an already-established session. Your best bet, as was pointed out previously, is to create a proxy and act as a man-in-the-middle interloper. Still not a trivial thing but doable.