I have a large Pandas DataFrame in Python that I would like to access in a Julia program (as a Julia DataFrames.DataFrame object). As I would like to avoid writing to disk for each file send from Python to Julia, it seems as though storing the DataFrame in an Apache Arrow/Feather file in a buffer and sending that via TCP from python to Julia is ideal.
I have tried extensively but cannot figure out how to
Write Apache Arrow/Feather files to memory (not storage)
Send them over TCP from python
Access them from the TCP port in Julia
Thanks for your help.
Hmmm, good question. I'm not sure using a TCP socket is necessarily the easiest, since you need one end to be the "server" socket and the other to be the client. So typically the TCP flow is: 1) server binds and listens to a port, 2) server calls to "accept" a new connection, 3) client calls "connect" on the port to initialize connection, 4) once server accepts, the connection is established, then server/client can write data to each other over connected socket.
I've had success doing something similar to what you've described by using mmapped files, though maybe you have a hard requirement to not touch disk at all. This works nicely though because both the python and Julia processes just "share" the mmapped file.
Another approach you could check out is what I setup to do "round trip" testing in the Arrow.jl Julia package: https://github.com/apache/arrow-julia/blob/main/test/pyarrow_roundtrip.jl. It's setup to use PyCall.jl from Julia to share the bytes between python and Julia.
Hope that helps!
Related
Need sugestions on how to implement/what to use/what or about what to read in order to know how, a mail server in Python, which would let me to generate temporary mailboxes.
Let’s say I have a domain test.com, I would like to be able to setup for example a mailbox temporary1#test.com, which would receive and store mails (even in txt files).
What do I need to be able to do that?
(I’m interested in links or explanations rather than implementations)
Python 3 ships with the smtpd module that can be used to implement this.
You'll probably want to trivially subclass smtpd.SMTPServer (maybe even derive from or at least be inspired by smtpd.DebuggingServer, but have it print out to files instead of standard output).
Once that's up and running (you can use the smtplib clients to test things out), you'll need to have that server process running on a machine on port 25 (the standard SMTP port), and you'll need to point the MX record of test.com to that machine's address.
Key points:
I need to send roughly ~100 float numbers every 1-30 seconds from one machine to another.
The first machine is catching those values through sensors connected to it.
The second machine is listening for them, passing them to an http server (nginx), a telegram bot and another program sending emails with alerts.
How would you do this and why?
Please be accurate. It's the first time I work with sockets and with python, but I'm confident I can do this. Just give me crucial details, lighten me up!
Some small portion (a few rows) of the core would be appreciated if you think it's a delicate part, but the main goal of my question is to see the big picture.
Main thing here is to decide on a connection design and to choose protocol. I.e. will you have a persistent connection to your server or connect each time when new data is ready to it.
Then will you use HTTP POST or Web Sockets or ordinary sockets. Will you rely exclusively on nginx or your data catcher will be another serving service.
This would be a most secure way, if other people will be connecting to nginx to view sites etc.
Write or use another server to run on another port. For example, another nginx process just for that. Then use SSL (i.e. HTTPS) with basic authentication to prevent anyone else from abusing the connection.
Then on client side, make a packet every x seconds of all data (pickle.dumps() or json or something), then connect to your port with your credentials and pass the packet.
Python script may wait for it there.
Or you write a socket server from scratch in Python (not extra hard) to wait for your packets.
The caveat here is that you have to implement your protocol and security. But you gain some other benefits. Much more easier to maintain persistent connection if you desire or need to. I don't think it is necessary though and it can become bulky to code break recovery.
No, just wait on some port for a connection. Client must clearly identify itself (else you instantly drop the connection), it must prove that it talks your protocol and then send the data.
Use SSL sockets to do it so that you don't have to implement encryption yourself to preserve authentication data. You may even rely only upon in advance built keys for security and then pass only data.
Do not worry about the speed. Sockets are handled by OS and if you are on Unix-like system you may connect as many times you want in as little time interval you need. Nothing short of DoS attack won't inpact it much.
If on Windows, better use some finished server because Windows sometimes do not release a socket on time so you will be forced to wait or do some hackery to avoid this unfortunate behaviour (non blocking sockets and reuse addr and then some flo control will be needed).
As far as your data is small you don't have to worry much about the server protocol. I would use HTTPS myself, but I would write myown light-weight server in Python or modify and run one of examples from internet. That's me though.
The simplest thing that could possibly work would be to take your N floats, convert them to a binary message using struct.pack(), and then send them via a UDP socket to the target machine (if it's on a single LAN you could even use UDP multicast, then multiple receivers could get the data if needed). You can safely send a maximum of 60 to 170 double-precision floats in a single UDP datagram (depending on your network).
This requires no application protocol, is easily debugged at the network level using Wireshark, is efficient, and makes it trivial to implement other publishers or subscribers in any language.
Using a Lantronix UDS-1100 serial to IP converter. The goal is to write a small proof of concept piece in Python to capture serial data output by this device over IP.
I've done a couple test projects using sockets in python, but they were all done between python processes (python > python): listen() on one end, and connect(), sendall() etc on the other.
I think I can use sockets for this project, but before I invest a bunch of time into it, wanted to make sure it is a viable solution.
Can python sockets be used to capture IP traffic when the traffic is originating from a non-python source? I have full control over the IP and port that the device sends the serial data to, but there will be no python connect() initiated by the client. I can pre-pend then serial data with some connect() string if needed.
If sockets won't work, please recommend another solution...guessing it will be REST or similar.
Of course. TCP/IP is supposed to be cross-platform and cross-language, so in theory you should be able to communicate with every kind of device as long as you manage to process and send the expected protocol.
I have an app that consists of a local "server" and a GUI client. The server is written in Python, while the GUI is meant to be changable, and is written in Flex 4. The client queries the local server for information and displays it accordingly. Both applications are meant to be run on the same computer, and will only communicate locally. At the moment, the client and the Python server communicates via a basic socket, the client writes a request to the socket, and the socket returns some data.
However, since I'm writing a desktop app, I thought it might be easier to maintain and to polish a system that uses standard streams instead of a socket. The server would listen for raw_input() continuously, and output according to anything written to stdin, and the client, in this case, would use AIR's NativeProcess class to read and write to stdout and stdin, rather than using sockets.
The client and server are separate processes, but are meant to be started at more or less the same time. I don't really have need for complex networking, just local cross-language communication.
What are the pros and cons of each approach? What would I gain or lose from using sockets, vs what I would gain or lose from using standard streams to communicate? Which one is more efficient? Which one is easier to maintain?
On UNIX-like platforms, using stdin/stdout is a socket, so there's no difference. That is, if you launch a process with its stdout redirected, that'll typically be done with socketpair, so making your own UNIX-domain socket to communicate is unnecessary. The Python classes for handling stdin/stdout won't give you access to the full flexibility of the underlying socket though, so you'll have to set it up yourself I think if you want to do a half-close, for example (Python can't offer that cross-platform because the Windows sys::stdin can't be a socket, for example, nor is it always on UNIX.)
The nice thing about a local TCP connection is that it's cross-platform, and gives predictable semantics everywhere. If you want to be able to close your output and still read from the input, for example, it's much simpler to do this sort of thing with sockets which are the same everywhere. Even without that, for simplicity's sake, using a TCP socket is always a reasonable way to work around the wacky Windows mess of named pipes, although Python shields you from that reasonably well.
I'm having a difficult time fully understanding the nature of a TCP server/client relationship when a JSON string is sent to the server. The information I need may be out there, but I'm perhpas not using the correct search paramaters as I'm looking.
I've built a Python TCP, JSON-RPC Server from the following examples:
https://github.com/joshmarshall/jsonrpclib
http://code.activestate.com/recipes/552751-json-rpc-server-and-client/
In both cases, I can communicate with the Python server from a Python console on a different computer, sending commands from one (the client) to the other (server). In all of the examples, I've had to install the libraries mentioned above on both the client and the server machines in order to facilitate the TCP communication.
So the background to my situation and question is, when does JSON enter the mix? This is what I want to do:
Setup a Python TCP server that accepts a JSON string from a remote client inside (or outside) the network. The server parses the JSON string, fetches the method and parameters from the objectified string, and executes the method. The server then sends a JSON string result to the calling client. In this case, the client is a mobile application (iPad, Android, etc) with a JavaScript library that I'll use to send the requests to the server.
Why would I need a Python client? From what I can gather, the client just needs to open a connection to the server and then send the JSON string, right? Why do all the code samples include Python client examples? Are they assuming a server machine is going to talk to a server machine, so they have included client code to help generate the JSON string that will be sent to the server?
If I assume that a Python client isn't really needed for anything, I've been sending JSON strings to the Python server from the iPad, but in each case the server is reporting a "Bad request syntax" error. I'll pen a new question on that issue if I'm understanding the current question correctly.
Insight is appreciated.
The JSON encoding is the lingua franca of your RPC protocol, so you can indeed use any client you like. The implementations you found for JSON-RPC use the HTTP protocol, a very specific communication protocol built on top of TCP/IP, but you can implement the same protocol over raw TCP-IP sockets if so required.
The examples include both the Python client and the server because they illustrate how to implement the JSON-RPC standard in Python, not in JavaScript or C or Lisp. They focus on the implementation in one language. The JSON-RPC standard however, is language agnostic. It doesn't matter what language you write either the server or the client in, as long as they use the same standard.