I need a way to simulate connectivity problems in an automated test suite, on Linux, and preferably from Python. Some sort of proxy that I can put in front of the web server that can hang or drop connections after one trigger or another (after X bytes transferred, etc) would be perfect.
It doesn't seem too hard to build, but I'd rather grab something pre-existing, if anyone has any good recommendations.
when i needed one, i found that building it yourself is the best thing..
start by raising a threaded server in python http://docs.python.org/dev/library/socketserver.html (you don't have to use the class itself).
and it's very simple:
in the new connection thread, you create a new socket and connects it to the real server.
then, you put both of them in a list and sends it to select.select (import select).
then, when socket x receive data - sends it to y. when socket y receives data sends it to x. (don't forget to close the socket when you receive empty string).
now you can do whatever you want..
if you need anything, i'm here..
Related
I am writing a React app with a Flask backend that I want to be able to receive data through a serial port, process this data, and graph it. Currently I am using the backend to send certain pieces of data (current time and available ports) to React. I want to set up a background Python thread that will run continuously to read data from the serial port using pyserial, process it, and send it to React, but I'm not sure what the best way to accomplish this is. My initial search brought me to Celery; however, I'm not sure if it's a good option for a continuous task. Any help is much appreciated!
The problem is that reading from a serial port is normally done in a blocking way. That means you do not poll periodically, but instead you open the port once and then read all the time, waiting for new data to come.
What you need is a separate thread. This is a part of the program that runs in parallel to your normal web server. Then you need some sort of data base to communicate between that thread and the web server. If you want your data to be persistent between device and server restarts, you should install a real database like Postgres. If not, you can simply use an array in your application memory.
In the thread, read from the serial port and write the values to the database/array.
In your REST endpoint, you output the last X values.
Then your client can poll against this endpoint.
(If you want to do it really fancy, you can use a more event-driven approach, but this would be more complicated to implement)
Key points:
I need to send roughly ~100 float numbers every 1-30 seconds from one machine to another.
The first machine is catching those values through sensors connected to it.
The second machine is listening for them, passing them to an http server (nginx), a telegram bot and another program sending emails with alerts.
How would you do this and why?
Please be accurate. It's the first time I work with sockets and with python, but I'm confident I can do this. Just give me crucial details, lighten me up!
Some small portion (a few rows) of the core would be appreciated if you think it's a delicate part, but the main goal of my question is to see the big picture.
Main thing here is to decide on a connection design and to choose protocol. I.e. will you have a persistent connection to your server or connect each time when new data is ready to it.
Then will you use HTTP POST or Web Sockets or ordinary sockets. Will you rely exclusively on nginx or your data catcher will be another serving service.
This would be a most secure way, if other people will be connecting to nginx to view sites etc.
Write or use another server to run on another port. For example, another nginx process just for that. Then use SSL (i.e. HTTPS) with basic authentication to prevent anyone else from abusing the connection.
Then on client side, make a packet every x seconds of all data (pickle.dumps() or json or something), then connect to your port with your credentials and pass the packet.
Python script may wait for it there.
Or you write a socket server from scratch in Python (not extra hard) to wait for your packets.
The caveat here is that you have to implement your protocol and security. But you gain some other benefits. Much more easier to maintain persistent connection if you desire or need to. I don't think it is necessary though and it can become bulky to code break recovery.
No, just wait on some port for a connection. Client must clearly identify itself (else you instantly drop the connection), it must prove that it talks your protocol and then send the data.
Use SSL sockets to do it so that you don't have to implement encryption yourself to preserve authentication data. You may even rely only upon in advance built keys for security and then pass only data.
Do not worry about the speed. Sockets are handled by OS and if you are on Unix-like system you may connect as many times you want in as little time interval you need. Nothing short of DoS attack won't inpact it much.
If on Windows, better use some finished server because Windows sometimes do not release a socket on time so you will be forced to wait or do some hackery to avoid this unfortunate behaviour (non blocking sockets and reuse addr and then some flo control will be needed).
As far as your data is small you don't have to worry much about the server protocol. I would use HTTPS myself, but I would write myown light-weight server in Python or modify and run one of examples from internet. That's me though.
The simplest thing that could possibly work would be to take your N floats, convert them to a binary message using struct.pack(), and then send them via a UDP socket to the target machine (if it's on a single LAN you could even use UDP multicast, then multiple receivers could get the data if needed). You can safely send a maximum of 60 to 170 double-precision floats in a single UDP datagram (depending on your network).
This requires no application protocol, is easily debugged at the network level using Wireshark, is efficient, and makes it trivial to implement other publishers or subscribers in any language.
I am trying to implement a multi-threading server which can handle with simultaneously read/write from client.
The server method:
The client connects to the server, when each message starts with the name of the user they want to send the message to, followed by '|'. It looks something like that: "USER_NAME|DATA".
After receiving the data, the server knows by a dictionary of {socket:username} where to send the data. Everything works great, except the fact that the client can't handle with simultaneously reading and writing. I searched for a method to handle that and i found the select() function, but with a lack of examples- i couldn't integrate that function in my code.
therefore I have 2 questions:
Is the select() function should be on the server side? will it be more efficient?
Is someone can demonstrate with a simple example how the select() method should look in the client side?
Thanks in advance!!!
Though select() will work, you have to use threads if you want to do other things while the system is blocked on the select.
Have a look at glib's GIO library. There you can connect callbacks to the actions you want to monitor or act on, for example the 'connect's from clients.
Just open a socket, and use its file descriptor to hang a gio.add_watch on. Here's a mini-tutorial on using giochannels.
I need to have a tcp socket client connected to a server to send data and receive.
But this socket must be always on and i cannot open another socket.
I have always some data to send over the time and then later process the answer to the data sent previously.
If i could open many sockets, i think it was more easy. But in my case i have to send everything on the same socket asynchronously.
So the question is, what do you recommend to use within the Python ecosystem? (twisted, tornado, etc)
Should i consider node.js or another option?
I highly recommend Twisted for this:
It comes with out-of-the-box support for many TCP protocols.
It is easy to maintain a single connection, there is a ReconnectingClientFactory that will deal with disconnections and use exponential backoff, and LoopingCall makes it easy to implement a heartbeat.
Stateful protocols are also easy to implement and intermingle with complex business logic.
It's fun.
I have a service that is exactly like the one you mention (single login, stays on all the time, processes data). It's been on for months working like a champ.
Twisted is possibly hard to get your head around, but the tutorials here are a great start. Knowing Twisted will get you far in the long run!
"i have to send everything on the same socket asynchronously"
Add your data to a queue, have a separate thread taking items out of the queue and sending via socket.send()
I have a client and a server, both written in Python 2.7.
Lets say I wanted to make a multiplayer game server (which I don't at the moment but I'm working towards it). I would need to keep the server up to date (and other clients) on my characters whereabouts, correct?
How would I do this with sockets? Send or request information only when it is needed (e.g the character moves, or another players character moves and the server sends the information to other clients) or would I keep a constant socket open to send data real-time of EVERYBODY's movement regardless of if they have actually done something since the last piece of data was sent or not.
I won't struggle coding it, I just need help with the concept of how I would actually do it.
With TCP sockets it is more typical to leave the connections open, given the teardown & rebuild cost.
Eventually when scaling you will do look into NewIO\RawIO.
If you do not, imagine that the game client might take a step & not get confirmation if sending it to the server & other players.
Definitely keep the socket open, but you should consider using something like ZeroMQ which gives you more kinds of sockets to work with.