I am trying to generate some traffic to a server by sending TCP messages to it.
For this, I am using a Python script which opens a TCP socket and then sends some data over it. After receiving a reply, the TCP connection gets closed.
Question: I would like to be able to predefine a rate with which the script will be sending the requests to the server, eg: 5 messages per second. However, I do not have a clue how to script this via Python :(.
Anyone an idea how to do this (a short example would be super ! ;) ?
Thanks in advance.
Note: I might need to add an extra difficulty: since the server has to reply,
I guess I have to make the script working asynchronously ... That way, I can
send the requests out without having to wait for a reply on the previous request...
What you're looking for is an implementation of the token bucket algorithm. It's analogous to a bucket with a fixed capacity, where each consumer can't perform the action until it gets a token, and the bucket is refilled at a fixed rate.
The algorithm is easy to implement, but the link below has an example:
http://code.activestate.com/recipes/511490-implementation-of-the-token-bucket-algorithm/
Related
I'm working on a debug latency problem of websocket.
I am trying to receive some price information from crypto-currency exchange with its websocket interface. The data packets we receive include the timestamp generatee on the exchange server. I log the time when we receive the tick information on our computer (the "client box") and compare the latency between the arrival time and the server generation time. Most of the ticks show a few tens of milliseconds which is more or less fine. But everyday we see a few times the latency becomes several seconds or even more then ten seconds and I would like to figure out where are these large latency come from.
The system is using Python programming language and the websocket module I'm using is websocket-client (https://pypi.org/project/websocket_client/, https://github.com/websocket-client/websocket-client), I tried to add logs inside the module and see if it is due to module processing time but still no luck.
One idea currently in my mind is to use tcpdump to capture the net traffic and record the time the tcp packet arrives my network card. If this time still presents the latency, I will have no way other than move the program to a co-located server. However, I encounters difficult here as the websocket connection is SSL-encrypted. Thus I cannot see the tick generation time packed inside the message.
Does anyone have some solution here ? In particular:
Is there any way to retrieve the private key of SSL from the websocket-client python package from client-end? (I assume the key should be available somewhere local side, otherwise the websocket-client cannot decrypt the data itself. And WireShark should be able to decrypt the message for TSL1.2 protocol)
if it is not easy to do this with websocket-client package, I'm happy to try other websocket lib written by python, C/C++.
Can tcpdump get the timestamp when the TCP data packet sent from server (even in server time)?
Any other advices are highly appreciated as well.
Thanks a lot!
Thanks #Eugène Adell
My tcpdump opened in WireShark is mostly like below
and I can see the TSval in TCP Option - Timestamps
Can these indicate something?
Sorry for probably basic questions, really lack of experience in this area & Thanks again.
EDIT
Can tcpdump get the timestamp when the TCP data packet sent from
server (even in server time)?
Open your capture and see if the packets have the TCP timestamps option (as defined in RFC 1323 but better explained in RFC 7323). If so, the very first SYN packet should already mention it.
Unluckily the meaning of the TSval (Timestamp value in milliseconds) given in these packets is not the real clock and is not always changing like a real clock (it depends on the implementation used by your computers). If the conversation with your server lasts for 60s for example, check if this TSval also moves from 60s, if so maybe can you use this field to track when the packets were sent.
Most IMAP email clients have the ability to receive emails automatically in real time, without refreshing on a fixed interval. This is because the email client keeps a connection open to the server, and if there isn't any email data exchanged to keep the connection alive, the email client will send a NOOP packet on a fixed interval (just like other TCP protocols have their own keepalive packets). In Thunderbird, this behavior can be controlled by the 'Allow immediate server notifications when new messages arrive' option.
I'm writing a Python program that needs to know instantly when emails come in. The standard procedure for receiving emails is (after connection and login) select('inbox'), search(None, 'ALL') (or some specific search term), and fetch(). I can't figure out how to listen for new emails in real time. Every guide I've read suggests running the search() and fetch() functions in a loop. I've tried running the read() function while sending myself emails, but I have never seen read() output any data. Perhaps the server needs to know to push emails to the client? I haven't been able to confirm or refute this theory.
How can I instantly receive new emails with imaplib, or a similar Python library?
While I did not know at the time of originally posting this question, I was looking for a solution that implements the IMAP IDLE command, as defined in RFC2177.
Since you want to fetch emails asynchronously you should use the below library
https://github.com/bamthomas/aioimaplib
I used aioimaplib in my code, but imaplib2 also supposedly supports IDLE; see: https://web.archive.org/web/20090814230913/http://blog.hokkertjes.nl/2009/03/11/python-imap-idle-with-imaplib2/
I'm using a service that sends me some data from user over webhooks. If there is any user interaction on this service, it hits my URL with HTTP request, with the data in POST/GET, and then expects text/json response to show back to the user. The response has to be in few seconds, otherwise the HTTP request times out and the service has no way of finding out what should be the response to the user.
The problem here is that now I'm not processing these data on my server with public IP, but I need to do it on my RPi, which keeps moving, which meains it has different IP every few hours, and mostly not public.
I'm sure I will still need to use the server with public IP to redirect these requests to my RPi, and I have few ideas, but I don't know what is reliable or if it even would work.
Let the API talk to my server and save the data. Then have the RPi constantly asking my server if there are any new data. Propably the dumbest idea - not ideal to use over metered connection, propably longer reply, and it will be harder to return the RPi's reply in the HTTP request made from API.
Having (Python) script running on my server, that will a) serve as socket server and RPi will connect to this socket, and b) have running SimpleHTTPRequestHandler to process requests from API and send them to the socket, the reply with RPi's reply. Propably easy way to keep connection between my server and RPi, allowing me to pass data in both directions.
Open SSH tunnel between the RPi and my server. This way, I could process the requests from service directly on my RPi. But how reliable is this solution? (Keeping it alive, opening the tunnel automatically, etc, propably question for superuser forum)
I'm thinking of going with choice 3 if it will be possible, but first I'd like to hear what you guys think. Is this a good and reliable idea? Or are there any better ways I don't know about? Or did anybody already faced this problem?
To sum it up:
Something sends HTTP request to public IP. I need to process this request (and reply to it) in Python script on device without public IP. I have a server with public IP that could be used as a bridge. I much don't care what will run on the server, if it will be able to redirect these requests.
Thanks
Key points:
I need to send roughly ~100 float numbers every 1-30 seconds from one machine to another.
The first machine is catching those values through sensors connected to it.
The second machine is listening for them, passing them to an http server (nginx), a telegram bot and another program sending emails with alerts.
How would you do this and why?
Please be accurate. It's the first time I work with sockets and with python, but I'm confident I can do this. Just give me crucial details, lighten me up!
Some small portion (a few rows) of the core would be appreciated if you think it's a delicate part, but the main goal of my question is to see the big picture.
Main thing here is to decide on a connection design and to choose protocol. I.e. will you have a persistent connection to your server or connect each time when new data is ready to it.
Then will you use HTTP POST or Web Sockets or ordinary sockets. Will you rely exclusively on nginx or your data catcher will be another serving service.
This would be a most secure way, if other people will be connecting to nginx to view sites etc.
Write or use another server to run on another port. For example, another nginx process just for that. Then use SSL (i.e. HTTPS) with basic authentication to prevent anyone else from abusing the connection.
Then on client side, make a packet every x seconds of all data (pickle.dumps() or json or something), then connect to your port with your credentials and pass the packet.
Python script may wait for it there.
Or you write a socket server from scratch in Python (not extra hard) to wait for your packets.
The caveat here is that you have to implement your protocol and security. But you gain some other benefits. Much more easier to maintain persistent connection if you desire or need to. I don't think it is necessary though and it can become bulky to code break recovery.
No, just wait on some port for a connection. Client must clearly identify itself (else you instantly drop the connection), it must prove that it talks your protocol and then send the data.
Use SSL sockets to do it so that you don't have to implement encryption yourself to preserve authentication data. You may even rely only upon in advance built keys for security and then pass only data.
Do not worry about the speed. Sockets are handled by OS and if you are on Unix-like system you may connect as many times you want in as little time interval you need. Nothing short of DoS attack won't inpact it much.
If on Windows, better use some finished server because Windows sometimes do not release a socket on time so you will be forced to wait or do some hackery to avoid this unfortunate behaviour (non blocking sockets and reuse addr and then some flo control will be needed).
As far as your data is small you don't have to worry much about the server protocol. I would use HTTPS myself, but I would write myown light-weight server in Python or modify and run one of examples from internet. That's me though.
The simplest thing that could possibly work would be to take your N floats, convert them to a binary message using struct.pack(), and then send them via a UDP socket to the target machine (if it's on a single LAN you could even use UDP multicast, then multiple receivers could get the data if needed). You can safely send a maximum of 60 to 170 double-precision floats in a single UDP datagram (depending on your network).
This requires no application protocol, is easily debugged at the network level using Wireshark, is efficient, and makes it trivial to implement other publishers or subscribers in any language.
I am using a server to send some piece of information to another server every second. The problem is that the other server response is few kilobytes and this consumes the bandwidth on the first server ( about 2 GB in an hour ). I would like to send the request and ignore the return ( not even receive it to save bandwidth ) ..
I use a small python script for this task using (urllib). I don't mind using any other tool or even any other language if this is going to make the request only.
A 5K reply is small stuff and is probably below the standard TCP window size of your OS. This means that even if you close your network connection just after sending the request and checking just the very first bytes of the reply (to be sure that request has been really received) probably the server already sent you the whole answer and the packets are already on the wire or on your computer.
If you cannot control (i.e. trim down) what is the server reply for your notification the only alternative I can think to is to add another server on the remote machine waiting for a simple command and doing the real request locally and just sending back to you the result code. This can be done very easily may be even just with bash/perl/python using for example netcat/wget locally.
By the way there is something strange in your math as Glenn Maynard correctly wrote in a comment.
For HTTP, you can send a HEAD request instead of GET or POST:
import urllib2
request = urllib2.Request('https://stackoverflow.com/q/5049244/')
request.get_method = lambda: 'HEAD' # override get_method
response = urllib2.urlopen(request) # make request
print response.code, response.url
Output
200 https://stackoverflow.com/questions/5049244/how-can-i-ignore-server-response-t
o-save-bandwidth
See How do you send a HEAD HTTP request in Python?
Sorry but this does not make much sense and is likely a violation of the HTTP protocol. I consider such an idea as weird and broken-by-design. Either make the remote server shut up or configure your application or whatever is running on the remote server on a different protocol level using a smarter protocol with less bandwidth usage. Everything else is hard being considered as nonsense.