I've coded a small raw packet syn port scanner to scan a list of ips and find out if they're online. (btw. for Debian in python2.7)
The basic intention was to simply check if some websites are reachable and speed up that process by preceding a raw syn request (port 80) but I stumbled upon something.
Just for fun I started trying to find out how fast I could get with this (fastest as far as i know) check technique and it turns out that despite I'm only sending raw syn packets on one port and listening for responses on that same port (with tcpdump) the connection reliability quite drops starting at about 1500-2000 packets/sec and shortly thereafter almost the entire networking starts blocking on the box.
I thought about it and if I compare this value with e.g. torrent seeding/leeching packets/sec the scan speed is quiet slow.
I have a few ideas why this happens but I'm not a professional and I have no clue how to check if I'm right with my assumptions.
Firstly it could be that the Linux networking has some fancy internal port forwarding stuff running to keep the sending port opened (maybe some sort of feature of iptables?) because the script seems to be able to receive syn-ack even with closed sourceport.
If so, is it possible to prevent or bypass that in some fashion?
Another guess is that the python library is simply too dumb to do real proper raw packet management but that's unlikely because its using internal Linux functions to do that as far as I know.
Does anyone have a clue why that network blocking is happening?
Where's the difference to torrent connections or anything else like that?
Do I have to send the packets in another way or anything?
Months ago I found out that this problem is well known as c10k problem.
It has to do amongst other things with how the kernel allocates and processes tcp connections internally.
The only efficient way to address the issue is to bypass the kernel tcp stack and implement various other low-level things by your own.
All good approaches I know are working with low-level async implementations
There are some good ways to deal with the problem depending on the scale.
For further information i would recommend to search for the c10k problem.
Related
I need to write a script to stress test the UDP server.It needs to simulate about 5000 online users and about 400 concurrent users.I couldn't find a similar function on Google, so I wrote a UDP client myself.But I had a problem simulating multiple clients.The solution I came up with:
One socket per client
How to mark online users and concurrent users when using multithreading and multiple sockets to simulate clients?
I encapsulate the client into classes,in this class __ init__ The method of adding one to a variable is used to record the of online users.In this way, concurrent operations cannot be performed successfully
Is it feasible to create 5000 sockets with threads? Is this a best practice? Good performance?
Other approaches?
Is there another approach I haven't thought of? Am I on the wrong track?
Is there a mature testing framework that can be used for reference?
Finally, English is not my mother tongue. Please forgive me for my typos or grammar.Thank you for your reading and look forward to your reply.
There is Apache JMeter tool which is free, open source and modular
There is UDP Request sampler plugin which adds support of the UDP protocol to JMeter, see
The "5000 online users and 400 concurrent users" requirement may be interpreted in the following manner: real users don't hammer the system under test non-stop, they need some time to "think" between operations, i.e. read text, type response, fill forms, take a phone call, etc. So you need to introduce realistic think times using JMeter Timers so you could come up with the configuration when:
5000 users are "online" (connected to the server)
4600 are not doing anything, just "sleeping"
400 are actively sending requests
As long as your machine is capable of doing this without running out of CPU, RAM, Network, etc - it should be fine, personally I would use something like greenlet
Key points:
I need to send roughly ~100 float numbers every 1-30 seconds from one machine to another.
The first machine is catching those values through sensors connected to it.
The second machine is listening for them, passing them to an http server (nginx), a telegram bot and another program sending emails with alerts.
How would you do this and why?
Please be accurate. It's the first time I work with sockets and with python, but I'm confident I can do this. Just give me crucial details, lighten me up!
Some small portion (a few rows) of the core would be appreciated if you think it's a delicate part, but the main goal of my question is to see the big picture.
Main thing here is to decide on a connection design and to choose protocol. I.e. will you have a persistent connection to your server or connect each time when new data is ready to it.
Then will you use HTTP POST or Web Sockets or ordinary sockets. Will you rely exclusively on nginx or your data catcher will be another serving service.
This would be a most secure way, if other people will be connecting to nginx to view sites etc.
Write or use another server to run on another port. For example, another nginx process just for that. Then use SSL (i.e. HTTPS) with basic authentication to prevent anyone else from abusing the connection.
Then on client side, make a packet every x seconds of all data (pickle.dumps() or json or something), then connect to your port with your credentials and pass the packet.
Python script may wait for it there.
Or you write a socket server from scratch in Python (not extra hard) to wait for your packets.
The caveat here is that you have to implement your protocol and security. But you gain some other benefits. Much more easier to maintain persistent connection if you desire or need to. I don't think it is necessary though and it can become bulky to code break recovery.
No, just wait on some port for a connection. Client must clearly identify itself (else you instantly drop the connection), it must prove that it talks your protocol and then send the data.
Use SSL sockets to do it so that you don't have to implement encryption yourself to preserve authentication data. You may even rely only upon in advance built keys for security and then pass only data.
Do not worry about the speed. Sockets are handled by OS and if you are on Unix-like system you may connect as many times you want in as little time interval you need. Nothing short of DoS attack won't inpact it much.
If on Windows, better use some finished server because Windows sometimes do not release a socket on time so you will be forced to wait or do some hackery to avoid this unfortunate behaviour (non blocking sockets and reuse addr and then some flo control will be needed).
As far as your data is small you don't have to worry much about the server protocol. I would use HTTPS myself, but I would write myown light-weight server in Python or modify and run one of examples from internet. That's me though.
The simplest thing that could possibly work would be to take your N floats, convert them to a binary message using struct.pack(), and then send them via a UDP socket to the target machine (if it's on a single LAN you could even use UDP multicast, then multiple receivers could get the data if needed). You can safely send a maximum of 60 to 170 double-precision floats in a single UDP datagram (depending on your network).
This requires no application protocol, is easily debugged at the network level using Wireshark, is efficient, and makes it trivial to implement other publishers or subscribers in any language.
As the title says, I would like to send data using an existing tcp connection. Said connection has already been established by a 3rd party program. I haven't been able to find much information about this, and it's safe to say I don't know how this will work at all.
The operating system is Windows. My preferred programming language is python - I'd prefer not to use 3rd party python modules, but I will if they make my life easier.
Just to clarify, in case you aren't sure what I want to do: I want to send data as if it were sent by a different program; pretty much like WPE pro's send function does.
Update:
Technically, couldn't I manually design the TCP packet and then tell the network device (or operating system) to send that packet? Wouldn't that be exactly the same thing an injected socket would do?
Edit: Wikipedia says the receiving host acknowledges packets it receives, which makes this a bit more difficult. But if can drop that acknowledge-packet before the 3rd party program receives it, then this should work. Right?
Scapy/Pcapy are pretty powerful tools for monitoring and injecting packets into a live network interface. I've used them for several projects. These tools are ideal for stimulus/response low-level network protocols (ie DHCP, DNS, etc) and anything non-stateful sent over simple UDP.
Unfortunately, the TCP layer is very complicated and stateful. So injecting something that makes sense into the stream will be more difficult. Moreover, Scapy/Pcapy do not currently have support for tcp.
A TCP session is not intended to be a many-to-one connection. Its a point-to-point stateful protocol which keeps track of packets that have been sent versus those that have been received by the other end. I don't believe you can inject yourself into an already-established session. Your best bet, as was pointed out previously, is to create a proxy and act as a man-in-the-middle interloper. Still not a trivial thing but doable.
This is the problem I'm trying to solve,
I want to write an application that will read outbound http request packets on the same machine's network card. This would then be able to extract the GET url from it.On basis of this information, I want to be able to stop the packet, or redirect it , or let it pass.
However I want my application to be running in promiscuous mode (like wireshark does), and yet be able to eat up (stop) the outbound packet.
I have searched around a bit on this..
libpcap / pcap.h allows to me read packets at the network card, however I haven't yet been able to figure out a way to stop these packets or inject new ones into the network.
Certain stuff like twisted or scapy in python, allows me set up a server that is listening on some local port, I can then configure my browser to connect to it, using proxy configurations. This app can then do the stuff.. but my main purpose of being promiscuous is defeated here..
Any help on how I could achieve this would be greatly appreciated ..
I'd suggest that you approach this at the application layer and use a transparent proxy (e.g. squid) and iptables based interception of outbound port-80 traffic.
The reason I suggest this is that that it will avoid issues with the request being split between packets.
However, if you still want to go ahead with packet interception, you can do it in userspace using netfilters in netlink. I believe there are python wrappers for libnl around.
Essentially you create an iptables rule pointing to "QUEUE" for the traffic you want to intercept and write a program using a netlink library to process the queue, accepting, rejecting and/or modifying packets.
Using pcap you cannot stop the packets, if you are under windows you must go down to the driver level... but you can stop only packets that your machine send.
A solution is act as a pipe to the destination machine: You need two network interfaces (without address possibly), when you get a packet that you does not found interesting on the source network card you simply send it on the destination network card. If the packet is interesting you does not send it, so you act as a filter. I have done it for multimedia performance test (adding jitter, noise, etc.. to video streaming)
You are confusing several things here:
"Promiscuous" usually refers to a mode of a hardware ethernet network card where it delivers all packets in its collision domain up to the kernel network stack and have it sort out delivery (vs. just unicast to given MAC, subscribed multicast, and broadcast in normal operating mode of the card). This is on the receive path.
All the data outbound from your machine will go through (one of) the network cards on the machine, so "promiscuous" does not at all apply here.
You are working on filtering TCP-based protocol (HTTP), but talk in terms of packets. This is wrong. TCP connection is a stream that could be (as far as socket readers and writers are concerned) arbitrarily split into IP datagrams. That URL from HTTP request header could be split across multiple link-layer frames. You would have to stitch them back together and parse the stream anyway. Then you have no chance even at that if SSL is in use.
If you are interested in HTTP filtering then read HTTP RFCs, and read existing open-source code, e.g. squid, nginx, etc.
If you are digging through network stack for better understaning then read W. Richard Stevens books, look into existing code in open-source operating systems, check out BPF and netlink.
Hope this clears it a little.
I have implemented this module in Windows by using two separate NICs and using a socket/pipe(whatever you like) between them in this thread
This whole topic is way out of my depth, so forgive my imprecise question, but I have two computers both connected to one LAN.
What I want is to be able to communicate one string between the two, by running a python script on the first (the host) where the string will originate, and a second on the client computer to retrieve the string.
What is the most efficient way for an inexperienced programmer like me to achieve this?
First, lets get the nomenclature straight. Usually the part that initiate the communication is the client, the parts that is waiting for a connection is a server, which then will receive the data from the client and generate a response. From your question, the "host" is the client and the "client" seems to be the server.
Then you have to decide how to transfer the data. You can use straight sockets, in which case you can use SocketServer, or you can rely on an existing protocol, like HTTP or XML-RPC, in which case you will find ready to use library packages with plenty of examples (e.g. xmlrpclib and SimpleXMLRPCServer)
There are about a million ways.
If I were doing it, I'd use the SocketServer library, because it's not too insane, fairly well documented, and most importantly, I've used it before.
There are a couple of examples here: http://docs.python.org/library/socketserver.html#examples
File share and polling filesystem every minute. No joke. Of course, it depends on what are requirements for your applications and what lag is acceptable but in practice using file shares is quite common.