This is the problem I'm trying to solve,
I want to write an application that will read outbound http request packets on the same machine's network card. This would then be able to extract the GET url from it.On basis of this information, I want to be able to stop the packet, or redirect it , or let it pass.
However I want my application to be running in promiscuous mode (like wireshark does), and yet be able to eat up (stop) the outbound packet.
I have searched around a bit on this..
libpcap / pcap.h allows to me read packets at the network card, however I haven't yet been able to figure out a way to stop these packets or inject new ones into the network.
Certain stuff like twisted or scapy in python, allows me set up a server that is listening on some local port, I can then configure my browser to connect to it, using proxy configurations. This app can then do the stuff.. but my main purpose of being promiscuous is defeated here..
Any help on how I could achieve this would be greatly appreciated ..
I'd suggest that you approach this at the application layer and use a transparent proxy (e.g. squid) and iptables based interception of outbound port-80 traffic.
The reason I suggest this is that that it will avoid issues with the request being split between packets.
However, if you still want to go ahead with packet interception, you can do it in userspace using netfilters in netlink. I believe there are python wrappers for libnl around.
Essentially you create an iptables rule pointing to "QUEUE" for the traffic you want to intercept and write a program using a netlink library to process the queue, accepting, rejecting and/or modifying packets.
Using pcap you cannot stop the packets, if you are under windows you must go down to the driver level... but you can stop only packets that your machine send.
A solution is act as a pipe to the destination machine: You need two network interfaces (without address possibly), when you get a packet that you does not found interesting on the source network card you simply send it on the destination network card. If the packet is interesting you does not send it, so you act as a filter. I have done it for multimedia performance test (adding jitter, noise, etc.. to video streaming)
You are confusing several things here:
"Promiscuous" usually refers to a mode of a hardware ethernet network card where it delivers all packets in its collision domain up to the kernel network stack and have it sort out delivery (vs. just unicast to given MAC, subscribed multicast, and broadcast in normal operating mode of the card). This is on the receive path.
All the data outbound from your machine will go through (one of) the network cards on the machine, so "promiscuous" does not at all apply here.
You are working on filtering TCP-based protocol (HTTP), but talk in terms of packets. This is wrong. TCP connection is a stream that could be (as far as socket readers and writers are concerned) arbitrarily split into IP datagrams. That URL from HTTP request header could be split across multiple link-layer frames. You would have to stitch them back together and parse the stream anyway. Then you have no chance even at that if SSL is in use.
If you are interested in HTTP filtering then read HTTP RFCs, and read existing open-source code, e.g. squid, nginx, etc.
If you are digging through network stack for better understaning then read W. Richard Stevens books, look into existing code in open-source operating systems, check out BPF and netlink.
Hope this clears it a little.
I have implemented this module in Windows by using two separate NICs and using a socket/pipe(whatever you like) between them in this thread
Related
Key points:
I need to send roughly ~100 float numbers every 1-30 seconds from one machine to another.
The first machine is catching those values through sensors connected to it.
The second machine is listening for them, passing them to an http server (nginx), a telegram bot and another program sending emails with alerts.
How would you do this and why?
Please be accurate. It's the first time I work with sockets and with python, but I'm confident I can do this. Just give me crucial details, lighten me up!
Some small portion (a few rows) of the core would be appreciated if you think it's a delicate part, but the main goal of my question is to see the big picture.
Main thing here is to decide on a connection design and to choose protocol. I.e. will you have a persistent connection to your server or connect each time when new data is ready to it.
Then will you use HTTP POST or Web Sockets or ordinary sockets. Will you rely exclusively on nginx or your data catcher will be another serving service.
This would be a most secure way, if other people will be connecting to nginx to view sites etc.
Write or use another server to run on another port. For example, another nginx process just for that. Then use SSL (i.e. HTTPS) with basic authentication to prevent anyone else from abusing the connection.
Then on client side, make a packet every x seconds of all data (pickle.dumps() or json or something), then connect to your port with your credentials and pass the packet.
Python script may wait for it there.
Or you write a socket server from scratch in Python (not extra hard) to wait for your packets.
The caveat here is that you have to implement your protocol and security. But you gain some other benefits. Much more easier to maintain persistent connection if you desire or need to. I don't think it is necessary though and it can become bulky to code break recovery.
No, just wait on some port for a connection. Client must clearly identify itself (else you instantly drop the connection), it must prove that it talks your protocol and then send the data.
Use SSL sockets to do it so that you don't have to implement encryption yourself to preserve authentication data. You may even rely only upon in advance built keys for security and then pass only data.
Do not worry about the speed. Sockets are handled by OS and if you are on Unix-like system you may connect as many times you want in as little time interval you need. Nothing short of DoS attack won't inpact it much.
If on Windows, better use some finished server because Windows sometimes do not release a socket on time so you will be forced to wait or do some hackery to avoid this unfortunate behaviour (non blocking sockets and reuse addr and then some flo control will be needed).
As far as your data is small you don't have to worry much about the server protocol. I would use HTTPS myself, but I would write myown light-weight server in Python or modify and run one of examples from internet. That's me though.
The simplest thing that could possibly work would be to take your N floats, convert them to a binary message using struct.pack(), and then send them via a UDP socket to the target machine (if it's on a single LAN you could even use UDP multicast, then multiple receivers could get the data if needed). You can safely send a maximum of 60 to 170 double-precision floats in a single UDP datagram (depending on your network).
This requires no application protocol, is easily debugged at the network level using Wireshark, is efficient, and makes it trivial to implement other publishers or subscribers in any language.
I need to generate a very high level of wifi activity for a study to see if very close proximity to a transceiver can have a negative impact on development of bee colonies.
I have tried to write an application which spawns several web-socket server-client pairs to continuously transfer mid-sized files (this approach hit >100MB). However, we want to run this on a single computer connected to a wifi router, so the packets invariably end up getting routed via the loopback interface, not the WLAN.
Alternatively I have tried using a either simple ping floods and curling the router, but this is not producing nearly the maximum bandwidth the router is capable of.
Is there a quick fix on linux to force the traffic over the network? The computer we are using has both an ethernet and a wireless interface, and I found one thread online which suggested setting up iptables to force traffic between the two interfaces and avoid the loopback.
Simply sending packets as fast as possible to a random destination (that is not localhost) should work.
You'll need to use udp (otherwise you need a connection acknowledge before you can send data).
cat /dev/urandom | pv | nc -u 1.1.1.1 9123
pv is optional (but nice).
You can also use /dev/zero, but there may be a risk of link-level compression.
Of course, make sure the router is not actually connected to the internet (you don't want to flood a server somewhere!), and that your computer has the router as the default route.
I've coded a small raw packet syn port scanner to scan a list of ips and find out if they're online. (btw. for Debian in python2.7)
The basic intention was to simply check if some websites are reachable and speed up that process by preceding a raw syn request (port 80) but I stumbled upon something.
Just for fun I started trying to find out how fast I could get with this (fastest as far as i know) check technique and it turns out that despite I'm only sending raw syn packets on one port and listening for responses on that same port (with tcpdump) the connection reliability quite drops starting at about 1500-2000 packets/sec and shortly thereafter almost the entire networking starts blocking on the box.
I thought about it and if I compare this value with e.g. torrent seeding/leeching packets/sec the scan speed is quiet slow.
I have a few ideas why this happens but I'm not a professional and I have no clue how to check if I'm right with my assumptions.
Firstly it could be that the Linux networking has some fancy internal port forwarding stuff running to keep the sending port opened (maybe some sort of feature of iptables?) because the script seems to be able to receive syn-ack even with closed sourceport.
If so, is it possible to prevent or bypass that in some fashion?
Another guess is that the python library is simply too dumb to do real proper raw packet management but that's unlikely because its using internal Linux functions to do that as far as I know.
Does anyone have a clue why that network blocking is happening?
Where's the difference to torrent connections or anything else like that?
Do I have to send the packets in another way or anything?
Months ago I found out that this problem is well known as c10k problem.
It has to do amongst other things with how the kernel allocates and processes tcp connections internally.
The only efficient way to address the issue is to bypass the kernel tcp stack and implement various other low-level things by your own.
All good approaches I know are working with low-level async implementations
There are some good ways to deal with the problem depending on the scale.
For further information i would recommend to search for the c10k problem.
Using a Lantronix UDS-1100 serial to IP converter. The goal is to write a small proof of concept piece in Python to capture serial data output by this device over IP.
I've done a couple test projects using sockets in python, but they were all done between python processes (python > python): listen() on one end, and connect(), sendall() etc on the other.
I think I can use sockets for this project, but before I invest a bunch of time into it, wanted to make sure it is a viable solution.
Can python sockets be used to capture IP traffic when the traffic is originating from a non-python source? I have full control over the IP and port that the device sends the serial data to, but there will be no python connect() initiated by the client. I can pre-pend then serial data with some connect() string if needed.
If sockets won't work, please recommend another solution...guessing it will be REST or similar.
Of course. TCP/IP is supposed to be cross-platform and cross-language, so in theory you should be able to communicate with every kind of device as long as you manage to process and send the expected protocol.
As the title says, I would like to send data using an existing tcp connection. Said connection has already been established by a 3rd party program. I haven't been able to find much information about this, and it's safe to say I don't know how this will work at all.
The operating system is Windows. My preferred programming language is python - I'd prefer not to use 3rd party python modules, but I will if they make my life easier.
Just to clarify, in case you aren't sure what I want to do: I want to send data as if it were sent by a different program; pretty much like WPE pro's send function does.
Update:
Technically, couldn't I manually design the TCP packet and then tell the network device (or operating system) to send that packet? Wouldn't that be exactly the same thing an injected socket would do?
Edit: Wikipedia says the receiving host acknowledges packets it receives, which makes this a bit more difficult. But if can drop that acknowledge-packet before the 3rd party program receives it, then this should work. Right?
Scapy/Pcapy are pretty powerful tools for monitoring and injecting packets into a live network interface. I've used them for several projects. These tools are ideal for stimulus/response low-level network protocols (ie DHCP, DNS, etc) and anything non-stateful sent over simple UDP.
Unfortunately, the TCP layer is very complicated and stateful. So injecting something that makes sense into the stream will be more difficult. Moreover, Scapy/Pcapy do not currently have support for tcp.
A TCP session is not intended to be a many-to-one connection. Its a point-to-point stateful protocol which keeps track of packets that have been sent versus those that have been received by the other end. I don't believe you can inject yourself into an already-established session. Your best bet, as was pointed out previously, is to create a proxy and act as a man-in-the-middle interloper. Still not a trivial thing but doable.