I have a project in which I need to setup a network that is essentially a bunch of Raspberry Pis connected through a router over ethernet, and have them talk to each other without using IP.
My challenge here is two folds, first, how can I write raw binary data to ethernet to pass my own custom payloads and have a custom parser on each end picking up and deserialising that data?
Second, and this is secondary for this post, if anyone has any ideas as to how I could use a router without using IP (aka setting up my own simple addressing protocol) this would be very welcomed. So far what I've sketched out is to procure myself a router than can be flashed, then have custom software on there running a custom protocol. However I'm not sure if this is even doable with off the shelf routers. Pointers are welcome.
Ideally I'd like to do all of this in python.
For your first question: asyncio comes as a standard library with Python. It can handle most of your communication needs, essentially acting as your communication stream for your binary data. Example implementation here.
For your second question: you can't go wrong with using IPv4. You could potentially implement something new but you'd probably go down a rabbit hole in doing so.
Related
As the title says, I would like to send data using an existing tcp connection. Said connection has already been established by a 3rd party program. I haven't been able to find much information about this, and it's safe to say I don't know how this will work at all.
The operating system is Windows. My preferred programming language is python - I'd prefer not to use 3rd party python modules, but I will if they make my life easier.
Just to clarify, in case you aren't sure what I want to do: I want to send data as if it were sent by a different program; pretty much like WPE pro's send function does.
Update:
Technically, couldn't I manually design the TCP packet and then tell the network device (or operating system) to send that packet? Wouldn't that be exactly the same thing an injected socket would do?
Edit: Wikipedia says the receiving host acknowledges packets it receives, which makes this a bit more difficult. But if can drop that acknowledge-packet before the 3rd party program receives it, then this should work. Right?
Scapy/Pcapy are pretty powerful tools for monitoring and injecting packets into a live network interface. I've used them for several projects. These tools are ideal for stimulus/response low-level network protocols (ie DHCP, DNS, etc) and anything non-stateful sent over simple UDP.
Unfortunately, the TCP layer is very complicated and stateful. So injecting something that makes sense into the stream will be more difficult. Moreover, Scapy/Pcapy do not currently have support for tcp.
A TCP session is not intended to be a many-to-one connection. Its a point-to-point stateful protocol which keeps track of packets that have been sent versus those that have been received by the other end. I don't believe you can inject yourself into an already-established session. Your best bet, as was pointed out previously, is to create a proxy and act as a man-in-the-middle interloper. Still not a trivial thing but doable.
This is the problem I'm trying to solve,
I want to write an application that will read outbound http request packets on the same machine's network card. This would then be able to extract the GET url from it.On basis of this information, I want to be able to stop the packet, or redirect it , or let it pass.
However I want my application to be running in promiscuous mode (like wireshark does), and yet be able to eat up (stop) the outbound packet.
I have searched around a bit on this..
libpcap / pcap.h allows to me read packets at the network card, however I haven't yet been able to figure out a way to stop these packets or inject new ones into the network.
Certain stuff like twisted or scapy in python, allows me set up a server that is listening on some local port, I can then configure my browser to connect to it, using proxy configurations. This app can then do the stuff.. but my main purpose of being promiscuous is defeated here..
Any help on how I could achieve this would be greatly appreciated ..
I'd suggest that you approach this at the application layer and use a transparent proxy (e.g. squid) and iptables based interception of outbound port-80 traffic.
The reason I suggest this is that that it will avoid issues with the request being split between packets.
However, if you still want to go ahead with packet interception, you can do it in userspace using netfilters in netlink. I believe there are python wrappers for libnl around.
Essentially you create an iptables rule pointing to "QUEUE" for the traffic you want to intercept and write a program using a netlink library to process the queue, accepting, rejecting and/or modifying packets.
Using pcap you cannot stop the packets, if you are under windows you must go down to the driver level... but you can stop only packets that your machine send.
A solution is act as a pipe to the destination machine: You need two network interfaces (without address possibly), when you get a packet that you does not found interesting on the source network card you simply send it on the destination network card. If the packet is interesting you does not send it, so you act as a filter. I have done it for multimedia performance test (adding jitter, noise, etc.. to video streaming)
You are confusing several things here:
"Promiscuous" usually refers to a mode of a hardware ethernet network card where it delivers all packets in its collision domain up to the kernel network stack and have it sort out delivery (vs. just unicast to given MAC, subscribed multicast, and broadcast in normal operating mode of the card). This is on the receive path.
All the data outbound from your machine will go through (one of) the network cards on the machine, so "promiscuous" does not at all apply here.
You are working on filtering TCP-based protocol (HTTP), but talk in terms of packets. This is wrong. TCP connection is a stream that could be (as far as socket readers and writers are concerned) arbitrarily split into IP datagrams. That URL from HTTP request header could be split across multiple link-layer frames. You would have to stitch them back together and parse the stream anyway. Then you have no chance even at that if SSL is in use.
If you are interested in HTTP filtering then read HTTP RFCs, and read existing open-source code, e.g. squid, nginx, etc.
If you are digging through network stack for better understaning then read W. Richard Stevens books, look into existing code in open-source operating systems, check out BPF and netlink.
Hope this clears it a little.
I have implemented this module in Windows by using two separate NICs and using a socket/pipe(whatever you like) between them in this thread
Which way is better?
Creating a while loop and then using the select module OR using ThreadedTCPServer with a custom class.
Im having problems with the Threaded TCP Server, although it could just be my coding.
My personal recommendation is to use Twisted. It's a Python-based framework intended primarily for writing event-driven network software. The documentation has a lot of great examples of how to create various types of servers and clients, as well.
I am sure there is no such a thing like the "correct" way.
If you want not, must not or cannot use any of the existing server implementations the general idea is (in pseudo code):
ss = serversocket()
ss.bind ()
while (True):
cs = ss.accept ()
spawnCommThread (cs)
In the CommThread for each client you take care of reading from the socket returned by accept, communicate with your client and die, when the client closes the connection or another criterion is given.
This whole topic is way out of my depth, so forgive my imprecise question, but I have two computers both connected to one LAN.
What I want is to be able to communicate one string between the two, by running a python script on the first (the host) where the string will originate, and a second on the client computer to retrieve the string.
What is the most efficient way for an inexperienced programmer like me to achieve this?
First, lets get the nomenclature straight. Usually the part that initiate the communication is the client, the parts that is waiting for a connection is a server, which then will receive the data from the client and generate a response. From your question, the "host" is the client and the "client" seems to be the server.
Then you have to decide how to transfer the data. You can use straight sockets, in which case you can use SocketServer, or you can rely on an existing protocol, like HTTP or XML-RPC, in which case you will find ready to use library packages with plenty of examples (e.g. xmlrpclib and SimpleXMLRPCServer)
There are about a million ways.
If I were doing it, I'd use the SocketServer library, because it's not too insane, fairly well documented, and most importantly, I've used it before.
There are a couple of examples here: http://docs.python.org/library/socketserver.html#examples
File share and polling filesystem every minute. No joke. Of course, it depends on what are requirements for your applications and what lag is acceptable but in practice using file shares is quite common.
The question: How do I create a python application that can connect and send packets over the internet to another computer running the same application? Is there any existing code/library I could use?
The background: I am pretty new to programming (HS senior). I've created a lot of simple things in python but I've recently decided to start on a bigger project. I'm considering creating a Magic: the Gathering booster draft simulator, but I'm not sure if it is feasible given my skill set so I'm asking around before I get started. The application would need to send data between computers about which cards are being picked/passed.
Thanks!
Twisted is a python event-driven networking engine licensed under MIT. Means that a single machine can communicate with one or more other machines, while doing other things between data being received and sent, all asynchronously, and running a in a single thread/process.
It supports many protocols out of the box, so you can just as well using an existing one. That's better because you get support for the protocol from 3rd party software (i.e. using HTTP for communication means middleware software that uses HTTP will be compatible: proxies etc.)
It also makes easy to create your own communication protocol, if that's what you want.
The documentation is filled with examples.
The standard library includes SocketServer (documented here), which might do what you want.
However I wonder if a better solution might be to use a message queue. Lots of good implementations already exist, including Python interfaces. I've used RabbitMQ before. The idea is that the computers both subscribe to the queue, and can either post or listen for events.
A great place to start looking is the Python Standard Library. In particular there are a few sections that will be relevant:
18. Interprocess Communication and Networking
19. Internet Data Handling
21. Internet Protocols and Support
Since you mentioned that you have no experience with this, I'd suggest starting with a HTTP based implementation. The HTTP protocol is fairly simple and easy to work with. In addition, there are nice frameworks to support this operation, such as webpy for the server and HTTPLib from the standard library for the client.
If you really want to dive into networking, then a socket based implementation might be educational. This will teach you the underlying concepts that are used in lots of networking code while resulting in an interface that is similar to a file stream.
Also, check out Pyro (Python remoting objects). Se this answer for more details.
Pyro basically allows you to publish Python object instances as services that can be called remotely. Pyro is probably the easiest way to implement Python-to-python process communication.
It's also worth looking at Kamaelia for this sort of thing - it's original usecase was network systems, and makes building such things relatively intuitive.
Some links: Overview of basic TCP system, Simple Chat server, Building a layered protocol, walk-through of how to evolve new components. Other extreme: P2P radio system: source, peer.
If it makes any difference, we've tested whether the system is accessible to novices through involvement in google summer of code 3 years in a row, actively taking on both experienced and inexperienced developers. All of them managed to build useful systems.
Essentially, if you've ever played with unix pipelines the ideas should be familiar.
Caveat: I wrote major chunks of Kamaelia :-)
If you want to learn how to do these things though, playing with a few different approaches makes sense, and you should definitely check out Twisted (the standard answer to this question), Pyro & the standard library tools. Each has a different approach, and learning them will definitely benefit you!
However, like nosklo, I would recommend against using the socket library directly and use a library instead - simply because it is much much harder to get sockets programming correct than people tend to realise.
Communication will take place with sockets, one way or another. Just a question of whether you use existing higher-level libraries, or roll your own.
If you're doing this as a learning experience, probably want to start as low-level as you can, to see the real nuts and bolts. Which means you probably want to start with a SocketServer, using a TCP connection (TCP is basically guaranteed delivery of data; UDP is not).
Google for some simple example code. Setting one up is very easy. But you will have to define all the details of your communications protocol: which end sends when and what, which end listens and when, what exactly the listener will expect, does it reply to confirm receipt, etc.