I have been creating an application using UDP for transmitting and receiving information. The problem I am running into is security. Right now I am using the IP/socketid in determining what data belongs to whom.
However, I have been reading about how people could simply spoof their IP, then just send data as a specific IP. So this seems to be the wrong way to do it (insecure). So how else am I suppose to identify what data belongs to what users? For instance you have 10 users connected, all have specific data. The server would need to match the user data to this data we received.
The only way I can see to do this is to use some sort of client/server key system and encrypt the data. I am curious as to how other applications (or games, since that's what this application is) make sure their data is genuine. Also there is the fact that encryption takes much longer to process than unencrypted. Although I am not sure by how much it will affect performance.
Any information would be appreciated. Thanks.
One solution is to use TCP because it is immune to spoofing the source address over the open internet because of the three-way-handshake (More information on why TCP source address spoofing is impossible.). If you still want to use UDP, you could have a simulated three way handshake to begin the connection. A session id could then be added to each UDP packet. This will increase the connection overhead by 2 packets and a few bits per packet, however you will still gain from UDP's speed for the rest of the session when compared to tcp.
However, using TCP or UDP as a transport layer still leaves you open to other attacks such as Sniffing and Man in The Middle attacks using arp spoofing or dns cache poising. Another problem is if both the attacker and the gamers are on the same local lan, such as a wireless network or another broadcast network then you are able to receive traffic regardless of the source/dest address and ONLY THEN does spoofing a three way handshake become possible (and an hmac can't help!). The best soltuion is to use SSL/TLS as your transport layer which solves all of these problems.
You should not reinvent the wheal, but if you need to encrypt UDP for some reason you should use a Stream Cipher like RC4-drop1024 or even better a Block Cipher like AES 256 in OFB Mode. This will save bandwidth over other modes of encryption because they round up to the largest block size.
EDIT:
Based on Marts comment for (Datagram Transport Layer Security)DTLS I did some digging and I found there is an official RFC and its supported by OpenSSL and should be exposed using the pyOpenSSL library. I recommend using the RC4-SHA cipher suite to reduce overhead, this suite is supported by SSL 3.0 (newest). However DTLS will probably have more overhead (LAG!) then TCP.
You can look at HMAC
Wikipedia:
In cryptography, HMAC (Hash-based
Message Authentication Code), is a
specific construction for calculating
a message authentication code (MAC)
involving a cryptographic hash
function in combination with a secret
key. As with any MAC, it may be used
to simultaneously verify both the data
integrity and the authenticity of a
message.
Each client would need to get a unique token which only they know. Every message they send, they'll make a hash based on the token and the message and send it along with the message itself. Then you can verify that the message came from a specific client.
If you absolutely need to verify that a particular user is a particular user then you need to use some form of encryption where the user signs their messages. This can be done pretty quickly because the user only needs to generate a hash of their message and then sign (encrypt) the hash.
For your game application you probably don't need to worry about this. Most ISPs wont allow their users to spoof IP addresses thus you need to only worry about users behind NAT in which you may have multiple users running from the same IP address. In this case, and the general one, you can fairly safely identify unique users based on a tuple containing ip address and UDP port.
DTLS is likely the best solution, however, it appears to be very poorly documented. I've been looking for a similar solution for a while now and all of the references I've seen to OpenSSL's DTLS implementation suggests that you'll be needing to dig through the OpenSSL examples & source code to figure out how to use it... which, to me, means I'm going to make 10 serious security mistakes when I try to set it up. Also, I don't believe the pyOpenSSL liberary exports this functionality.
An alternative I've been considering is the Secure Remote Password Protocol. The advantage of this solution is that it gives you strong mutual authentication (on par with the security of Kerberos according to the docs) and, just as importantly in your case, it provides both ends with a shared session key that can be used for encryption.
Given the shared key, each packet could contain AES256_CBC( <random starter block for CBC><user-id><sequence_number><application data> ) If the decryption succeeds in providing the anticipated user-id, the packet is authenticated as coming from your user and the sequence number can be used for avoiding replay attacks.
One downside to SRP is that, in Python, the number crunching is pretty slow. I modified the demo Python code into something a bit more usable and found that it took about 300ms to perform a single client-server SRP exchange (2Ghz CPU). However, a straight-forward implementation in C++ of the SRP algorithim using the BigNumber support in the OpenSSL took only 2ms. So, if you intend to go this route, I'd highly recommend using a C/C++ implementation of the algorihim for production code. Otherwise, you'll likely only be able to handle a few logins per second.
I'm breaking this down into four levels of security.
Extremely Insecure - Anyone on the network can spoof a valid request/response with generally available prior knowledge. (ie syslog)
Very Insecure - Anyone on the network can spoof a valid request/response only if they have at least read access to the wire. (Passive MITM) (ie http accessable forum with browser cookies)
Somewhat Insecure - Anyone in the network can spoof a valid request/response if they can read AND make changes to the wire (Active MITM) (ie https site with self-signed cert)
Secure - Requests/Responses cannot be spoofed even with full access to the
wire. (ie https accessable ecommerce site)
For Internet games the very insecure solution might actually be acceptable (It would be my choice) It requires no crypto. Just a field in your apps UDP packet format with some kind of random practically unguessable session identifier ferried around for the duration of the game.
Somewhat insecure requires a little bit of crypto but none of the trust/PKI/PSK needed to prevent Active-MITM of the secure solution. With somewhat insecure if the data payloads were not sensitive you could use an integrity only cipher with (TCP) TLS/ (UDP) DTLS to reduce processing overhead and latency at the client and server.
For games UDP is a huge benefit because if there is packet loss you don't want the IP stack to waste time retransmitting stale state - you want to send new state. With UDP there are a number of clever schemes such as non-acknowledged frames (world details which don't matter so much if their lost) and statistical methods of duplicating important state data to counter predictable levels of observed packet loss.
At the end of the day I would recommend go very insecure or somewhat insecure /w DTLS integrity only.
I would look into the Garage Games networking library. It is written in C++ and uses UDP. It is designed for low latency and is considered one of the best for games.
If I remember correctly they would actually calculate the likely position of the player both on the client side and the server side. It would do this for many aspects to ensure integrity of the data. It also would do a crc check on the client software and compare against the server software to make sure they matched.
I am not sure you can license it separately anymore so you may have to license the game engine (100 bucks). It would at least give you some insight on a proven approach to UDP for games. Another possibility is looking into the PyGame networking code. It may have already addressed the issues you are facing.
Related
I have built a python script that uses python socket to build a connection between my python application and my python server. I have encrypted the data sent between the two systems. I was wondering if I should think of any other things related to security against hackers. Can they do something that could possibly steal data from my computer.
thanks in advance for the effort.
I have encrypted the data sent between the two systems.
Encryption is generally a good step, but there are still some subtle concerns, e.g.:
An attacker can capture an encrypted message and replay it by resending the ciphertext without knowing the encryption key. If it causes a command (such as turning on a lamp or coffee machine, rebooting, etc) the attacker can rerun the command.
Similarly, certain types of encryption are vulnerable to an attacker piecing together pieces of ciphertexts to create a frankenmessage that will decrypt properly (e.g. with AES-ECB).
Your handshake (per your comment) seems to be more security-by-obscurity than a reviewed means of security.
There are off-the-shelf protocols, like the well-known TLS, that provide fairly comprehensive protection. If you can easily add this layer to your sockets (even with hardcoded, self-signed certificates that you distribute to both machines and verify) you already gain significant security over DIY encryption. As you adopt more of the TLS ecosystem, such as a certificate authority and PKI, you may be able to gain further security benefits for some threat models.
There are other theoretical risks, such as an attacker taking advantage of buffer overflow issues to try to gain remote control of the server. Python 3 is generally a good language as far as memory safety, but it's a good idea to make sure that your libraries and machine stay up to date.
If your threat model isn't concerned about this, then you're likely fine. Further, if this is a personal project, you may even want to try to deploy it, and then break into it yourself (knowing everything other than the encryption key) as a further learning exercise.
If the data is encrypted using a good decryption (AES for example) and the decryption is key is send safely your data is safe. The only other thing I can think about is adding a password or another authentication before accepting data sent to you via socket.
Edit: If you keep the connection open, it's always a good sign to create authentication method so random people won't be able to send you random data.
Key points:
I need to send roughly ~100 float numbers every 1-30 seconds from one machine to another.
The first machine is catching those values through sensors connected to it.
The second machine is listening for them, passing them to an http server (nginx), a telegram bot and another program sending emails with alerts.
How would you do this and why?
Please be accurate. It's the first time I work with sockets and with python, but I'm confident I can do this. Just give me crucial details, lighten me up!
Some small portion (a few rows) of the core would be appreciated if you think it's a delicate part, but the main goal of my question is to see the big picture.
Main thing here is to decide on a connection design and to choose protocol. I.e. will you have a persistent connection to your server or connect each time when new data is ready to it.
Then will you use HTTP POST or Web Sockets or ordinary sockets. Will you rely exclusively on nginx or your data catcher will be another serving service.
This would be a most secure way, if other people will be connecting to nginx to view sites etc.
Write or use another server to run on another port. For example, another nginx process just for that. Then use SSL (i.e. HTTPS) with basic authentication to prevent anyone else from abusing the connection.
Then on client side, make a packet every x seconds of all data (pickle.dumps() or json or something), then connect to your port with your credentials and pass the packet.
Python script may wait for it there.
Or you write a socket server from scratch in Python (not extra hard) to wait for your packets.
The caveat here is that you have to implement your protocol and security. But you gain some other benefits. Much more easier to maintain persistent connection if you desire or need to. I don't think it is necessary though and it can become bulky to code break recovery.
No, just wait on some port for a connection. Client must clearly identify itself (else you instantly drop the connection), it must prove that it talks your protocol and then send the data.
Use SSL sockets to do it so that you don't have to implement encryption yourself to preserve authentication data. You may even rely only upon in advance built keys for security and then pass only data.
Do not worry about the speed. Sockets are handled by OS and if you are on Unix-like system you may connect as many times you want in as little time interval you need. Nothing short of DoS attack won't inpact it much.
If on Windows, better use some finished server because Windows sometimes do not release a socket on time so you will be forced to wait or do some hackery to avoid this unfortunate behaviour (non blocking sockets and reuse addr and then some flo control will be needed).
As far as your data is small you don't have to worry much about the server protocol. I would use HTTPS myself, but I would write myown light-weight server in Python or modify and run one of examples from internet. That's me though.
The simplest thing that could possibly work would be to take your N floats, convert them to a binary message using struct.pack(), and then send them via a UDP socket to the target machine (if it's on a single LAN you could even use UDP multicast, then multiple receivers could get the data if needed). You can safely send a maximum of 60 to 170 double-precision floats in a single UDP datagram (depending on your network).
This requires no application protocol, is easily debugged at the network level using Wireshark, is efficient, and makes it trivial to implement other publishers or subscribers in any language.
I need to build a Python application that receives highly secure data, decrypts it, and processes & stores in a database. My server may be anywhere in the world so direct connection is not feasible. What is the safest/smartest way to securely transmit data from one server to another (think government/bank-level security). I know this is quite vague but part of the reason for that is to not limit the scope of answers received.
Basically, if you were building an app between two banks (this has nothing to do with banks but just for reference), how would you securely transmit the data?
Sorry, I should also add SFTP probably will not cut it since this python app must fire when it is pinged from the other server with a secure data transmission.
What is the safest/smartest way to securely transmit data from one server to another (think government/bank-level security)
It depends on your threat model, but intrasite VPN is sometimes used to tunnel traffic like this.
If you want to move up in the protocol stack, then mutual authentication with the client pinning the server's public key would be a good option.
In contrast, I used to perform security architecture work for a US investment bank. They did not use anything - they felt the leased line between data centers provided enough security.
Transmission and encryption need not happen together. You can get away with just about any delivery method, if you encrypt PROPERLY!
Encrypting properly means using a large, randomly generated keys, using HMACs (INSIDE! the encryption) and checking for replay attacks. There may also be a denial of service attack, timing attacks and so forth; though these may also apply to any encrypted connection. Check for data coming in out of order, late, more than once. There is also the possibility (again, depending on the situation) that your "packets" will leak data (e.g. transaction volumes, etc).
DO NOT, UNDER ANY CIRCUMSTANCES, MAKE YOUR OWN ENCRYPTION SCHEME.
I think that public key encryption would be worthwhile; that way if someone collects copies of the encrypted data, then attacks the sending server, they will not have the keys needed to decrypt the data.
There may be standards for your industry (e.g. banking industry), to which you need to conform.
There are VERY SERIOUS PITFALLS if you do not implement this sort of thing correctly. If you are running a bank, get a security professional.
There are several details to be considered, and I guess the question is not detailed enough to provide a single straight answer. But yes, I agree, the VPN option is definitely a safe way to do it, provided you can set up a VPN.If not, the SFTP protocol (not FTPS) would be the next best choice, as it is PCI-DSS compliant (secure enough for banking) and HIPAA compliant (secure enough to transfer hospital records) and - unlike FTPS - the SFTP protocol is a subsystem of SSH and it only requires a single open TCP port on the server side (22).
I have been playing with the Python implementation of Heartbleed on a couple of servers and got all sorts of data in response.
Amongst the data received, I have seen:
- Cookies values (SESSIONID, etc)
- Random characters that make no sense
- HTML
- ...?
I know I'm being a script kiddie here but anyway, I was wondering where this data comes from (RAM?) what apps put the data there (apache? openSSL?) and generally hoping to get a broader picture of what is going on.
Any help?
According to heartbleed.com, a vulnerable machine can leak 64kb of memory contents per heartbeat request, but an attacker can make those requests arbitrarily many times. I've seen comments to the effect that an attacker could theoretically recover enough data to fully reconstruct the RAM contents of the target machine - or, at least, however much of it Apache (or whichever program is involved in the SSL session) could see. That can include, for example, cookies, files that are being served, and from data coming in - especially usernames and passwords.
One particular problem is that any process doing SSL needs to have enough information to decrypt incoming data and sign outgoing data - that is to say, the private key. Leaking that opens you to MITM attacks and potentially (not always) retrospective decryption of intercepted data. The potential for keys to leak undetected is why the advised response is to patch OpenSSL and then rekey - patching OpenSSL protects you against future attacks, but you have no way of knowing whether your encryption keys are already compromised.
Here's an explaination by Jacob originally posted on Superuser:
In RFC 5246, which set the standards for Transport Layer Security (TLS), there is a function known as a heartbeat. The client and server send some data back and forth to keep the connection alive so that it can be used later. Now in practice the client will send some data and the server will just send it back, and everything is great. However in the affected OpenSSL versions there is no check to see if the client actually sent the amount of data that it did. So if I send it 1B and tell the server that I actually sent it 64kB then it is going to happily send me back 64kB. Where do those other Bytes come from? That's the key right there. OpenSSL is going to send you back 64kB-1B of memory that the process has access to and that you originally didn't send, depending on where your 1B is stored. These extra Bytes from memory are the problem as they can contain valuable information such as Private key material and information that the server is decrypting to use. Examples of this would be: passwords, credit card information, and/or PINs.
This is the problem I'm trying to solve,
I want to write an application that will read outbound http request packets on the same machine's network card. This would then be able to extract the GET url from it.On basis of this information, I want to be able to stop the packet, or redirect it , or let it pass.
However I want my application to be running in promiscuous mode (like wireshark does), and yet be able to eat up (stop) the outbound packet.
I have searched around a bit on this..
libpcap / pcap.h allows to me read packets at the network card, however I haven't yet been able to figure out a way to stop these packets or inject new ones into the network.
Certain stuff like twisted or scapy in python, allows me set up a server that is listening on some local port, I can then configure my browser to connect to it, using proxy configurations. This app can then do the stuff.. but my main purpose of being promiscuous is defeated here..
Any help on how I could achieve this would be greatly appreciated ..
I'd suggest that you approach this at the application layer and use a transparent proxy (e.g. squid) and iptables based interception of outbound port-80 traffic.
The reason I suggest this is that that it will avoid issues with the request being split between packets.
However, if you still want to go ahead with packet interception, you can do it in userspace using netfilters in netlink. I believe there are python wrappers for libnl around.
Essentially you create an iptables rule pointing to "QUEUE" for the traffic you want to intercept and write a program using a netlink library to process the queue, accepting, rejecting and/or modifying packets.
Using pcap you cannot stop the packets, if you are under windows you must go down to the driver level... but you can stop only packets that your machine send.
A solution is act as a pipe to the destination machine: You need two network interfaces (without address possibly), when you get a packet that you does not found interesting on the source network card you simply send it on the destination network card. If the packet is interesting you does not send it, so you act as a filter. I have done it for multimedia performance test (adding jitter, noise, etc.. to video streaming)
You are confusing several things here:
"Promiscuous" usually refers to a mode of a hardware ethernet network card where it delivers all packets in its collision domain up to the kernel network stack and have it sort out delivery (vs. just unicast to given MAC, subscribed multicast, and broadcast in normal operating mode of the card). This is on the receive path.
All the data outbound from your machine will go through (one of) the network cards on the machine, so "promiscuous" does not at all apply here.
You are working on filtering TCP-based protocol (HTTP), but talk in terms of packets. This is wrong. TCP connection is a stream that could be (as far as socket readers and writers are concerned) arbitrarily split into IP datagrams. That URL from HTTP request header could be split across multiple link-layer frames. You would have to stitch them back together and parse the stream anyway. Then you have no chance even at that if SSL is in use.
If you are interested in HTTP filtering then read HTTP RFCs, and read existing open-source code, e.g. squid, nginx, etc.
If you are digging through network stack for better understaning then read W. Richard Stevens books, look into existing code in open-source operating systems, check out BPF and netlink.
Hope this clears it a little.
I have implemented this module in Windows by using two separate NICs and using a socket/pipe(whatever you like) between them in this thread