We Keep Coding

Socket Fragmented Received Data

I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1

As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.

One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.

You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()

How to check if socket connection is working and buffering or is the request incorrect?

I have been trying to figure this out and can't seem to wrap my head around it.
import socket
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server = '192.168.1.7'
port = 8000
buffer_size = 4096
request = '1|gkdgdfg49|sdfsdfsd*|sdsfsf*|'+server
request = request.encode()
s.connect((server,port))
s.send(request)
x = s.recv(buffer_size)
The documentation for the server side is:
https://www.sharekhan.com/Upload/General/TradeTigerAPIForClient.pdf
The log of api server shows:
[I] "API Client Socket Connected with error Code=0"2018-09-22 16:07:23.555+05:30
[I] "Hurraay You have Connected."2018-09-22 16:07:23.555+05:30
[I] "Received Bytes Count = 49 Data Length = 1801944113 Thread State = Background, WaitSleepJoin , Queue Count = 0"2018-09-22 16:07:23.555+05:30
[I] "API Client Request Data Length 1801944113 bytes needs to read"
2018-09-22 16:08:21.984+05:30
[I] "Received Bytes Count = 49 Data Length = 1801944113 Thread State = Background, WaitSleepJoin , Queue Count = 0"
2018-09-22 16:08:21.984+05:30
[I] "API Client Request Data Length 1801944113 bytes needs to read"|Reserved = |
Is the request correct? Is the data being transferred?
If I rerun
s.send(request)
x = s.recv(buffer_size)
Nothing happens. The interpreter is stuck at running the above two lines.
I would appreciate any help. Very new to `sockets and the documentation is not very helpful.

As I pointed out in your other question, this is a binary protocol.
You're sending it ASCII text. Note that big number (1801944113), when interpreted as text, is equal to 'kg|1'. This is (the big-endian ordering of) the first four characters you sent it. In other words, it's taken the first four bytes you sent it, interpreted them as a binary value, and is now using it as the message length value -- which is obviously all wrong.
You need to use the python struct module's pack and unpack methods to encode and decode binary values to build up your buffer.
That would look something like this:
import struct
trans_code = 1 # LOGIN
login_id = b'MyName'
mpassword = b'MyPass'
tpassword = b'MyTradingPass'
my_ip = b'192.168.1.31' # No idea what the point of this is
reserved = b''
msg_length = 196
msg_buffer = struct.pack("!IH30s20s20s20s100s", msg_length, trans_code,
login_id, mpassword, tpassword, my_ip, reserved)
assert len(msg_buffer) == msg_length
print("Login request to send '{}'".format(msg_buffer))
You will need to learn to use the struct module. It's well-specified in the python documentation.
Your results (in particular that big number) establish that the server is expecting numeric fields to be encoded in big-endian byte order (AKA "network byte order").

Python raw socket receive issue

I am using the following python script for raw socket packet transfer. Packet transfer is fine, but I am not able to print the incoming packet from the other end.
from socket import socket, AF_PACKET, SOCK_RAW
s = socket(AF_PACKET, SOCK_RAW)
s.bind(("eth0", 0))
src_addr = "\x54\xbe\xf7\x40\xf5\x82"
dst_addr = "\xff\xff\xff\xff\xff\xff"
payload = ("[("*30)+"Hello"+("]"*30)
checksum = "\x1a\x2b\x3c\x4d"
data = payload+checksum
s.send(dst_addr+src_addr+data)
#for receive function
response=s.recv(4096)
print response
s.close()

There is a third argument to the socket function: protocol. If not given, it's defaulting to 0. For AF_PACKET / SOCK_RAW, the protocol argument specifies what kind of packets you're interested in receiving. The values are documented in the packet(7) man page: http://man7.org/linux/man-pages/man7/packet.7.html
I don't think the values are actually defined anywhere in the core python2 modules. Some of them can be found in scapy (http://www.secdev.org/projects/scapy/), or you can just hunt up the linux header file where they are defined (/usr/include/linux/if_ether.h).
So, to fix this, change your code to:
from socket import socket, AF_PACKET, SOCK_RAW, htons
ETH_P_ALL = 3
ETH_P_IP = 0x800 # Alternatively using this will receive the next IP packet
s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))
...
Other comments on your code:
As written, the packet you're sending is unlikely to be comprehensible by anyone. You've got a dst and src MAC address, but then you're not providing an EtherType. Instead the first "[(" will be seen as the EtherType. That probably won't make sense to any receiver of the packet so it will just be discarded.
Also, you should understand that with a raw socket, you're going to receive the next packet of the type you've specified in the protocol. That isn't necessarily (and in fact probably won't be) a response to the packet you just sent.

Thanks everyone now I am able receive the packet with the following the script. But still I am facing issue with printing multiple response packets(While doing in loop).
from socket import socket, AF_PACKET, SOCK_RAW, htons
from struct import *
import select
import time
ETH_P_ALL = 3
ETH_P_IP = 0x800
s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))
s.bind(("eth0", 0))
src_addr = "\x54\xbe\xf7\x40\xf7\x82"
dst_addr = "\xff\xff\xff\xff\xff\xff"
l = "\x00\x21"
ethertype = "\x08\x01"
a ="\x00\x10\x00\x10\x00\x10"
b = "\x00\x11\x00\x11\x00\x11"
payload = ethertype + l + a + b
for i in range(5):
time.sleep(5)
s.send(dst_addr+src_addr+ payload)
message=s.recv(4096)
print message

when i made a 3 handshake with ubuntu in VMware return package R

#!/usr/bin/python
from scapy.all import *
def findWeb():
a = sr1(IP(dst="8.8.8.8")/UDP()/DNS(qd=DNSQR(qname="www.google.com")),verbose=0)
return a[DNSRR].rdata
def sendPacket(dst,src):
ip = IP(dst = dst)
SYN = TCP(sport=1500, dport=80, flags='S')
SYNACK = sr1(ip/SYN)
my_ack = SYNACK.seq + 1
ACK = TCP(sport=1050, dport=80, flags='A', ack=my_ack)
send(ip/ACK)
payload = "stuff"
PUSH = TCP(sport=1050, dport=80, flags='PA', seq=11, ack=my_ack)
send(ip/PUSH/payload)
http = sr1(ip/TCP()/'GET /index.html HTTP/1.0 \n\n',verbose=0)
print http.show()
src = '10.0.0.24'
dst = findWeb()
sendPacket(dst,src)
I'm trying to do HTTP packets with SCAPY
I am using UBUNTU on VMwaer
The problem is that every time I send messages I have RESET
How do we fix it?
Thanks
sniff package image

Few things I notice wrong.
1. You have your sequence number set statically (seq=11) which is wrong. Sequence numbers are always randomly generated and they must be used as per RFC793. So the sequence should be = SYNACK[TCP].ack
You set your source port as 1500 during SYN packet, but then you use it as 1050 (typo?)
You don't need extra payload/PUSH.
Also, have a look at these threads:
How to create HTTP GET request Scapy?
Python-Scapy or the like-How can I create an HTTP GET request at the packet level

How to create HTTP GET request Scapy?

I need to create HTTP GET request and save the data response.
I tried to use this:
syn = IP(dst=URL) / TCP(dport=80, flags='S')
syn_ack = sr1(syn)
getStr = 'GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n'
request = IP(dst='www.google.com') / TCP(dport=80, sport=syn_ack[TCP].dport,
seq=syn_ack[TCP].ack, ack=syn_ack[TCP].seq + 1, flags='A') / getStr
reply = sr1(request)
print reply.show()
But when I print reply I don't see any data response.
In addition, when I checked in 'Wireshark' I got SYN, SYN/ACK but I didn't get an ACK.
Image:
Edit:
I try to do that now:
# Import scapy
from scapy.all import *
# Print info header
print "[*] ACK-GET example -- Thijs 'Thice' Bosschert, 06-06-2011"
# Prepare GET statement
get='GET / HTTP/1.0\n\n'
# Set up target IP
ip=IP(dst="www.google.com")
# Generate random source port number
port=RandNum(1024,65535)
# Create SYN packet
SYN=ip/TCP(sport=port, dport=80, flags="S", seq=42)
# Send SYN and receive SYN,ACK
print "\n[*] Sending SYN packet"
SYNACK=sr1(SYN)
# Create ACK with GET request
ACK=ip/TCP(sport=SYNACK.dport, dport=80, flags="A", seq=SYNACK.ack, ack=SYNACK.seq + 1) / get
# SEND our ACK-GET request
print "\n[*] Sending ACK-GET packet"
reply,error=sr(ACK)
# print reply from server
print "\n[*] Reply from server:"
print reply.show()
print '\n[*] Done!'
but its print me in reply from server;
0000 IP / TCP 192.168.44.130:23181 > 216.58.208.164:http A / Raw ==>
IP / TCP 216.58.208.164:http > 192.168.44.130:23181 A / Padding None
And I need Line-based text data: text/html.

You are sending a RST segment in response to the SYN-ACK because your kernel has no knowledge of the SYN you sent via Scapy (see here). This could be solved with an iptable rule:
iptables -A OUTPUT -p tcp --tcp-flags RST RST -s <your ip> -j DROP
Because you are ending the connection with that RST segment, when you send your HTTP request, the endpoint answers with a RST too because connection is not established and so you are using show() on a RST segment with no data, that is why you do not see anything.

You are sending a SYN and correctly receiving a SYN_ACK. At this point, you should generate and send an ACK based on the SYN_ACK that you've received, and THEN finally transmit the HTTP GET request. It seems that you are somewhat confused about the TCP 3-way handshake mechanism. In short, you are not supposed to 'get' an ACK, you are supposed to generate and send this yourself.

After setting the rule in your iptables as has been suggested above, you could do the following :
from scapy.all import *
seq = 12345
sport = 1040
dport = 80
ip_packet = IP(dst='192.168.56.107')
syn_packet = TCP(sport=sport, dport=dport, flags='S', seq=seq)
packet = ip_packet/syn_packet
synack_response = sr1(packet)
next_seq = seq + 1
my_ack = synack_response.seq + 1
ack_packet = TCP(sport=sport, dport=dport, flags='A', seq=next_seq, ack=my_ack)
send(ip_packet/ack_packet)
payload_packet = TCP(sport=sport, dport=dport, flags='A', seq=next_seq, ack=my_ack)
payload = "GET / HTTP/1.0\r\nHOST: 192.168.56.107\r\n\r\n"
reply, error = sr(ip_packet/payload_packet/payload, multi=1, timeout=1)
for r in reply:
r[0].show2()
r[1].show2()
Hope this helps. Basically, the first response you get back does not really hold the HTTP response data. I tested the script against an INETSIM simulated HTTP server and in that case (at least) the first packet (after the 3-way TCP handshake) that the server responded with was a series of NULL (0x00) bytes. Hence using multi somehow did the stuff in my case.