got a serial reading problem that drives me nuts. Current setup is an Arduino connected to a RaspberryPi (standard USB connection). Trying to read from serial info that Arduino sends via serial to Raspberry on /dev/ttyACM0
When trying a pretty basic monitoring, raw read:
$ (stty raw; cat) < /dev/ttyACM0
RFBee ready to receive!
: 03 Packet: 80 0F EA 1D D9 00 F3 15 FF FF RSSI: -60 LQI: 30 Error: 7 Next: 64 Packets: 4
RFBee ready to receive!
Index: 00 Packet: E0 0E F2 C6 01 02 B1 91 FF FF RSSI: -60 LQI: 2F Error: 7 Next: 64 Packets: 1
Index: 01 Packet: 50 0D E7 FF 73 04 56 C4 FF FF RSSI: -60 LQI: 2F Error: 7 Next: 64 Packets: 2
Index: 02 Packet: 80 0D F6 1D D8 0C 11 BE FF FF RSSI: -61 LQI: 2F Error: 7 Next: 64 Packets: 3
Index: 03 Packet: 20 0D E7 D4 C1 86 AC CA FF FF RSSI: -61 LQI: 2E Error: 7 Next: 64 Packets: 4
Index: 04 Packet: E0 0D EF C6 03 0C FA CE FF FF RSSI: -61 LQI: 2F Error: 7 Next: 64 Packets: 5
Index: 00 Packet: 50 0B DF FF 71 06 74 4B FF FF RSSI: -60 LQI: 2E Error: 0 Next: 64 Packets: 6
Index: 01 Packet: 80 0B EA 1D D8 06 29 E4 FF FF RSSI: -60 LQI: 2E Error: 0 Next: 64 Packets: 7
Index: 02 Packet: 70 0A F8 AC 03 8C D4 16 FF FF RSSI: -61 LQI: 2E Error: 0 Next: 64 Packets: 8
It all seems to be working more or less fine (except for the double RFBee ready to receive line), but then catches on and prints data line by line.
Minicom works the same way, reads data, all fine.
With PySerial, using this code:
#!/usr/bin/python -u
import serial
while True:
with serial.Serial('/dev/ttyACM0', 115200) as ser:
line = ser.readline()
if line != '':
print(line)
It just prints out these lines:
$ python davis_collect.py
B DF CA FF FF RSSI: -60 LQI: 2E Error: 0 Next: 64 Packets: 9
RFBee ready to receive!
RFBee ready to receive!
RFBee ready to receive!
RFBee ready to receive!
RFBee ready to receive!
So it catches whatever part of the buffer that it should at first, but then, for some reason, cannot stop printing that beautiful line.
To add on top, some basic information: Tried this on my notebook, PySerial worked. The serial communication appareently works via RasPi as well, as it is readable via standard utilities.
Python3 output:
python3 davis_collect.py
b'RFBee ready to receive!\r\n'
b'RFBee ready to receive!\r\n'
b'RFBee ready to receive!\r\n'
b'RFBee ready to receive!\r\n'
b'RFBee ready to receive!\r\n'
Didn't find anything similar so far. Any ideas, links? (First post here, if I need to add something, please, you, the more experienced guys, leave suggestions)
I am officially a moron, as gre_gor (thanks for the kick in the head) noted above, the loop was called and connected each time to the serial port anew.
This does not pose a problem on most of the machines that you normally come across, but because of default settings on the RasPi - buffer handling - the first line was read in a loop after each connect.
It is enough to say, that using (note the endless loop AFTER I connect):
#!/usr/bin/python -u
import serial
with serial.Serial('/dev/ttyACM0', 115200) as ser:
while True:
line = ser.readline()
if line != '':
print(line)
instead, produced expected results. And I jsut add: Doh! - oviously!
Related
I am trying to extract a specific byte from each packet in a pcap file. All packets are ICMP.
In the data section, there is a byte that changes each packet. It is in the same position for each one. I would like to extract that byte.
Using scapy:
pkts = rdpcap('test.pcap')
pl = PacketList([p for p in pkts])
bytes(pl[12].payload)
returns the following:
b'E\x00\x00T]\xa7\x00\x00***J***\x01!A\xc0\xa88\x01\xc0\xa88o\x08\x004\xe9\xbf2\x00\x00^"\x87\xbe\x00\x0c2\xf4\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./01234567'
I have enclosed the byte that I want to extract within three stars. However, if I print out the bytes for each packet, the byte that I want to extract will be in a different offset.
If I run a hexdump for each packet as follows:
hexdump(bytes(pl[12].payload))
The specific byte I want to extract is always in the same position, but I don't know how to extract it.
How can I extract a specific byte from a pcap using scapy?
Following this answer here: Get specific bytes in payload from a pcap file
If I execute the same command, it doesn't do anything useful:
>>> hexdump(pkts[14][2].load[8])
0000 00 00 00 00 00 00 00 00 ........
>>>
You want the TTL?
Lets start at the high level, and move down.
Scapy is giving you the constructed packet. If you want the TTL of the packet, call the attribute:
>>> plist[182].ttl
64
If you want to get the specific byte of the packet, lets look at the hexdump:
>>> hexdump(plist[182])
0000 AA BB CC 66 42 DE AA BB CC 3F 52 A3 08 00 45 00 .a.lM..M.AK...E.
0010 00 5B 58 B9 40 00 40 06 64 96 C0 A8 01 28 AC D9 .[X.#.#.d....(..
...
These are in hex, the first field 0000 is the offset, then 16 bytes in hex, then ascii.
Offset Bytes ASCII
====== =============================================== ================
0000 AA BB CC 66 42 DE AA BB CC 3F 52 A3 08 00 45 00 .a.lM..M.AK...E.
Things start at 0, so the the byte addresses are 0..15 for the first line.
The second line the offset is 16 (16 * 1). So the byte addresses are 16..31.
The third line, the offset is 32 (16 * 2). So the byte addresses are 32..47
You highlighted the 7th byte on row 2:
Offset Bytes ASCII
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
====== =============================================== ================
0010 00 5B 58 B9 40 00 40 06 64 96 C0 A8 01 28 AC D9 .[X.#.#.d....(..
That address is:
offset + byte_address.
offset = 16 * 1
byte_address = 6
Which gives us:
16 + 6 = 22
With that, we can now get byte address 22 out of the raw packet:
>>> b = raw(plist[182])
>>> b[22]
64
Note that wireshark packet numbers start at 1. The packets in python are going to start at 0. So in my example, packet 182 corresponded to packet 183 in Wireshark.
plist[182].payload gives you the IP portion of the packet, so the offsets are going to be different since we aren't looking at the whole packet anymore. We can get the same value using the '.ttl' attribute. Or, knowing that the address is byte 8 in the IP header:
>>> plist[182].payload.ttl
64
>>> raw(plist[182].payload)[8]
64
Good day,
I am not sure where I should place this question , I am learning about DNS and how it works and as I understand I send a request out on UDP port 53 to a server and the host should respond to me on that port correct?
Here is a script that I am working with and it works and accurately describes the DNS message query and usage and even gets a DNS answer back for me.
How is this possible if it cannot listen on port 53 with out having root on a system?
DNS PACKET DETAILS
;DNS HEADER;
; AA AA - ID
; 01 00 - Query parameters
; 00 01 - Number of questions
; 00 00 - Number of answers
; 00 00 - Number of authority records
; 00 00 - Number of additional records
; DNS QUESTION --
; 07 - 'example' has length 7, ;so change this to be the length of domain ; keep in ming there are not '.' in the question.
; 65 - e
; 78 - x
; 61 - a
; 6D - m
; 70 - p
; 6C - l
; 65 - e
; 03 - subdomain '.com' length 03 ; change this to be the length of type.
; 63 - c
; 6F - o
; 6D - m
CODE :
import binascii
import socket
def send_udp_message(message, address, port):
"""send_udp_message sends a message to UDP server
message should be a hexadecimal encoded string
"""
message = message.replace(" ", "").replace("\n", "")
server_address = (address, port)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
sock.sendto(binascii.unhexlify(message), server_address)
data, _ = sock.recvfrom(4096)
finally:
sock.close()
return binascii.hexlify(data).decode("utf-8")
def format_hex(hex):
"""format_hex returns a pretty version of a hex string"""
octets = [hex[i:i+2] for i in range(0, len(hex), 2)]
pairs = [" ".join(octets[i:i+2]) for i in range(0, len(octets), 2)]
return "\n".join(pairs)
message = "AA AA 01 00 00 01 00 00 00 00 00 00 " \
"07 65 78 61 6d 70 6c 65 03 63 6f 6d 00 00 01 00 01"
response = send_udp_message(message, "8.8.8.8", 53)
print(format_hex(response))
RESPONSE:
aa aa
81 80
00 01
00 01
00 00
00 00
07 65
78 61
6d 70
6c 65
03 63
6f 6d
00 00
01 00
01 c0
0c 00
01 00
01 00
00 32
98 00
04 5d
b8 d8
22
If you look at the last four bytes you'll see that this is the IP for example.com in hex 5db8d822
You can go here to check it out.
HEX to IP converter Online
No, your source port is not port 53. User processes are allocated outbound port numbers above 1023, which are unprivileged.
A simple synchronous Python DNS client will basically block and hold the same port open until the server responds. The IP packet you send contains the information that the server needs in order to know where to reply (this is in the headers of the IP packet itself, before the DNS query payload).
I am trying to follow Telegram's authorization sample but do not get any response from the server:
msg = """
0000 | 00 00 00 00 00 00 00 00 4A 96 70 27 C4 7A E5 51
0010 | 14 00 00 00 78 97 46 60 3E 05 49 82 8C CA 27 E9
0020 | 66 B3 01 A4 8F EC E2 FC
"""
import re
msg = re.sub('\d{4} \| ', '', msg)
msg = re.sub('\s+', '', msg)
msg = bytes.fromhex(msg)
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('149.154.167.40', 443))
s.send(msg)
print(s.recv(1024)) # prints: b''
s.close()
Why?
Yes, #habnabit, it was the TCP headers that were missing. I thought they were handled by Python since I haven't seen that part in other clients. Found it in Telethon, thanks for the tip.
import struct
import zlib
msg = struct.pack('<ii', len(msg) + 12, counter) + msg
msg += struct.pack('<I', zlib.crc32(msg))
I have the following contents from data.log file. I wish to extract the ts value and part of the payload (after deadbeef in the payload, third row, starting second to last byte. Please refer to expected output).
data.log
print 1: file offset 0x0
ts=0x584819041ff529e0 2016-12-07 14:13:24.124834649 UTC
type: ERF Ethernet
dserror=0 rxerror=0 trunc=0 vlen=0 iface=1 rlen=96 lctr=0 wlen=68
pad=0x00 offset=0x00
dst=aa:bb:cc:dd:ee:ff src=ca:fe:ba:be:ca:fe
etype=0x0800
45 00 00 32 00 00 40 00 40 11 50 ff c0 a8 34 35 E..2..#.#.P...45
c0 a8 34 36 80 01 00 00 00 1e 00 00 08 08 08 08 ..46............
08 08 50 e6 61 c3 85 21 01 00 de ad be ef 85 d7 ..P.a..!........
91 21 6f 9a 32 94 fd 07 01 00 de ad be ef 85 d7 .!o.2...........
print 2: file offset 0x60
ts=0x584819041ff52b00 2016-12-07 14:13:24.124834716 UTC
type: ERF Ethernet
dserror=0 rxerror=0 trunc=0 vlen=0 iface=1 rlen=96 lctr=0 wlen=68
pad=0x00 offset=0x00
dst=aa:bb:cc:dd:ee:ff src=ca:fe:ba:be:ca:fe
etype=0x0800
45 00 00 32 00 00 40 00 40 11 50 ff c0 a8 34 35 E..2..#.#.P...45
c0 a8 34 36 80 01 00 00 00 1e 00 00 08 08 08 08 ..46............
08 08 68 e7 61 c3 85 21 01 00 de ad be ef 86 d7 ..h.a..!........
91 21 c5 34 77 bd fd 07 01 00 de ad be ef 86 d7 .!.4w...........
print 3806: file offset 0x592e0
ts=0x584819042006b840 2016-12-07 14:13:24.125102535 UTC
type: ERF Ethernet
dserror=0 rxerror=0 trunc=0 vlen=0 iface=1 rlen=96 lctr=0 wlen=68
pad=0x00 offset=0x00
dst=aa:bb:cc:dd:ee:ff src=ca:fe:ba:be:ca:fe
etype=0x0800
45 00 00 32 00 00 40 00 40 11 50 ff c0 a8 34 35 E..2..#.#.P...45
c0 a8 34 36 80 01 00 00 00 1e 00 00 08 08 08 08 ..46............
08 08 50 74 73 c3 85 21 01 00 de ad be ef 62 e6 ..Pts..!......b.
91 21 ed 4a 8c df fd 07 01 00 de ad be ef 62 e6 .!.J..........b.
My expected output
0x584819041ff529e0,85d79121
0x584819041ff52b00,86d79121
0x584819042006b840,62e69121
What I have tried so far
I am able to extract the ts value. I used
awk -v ORS="" '$NF == "UTC"{print sep$1; sep=","} END{print "\n"}' data.log
>> ts=0x584819041ff529e0,ts=0x584819041ff52b00
But didn't succeed in extracting the payload contents.
Any help is much appreciated.
Here's one way to get it done:
awk -F '=| ' '/^ts=/{printf $2","} /de ad be ef/{if(!a){printf $15$16;a=1}else{print $1$2;a=0}}' data.log
Output:
0x584819041ff529e0,85d79121
0x584819041ff52b00,86d79121
Explanation:
-F '=| ' : set the field seperator to both '=' and 'space'
/^ts=/{printf $2","} : if pattern 'ts=' found at line beginning, print the second field
/de ad be ef/{something} : if pattern 'de ad be ef' found, do 'something'
Initially variable a will be equal to 0. if pattern de ad be ef is found for the first time, if(!a) would succeed and hence print the 15th and 16th fields. Now set a to 1. So when de ad be ef pattern is matched in the next line, if(!a) check would fail and hence print the 1st and 2nd fields. Now, reset a to 0 and continue the same process for the rest of the file.
If you want sed:
sed -n -e '/^ts/ {s/^ts=\([^ ]*\) \(.*\)/\1/; H;};' \
-e '/de ad be ef/ {N; s/\(.*\)de ad be ef \([0-9a-f]\+\) \([0-9a-f]\+\) \(.*\) \([0-9a-f]\+\) \([0-9a-f]\+\) \(.*\)/,\2\3\5\6/; H;};' \
-e '$ {x; s/\n,/,/g p;}' file
If you are interested in further infos, just ask.
awk variant using deadbeef as switch
awk -F '[= ]' '/^ts/{s=$2",";a=15} /de ad be ef/{s=s $a $(a+1);if(a==1)print s;a=1}' data.log
and a sed variant
sed -n -e '/^ts=/{h;b^J}' -e "/de ad be ef/,//{H;g;s/ts=\([^ ]*\).*\n*de ad be ef \(..\) \(..\).*\n\(..\) \(..\).*/\1,\2\3\4\4/p;}" data.log
info: "^J" is a CTRL+J (new line carractere) in posix version and a ";" in GNU version
With GNU awk for gensub():
$ awk -v RS= '{
gsub(/( |\t)+[^\n]*(\n|$)/," ")
print gensub(/.*\nts=(\S+).*de ad be ef (..) (..) (..) (..).*/,"\\1,\\2\\3\\4\\5\\6",1)
}' data.log
0x584819041ff529e0,85d79121
0x584819041ff52b00,86d79121
0x584819042006b840,62e69121
The above will work even if deadbeef is split across lines.
I am trying to write a simple program in python to use telegram api, (not bot api, main messaging api) Now i have written this code
#!/usr/bin/env python
import socket
import random
import time
import struct
import requests
def swap64(i):
return struct.unpack("<L", struct.pack(">L", i))[0]
MESSAGE = '0000000000000000'+format(swap32(int(time.time()*1000%1000)<<21|random.randint(0,1048575)<<3|4),'x')+format(swap32(int(time.time())),'x')+'140000007897466068edeaecd1372139bbb0394b6fd775d3'
res = requests.post(url='http://149.154.167.40',
data=bytes.fromhex(MESSAGE),
headers={'connection': 'keep-alive'})
print("received data:", res)
For payload of post data i used the source code of telegram web, The 0 auth id, message id is generated using the algo in telegram web, next is length (14000000) just like in the source and main doc and then the method and so on,
When i run this code i get received data: <Response [404]> i have used both tcp and http transport with this and tcp one gives me nothing as answer from server, i don't know where i'm wrong in my code
i would be glad if someone can show the error in my code
btw here is hex dump of my generated req:
0000 34 08 04 17 7a ec 48 5d 60 84 ba ed 08 00 45 00
0010 00 50 c6 07 40 00 40 06 76 28 c0 a8 01 0d 95 9a
0020 a7 28 c9 62 00 50 0d 1a 3b df 41 5a 40 7f 50 18
0030 72 10 ca 39 00 00 00 00 00 00 00 00 00 00 6c 28
0040 22 4a 94 a9 c9 56 14 00 00 00 78 97 46 60 68 ed
0050 ea ec d1 37 21 39 bb b0 39 4b 6f d7 75 d3
i have already read this and this and many other docs but cant find out my problem
thanks in advance
update
i used this code as suggested
TCP_IP = '149.154.167.40'
TCP_PORT = 80
MESSAGE = 'ef0000000000000000'+"{0:0{1}x}".format(int(time.time()*4294.967296*1000),16)+'140000007897466068edeaecd1372139bbb0394b6fd775d3'
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(bytes.fromhex(MESSAGE))
data = s.recv(BUFFER_SIZE)
s.close()
and i still get no response
hex dump of my request:
0000 34 08 04 17 7a ec 48 5d 60 84 ba ed 08 00 45 00
0010 00 51 e1 44 40 00 40 06 5a ea c0 a8 01 0d 95 9a
0020 a7 28 df 8c 00 50 e4 0d 12 46 e2 98 bf a3 50 18
0030 72 10 af 66 00 00 ef 00 00 00 00 00 00 00 00 00
0040 16 37 dc e1 28 39 23 14 00 00 00 78 97 46 60 68
0050 ed ea ec d1 37 21 39 bb b0 39 4b 6f d7 75 d3
Fixed code
Finally got it working with this code
import socket
import random
import time
import struct
import requests
def swap32(i):
return struct.unpack("<L", struct.pack(">L", i))[0]
TCP_IP = '149.154.167.40'
TCP_PORT = 80
z = int(time.time()*4294.967296*1000000)
z = format(z,'x')
q = bytearray.fromhex(z)
e = q[::-1].hex()
MESSAGE = 'ef0a0000000000000000'+e+'140000007897466068edeaecd1372139bbb0394b6fd775d3'
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(bytes.fromhex(MESSAGE))
data = s.recv(BUFFER_SIZE)
s.close()
print(data)
here is sample data from a simple TCP handshake with Telegram Servers:
Connect:Success:0
Connected to 149.154.167.40:443
raw_data: 000000000000000000F011DB3B2AA9561400000078974660A9729A4F5B51F18F7943F9C0D61B1315
auth_key_id: 0000000000000000 0
message_id: 56A92A3BDB11F000 6244568794892726272
data_length: 00000014 20
message_data: 78974660A9729A4F5B51F18F7943F9C0D61B1315
message_type: 60469778
>> EF0A000000000000000000F011DB3B2AA9561400000078974660A9729A4F5B51F18F7943F9C0D61B1315
Send:Success:42
Receive:Success:85
<< 15000000000000000001CC0CC93D2AA9564000000063241605A9729A4F5B51F18F7943F9C0D61B1315B4445B94718B3C6DD4136466FAC62DCD082311272BE9FF8F9700000015C4B51C01000000216BE86C022BB4C3
raw_data: 000000000000000001CC0CC93D2AA9564000000063241605A9729A4F5B51F18F7943F9C0D61B1315B4445B94718B3C6DD4136466FAC62DCD082311272BE9FF8F9700000015C4B51C01000000216BE86C022BB4C3
auth_key_id: 0000000000000000 0
message_id: 56A92A3DC90CCC01 6244568803180334081
data_length: 00000040 64
message_data: 63241605A9729A4F5B51F18F7943F9C0D61B1315B4445B94718B3C6DD4136466FAC62DCD082311272BE9FF8F9700000015C4B51C01000000216BE86C022BB4C3
message_type: 05162463
classid: resPQ#05162463
nonce: A9729A4F5B51F18F7943F9C0D61B1315
server_nonce: B4445B94718B3C6DD4136466FAC62DCD
pq: 2311272BE9FF8F97 2526843935494475671
count: 00000001 1
fingerprints: C3B42B026CE86B21 14101943622620965665
Lets break it down:
We are using the TCP abridged version, so we start off with 0xEF
The format for plain-text Telegram messages is auth_ke_id + msg_id + msg_len + msg
auth_key_id is always 0 for plain-text messages hence we always start with 0000000000000000
msg_id must approximately equal unixtime*2^32(see here) I have also seen that some variant of this works quite well for msg_id in any language on any platform: whole_part_of(current_micro_second_time_stamp * 4294.967296)
The first message you start with for Auth_key generation is reqPQ which is defined as: reqPQ#0x60469778 {:nonce, :int128} so it is simply a TL-header + a 128-bit random integer the total length will always be 4 + 16 = 20 encoded as little-endian that would be msg_len = 14000000
say we have a 128-bit random integer= 55555555555555555555555555555555, then our reqPQ message would be 7897466055555555555555555555555555555555, which is simply TL-type 60469778 or 78974660 in little-endian followed by your randomly chooses 128-bit nonce.
Before you send out the packet, again recall that TCP-abridged mode required you to include the total packet length in front of the other bytes just after the initial 0xEA . This packet length is computed as follows
let len = total_length / 4
a) If len < 127 then len_header = len as byte
b) If len >=127 then len_header = 0x7f + to_3_byte_little_endian(len)
finally we have:
EF0A000000000000000000F011DB3B2AA956140000007897466055555555555555555555555555555555
or
EF0A
0000000000000000
00F011DB3B2AA956
14000000
78974660
55555555555555555555555555555555
compared to yours:
0000000000000000
6C28224A94A9C956
14000000
78974660
68EDEAECD1372139BBB0394B6FD775D3
I would say, try using TCP-abriged mode by include the 0xEF starting bit and re-check your msg_id computation
cheers.