We Keep Coding

Unable to get packet parameters with Scapy

Not sure if this is the best way, but I'm trying to read a .pcap file using Scapy and get all DHCP6 Solicit request & Advertise response parameters and the corresponding values using the following snippet:
from scapy.all import *
packets = rdpcap('DHCPv6.cap')
for packet in packets:
if packet.haslayer(DHCP6_Advertise):
print(packet.getlayer(DHCP6_Advertise))
print(packet.getlayer(DHCP6OptClientId).show())
The output shows all the fields that can be retrieved, but I noticed some fields are missing as compared to what is shown by the Wireshark output(attached image).
Is there any other way to do the same?
DHCP6_Advertise / DHCP6OptIA_PD / DHCP6OptClientId / DHCP6OptServerId
###[ DHCP6 Client Identifier Option ]###
optcode = CLIENTID
optlen = 14
\duid \
|###[ DUID - Link-layer address plus time ]###
| type = Link-layer address plus time
| hwtype = Ethernet (10Mb)
| timeval = Fri, 02 Jan 2015 21:52:08 +0000 (1420235528)
| lladdr = 08:00:27:fe:8f:95
###[ DHCP6 Server Identifier Option ]###
optcode = SERVERID
optlen = 14
\duid \
|###[ DUID - Link-layer address plus time ]###
| type = Link-layer address plus time
| hwtype = Ethernet (10Mb)
| timeval = Thu, 01 Jan 2015 15:36:08 +0000 (1420126568)
| lladdr = 08:00:27:d4:10:bb

How to fetch date and time from internet [duplicate]

I need to get the time for the UK from an NTP server. Found stuff online however any time I try out the code, I always get a return date time, the same as my computer. I changed the time on my computer to confirm this, and I always get that, so it's not coming from the NTP server.
import ntplib
from time import ctime
c = ntplib.NTPClient()
response = c.request('uk.pool.ntp.org', version=3)
response.offset
print (ctime(response.tx_time))
print (ntplib.ref_id_to_text(response.ref_id))
x = ntplib.NTPClient()
print ((x.request('ch.pool.ntp.org').tx_time))

This will work (Python 3):
import socket
import struct
import sys
import time
def RequestTimefromNtp(addr='0.de.pool.ntp.org'):
REF_TIME_1970 = 2208988800 # Reference time
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
data = b'\x1b' + 47 * b'\0'
client.sendto(data, (addr, 123))
data, address = client.recvfrom(1024)
if data:
t = struct.unpack('!12I', data)[10]
t -= REF_TIME_1970
return time.ctime(t), t
if __name__ == "__main__":
print(RequestTimefromNtp())

The timestamps returned as call to the NTP server returns time in seconds.
ctime() provides datetime format based on local machine's timezone settings by default. Thus, for uk timezone you need to convert tx_time using that timezone. Python's in-built datetime module contains function for this purpose
import ntplib
from datetime import datetime, timezone
c = ntplib.NTPClient()
# Provide the respective ntp server ip in below function
response = c.request('uk.pool.ntp.org', version=3)
response.offset
print (datetime.fromtimestamp(response.tx_time, timezone.utc))
UTC timezone used here. For working with different timezones you can use pytz library

This is basically Ahmads answer but working for me on Python 3. I am currently keen on Arrow as simplifying times and then you get:
import arrow
import socket
import struct
import sys
def RequestTimefromNtp(addr='0.de.pool.ntp.org'):
REF_TIME_1970 = 2208988800 # Reference time
client = socket.socket( socket.AF_INET, socket.SOCK_DGRAM )
data = b'\x1b' + 47 * b'\0'
client.sendto( data, (addr, 123))
data, address = client.recvfrom( 1024 )
if data:
t = struct.unpack( '!12I', data )[10]
t -= REF_TIME_1970
return arrow.get(t)
print(RequestTimefromNtp())

The following function is working well using python 3:
def GetNTPDateTime(server):
try:
ntpDate = None
client = ntplib.NTPClient()
response = client.request(server, version=3)
ntpDate = ctime(response.tx_time)
print (ntpDate)
except Exception as e:
print (e)
return datetime.datetime.strptime(ntpDate, "%a %b %d %H:%M:%S %Y")

I used ntplib server and get date and change format in dd-mm-yyyy

Take specifics date in log file and process it

I want to process a log file that contains events log, but only today logs.
The log file looks like this:
Aug 23 07:23:05 iZk1a211s8hkb4hkecu7w1Z sshd[19569]: Invalid user test from 10.148.0.13 port 48382
...
Sep 20 07:23:06 iZk1a211s8hkb4hkecu7w1Z sshd[19569]: Failed password for invalid user test from 10.148.0.13 port 48382 ssh2
...
Aug 23 07:23:07 iZk1a211s8hkb4hkecu7w1Z sshd[19564]: Failed password for invalid user sysadm from 10.148.0.13 port 48380 ssh2
...
Oct 15 07:23:09 iZk1a211s8hkb4hkecu7w1Z sshd[19573]: Invalid user sinusbot from 10.148.0.13 port 48384
...
Sep 08 07:23:11 iZk1a211s8hkb4hkecu7w1Z sshd[19573]: Failed password for invalid user sinusbot from 10.148.0.13 port 48384 ssh2
...
Nov 01 07:23:16 iZk1a211s8hkb4hkecu7w1Z sshd[19587]: Invalid user smkim from 10.148.0.13 port 48386
...
Nov 12 07:23:18 iZk1a211s8hkb4hkecu7w1Z sshd[19587]: Failed password for invalid user smkim from 10.148.0.13 port 48386 ssh2
How to grab the today line in the log?
I've tried this and got stuck in finding the patterns:
from datetime import date
today = date.today()
today = today.strftime("%B %d")
with open('file.log','r') as f:
for line in f:
date = line.find("*idk I'm stuck at this point*")
if date = today:
`*run my process script*`
Does anyone have any suggestions?

You need to extract the part of the string containing the date, parse it as datetime and convert it to a date:
from datetime import date
today: date = date.today()
with open('file.log','r') as f:
for line in f:
date: date = datetime.strptime(line[:15], "%b %d %H:%M:%S").date().replace(year=today.year)
if date == today:
`*run my process script*`

Python Error: “ValueError: need more than 1 value to unpack” in socket programming

In Python, when I run this code:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import socket
s=socket.socket()
s.connect(('www.sina.com.cn',80))
s.send(b'GET /HTTP/1.1\r\nHost: www.sina.com.cn\r\nConnection: close\r\n\r\n')
buffer=[]
while True:
d=s.recv(1024)
if d:
buffer.append(d)
else:
break
data=b''.join(buffer)
s.close()
header,html = data.split(b'\r\n\r\n',1)
print(header.decode('utf-8'))
with open('sina_test.html','wb') as f:
f.write(html)
I get this error:
line 19, in (header,html,h) = data.split(b'\r\n\r\n',1)
ValueError: need more than 1 value to unpack
What does that error mean?

The second argument to split method limits how many items the method will return
header,html = data.split(b'\r\n\r\n',1)
Here you are trying to unpack more than 1 even though you specified that split should only return 1 item

There's a SPACE before the HTTP
# wrong
s.send(b'GET /HTTP/1.1\r\nHost: www.sina.com.cn\r\nConnection: close\r\n\r\n')
# correct
s.send(b'GET / HTTP/1.1\r\nHost: www.sina.com.cn\r\nConnection: close\r\n\r\n')
The wrong usage maybe send you this information:
print(data)
<HTML>\n<HEAD>\n<TITLE>Not Found on Accelerator</TITLE>\n</HEAD>\n\n<BODY BGCOLOR="white" FGCOLOR="black">\n<H1>Not Found on Accelerator</H1>\n<HR>\n\n<FONT FACE="Helvetica,Arial"><B>\nDescription: Your request on the specified host was not found.\nCheck the location and try again.\n</B></FONT>\n<HR>\n</BODY>\n
The right information is just like this:
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 23 Jan 2018 03:28:38 GMT
Content-Type: text/html
Content-Length: 605221
Connection: close
Last-Modified: Tue, 23 Jan 2018 03:27:02 GMT
Vary: Accept-Encoding
Expires: Tue, 23 Jan 2018 03:29:37 GMT
Cache-Control: max-age=60
X-Powered-By: shci_v1.03
Age: 1
Via: http/1.1 ctc.jiangsu.ha2ts4.82 (ApacheTrafficServer/6.2.1 [cHs f ])
X-Cache: HIT.82
X-Via-CDN: f=edge,s=ctc.jiangsu.ha2ts4.83.nb.sinaedge.com,c=58.213.91.6;f=Edge,s=ctc.jiangsu.ha2ts4.82,c=61.155.142.83
X-Via-Edge: 1516678118627065bd53afa8e9b3d553f23b9
That's why ValueError occurs.

This error means that your string (data) does not contain the regex you are trying to split accordingly and therefore - data.split(b'\r\n\r\n',1) == data which cannot be assigned to header and html.

Python Log file count per Hour per IP

This script that displays how many attacks occur per hour per day. I want it to also count by IP address so it will show the IP addresses that were attacked per hour, per day.
from itertools import groupby
#open the auth.log for reading
myAuthlog=open('auth.log', 'r')
# Goes through the log file line by line and produces a list then looks for 'Failed password for'
myAuthlog = (line for line in myAuthlog if "Failed password for" in line)
# Groups all the times and dates together
for key, group in groupby(myAuthlog, key = lambda x: x[:9]):
month, day, hour = key[0:3], key[4:6], key[7:9]
# prints the results out in a format to understand e.g date, time then amount of attacks
print "On%s-%s at %s:00 There was %d attacks"%(day, month, hour, len(list(group)))
The Log File looks like This
Feb 3 13:34:05 j4-be02 sshd[676]: Failed password for root from 85.17.188.70 port 48495 ssh2
Feb 3 21:45:18 j4-be02 sshd[746]: Failed password for invalid user test from 62.45.87.113 port 50636 ssh2
Feb 4 08:39:46 j4-be02 sshd[1078]: Failed password for root from 1.234.51.243 port 60740 ssh2
A Example outcome of the code i have is:
On 3-Feb at 21:00 There was 1 attacks
On 4-Feb at 08:00 There was 15 attacks
On 4-Feb at 10:00 There was 60 attacks

from itertools import groupby
import re
myAuthlog=open('dict.txt', 'r')
myAuthlog = (line for line in myAuthlog if "Failed password for" in line)
for key, group in groupby(myAuthlog, key = lambda x: x[:9] + re.search('from(.+?) port', x).group(1)):
month, day, hour, ip = key[0:3], key[4:6], key[7:9] , key[10:]
print "On%s-%s at %s:00 There was %d attacks FROM IP %s"%(day, month, hour, len(list(group)), ip)
Log file:
Feb 3 13:34:05 j4-be02 sshd[676]: Failed password for root from 85.17.188.70 port 48495 ssh2
Feb 3 21:45:18 j4-be02 sshd[746]: Failed password for invalid user test from 62.45.87.113 port 50636 ssh2
Feb 4 08:39:46 j4-be02 sshd[1078]: Failed password for root from 1.234.51.243 port 60740 ssh2
Feb 4 08:53:46 j4-be02 sshd[1078]: Failed password for root from 1.234.51.243 port 60740 ssh2
output:
On 3-Feb at 13:00 There was 1 attacks FROM IP 85.17.188.70
On 3-Feb at 21:00 There was 1 attacks FROM IP 62.45.87.113
On 4-Feb at 08:00 There was 2 attacks FROM IP 1.234.51.243

Since you already know how to get the log lines per hour per day, use the following to count the IPs per hour per day. This is not a complete solution.
from collections import defaultdict
import re
ip_count = defaultdict(int)
with open('logfile') as data:
for line in data:
ip_count[re.findall(r'.*from (.*) port.*', line)[0]] += 1
for ip, count in ip_count.iteritems():
print ip, count