I have this HTTP Request and I want to display only the Authorization section (base64 Value) : any help ?
This Request is stored on a variable called hreq
I have tried this :
reg = re.search(r"Authorization:\sBasic\s(.*)\r", hreq)
print reg.group()
but doesn't work
Here is the request :
HTTP Request:
Path: /dynaform/custom.js
Http-Version: HTTP/1.1
Host: 192.168.1.254
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://domain.com/userRpm/StatusRpm.htm
Authorization: Basic YWhtEWa6MDfGcmVlc3R6bGH
I want to display the value YWhtEWa6MDfGcmVlc3R6bGH
Please I need your help
thanks in advance experts
You can get rid of the \r at the end of the regex, in Linux it is a \n and it might break your script since you were expecting \r instead of \n:
>>> reg = re.search(r"Authorization:\sBasic\s(.*)", a)
>>> reg.groups()
('YWhtEWa6MDfGcmVlc3R6bGH ',)
The \r is probably throwing things off; there probably isn't a carriage return at the end of the request (but it's hard to say from this end). Try removing it or using $ (end-of-input) instead.
You can use this online Python regex tester to try your inputs by hand before putting them in your code.
If that is your whole input text maybe /(\b[ ].*)$/ could help
Online Demo
[ ] match a space character present in the text. followed by
.* any character (except newline) followed by
$ the end of the string
Thanks for your answers :), actually I got an error msg
but anyway I'm gonna show what exactly I wanna do,
here is the script :
!/usr/bin/python
from scapy.all import *
from scapy.error import Scapy_Exception
from scapy import HTTP
my_iface="wlan0"
count=0
def pktTCP(pkt):
global count
count=count+1
if HTTP.HTTPRequest or HTTP.HTTPResponse in pkt:
src=pkt[IP].src
srcport=pkt[IP].sport
dst=pkt[IP].dst
dstport=pkt[IP].dport
test=pkt[TCP].payload
if HTTP.HTTPRequest in pkt:
print "HTTP Request:"
print test
print "======================================================================"
if HTTP.HTTPResponse in pkt:
print "HTTP Response:"
print test
print "======================================================================"
sniff(filter='tcp and port 80',iface=my_iface,prn=pktTCP)
Related
What I want:
I want my Raspberry Pi to act as a NFC-reader that can trigger the URL record from a NFC tag.
My setup is a Raspberry Pi with a PN532 NFC HAT and nfcpy. I am using the example tagtool.py and right now I am able to scan the NFC-tag and then show the URL (+ some extra data)
But I want the system to run the URL which triggers a webhook on IFTTT (which then triggers a playlist on Spotify...)
What I have done so far:
I have used setup.py to install nfcpy and experimented a bit with the commands. But when I run the command
python3 tagtool.py --device tty:S0:pn532 -d nfc.ndef.UriRecord -l
It first returns this
[main] enable debug output for 'nfc.ndef.UriRecord'
[nfc.clf] searching for reader on path tty:S0:pn532
[nfc.clf] using PN532v1.6 at /dev/ttyS0
** waiting for a tag **
and then when I scan one of my NFC tags - which have a URL in URI Record - with the reader I get this message.
Type2Tag 'NXP NTAG213' ID=04EA530A3E4D80
NDEF Capabilities:
readable = yes
writeable = yes
capacity = 137 byte
message = 67 byte
NDEF Message:
record 1
type = 'urn:nfc:wkt:U'
name = ''
data = b'\x04maker.ifttt.com/trigger/Playlist_022/with/key/bVTin_XXEEEDDDDEEEEEE'
[main] *** RESTART ***
[nfc.clf] searching for reader on path tty:S0:pn532
[nfc.clf] using PN532v1.6 at /dev/ttyS0
** waiting for a tag **
As you can see the URL is right there under data (+b\x04 but without https:\ but I guess thats quite easy to change). So basically I just need to trigger it.
I read somewhere that I could use curlify so I have used the command 'pip3 install curlify' and made some changes to tagtool.py.
The original tagtool.py (which I believe is the most important part for what I am trying to do) looks like this
if tag.ndef:
print("NDEF Capabilities:")
print(" readable = %s" % ("no", "yes")[tag.ndef.is_readable])
print(" writeable = %s" % ("no", "yes")[tag.ndef.is_writeable])
print(" capacity = %d byte" % tag.ndef.capacity)
print(" message = %d byte" % tag.ndef.length)
if tag.ndef.length > 0:
print("NDEF Message:")
for i, record in enumerate(tag.ndef.records):
print("record", i + 1)
print(" type =", repr(record.type))
print(" name =", repr(record.name))
print(" data =", repr(record.data))
In the new tagtool2.py I have added this to the start of the document
import curlify
import requests
And then I have added this line
response = requests.get("https://repr(record.data)")
print(curlify.to_curl(response.request))
Which means it looks like this. And this is probably wrong in several ways:
if tag.ndef:
print("NDEF Capabilities:")
print(" readable = %s" % ("no", "yes")[tag.ndef.is_readable])
print(" writeable = %s" % ("no", "yes")[tag.ndef.is_writeable])
print(" capacity = %d byte" % tag.ndef.capacity)
print(" message = %d byte" % tag.ndef.length)
if tag.ndef.length > 0:
print("NDEF Message:")
for i, record in enumerate(tag.ndef.records):
print("record", i + 1)
print(" type =", repr(record.type))
print(" name =", repr(record.name))
print(" data =", repr(record.data))
response = requests.get("https://repr(record.data)")
print(curlify.to_curl(response.request))
Because when I try to trigger the URL with a NFC tag I get this message:
Type2Tag 'NXP NTAG213' ID=04EA530A3E4D80
NDEF Message:
record 1
type = 'urn:nfc:wkt:U'
name = ''
data = b'\x04maker.ifttt.com/trigger/Metal1/with/key/bVTin_XXEEEDDDDEEEEEE'
[urllib3.connectionpool] Starting new HTTPS connection (1): repr(record.data):443
[nfc.clf] HTTPSConnectionPool(host='repr(record.data)', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0xb579a650>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Can anyone tell me what I am doing wrong? And if curlify is the right way to go?
What you are doing wrong is that the data stored in the NDEF message is encoded, you cannot just open a connection using the raw data you have to decode it first using the correct type. (as the encoded value has a Hex number in it)
It is also encoded in utf-8 so Python treats it as bytes not as a string type object.
So the type says it is a URI record type as you used 'nfc.ndef.UriRecord' (Don't know why call it urn instead)
So the Hex number \x04 means https://
Unfortunately I don't think anybody has written a decoder method for the NFC Uri specification only encoders.
Here is a link full spec for the NDEF URI record type
once you have replaced the Hex character in the data with the correct decoded value you will get the URL https://maker.ifttt.com/trigger/Metal1/with/key/bVTin_XXEEEDDDDEEEEEE
a simple example (where a stores the value instead of record.data)
import re
a = b'\x04maker.ifttt.com/trigger/Metal1/with/key/bVTin_XXEEEDDDDEEEEEE'
a_text = a.decode('utf-8')
x = re.sub('\x04', 'https://', a_text)
print(x)
requests.get(x)
Then you can use requests.get() on the decoded value
This turned out to be the simple but good solution for my needs.
if tag.ndef:
if tag.ndef.length > 0:
#print("NDEF Message:")
for i, record in enumerate(tag.ndef.records):
print(record.uri)
response = requests.get(record.uri)
print(curlify.to_curl(response.request))
I started out with a way more complicated solution. I am keeping it here if anybody runs into similar problems.
Since the first take, I have cleaned out some of the print() lines as well, and I know I can clean out some more lines, but I am keeping them here to make it easier to see whats happening.
Its especially worth noticing the y variable. I was left with an almost perfect URL, but I kept getting errors because of an extra ' at the end of the URL.
if tag.ndef:
if tag.ndef.length > 0:
for i, record in enumerate(tag.ndef.records):
print(repr(record.data))
print(str(record.data))
org_string = str(record.data)
mod_string = org_string[6:]
y = mod_string.rstrip(mod_string[-1])
w = "https://"
print(mod_string)
print(y)
print(w)
response = requests.get(w + y)
print(curlify.to_curl(response.request))
The code can be improved, but it works and it gives me this message and - more important - it triggers the URL on the NFC tag (I have scrambled the IFTTT webhook).
Type2Tag 'NXP NTAG215' ID=04722801E14103
b'\x04maker.ifttt.com/trigger/python_test/with/key/bEDoi_gUT5x5uDsdsaR3Ao'
b'\x04maker.ifttt.com/trigger/python_test/with/key/bEDoi_gUT5x5uDsdsaR3Aoo'
maker.ifttt.com/trigger/python_test/with/key/bEDoi_gUT5x5uDsdsaR3Ao'
maker.ifttt.com/trigger/python_test/with/key/bEDoi_gUT5x5uDsdsaR3Ao
https://
[urllib3.connectionpool] Starting new HTTPS connection (1): maker.ifttt.com:443
[urllib3.connectionpool] https://maker.ifttt.com:443 "GET maker.ifttt.com/trigger/python_test/with/key/bEDoi_gUT5x5uDsdsaR3Ao HTTP/1.1" 200 69
curl -X GET -H 'Accept: */*' -H 'Accept-Encoding: gzip, deflate' -H 'Connection: keep-alive' -H 'User-Agent: python-requests/2.21.0' https://maker.ifttt.com/trigger/python_test/with/key/bEDoi_gUT5x5uDsdsaR3Ao
[main] *** RESTART ***
[nfc.clf] searching for reader on path tty:S0:pn532
[nfc.clf] using PN532v1.6 at /dev/ttyS0
** waiting for a tag **
In the following code the HTTP protocol needs two newline character but what is the need of the \r there. Why can't we just add two \n and send the request?
import socket
mysock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
mysock.connect(("data.pr4e.org",80))
cmd = 'GET http://data.pr4e.org/romeo.txt HTTP/1.0\r\n\r\n'.encode() # here
mysock.send(cmd)
while True:
data = mysock.recv(512)
if len(data) > 0:
print(data.decode())
else :
break
mysock.close()
Because that's how the HTTP protocol is defined. More specifically, HTTP 1.0 defines a request like this:
Request = Simple-Request | Full-Request
Full-Request = Request-Line
*( General-Header
| Request-Header
| Entity-Header )
CRLF
[ Entity-Body ]
Request-Line = Method SP Request-URI SP HTTP-Version CRLF
Full-Request, which is what should be used by any HTTP 1.0 compatible client (simple request is HTTP 0.9 and deprecated) needs to have two CRLF tokens (one is in Request-Line). A CRLF token is the two bytes \r\n. Hence the need to end the string in your example with \r\n\r\n.
This design choice was kept in HTTP 1.1.
Because that is how the HTTP protocol works.
The request/status line and headers must all end with <CR><LF> (that is, a carriage return followed by a line feed). The empty line must consist of only <CR><LF> and no other whitespace.
https://en.wikipedia.org/wiki/HTTP_message_body
Hope someone might be able to help. I have a log been sent from a syslog server to python which looks like this:
{'Raw': 'Nov 26 00:23:07 TEST 23856434232342 (2016-11-26T00:23:07) http-proxy[2063]: Allow 1-Trusted 0-External tcp 192.168.0.1 2.3.4.5 57405 80 msg="HTTP Request" proxy_act="HTTP-TEST" op="POST" dstname="www.google.com" arg="/" sent_bytes="351" rcvd_bytes="1400" (HTTP-proxy-TEST-00)'}
I need to be able to extract the IP address, dstname=, sent_bytes= and dcvd_bytes= and if possible parse to json. I started trying to use REGEX (["'])(?:(?=(\\?))\2.)*?\1 to match the double quotes but its not working correctly.
Any ideas how I might get the data I need? Or how to parse the above to json?
Thanks
Assuming IP, dstname sent_bytes and rcvd_bytes are always in order, use re.findall to get them all
import re
s = r"""{'Raw': 'Nov 26 00:23:07 TEST 23856434232342 (2016-11-26T00:23:07) http-proxy[2063]: Allow 1-Trusted 0-External tcp 192.168.0.1 2.3.4.5 57405 80 msg="HTTP Request" proxy_act="HTTP-TEST" op="POST" dstname="www.google.com" arg="/" sent_bytes="351" rcvd_bytes="1400" (HTTP-proxy-TEST-00)'}"""
match = re.findall('(?:tcp |dstname=|sent_bytes=|rcvd_bytes=)"?([^\s"]+)', s)
# match = ['192.168.0.1', 'www.google.com', '351', '1400']
(ip, dstname, sent_bytes, rcvd_bytes) = match
# use this to parse to json
My Goal is that my SSI (shtml) will parse POST data parameters and return a long encrypted string which will be based on those parameters.
I am using IIS as the server.
My constraints:
I can't change the url - /document.shtml
The shtml need to return only the base 64 string result.
What i tried:
Use of "exec cgi" in the shtml, this way:
document.shtml:
<!--#exec cgi="/pythonApp/test.py?sdfsdf=sdfsdf" -->
test.py (simplified version , without the encryption logic behind):
import cgitb
cgitb.enable()
import base64,urllib,json,os,sys
import cgi
print "Content-Type: text/plain;charset=utf-8"
print
cl, _ = cgi.parse_header(os.environ['CONTENT_LENGTH'])
if (int(cl)>0):
data = sys.stdin.read(int(cl))
input_j = json.loads(data)
print "AB : ",input_j["AB"]
else:
print "Failed to read POST request"
The problem here, is that if i send post request to .../test.py , then it works,
but if i send the request to document.shtml, the content length will still be positive as described by the environment variable, but the stdin will get an error.
My other idea was to read the POST data , somehow from the shtml itself and pass it to the cgi , this way:
<!--#exec cgi="/pythonApp/test.py?sdfsdf=sdfsdf" -->
The problem , is that i looked over the net and haven't found a way to do it from the shtml.
Any suggestions? am i missing something simple?
I haven't found away to do it using SSI directive , i did understand that the stdin is not passed to the cgi from the main script.
So i found other solution which still answers my constrains.
In the IIS Handler Mapping for SHTML , i configured the executable to be php.
and in the shtml it self i wrote a php code (showing here just the sample that prints the json string):
<?php var_dump($HTTP_RAW_POST_DATA); ?>
I'm searching for help understanding how to develope a complete and functional NTRIP Client in order to receive RTCM corrections.
I'm using Python 3.4, for now on Windows 7. Searching the net, I found some sample code and I used it to write a basic client. The problem is... it doesn't work.
I have access to a rtk correction service. The service is active and functioning.
This is a snippet of my code.
dummyNMEA = "$GPGGA,143741.356,7839.493,S,07627.626,W,0,00,,,M,,M,,*45"
username = my_username #username for RTCM correction service
password = my_password #password for RTCM correction service
port = 2101 #port for the service
'''Generate an encoding of the username:password for the service.
The string must be first encoded in ascii to be correctly parsed by the
base64.b64encode function.'''
pwd = base64.b64encode("{}:{}".format(username, password).encode('ascii'))
#The following decoding is necessary in order to remove the b' character that
#the ascii encoding add. Othrewise said character will be sent to the net and misinterpreted.
pwd = pwd.decode('ascii')
print("Header sending... \n")
header =\
"GET /mountpoint HTTP/1.1\r\n" +\
"Host my_host\r\n" +\
"Ntrip-Version: Ntrip/1.0\r\n" +\
"User-Agent: ntrip.py/0.1\r\n" +\
"Accept: */*" +\
"Connection: close\r\n" +\
"Authorization: Basic {}\r\n\r\n".format(pwd)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((my_host,int(port)))
s.send(header.encode('ascii'))
print("Waiting answer...\n")
data = s.recv(2048).decode('ascii')
print(data)
s.send(dummyNMEA.encode('ascii'))
data = s.recv(2048).decode('ascii')
print(data)
s.close()
Now, the code is partially funcitoning. The request goes to the rtk server and I am correctly authenticated. I receive the correct answer as from ntrip protocol:
ICY 200 OK
Server: "Server of the mountpoint"
Date: "The date"
After this, I have to send a NMEA GGA sentence, in order to start receiving the RTCM corrections. I created various dummy NMEA sentences with a generator and tested sending them. I send the sentence and.... nothing happens. I receive no answer from the server.
Somebody has some idea? Perhaps I do something wrong when encoding the sentence?
I read that perhaps I should send the NMEA sentence continuosly, but I'm new in Python programming and I am not sure how to do that with sockets.
English is not my mother language, so please excuse my errors :)
Thnak you everyone.
When you are sending the GGA is it a position that has coverage on your rtk correction service? I have done this before, sent dummy positions that are outside the network coverage and nothing was returned, not even an error message, just no corrections.
Cheers,
Steve.
You need to add '\r\n' to the end of the NMEA string.
dummyNMEA = "$GPGGA,143741.356,7839.493,S,07627.626,W,0,00,,,M,,M,,*45\r\n"
You have to create an Header for the NMEA as well.
dummyHeader = \
"Ntrip-GGA: {}\r\n".format(dummyNMEA)
Then you should get an answer.