I'm using my own resolver and would like to use urllib2 to just connect to the IP (no resolving in urllib2) and I would like set the HTTP Host-header myself. But urllib2 is just ignoring my Host-header:
txheaders = { 'User-Agent': UA, "Host: ": nohttp_url }
robots = urllib2.Request("http://" + ip + "/robots.txt", txdata, txheaders)
You have included ": " in the "Host" string.
txheaders = { "User-Agent": UA, "Host": nohttp_url }
robots = urllib2.Request("http://" + ip + "/robots.txt", txdata, txheaders)
Related
I have been on this site and cannot find a suitable resolution to this problem.
I can connect to my system via Powershell using the following;
$auth = '{"username":' + '"' + $user + '","password":' + '"' + $Pass + '"}'
$body = $auth
$hdrs = #{}
$hdrs.Add("X-API-KEY", "???")
$r = Invoke-RestMethod -Uri http://$URLSystem/login -Method Post -Body $body -ContentType 'application/json' -Headers $hdrs
I get a response back of 200 and I can get session keys etc...
I have tried a number of things in Python to connect to the same system. I tried this basic approach;
import requests
from requests.auth import HTTPBasicAuth
basic = HTTPBasicAuth('user1','pass1')
r = requests.get("http://URLSystem/login", auth=basic)
print(r.headers)
print(r)
I get a 405 response code. I have tried changing the get to a POST and get a 415 error response.
I am new to Python and having a little difficulty getting this going. Any help would be greatly appreciated.
Thank you for the response.
I will look at the resources you pointed out - this was helpful.
Yes, I thought I should be using request.post(...) but could not get the right format for headers and params to use.
I did find this post #this web site and it worked for me with some slight modification, so I am good for now...
Posting the solution here for anyone else if they have similar issues.
import requests
import json
URL = 'http://your-url'
headers = {
"accept": "application/json",
"Content-Type": "application/json"
}
params = {
"username": "yourusername",
"password": "yourpassword"
}
resp = requests.post(URL, headers = headers ,data=json.dumps(params))
session = json.loads(resp.text)['SessionToken']
if resp.status_code != 200:
print('error: ' + str(resp.status_code))
else:
print('Response Code: ' + str(resp.status_code) + ', Session-Token: ' + str(session))
I am trying to send a get request with python like this:
import requests
url = "internal_url" # I replaced all internal urls
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0", "Accept": "*/*", "Accept-Language": "en-GB,en;q=0.5", "Accept-Encoding": "gzip, deflate", "X-Requested-With": "XMLHttpRequest", "Connection": "close", "Referer": "internal url"}
r = requests.get(url , headers=header)
print(r.text)
As reponse I am expecting json data. But instead I get this:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta http-equiv="Content-Script-Type" content="text/javascript">
<script type="text/javascript">
function getCookie(c_name) { // Local function for getting a cookie value
if (document.cookie.length > 0) {
c_start = document.cookie.indexOf(c_name + "=");
if (c_start!=-1) {
c_start=c_start + c_name.length + 1;
c_end=document.cookie.indexOf(";", c_start);
if (c_end==-1)
c_end = document.cookie.length;
return unescape(document.cookie.substring(c_start,c_end));
}
}
return "";
}
function setCookie(c_name, value, expiredays) { // Local function for setting a value of a cookie
var exdate = new Date();
exdate.setDate(exdate.getDate()+expiredays);
document.cookie = c_name + "=" + escape(value) + ((expiredays==null) ? "" : ";expires=" + exdate.toGMTString()) + ";path=/";
}
function getHostUri() {
var loc = document.location;
return loc.toString();
}
setCookie('STRING redacted', IP-ADDRESS redacted, 10);
try {
location.reload(false);
} catch (err1) {
try {
location.reload();
} catch (err2) {
location.href = getHostUri();
}
}
</script>
</head>
<body>
<noscript>This site requires JavaScript and Cookies to be enabled. Please change your browser settings or upgrade your browser.</noscript>
</body>
</html>
When I changed the request to use the burp suite proxy so I can see the request, it suddenly works and I get the correct response:
proxies = {"http": "127.0.0.1:8080", "https": "http://127.0.0.1:8080"}
r = requests.get(url, headers=headers, verify=False, proxies=proxies)
My browser displays the correct results as text when I visit the link itself. Burp suite proxy not needed.
I think its possible that it has to do with the company proxy.
But even when I tried to run the request with company proxies supplied it still does not work.
Is there something I am missing?
EDIT:
After some more searching it seems like I get redirected when I dont use any proxies in python. That doesnt happen when I go over the burp suite proxy.
After a few days and some outside help I finally found the solution. Posting it here for the future.
My problem was that I was using a partially qualified domain name instead of a fully qualified domain name
So for example: myhost instead of myhost.example.com
Burp suite or the browser were handling the translation for me but in python I had to do it myself.
I want to log in to a website with requests.Session.post. But when I had already the homepage login, and then go into the account page. It seems the cookies had not saved, because the cookies are empty. And I can not get to the right account page.
import requests
from bs4 import BeautifulSoup
header = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
'AppleWebKit/537.36 (KHTML, like Gecko)'
'Chrome/86.0.4240.111 Safari/537.36',
'origin': 'https://selma.tu-dresden.de'
}
logdata = {
'usrname': '******',
'pass': '******',
'APPNAME': 'CampusNet',
'PRGNAME': 'LOGINCHECK',
'ARGUMENTS': 'clino,usrname,pass,menuno,menu_type,browser,platform',
'clino': '000000000000001',
'menuno': '000155',
'menu_type': 'Classic',
'browser':'',
'platform':''
}
# post log in data
session = requests.Session()
post_url = 'https://selma.tu-dresden.de/APP'
html = session.post(post_url, data=logdata, headers=header)
print(html.status_code)
print(html.cookies)
# check if we already log in
rout = 'https://selma.tu-dresden.de/APP/EXTERNALPAGES/-N904200466705967,-N000108,-AEXT_Bewerbung'
konto_html = requests.get(rout, allow_redirects=False, cookies=html.cookies)
print(konto_html.cookies)
the result is:
200
<RequestsCookieJar[<Cookie ARRAffinity=f220f466b0cc86d57e60f7469bcce5940d9291ac90e2d500fa9a70c1375769a2 for .selma.tu-dresden.de/>, <Cookie cnsc=5FCB07D7E3960C2309380EF88E0684AE for selma.tu-dresden.de/>]>
<RequestsCookieJar[]>
In fact, I did not log in my account, because in the html.text show that
<title>Zugang verweigert
It means failure.
I don't know why.🤨🤨🤨
The URL, that you are routing to, has arguments. The first one "-N904200466705967" is the session ID. You are trying to rout to another session. This is why you aren't routing to that site and the web application is denying you access.
Here is a Code Snippet, of how submitting the form creates the URL, you are redirected to.
reloadpage = {
submitForm:function(formName){
/*function to submit a form, without editing the reload params*/
var form = document.getElementById(formName);
if (form != null)
{
$('input[name="Refresh"]').attr("disabled",true);
$('input[name="Refresh"]').addClass("cnm_disabled");
form.submit();
}
},
createUrlAndReload:function(dispatcher, applicationName, programName, sessionNo, menuId,args){
var temp_args;
if(args!=''){ temp_args =','+args};
$('input[name="Refresh"]').attr("disabled",true);
$('input[name="Refresh"]').addClass("cnm_disabled");
window.location.href = dispatcher + "?APPNAME=" + applicationName + "&PRGNAME=" + programName + "&ARGUMENTS=-N" + sessionNo + ",-N" + menuId + temp_args;
}
}
I have PTZ camera and I'm trying different way to access that camera via CURL. I'm intended to access preset position in the camera via web interface.
The logic to access preset of PTZ camera, based on browser debugger is like this:
Login using POST method
Select preset position using POST method
Submit using PUT method
Following is source code using shell script:
echo "Set PTZ"
echo $1 #IP address
echo $2 #preset
url_login='http://'$1'/login/login/'
url_preset='http://'$1'/ptz/presets.html'
curl -c cookies.txt -s -X POST $url_login --data "user=admin&pass=admin&forceLogin=on"
curl -b cookies.txt -s -X POST $url_preset --data 'PTZInterface!!ptzPositions='$2
curl -b cookies.txt -s -X PUT $url_preset --data 'autobutton=GotoCurrVirtualPreset&object=PTZInterface&id='
I have succeed using shell script, accessing the camera and go to preset.
But my main purpose is to create a program using python. Following is my python using requests:
import requests
URL_LOGIN = "/login/login/"
PARAMS_LOGIN = {"user": "admin", "pass": "admin", "forceLogin": "on"}
URL_PRESET = "/ptz/presets.html"
HEADERS = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
'Accept': '*/*', 'Accept-Language': 'en-US,en;q=0.5',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'X-Requested-With': 'XMLHttpRequest',
'Connection': 'keep-alive', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache'}
def set_ptz(arg_camera = None, arg_preset = None):
url_login = "http://" + arg_camera + URL_LOGIN
url_preset = "http://" + arg_camera + URL_PRESET
HEADERS['Host'] = arg_camera
HEADERS['Referer'] = 'http://' + arg_camera + '/'
params = {}
params["PTZInterface!!ptzPositions"] = arg_preset
params_put = {}
params_put["autobutton"] = "GotoCurrVirtualPreset"
params_put["object"] = "PTZInterface"
params_put["id"] = ""
s = requests.Session()
r1 = s.post(url_login, data = PARAMS_LOGIN) # Login -> success
var_cookies = r1.cookies
r2 = s.post(url_preset, cookies = var_cookies, headers = HEADERS, data = params) # Post preset position -> failed
r3 = s.put(url_preset, cookies = var_cookies, headers = HEADERS, data = params_put) # Put execution -> success
print r1.headers
print var_cookies
print r2.headers
print r3.headers
print r3.text
print r1.status_code
print r2.status_code
print r3.status_code
set_ptz('10.26.1.3.61', 1)
I'm succeed to login and submit using PUT, but failed to POST the preset position. What's wrong in my python code? I thought that the result should be same.
Thank you for help.
requests is escaping the exclamation points in the POST data:
In [1]: import requests
In [2]: requests.post(..., data={"PTZInterface!!ptzPositions": '1'}).request.body
Out[2]: 'PTZInterface%21%21ptzPositions=1'
cURL just sends them as-is. You can pass data directly as a string:
In [3]: requests.post(..., data="PTZInterface!!ptzPositions=1").request.body
Out[3]: 'PTZInterface!!ptzPositions=1'
Or use urllib.parse.urlencode's safe parameter to build it:
In [13]: urllib.parse.urlencode({'PTZInterface!!ptzPositions': 1}, safe='!')
Out[13]: 'PTZInterface!!ptzPositions=1'
I'm trying to scrape the search results of this ASP.NET website using Python requests to send a POST request. Even though I use a GET request to get the requestverificationtoken and include it in my header I get just get this reply:
{"Token":"Y2VgsmEAAwA","Link":"/search/Y2VgsmEAAwA/"}
which is not the valid link. It's the total search results with no defined arrival data or area as included in my POST request. What am I missing? Who do I scrape a site like this that generates a (session?) ID for the URL?
Thank you so much in advance to all of you!
My python script:
import json
import requests
from bs4 import BeautifulSoup
r = requests.Session()
# GET request
gr = r.get("http://www.feline.dk")
bsObj = BeautifulSoup(gr.text,"html.parser")
auth_string = bsObj.find("input", {"name": "__RequestVerificationToken"})['value']
#print(auth_string)
#print(gr.url)
# POST request
search_request = {
"Geography.Geography":"Danmark",
"Geography.GeographyLong=":"Danmark (Ferieområde)",
"Geography.Id":"da509992-0830-44bd-869d-0270ba74ff62",
"Geography.SuggestionId": "",
"Period.Arrival":"16-1-2016",
"Period.Duration":7,
"Period.ArrivalCorrection":"false",
"Price.MinPrice":None,
"Price.MaxPrice":None,
"Price.MinDiscountPercentage":None,
"Accommodation.MinPersonNumber":None,
"Accommodation.MinBedrooms":None,
"Accommodation.NumberOfPets":None,
"Accommodation.MaxDistanceWater":None,
"Accommodation.MaxDistanceShopping":None,
"Facilities.SwimmingPool":"false",
"Facilities.Whirlpool":"false",
"Facilities.Sauna":"false",
"Facilities.InternetAccess":"false",
"Facilities.SatelliteCableTV":"false",
"Facilities.FireplaceStove":"false",
"Facilities.Dishwasher":"false",
"Facilities.WashingMachine":"false",
"Facilities.TumblerDryer":"false",
"update":"true"
}
payload = {
"searchRequestJson": json.dumps(search_request),
}
header ={
"Accept":"application/json, text/html, */*; q=0.01",
"Accept-Encoding":"gzip, deflate",
"Accept-Language":"da-DK,da;q=0.8,en-US;q=0.6,en;q=0.4",
"Connection":"keep-alive",
"Content-Length":"720",
"Content-Type":"application/x-www-form-urlencoded; charset=UTF-8",
"Cookie":"ASP.NET_SessionId=ebkmy3bzorzm2145iwj3bxnq; __RequestVerificationToken=" + auth_string + "; aid=382a95aab250435192664e80f4d44e0f; cid=google-dk; popout=hidden; __utmt=1; __utma=1.637664197.1451565630.1451638089.1451643956.3; __utmb=1.7.10.1451643956; __utmc=1; __utmz=1.1451565630.1.1.utmgclid=CMWOra2PhsoCFQkMcwod4KALDQ|utmccn=(not%20set)|utmcmd=(not%20set)|utmctr=(not%20provided); BNI_Feline.Web.FelineHolidays=0000000000000000000000009b84f30a00000000",
"Host":"www.feline.dk",
"Origin":"http://www.feline.dk",
#"Referer":"http://www.feline.dk/search/Y2WZNDPglgHHXpe2uUwFu0r-JzExMYi6yif5KNswMDBwMDAAAA/",
"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36",
"X-Requested-With":"XMLHttpRequest"
}
gr = r.post(
url = 'http://www.feline.dk/search',
data = payload,
headers = header
)
#print(gr.url)
bsObj = BeautifulSoup(gr.text,"html.parser")
print(bsObj)
After multiples tries, I found that your search request is misformatted (need to be URL Encoded and not JSON), and cookies informations are overwrited in headers (Just let session make the work).
I simplified the code like that and I get the desired result
r = requests.Session()
# GET request
gr = r.get("http://www.feline.dk")
bsObj = BeautifulSoup(gr.text,"html.parser")
auth_string = bsObj.find("input", {"name": "__RequestVerificationToken"})['value']
# POST request
search_request = "Geography.Geography=Hou&Geography.GeographyLong=Hou%2C+Danmark+(Ferieomr%C3%A5de)&Geography.Id=847fcbc5-0795-4396-9318-01e638f3b0f6&Geography.SuggestionId=&Period.Arrival=&Period.Duration=7&Period.ArrivalCorrection=False&Price.MinPrice=&Price.MaxPrice=&Price.MinDiscountPercentage=&Accommodation.MinPersonNumber=&Accommodation.MinBedrooms=&Accommodation.NumberOfPets=&Accommodation.MaxDistanceWater=&Accommodation.MaxDistanceShopping=&Facilities.SwimmingPool=false&Facilities.Whirlpool=false&Facilities.Sauna=false&Facilities.InternetAccess=false&Facilities.SatelliteCableTV=false&Facilities.FireplaceStove=false&Facilities.Dishwasher=false&Facilities.WashingMachine=false&Facilities.TumblerDryer=false"
gr = r.post(
url = 'http://www.feline.dk/search/',
data = search_request,
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
)
print(gr.url)
Result :
http://www.feline.dk/search/Y2U5erq-ZSr7NOfJEozPLD5v-MZkw8DAwMHAAAA/
Thank you Kantium for your answer, in my case, i found that the RequestVerificationToken was actually generated in a JS script inside the page.
1 - Call the first page that generates the code, in my case it returned something like this inside the HTML:
<script>
Sys.Net.WebRequestManager.add_invokingRequest(function (sender, networkRequestEventArgs) {
var request = networkRequestEventArgs.get_webRequest();
var headers = request.get_headers();
headers['RequestVerificationToken'] = '546bd932b91b4cdba97335574a263e47';
});
$.ajaxSetup({
beforeSend: function (xhr) {
xhr.setRequestHeader("RequestVerificationToken", '546bd932b91b4cdba97335574a263e47');
},
complete: function (result) {
console.log(result);
},
});
</script>
2 - Grab the RequestVerificationToken code and then add it to your request along with the cookie from set-cookie.
let resp_setcookie = response.headers["set-cookie"];
let rege = new RegExp(/(?:RequestVerificationToken", ')(\S*)'/);
let token = rege.exec(response.body)[1];
I actually store them in a global variable, and later in my Nodejs Request i would add this to the request object:
headers.Cookie = gCookies.cookie;
headers.RequestVerificationToken = gCookies.token;
So that the end request would look something like this:
Remember that you can monitor requests sent using:
require("request-debug")(requestpromise);
Good luck !