I'm trying to login in to a jsp form with selenium webdriver in python. I'm trying to login by posting the parameters but I cannot reach the form or anything beyond the body tag for that matter. What am i doing wrong? Below is my code and below that is the page source - since it is a non-public web page:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
def my_method():
driver = webdriver.PhantomJS()
driver.get("https://<URL>.se:20443/snl/login.jsp")
password = driver.find_element_by_xpath("//input[#id='j_password']")
driver.close()
my_method()
Page source:
<!DOCTYPE html>
<!-- WARNING : modifying login.jsp may affect the login layout for Mobile, Embedded and Desktop Version. -->
<html>
<head>
<!-- Import header for script/style ... -->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="Cache-Control" content="no-cache"/>
<meta http-equiv="Expires" content="-1" />
<meta http-equiv="Content-type" content="text/html;charset=UTF-8">
<link type="image/x-icon" rel="shortcut icon" href="resources/images/favicon.ico">
<link type="text/css" rel="stylesheet" href="resources/style/login.css">
<style type="text/css">
body{
background: url(resources/images/gradient_body_login.png) repeat-x;
background-color: #d3d3d3;
}
#container{
background: url(resources/images/background_body.png) no-repeat top;
width:1090px;
min-height:770px;
height:auto;
}
#container{
margin:auto;
margin-top:0;
}
#login input[type="submit"],#login input[type="button"]{
margin:10px 5px 10px 0;
}
#mask{
opacity: 0.5;
filter: alpha(opacity = 50);
}
</style>
<script type="text/javascript" src="resources/script/live.js"></script>
<script type="text/javascript">
<!-- Against XFS attack -->
if(top != self)
{top.location=self.location;}
<!-- Against XFS attack -->
function Start(){
DisplayContent();
GiveFocus("j_username");
<!-- Check login failed -->
var vars = getUrlParameters();
var loginLabel = document.getElementById("failureIndication");
var authFailed = vars["authfailed"];
if (authFailed === "true")
loginLabel.innerHTML = "Login Failed";
<!-- Check login failed -->
}
function DisplayContent() {
var mainFrame = document.getElementById("hasJavascript");
mainFrame.style.display = 'block';
}
function GiveFocus(id){
document.getElementById(id).focus();
}
function setSubmitUrl(form){
var hash = getUrlHash();
if((hash != null) && (hash != "")) {
form.action = "j_spring_security_check#" + hash;
}else {
form.action = "j_spring_security_check";
}
return true;
}
</script>
<!-- Sample of custom logo -->
<style type="text/css">
h2 {
background: url("resources/large.png") no-repeat 20px 0 transparent;
line-height: 20px;
padding-left: 170px;
background-size: 75px 33px !important;
background-position: 35px 0px !important;
}
body {
font-family:Verdana;font-size:12px;color:#444;margin:0;padding:0;width:100%;height:100%;
background-color: #5BBF19 !important;
background-attachment: fixed !important;
background-repeat: no-repeat !important}
}
#mask {
display: none !important;
}
#login{
border:4px solid #008800 !important
}
</style>
<title>
Portal Login
</title>
</head>
<body OnLoad="Start();">
<noscript>
<div class="noJavascriptBox">
Your web browser must have JavaScript enabled
in order for this application to display correctly.
</div>
</noscript>
<div id="hasJavascript" class="hidden contentContainer">
<div id="container">
<div id="mask"></div>
<div id="login">
<h2>Portal Live Login</h2>
<form id="login_form" method="POST" onSubmit="return setSubmitUrl(this);">
<label for="j_username">Username:</label>
<input type="text" id="j_username" name="j_username" autocapitalize="off" autocorrect="off"/>
<label for="j_password">Password:</label>
<input type="password" id="j_password" name="j_password" autocapitalize="off" autocorrect="off"/>
<label id="failureIndication"> </label>
<input type="submit" value="OK"/>
</form>
</div>
</div>
</div>
</body>
I think you need to wait for the page to load or, to be more specific, wait for the visibility of the password field:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
password = wait.until(EC.visibility_of_element_located((By.ID, "j_password")))
password.send_keys("password")
I solved the problem by passing a service argument to ignore ssl errors (service_args=['--ignore-ssl-errors=true']) like this:
driver = webdriver.PhantomJS(desired_capabilities=dcap,service_args=['--ignore-ssl-errors=true'])
Then it worked!
Related
I am looking to scrape prices for different products from Metro's online grocery store. To do this, I need to set a particular store as a "favourite" so that Metro knows which products to show. I'm currently using Selenium to automate this part and return the cookies after selecting a particular store. However, I am still getting 403 errors when passing the cookies to a Request despite the fact that I can access other pages on Metro's website.
import requests
import time
from user_agent import generate_user_agent
from selenium import webdriver
from selenium_stealth import stealth
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
user_agent = generate_user_agent(navigator="chrome")
header = {"User-Agent": user_agent}
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
def getMetroCookies(store_url):
browser = webdriver.Chrome(options=options, executable_path="C:/Users/XXXX/Documents/chrome_driver/chromedriver.exe")
browser.delete_all_cookies()
stealth(browser,
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
webgl_vendor="Intel Inc.",
renderer="Intel Iris OpenGL Engine",
fix_hairline=True,
)
browser.get(store_url)
time.sleep(1.5)
cookie_button = browser.find_element_by_xpath("/html/body/div[4]/div/div[3]/button")
cookie_button.click()
WebDriverWait(browser, 10).until(EC.invisibility_of_element_located((By.XPATH, "/html/body/div[4]/div/div[3]/button")))
store_button = browser.find_element_by_xpath("/html/body/div[1]/div[2]/div[1]/div[2]/div[3]/div/div/div/div[1]/div/div[3]/button")
time.sleep(1)
store_button.click()
time.sleep(3)
driver_cookies = browser.get_cookies()
c = {c['name']:c['value'] for c in driver_cookies}
browser.close()
return(c)
store_url = "https://www.metro.ca/en/find-a-grocery/164"
cookies = getMetroCookies(store_url)
base_url = "https://www.metro.ca/en/online-grocery/search?filter="
search_item = "chicken"
search_url = base_url+search_item
page = requests.get(search_url, headers=header, cookies=cookies)
content = BeautifulSoup(page.text, 'html.parser')
This gives me a 403 error along with the following page content.
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Attention Required! | Cloudflare</title>
<meta id="captcha-bypass" name="captcha-bypass"/>
<meta charset="utf-8"/>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
<meta content="IE=Edge,chrome=1" http-equiv="X-UA-Compatible"/>
<meta content="noindex, nofollow" name="robots"/>
<meta content="width=device-width,initial-scale=1" name="viewport"/>
<link href="/cdn-cgi/styles/cf.errors.css" id="cf_styles-css" media="screen,projection" rel="stylesheet" type="text/css"/>
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>
<!--[if gte IE 10]><!-->
<script>
if (!navigator.cookieEnabled) {
window.addEventListener('DOMContentLoaded', function () {
var cookieEl = document.getElementById('cookie-alert');
cookieEl.style.display = 'block';
})
}
</script>
<!--<![endif]-->
<script type="text/javascript">
//<![CDATA[
(function(){
window._cf_chl_opt={
cvId: "2",
cType: "interactive",
cNounce: "94024",
cRay: "6657c0090c70ecee",
cHash: "f2ab1c66a7c7fb9",
cFPWv: "g",
cTTimeMs: "4000",
cLt: "n",
cRq: {
ru: "aHR0cHM6Ly93d3cubWV0cm8uY2EvZW4vb25saW5lLWdyb2Nlcnkvc2VhcmNoP2ZpbHRlcj1jaGlja2Vu",
ra: "TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV09XNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIENocm9tZS81NC4wLjI4ODIuNzQgU2FmYXJpLzUzNy4zNg==",
rm: "R0VU",
d: "EI8002UISMNZV4/wX/5oFZrkU66iZFrjnbrNYKgh3Ttb0AlT4tTYpyyzbKdGR4wfseBSZjcF8rJrwqdQEMKdIRBqLQjf0JlIowEseVWSf0dY03uEBGR+076Co1cm3pAeU83GN1kzFNq/sMe832Ng4oWK/pCpJ6XdIvbGWpk1l8Qtrwbi/hVtj3R1BXapeIgGrJRGlUjcsa72BbNFXOb97CsKqFb+6xMTSO9D/nTxFlouAqHyvbrkTG+CeGvImNQTqu9AVSsZiibNCRQ9C/IlNzCwn0tEvnJ6dZ6WA5RaS4riPmOdbpVGDcS2hIOjIfeGK4/Xj0dho0VkraSq+NPcFTfs18YuqtvQq/h7+V7uST5whKYXu1DM5F1TwPLbzM3KpB/KeYlad+JgxDcOaz1k0H/t52rfMhz8PYAjNvn7SwUXSJMRDeQavS6428g8IWtveqSUj4gnn6d4wGdTTNRpqnUm+m9SJARft2IjidMpvvBtUUzZe6srQs4JPZ9XzjfH+X/kMWgQT3X2pZVDrZZC9Od7P+sqyXPKoFNuZRPrWP15XogncIKTjt5MJLQUV42MGcaGlQ5w1PAvLNGGyNeMFG8wCfhuc/vLzodD+DP3bgIi7tjx8d5zhP3jMPAsUPxAxcJpZkBtuMBuKDNQO50dYHD2wwyOhx9HMcqHWCssMWN4qUzYKOth1KNlg0zlA/qzry1csYQqILH1F1b9O5QypPa2OA5gGmJNhar8svffekU9CXsqgtHDphJgEwsqrP1qSZzQ6wq1s5McDp6pPKijdPGbBrK4q2pxbJaVHu0lRn58gStP6HGEY8BLV/kEpygG27T4Vq4dp4uWLZDKw2oxk8ezrOIgv/lq7yXkZmhZs1GzHd4XWVXJvZ5dTI3rT1zrXMOTpInw4RWXULnazZn3HofZYOm0mUJvsofwzjaG88A=",
t: "MTYyNDcyNDI5Mi4wMTcwMDA=",
m: "SuNqM4NyxmnA1WU+nYefP0zkF5LxO+2HK+JlYjzu4dw=",
i1: "Z/V7+yIdblkqF9PRfarDwA==",
i2: "iMe97FeUtyqejNZ6Ziyc8w==",
zh: "/vdKLh0CrKHrnBUka1HcvI1mkhoFozUewI640Q15E4c=",
uh: "wSvBDgWWw4CCletn46YSZpWn4A/qjMkCb4uV9eAjmfA=",
hh: "56bTGUAA35o0NPPIwaihW3gLWiRsmO2PeArMwpTuU9E=",
}
};
}());
//]]>
</script>
<style type="text/css">
#cf-wrapper #spinner {width:69px; margin: auto;}
#cf-wrapper #cf-please-wait{text-align:center}
.attribution {margin-top: 32px;}
.bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; }
#cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; }
#cf-hcaptcha-container { text-align:center;}
#cf-hcaptcha-container iframe { display: inline-block;}
#keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
#cf-wrapper #cf-bubbles { width:69px; }
#-webkit-keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
#cf-bubbles > .bubbles { animation: fader 1.6s infinite;}
#cf-bubbles > .bubbles:nth-child(2) { animation-delay: .2s;}
#cf-bubbles > .bubbles:nth-child(3) { animation-delay: .4s;}
</style>
</head>
<body>
<div id="cf-wrapper">
<div class="cf-alert cf-alert-error cf-cookie-error" data-translate="enable_cookies" id="cookie-alert">Please enable cookies.</div>
<div class="cf-error-details-wrapper" id="cf-error-details">
<div class="cf-wrapper cf-header cf-error-overview">
<h1 data-translate="challenge_headline">One more step</h1>
<h2 class="cf-subheadline"><span data-translate="complete_sec_check">Please complete the security check to access</span> www.metro.ca</h2>
</div>
<div class="cf-section cf-highlight cf-captcha-container">
<div class="cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<div class="cf-highlight-inverse cf-form-stacked">
<form action="/en/online-grocery/search?filter=chicken&__cf_chl_captcha_tk__=0641319015a45358b1db60468c92bf88af4a70ea-1624724292-0-ATxUvClOko_GDrF_ejLwzZX-kuPRpoh1BFTlbPpgnM7UZS0tt0LcTa6u0ksaDrdsCuFkwxbyL7QYwbUeX6srjGPdlhXjLsQNqAH5sr4WHG8JX55aU2kRJzjzY9HulNoXyr6MuhmU1HzLv1ZvLss4X5hP-lABtnHTc5waDyQNzn3zxVHYetOu-uA7COqv76by9yx8dhQAWX0pT8cgjYQ2QwRLhrAw49GqhCux2EluSfziYo-Zqncf4uDyMe0Pb7Hb1csz2l9E_L26erOLQTrM_U2c1sYY0T-4ofJdQNEVLFA7e1FkGspeuGaFRRNmcXhCNPB7YKEiHlkROpAr2nxQeepJuefHBMdzbixJRXE5glhNCX9XXJ5nbpo8OzLY7pnMrJgaW6_YucjLh0fJs4c0bfBHAHZLWQeGxvcG7_AeM3zY6MIXngvnXg64GyrpxmYfADy_znyKmVlTCvVwdc8VEBZo27I4iGoqhJWaG0E1Q0Dw9a6dTU7bOWCSpoaxSNUmNkuwL5VsBAk3paSDIwYaewFLHijU-PUdeGw9hcLFsNbD95qUGlVEHZsdUMg176NYJ1VyZho1MMbNj8bVVC2kDKyZOu1IqcMe0TTqVwV5p9j_zZU6ODLXhn_d2VFULBMQTs9eIZUIz3j6uMZdEYV2o53P421SCx-MPPD5rALfYHdTRmSBDCLeW7gUG5-UvnWh87p87HJH__7plEmoJhFkW8crBpUeKBhwt7JQR_huvqOW" class="challenge-form interactive-form" enctype="application/x-www-form-urlencoded" id="challenge-form" method="POST">
<div id="cf-please-wait">
<div id="spinner">
<div id="cf-bubbles">
<div class="bubbles"></div>
<div class="bubbles"></div>
<div class="bubbles"></div>
</div>
</div>
<p data-translate="please_wait" id="cf-spinner-please-wait">Please stand by, while we are checking your browser...</p>
<p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">Redirecting...</p>
</div>
<input name="r" type="hidden" value="86274ebf891ca5903cedef6f5476291f7a3f2375-1624724292-0-AYyBXOiOkLO6sEJbZAhxnborgsqm+9Myz1E+TgVNFE0OKQcJs14P/RNNa9jSf5uTx9Eo4AxksOkzMWys+5Roo/xz2LZWFQybup/QSTYAEX6Oz5WVB05OtClBu7NY+EMGVabeM1OM2Q3cc1qgrnOH4h4UWw/tFTEYmY0tOXDYpe93zmxREYOxBU/vxCsLtda3YAAodT9qhQyO7oiTEgWMNC595Rjao12av3f+TtLrX41QyH/qiSfKJYRQf616Yvk7IEzTwc/n8ZvMc8wnGm5j+9lM0bzc6kRGoCfHVj1r0eJJxEV9aF15A+pKYuIzupkw/QOT8rUZE6UtL3yGB9UYVwqmcvtvIIO4ILPVnQV8fXxnvnXpvCVXKr0PgxPF4p8Drl1Vb95PdVn7ZvQ0jr6xGiqhbPFu2//9mwUnSjBQt8SXfei/Zq/Z0uL0TD/513/bBF1Jp/QojGEJjVGfs1Wo4L+usEpn9O0Z6gWaZXPfQgqTwiO9uboO+Z9V8pdeK3egZMneaXfCjhwgrNzmTilR90jQlGbsMOXhUokOQaxqhJ/khmBgnu5UfJ3OFxG5e2zQylxXkK88T38DE7DysMBuXE3wv10Pf4Dl6lEPMYbXqUB1Vp/hT+ShzNvaG1wpRQD6XA1WIzKNVINAg9QIffi30ojuKxldogRE0rpTAzZfgzRN8kiXFsxwQfMfTYMvdtoJsEbBP6CrvsNNNOmzN9exuM5WSbj+UXSa4/ZSlkHp0SVEJZOccYYYT6C9kAV0srfDmysEkDpfYQcap8AhFh7Ub8pYA9CedTD31+ghxXqlBphJj6zAQQfJyawwyFv4dwctjYJxduR3p6yG/7fyhTh8/B7U47sR30cP2mRA2sUMRLAYrLp2bd8yiz+jxsuoD6JxikOSYLTOl9e862isXFOg4RSspNB1RqCtb/154pnoP3bRghEl6vTSpj6dSH8GUxBjSQPWxbZuSwSMGTHHPxevAZFSpct5SNv05aU6rFvwPcna2h6UwjqcZOenMDr53xh2NVjHVWIcsDMXDReVncZyb28PIBqmBOdx4Fui/4KNXJuM1kuk5SMnN9zc1H9ZKgJnpXKIuvGFqS+Ifb56RCH/XWaoVPOG6tMwbitv8I0BkDvCSWBFXIgHw2tDuS3i/CxrELTCK6QURZZdFgZQKITtC/FvsxnDvPmnaON0dxzhJufdiBGiRCJAsLNJUkE1RfJCg8pApT+REE6IvrKb1r/1DIjETBWN0ntGE/J8fzZXOXaJDmmX2ZxWfBQCJ66RisEmJTzwRU9ModQMXfUeXxYx30IZN4H1BML6G/qzRiwN4mgO1aTeG0ic1Pv8NGdLlWP66gxvlxVTdNTuo1GTR8zyBs0AIwP3ohZrUH2KBH/r/NCVmKbxW7jswpl2kK9dzcPQi48TgeOyV8080BDzWkDOZj1agmycaGFobNAMdFhZhCSfYg+6+Y6rHba2CXKi2IioAGLh9/iMOvTGlMRZsw/dSd//ihW+otU033+sCxNjv/xK7RyAicZMk1MVDCDYbaEwzy6MAluXTpSSto5MHUDBWb+qwDlQYVqbkU5TO5ivbbBWpq1+8YFeq5zcrBfU9r+8ttj3qR8MpLcIAF18q9Ll1rE02opU/J6cCMNoFRmBecQZLmcSFoDWmS5n0nca51/KYQdJJDEpq63RKfc7KrizwX0lHfM+vwW4P3zYlGjXRdCjrNIf9Oae8nZpcB6itAjhzeu4n+gx24EHQmdNeg8AJ3B519bjqCA+aYooSfUzgUrNF+YDQBbI7Nq/sErOM6RanUuFaoMS3jnNCS4tP3TWdjJraHEY53wpBg+oqXpsJzdhfesM/KjNpxBxX9OT9v4vXyi8xzDPJB0EiZ8I6OihO7odnVW+gUdFLr7aMCtPsx5LbTFwLvE8ESTtdCfXWKSGB0GmQdJe5KmGsGQ1pxQEiVW0KUw+PCzvudBYngQF4N+UQcthlmt2pRt73ULhy2abmRa+JHOLWdvOgOQASxb8DW1k/htFRdj6FFmLygYJx+NBMD0kcQO32768SuU1S18wh2Vi3b3LXZrjpc/tPfbvADi2BVyMiEfs3cKtLwZjK2mrEONy5xq1BAL0UzLJCZCpVDc8IoxIQ3LpTxOAoQ9sw92LQdfvq/CyhMF8sAhMxQamvsWklrv5seJlNWvoNlvgfeaNxI/ugceoW9IwiZCb26d5ySpiySIgANeZwV//k5eGECYr8gLB37o+dGblgHjr+onK4UG2nHLAkIbhXBI1ZAlfE4f6YyruB2Z/35lxayZkRE/YYXJrYtpYJRU/ssl7S0VGY8SPh7aRdx8N9sw+F3XKQ63Y2pxO1KAm/Xf1CElhz86alEXlAdA24LZRz8cVcuHvk9mKM2j/YmUlYX+1uF2Zul+101PVpuvCypZtAa0nhlGTiB+st00ohFe6HmhK6d2T4UWISX6JiubywIJ0oLEF4hzecd1hB0/2Vdpl5Z9y/jhuOxPWceYGhriP3JYP9cS+MFbC36wOkF7hYpsdg9NEgFIDLFxzSYeEFkPeIuE13M1hwZHjjW8Zf6REdiPnQrDZHAKRDldWwzwBrs36guuJ4AiNju+Mx8Lr8wB6Krcd1+HriLm4uUFVM2DLeuusRkrSojUWkdWc2dpBrkLZ0tQw7wa6ZXVRt1nsWr5/ApEuzcC1+BaCGNdl1UzNd3NnGlPDYtYPFNPsuyUJIWjUTcB0rk/CfFP6JLoROVSP2l3WFVbktqw3m+mcwa6bw7Aew7YU4N/O2yJ8ab8a13/tV01Dfi61AIdKB2APWbVNRGxinXj++7fTKmqLB4B7usJC9EYqYbqq7ntAnjV9b1jI3iut/E6qDPZ58j9021JY4k2dfKY3Ry6GbIPAhvd/aKcN5Y6x79KItMsijXvAhBSILkbOwGQXccjo7lEIeh8Z1M+e3X0j2B811qcNjCvJDeMYb57+7jVkCzuL3ICADL1IjHGftYzjBPhPwl2UiZ4qD7tqc7Q2/Ol7BgYsIuddbNV72tof1/akffCEltbezCynu7P0hoDFCjDJAPmv4hGLFZZrLCU69jxLGYKU/ol6l8EEQA=="/>
<input name="cf_captcha_kind" type="hidden" value="h"/>
<input name="vc" type="hidden" value="cb7d9f733e82b2a322f24468dd51d0a0"/>
<noscript class="cf-captcha-info" id="cf-captcha-bookmark">
<h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1>
</noscript>
<div class="cookie-warning" data-translate="turn_on_cookies" id="no-cookie-warning" style="display:none">
<p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload the page.</p>
</div>
<script type="text/javascript">
//<![CDATA[
var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
b(function(){
var cookiesEnabled=(navigator.cookieEnabled)? true : false;
if(!cookiesEnabled){
var q = document.getElementById('no-cookie-warning');q.style.display = 'block';
}
});
//]]>
</script>
<div id="trk_captcha_js" style="background-image:url('/cdn-cgi/images/trace/captcha/nojs/h/transparent.gif?ray=6657c0090c70ecee')"></div>
</form>
<script type="text/javascript">
//<![CDATA[
(function(){
var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
var trkjs = isIE ? new Image() : document.createElement('img');
trkjs.setAttribute("src", "/cdn-cgi/images/trace/captcha/js/transparent.gif?ray=6657c0090c70ecee");
trkjs.id = "trk_captcha_js";
trkjs.setAttribute("alt", "");
document.body.appendChild(trkjs);
var cpo=document.createElement('script');
cpo.type='text/javascript';
cpo.src="/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1?ray=6657c0090c70ecee";
document.getElementsByTagName('head')[0].appendChild(cpo);
}());
//]]>
</script>
</div>
</div>
<div class="cf-column">
<div class="cf-screenshot-container">
<span class="cf-no-screenshot"></span>
</div>
</div>
</div>
</div>
</div>
<div class="cf-section cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<h2 data-translate="why_captcha_headline">Why do I have to complete a CAPTCHA?</h2>
<p data-translate="why_captcha_detail">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>
</div>
<div class="cf-column">
<h2 data-translate="resolve_captcha_headline">What can I do to prevent this in the future?</h2>
<p data-translate="resolve_captcha_antivirus">If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.</p>
<p data-translate="resolve_captcha_network">If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.</p>
<p data-translate="resolve_captcha_privacy_pass"> Another way to prevent getting this page in the future is to use Privacy Pass. You may need to download version 2.0 now from the Chrome Web Store.</p>
</div>
</div>
</div>
<div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
<p class="text-13">
<span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">6657c0090c70ecee</strong></span>
<span class="cf-footer-separator sm:hidden">•</span>
<span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: 2607:fa49:3801:a800:6901:b6b5:6c3a:ec5</span>
<span class="cf-footer-separator sm:hidden">•</span>
<span class="cf-footer-item sm:block sm:mb-1"><span>Performance & security by</span> Cloudflare</span>
</p>
</div><!-- /.error-footer -->
</div>
</div>
<script type="text/javascript">
window._cf_translation = {};
</script>
</body>
</html>
My guess is that I'm doing something wrong when extracting the cookies as I am able to access pretty much any part of Metro's website using requests, but I'm pretty new to this so I'm not entirely sure. Any help would be much appreciated!
The website uses Cloudflare services so that it will prevent the request without browser interaction. When you send a request without browser interaction (JavaScript), it will activate a captcha to check whether you are a bot or not. You can use selenium to scrape the information from the website.
from selenium import webdriver
from bs4 import BeautifulSoup
import requests
link = 'https://www.metro.ca/en'
chrome_driver = 'C:/Users/XXXX/Documents/chrome_driver/chromedriver.exe'
driver = webdriver.Chrome(executable_path=chrome_driver)
driver.implicitly_wait(10)
driver.get(link)
cookie = [f"{c['name']}={c['value']};" for c in driver.get_cookies()]
cookie = ' '.join([elem for elem in cookie])
search = driver.find_element_by_css_selector('#header--search--input')
search.send_keys("chicken")
submitButton = driver.find_element_by_css_selector("#header--search--button")
submitButton.click()
driver.implicitly_wait(10)
content = BeautifulSoup(driver.page_source, 'html.parser')
print(content)
Using requests
from selenium import webdriver
from bs4 import BeautifulSoup
import requests
link = 'https://www.metro.ca/en'
chrome_driver = 'C:/Users/XXXX/Documents/chrome_driver/chromedriver.exe'
driver = webdriver.Chrome(executable_path=chrome_driver)
driver.implicitly_wait(10)
driver.get(link)
cookie = [f"{c['name']}={c['value']};" for c in driver.get_cookies()]
cookie = ' '.join([elem for elem in cookie])
def using_request():
header = {
'Host': 'www.metro.ca',
'Connection': 'close',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Client-Version': 'web version 2.0',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36',
'Origin': 'https://www.metro.ca/en',
'Referer': 'https://www.metro.ca/en',
'Accept-Encoding': 'gzip, deflate',
'Cookie': f"{cookie}"
}
search_item = "chicken"
base_url = f"https://www.metro.ca/en/search?filter={search_item}&freeText=true"
page = requests.get(base_url, headers=header)
content = BeautifulSoup(page.text, 'html.parser')
print(content)
using_request()
I'm able to log in to a session using request.
I think I'm correct because the content of the HTML being parsed is showing the result after I manually inspect the page. Below is what I've worked so far.
import requests
from bs4 import BeautifulSoup as bs
import lxml
from selenium import webdriver
import time
### login website
### try
login_url='https://loancare.wd.lendingsvcs.com/smagent/forms/DynamicRetry.lfcc?SMQUERYDATA=-SM-YqDWGVYPqWY8B5OeWH3aAAimesXFQJ%2fJZl7sRyuKfOaqsAKdA%2fdfrgNl5ApC3q62OYvR98r%2fBHIytJa3QlVruC7F0wmsqUXFk4lh9whGJQFJPsLSSZE1suQW3mFFgRjDcwCPdEunMEwu%2fvEglLGa1W5z5Oq8ww53oXmtxyW2hfl3mI%2foNTbyya4DpptRTu5rLC1QthHeCjvmU8Ss5PumMI96cTCefMdmFwowyYDGKMkK76h9FIe8BJFwoBECSWPsnUOK0YEAMM4XFEh6yyaJcurlgzrnWhUOjVMw9gcBwaEqw3Y0gFyT0JZ6LpjkrG%2fNfauWm8%2fdU7R6W20WHCwUUAObQgRTxBsxooG2u37kzQ%2b7vzUcgmNwoi551t27rG1zhJbFOQAh42ZfujyTiAeuveIH5uyH0C4YK%2fn5YSh5LGQ6GaCpnZR0cj7T%2b965f8f3'
### fill in headers
headers = {
"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"}
### find hidden input tags
r = requests.get(login_url, headers=headers)
soup = bs(r.text)
itags = soup.find_all(name="input")
for tag in itags:
print(tag)
print('\n')
### account info
username = 'myusername'
password = 'mypassword'
### fill in input tags
data = {
"USERINPUT":username,
"PASSWORD":password,
"SMENC":"ISO-8859-1",
"SMLOCALE":"US-EN",
"UserCompany":"loancare",
"USER": "carrie.mcgahan,o=loancare",
"smauthreason":"0",
"smagentname":"sKydreaDKgXaN3m7DBUPd3z8oTpsSTRHjK/Eb2ArY4ip5nyfodVHKy8fvnIR3ghL",
"postpreservationdata":""
}
### start session.
session = requests.session()
### login
r = session.post(url=login_url, headers= headers, data=data)
# print(r.url)
print('\n')
### try one link first
scrape_url ='https://loancare.wd.lendingsvcs.com/webdirect/WebDirect.html#taskListView?loanNumber=24832974&clientId=623&providerName=loancare&systemId=prod'
### request page want to scrape.
url = session.get(url=scrape_url, headers=headers, allow_redirects=False)
time.sleep(10)
print(url.status_code)
print('\n')
print(url.url)
print('\n')
print(url.content)
print('\n')
soup = bs(url.text,'lxml')
print(soup)
print('finished')
session.close()
The issue is the parsed page doesn't show all the tags but it shows parts of the tags in below.
<!DOCTYPE html>
<html>
<head>
<title>Web Direct</title>
<meta content="no-cache" http-equiv="Pragma" />
<meta content="-1" http-equiv="Expires" />
<meta content="-1;URL=../index.html" http-equiv="Refresh" />
<meta content="IE=11" http-equiv="X-UA-Compatible" />
<link href="webdirect/reset.css" rel="stylesheet" type="text/css" />
<link href="css/gxt-all.css" rel="stylesheet" type="text/css" />
<link href="WebDirectApp.css" rel="stylesheet" type="text/css" />
<link href="images/favicon.ico" rel="icon" type="image/x-icon" />
<link href="images/favicon.ico" rel="shortcut icon" type="image/x-icon"
/>
</head>
<style>
* {
margin: 0px;
padding: 0px;
}
#loading {
position: absolute;
left: 45%;
top: 40%;
margin-left: -45px;
padding: 2px;
z-index: 20001;
height: auto;
border: 1px solid #ccc;
}
#loading a {
color: #225588;
}
#loading .loading-indicator {
background: white;
color: #444;
font: bold 13px tahoma, arial, helvetica;
padding: 10px;
margin: 0;
height: auto;
}
#loading .loading-indicator img {
margin-right: 8px;
float: left;
vertical-align: top;
}
#loading-msg {
font: normal 10px arial, tahoma, sans-serif;
}
<body style="overflow: hidden">
<div id="loading">
<div class="loading-indicator" style="text-align: center;"><img height="32" src="images/default/shared/large-loading.gif" style="margin-right:8px;float:left;vertical-align:top;" width="32" /><span id="loading-msg" style="font-size: 10pt;"><br/>Loading WebDirect...</span></div>
</div>
<!-- OPTIONAL: include this if you want history support -->
<iframe id="__gwt_historyFrame" src="javascript:''" style="width:0;height:0;border:0"></iframe>
<script src="webdirect/webdirect.nocache.js" type="text/javascript"></script>
<script src="js/rsa.js" type="text/javascript"></script>
</body>
The correct content I hope to parse is like in the screenshot next to the blue ink.
Then I found the content I have is different than the content in the picture.
I have
<div id="loading">
But the picture has
<div id="x-auto-6">
I've tried using different parsers like lxml and html5lib but still don't work out for me. Any suggestions? Thank you so much in advance.
I can't locate this element.. I'm trying to un-check the history box and dl box (they're checked by default)
from selenium import webdriver
import time
chrome_path = r"C:\Users\Skid\Desktop\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("chrome://settings/clearBrowserData")
driver.find_element_by_xpath("""//*[#id=delete-browsing-history-checkbox"]""") #unchecks history
driver.find_element_by_xpath("""//*[#id="delete-download-history-checkbox"]""") #unchecks dl history
This is the page source that someone wanted me to update.
<!DOCTYPE html><html xmlns="http://www.w3.org/1999/xhtml" id="uber" class="" i18n-values="dir:textdirection;lang:language" dir="ltr" lang="en" i18n-processed=""><head>
<meta charset="utf-8" />
<title i18n-content="pageTitle">Settings - Clear browsing data</title>
<link id="favicon" rel="icon" type="image/png" sizes="16x16" href="chrome://theme/IDR_SETTINGS_FAVICON" />
<link id="favicon2x" rel="icon" type="image/png" sizes="32x32" href="chrome://theme/IDR_SETTINGS_FAVICON#2x" />
<link rel="stylesheet" href="chrome://resources/css/chrome_shared.css" />
<style>/* Copyright (c) 2012 The Chromium Authors. All rights reserved.
* Use of this source code is governed by a BSD-style license that can be
* found in the LICENSE file. */
body {
/* http://crbug.com/129406 --- horizontal scrollbars flicker when changing
* sections. */
overflow-x: hidden;
}
#navigation {
height: 100%;
left: 0;
/* This is a hack to prevent the navigation bar from occluding pointer events
* from the bottom scroll bar (which shows when one needs to horizontally
* scroll). Corresponding padding-top to offset this is in uber_frame.css */
margin-top: -20px;
position: absolute;
/* This value is different from the left value to compensate for the scroll
* bar (which is always on and to the right) in RTL. */
right: 15px;
width: 155px;
z-index: 3;
}
#navigation.background {
z-index: 1;
}
#navigation.changing-content {
-webkit-transition: -webkit-transform 100ms, width 100ms;
}
.iframe-container {
-webkit-margin-start: -20px;
-webkit-transition: margin 100ms, opacity 100ms;
bottom: 0;
left: 0;
opacity: 0;
position: absolute;
right: 0;
top: 0;
z-index: 1;
}
.iframe-container.selected {
-webkit-margin-start: 0;
-webkit-transition: margin 200ms, opacity 200ms;
-webkit-transition-delay: 100ms;
opacity: 1;
z-index: 2;
}
.iframe-container.expanded {
left: 0;
}
iframe {
border: none;
display: block;
height: 100%;
width: 100%;
}
</style>
<script src="chrome://resources/js/cr.js"></script>
<script src="chrome://resources/js/cr/ui/focus_manager.js"></script>
<script src="chrome://resources/js/load_time_data.js"></script>
<script src="chrome://resources/js/util.js"></script>
<script src="chrome://chrome/uber.js"></script>
<script src="chrome://chrome/uber_utils.js"></script>
</head>
<body>
<div id="navigation" data-width="155" class="changing-content background" style="transform: translateX(0px);"><iframe src="chrome://uber-frame/" name="chrome" role="presentation" tabindex="-1" aria-hidden="true"></iframe></div>
<div class="iframe-container" i18n-values="id:historyHost; data-url:historyFrameURL;" data-favicon="IDR_HISTORY_FAVICON" id="history" data-url="chrome://history-frame/" hidden="" aria-hidden="true"></div>
<div class="iframe-container" i18n-values="id:extensionsHost; data-url:extensionsFrameURL;" data-favicon="IDR_EXTENSIONS_FAVICON" id="extensions" data-url="chrome://extensions-frame/" hidden="" aria-hidden="true"></div>
<div class="iframe-container selected" i18n-values="id:settingsHost; data-url:settingsFrameURL;" data-favicon="IDR_SETTINGS_FAVICON" id="settings" data-url="chrome://settings-frame/" aria-hidden="false" data-title="Settings - Clear browsing data"><iframe name="settings" role="presentation" src="chrome://settings-frame/clearBrowserData" data-ready="true"></iframe></div>
<div class="iframe-container" i18n-values="id:helpHost; data-url:helpFrameURL;" data-favicon="IDR_PRODUCT_LOGO_16" id="help" data-url="chrome://help-frame/" hidden="" aria-hidden="true"></div>
<script src="chrome://chrome/strings.js"></script>
<script src="chrome://resources/js/i18n_template.js"></script>
</body></html>
driver.find_element_by_xpath is just looking for the checkbox and returning it as WebElement. You want to click on it to unchecked it
driver.find_element_by_xpath("""//*[#id="delete-browsing-history-checkbox"]""").click()
Also, you forgot apostrophes in the first xpath after #id=. It should be like in the example above.
Edit
You can try locating the checkbox by id
driver.find_element_by_id("delete-browsing-history-checkbox").click()
Edit 2
The checkbox are inside iframe. You need to switch to it first
driver.switch_to.frame("settings") # switch to the iframe by name attribute
# driver.switch_to.frame(driver.find_element_by_name("settings")) # should also work
driver.find_element_by_id("delete-browsing-history-checkbox").click()
driver.switch_to.default_content() # switch back to main window
Can you add to your question what you get as body from selenium?
driver.get("chrome://settings/clearBrowserData")
driver.page_source
If I check the source code in Google Chrome of this page I get:
view-source:chrome://chrome/settings/clearBrowserData
<body>
<div id="navigation"><iframe src="chrome://uber-frame/" name="chrome" role="presentation"></iframe></div>
<div class="iframe-container"
i18n-values="id:historyHost; data-url:historyFrameURL;"
data-favicon="IDR_HISTORY_FAVICON"></div>
<div class="iframe-container"
i18n-values="id:extensionsHost; data-url:extensionsFrameURL;"
data-favicon="IDR_EXTENSIONS_FAVICON"></div>
<div class="iframe-container"
i18n-values="id:settingsHost; data-url:settingsFrameURL;"
data-favicon="IDR_SETTINGS_FAVICON"></div>
<div class="iframe-container"
i18n-values="id:helpHost; data-url:helpFrameURL;"
data-favicon="IDR_PRODUCT_LOGO_16"></div>
<script src="chrome://chrome/strings.js"></script>
<script src="chrome://resources/js/i18n_template.js"></script>
</body>
It might be necessary to find another way to do it, if your driver cannot see this node.
Edit
In the source code you posted as page_source returned from selenium, there isn't the node you are trying to find.
After doing a find_element_by... all you get is the element. You also need to have a .click() on that element.
Either:
elem = driver.find_element_by_xpath("""//*[#id=delete-browsing-history-checkbox"]""")
elem.click()
or:
driver.find_element_by_xpath("""//*[#id=delete-browsing-history-checkbox"]""").click()
Btw, you could just use find_element_by_id("delete-browsing-history-checkbox") in your case.
Also, I don't think selenium works on non-web pages. So chrome settings and Firefox's about:config pages (for example) don't work with selenium.
i need to know how to redirect the user in Django views to a certain page after he logs in.
let's say we have 3 types of users and 3 types of pages, i want each type to be directed to a certain page, and in the same time doesn't has the permission to view the other pages.
You can do something like this:
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
from django.contrib.auth.decorators import login_required
#login_required
def home(request):
return HttpResponseRedirect(
reverse(custom_view,
args=[request.user.username]))
Here, custom_view should be your user specific view. This is assuming you have:
LOGIN_REDIRECT_URL = '/profiles/home'
and you have configured a urlpattern:
(r'^profiles/home', 'myapp.views.home')
You can add a check for account type and redirect to the correct view.
I wrote some similar functionality recently by sub-classing the LoginView provided by django.contrib.auth. Okay, make your first page from the root directory, call it login:
python manage.py startapp login. Make this file exist <projectRoot>/login/templates/registration/login.html Add this code inside said file, it's more or less cut and pasted from bootstrap's login form, but with some standard django template language to bind to the expected AuthenticationForm fields.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<meta name="description" content="">
<meta name="author" content="">
<link rel="icon" href="../../favicon.ico">
<title>Signin to yourWebsite.com</title>
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js" integrity="sha384-ApNbgh9B+Y1QKtv3Rn7W3mgPxhU9K/ScQsAP7hUibX39j7fakFPskvXusvfa0b4Q" crossorigin="anonymous"></script>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js" integrity="sha384-JZR6Spejh4U02d8jOt6vLEHfe/JQGiRRSQQxSfFWpi1MquVdAyjUar5+76PVCmYl" crossorigin="anonymous"></script>
</head>
<body class="text-center">
<form class="form-signin" method="post" action="{% url 'login' %}?next=clockin">
{% csrf_token %}
<img class="mb-4" src="https://getbootstrap.com/assets/brand/bootstrap-solid.svg" alt="" width="72" height="72">
<h1 class="h3 mb-3 font-weight-normal">Please sign in</h1>
<label for="inputEmail" class="sr-only">Email address</label>
<!--input id="inputEmail" class="form-control" placeholder="Email address" required="True" autofocus="" type="email"-->
{{form.username}}
<label for="id_password" class="sr-only">Password</label>
{{form.password}}
<!--input id="inputPassword" class="form-control" placeholder="Password" required="True" type="password"-->
<div class="checkbox mb-3">
<label>
<input value="remember-me" type="checkbox"> Remember me
</label>
</div>
<button class="btn btn-lg btn-primary btn-block" type="submit">Sign in</button>
<p class="mt-5 mb-3 text-muted">© 2018</p>
</form>
</body>
</html>
<script>
$("#id_password").attr({
"class":"form-control",
"placeholder":"Password",
"required":"True",
"type":"password",
});
$("#id_username").attr({"class":"form-control",
"placeholder":"Email address",
"required":"True",
"type":"email",
});
</script>
<style>
html,
body {
height: 100%;
}
body {
display: -ms-flexbox;
display: -webkit-box;
display: flex;
-ms-flex-align: center;
-ms-flex-pack: center;
-webkit-box-align: center;
align-items: center;
-webkit-box-pack: center;
justify-content: center;
padding-top: 40px;
padding-bottom: 40px;
background-color: #f5f5f5;
}
.form-signin {
width: 100%;
max-width: 330px;
padding: 15px;
margin: 0 auto;
}
.form-signin .checkbox {
font-weight: 400;
}
.form-signin .form-control {
position: relative;
box-sizing: border-box;
height: auto;
padding: 10px;
font-size: 16px;
}
.form-signin .form-control:focus {
z-index: 2;
}
.form-signin input[type="email"] {
margin-bottom: -1px;
border-bottom-right-radius: 0;
border-bottom-left-radius: 0;
}
.form-signin input[type="password"] {
margin-bottom: 10px;
border-top-left-radius: 0;
border-top-right-radius: 0;
}
</style>
Next, subclass the built in view and override the redirect part. Inside login.views.py, add this:
from django.contrib.auth.views import LoginView
class CustomLoginview(LoginView):
def get_redirect_url(self):
if self.request.user.groups.filter(name="customer").exists():
return 'invoice'
return super().get_redirect_url()
Finally, update urls.py:
from login.views import CustomLoginview
urlpatterns = [
path('', CustomLoginview.as_view(), name='login'),
I'm telling the login page to go to my next app 'invoice' if the user is a customer, otherwise it goes to the default that I specified in the settings file. Obviously you could expound upon the concept for 3 types of users, which is different than routing based on the names of users, not sure how that got 2 upvotes.
I am attempting to set up django view for my web application which redirects the page once a file upload is complete, and the status bar showing the upload progress reaches 100%. I have looked around online and attempted to do this in several ways but nothing seems to be working. When I use
render(request, 'template_name')
The application simply returns the plain text of 'template_name' to the console rather than rendering it in the browser window. The original page of the loading bar stays in place after this plain text is returned.
My view looks like the following
def barUpdate(request):
importid = request.GET.get('impid')
response_data = {}
import_status_dict = get_import_status(importid)
status_id = import_status_dict['returnval']
import_status_info = import_status_dict['data_row']
import_status_info = import_status_info[0]
total_rows = import_status_info['total_data_rows']
rows_analyzed = import_status_info['number_of_rows_analyised']
if status_id != 2:
if (rows_analyzed != None and total_rows != None):
percent_complete = int((float(rows_analyzed)/total_rows)*100)
response_data['value'] = percent_complete
if 'percent_complete' in locals():
if response_data['value'] >= 100:
#return render(request,'statustool/completed.html',{'importid':importid,'username':username,'failedparameters':new_failed_param_group,'failedsources':failed_sources,'failedparametergroups':failed_parameters_group,'failedsitegroups':failed_sites_group,'sources':get_sources(), 'failedunits':failed_units})
#Right here I would like to render a new template in my browser, although this is just a dummy template I created for testing
return render(request,'statustool/test.html')
response = HttpResponse(json.dumps(response_data), content_type="application/json")
return response
else:
response_data['value'] = 0
response = HttpResponse(json.dumps(response_data), content_type="application/json")
return response
My dummy template is the following which contains no variables to be passed in from the view
<html>
<head>
test
</head>
<body>
<h1>Finished with data insert!</h1>
</body>
</html>
Is there something I am missing?
If it helps, the current page with the status bar looks like the following and uses a javascript function called status to make GET requests every second to find the upload status for the status bar
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>CACW: Status - Processing</title>
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<!-- Le styles -->
<link href="{{ STATIC_URL }}css/bootstrap.css" rel="stylesheet">
<link href="{{ STATIC_URL }}css/boostrap-responsive.css" rel="stylsheet">
<style>
body,html{
padding-top: 30px; /* 60px to make the container go all the way to the bottom of the topbar */
}
.container{
min-height:100%;
}
.footer{
height:40px;
margin-top:-25px;
}
.barcontainer{
width: 100px;
color: blue;
}
progress {
background-color: whiteSmoke;
border-radius: 2px;
box-shadow: 0 2px 3px rgba(0, 0, 0, 0.25) inset;
width: 250px;
height: 20px;
position: relative;
display: block;
}
</style>
<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script>
<script src="http://code.jquery.com/ui/1.10.3/jquery-ui.js"></script>
<script type="text/javascript" src="{{ STATIC_URL }}js/d3examples.js"></script>
<script type="text/javascript">
var importNum = {{importid}}
function status(){
var barProgress = window.setInterval("getProgress(importNum);", 1000);
}
var url=api_server_address+"import/status/update/";
var getProgress = function(importid) {
$.ajax({
url: "https://cacw.luc.edu/status/update/",
data: { impid: importid },
type: "GET",
dataType: "json"
})
.done(function(data){
$('#progressBar').val(data['value']);
console.log(data);
});
}
</script>
</head>
<div class="navbar navbar-inverse navbar-fixed-top">
<div class="navbar-inner">
<div class="container">
<a class="btn btn-navbar" data-toggle="collapse" data-target=".nav-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</a>
<a class="brand" href="#">CACW</a>
<div class="nav-collapse collapse">
<ul class="nav">
<li class="active">Home</li>
<li>Wiki</li>
<li>Contact</li>
</ul>
<a class="btn btn-primary pull-right" href="/logout">Logout</a>
</div><!--/.nav-collapse -->
</div>
</div>
</div>
<body onload="status({{importid}});">
<div class="container">
<div class="page-header">
<p><h2>Import {{ importid }} Status.</h2></p>
{{percent_complete}}
<progress id="progressBar" value={{status}} max="100"></progress>
</div>
</div>
<div class="footer">
<div class="navbar-fixed-bottom">
<hr>
<div class = "container" style="text-align: center">
<p> Help - Information - Contact - Wiki <p>
<img src="{{ STATIC_URL }}img/luc_logo.jpg"></img>
</div>
</div>
</div>
</body>
</html>
Since you are just getting the data in an AJAX call, this will never update your page (from the server side). What you can do is add a flag/object/parameter to your servers response to indicate when the upload is done, then on the client side, redirect to that location when the upload is finished.
Server side:
# code shortened a bit... continues from after line defining percent complete
response_data['value'] = percent_complete if 'percent_complete' in locals() else 0
response_data['done'] = response_data['value'] >= 100
return HttpResponse(json.dumps(response_data), content_type="application/json")
Client Side:
var getProgress = function(importid) {
$.ajax({
url: "https://cacw.luc.edu/status/update/",
data: { impid: importid },
type: "GET",
dataType: "json"
})
.done(function(data) {
if(data['done']) {
// I forget if this is how to do a redirect but it's where you put it
location.href('whatever/your/url/is');
} else {
$('#progressBar').val(data['value']);
console.log(data);
}
});
}