I need to extract the source code from the website that requires loggin.
I can access and open the web page by clicking the link, because I already logged into the page before and there are stored cookie.Click here to view the Manual Process diagram
However, if I tried to use python with the same link
import urllib
link = "http://www.somesite.com/details.pl?urn=2344"
f = urllib.urlopen(link)
myfile = f.read()
print myfile
The result always returns the source code of the login page
Can someone help with me this?Thanks a lot
UPDATE 1.0:
I have tried the stackoverflow.com/a/13955538/968442 provided by itsneo and it works perfectly with my Reddit account. However, after I update the user name, password and the URL (have double checked on these values), I still stuck at the login page that I want to access with.
Following is the login page I tried to access, do I need to add any additional attribute to the code?
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<title></title>
<link href="/stylesheets/ab.css?1447805380" media="screen" rel="stylesheet" type="text/css" />
<link href="/stylesheets/login.css?1412579810" media="screen" rel="stylesheet" type="text/css" />
</head>
<body>
<span class="centerblockabsolute login">
<div class="base-layer">
<div id="banner">
active billing
</div>
<form action="/login/login" method="post">
<div id="leftcontent">
<div>
</div>
<span class="error"></span><br />
<table>
<tr>
<td><label for="username">user name:</td>
<td><input id="username" name="username" size="20" type="text" /></td>
</tr>
<tr>
<td><label for="passsword">Password: </label></td>
<td><input id="password" name="password" size="22" type="password" /></td>
</tr>
<tr>
<td align="right" colspan="2">
<input name="submit" src="/images/buttons/login.gif?1412579810" type="image" value="submit" />
</td>
</tr>
</table>
</div>
<div id="rightcontent">
<img alt="Logo-ab" src="/images/logo-ab.png?1412579810" />
<input id="redirect_url" name="redirect_url" type="hidden" value="https://eutility.activebilling.com.au/" />
</div>
</form>
</div>
</span>
</body>
</html>
Related
I want to use requests.Sessions() to deliver my login information to a website. Once logged in I want to navigate to a second URL that can only be accessed once logged in. In order to scrape data from the second URL.
I am new to scraping and don't really have any experience with HTML
I am working in collaboratory if that makes any difference.
This is my code and the outputs:
import requests
page = requests.get("https://app.gristanalytics.com/Account/Login")
page
<Response [200]>
page.status_code
200
from bs4 import BeautifulSoup
soup = BeautifulSoup(page.content, 'html.parser')
print(soup.prettify())
This is the output:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/>
<link href="/lib/bootstrap/css/bootstrap.min.css" rel="stylesheet"/>
<link href="/lib/fontawesome/css/all.min.css" rel="stylesheet"/>
<link href="/lib/datetimepicker/bootstrap-datetimepicker.min.css" rel="stylesheet"/>
<link href="/lib/vue-multiselect/vue-multiselect.min.css" rel="stylesheet"/>
<link href="/css/site.css" rel="stylesheet"/>
<title>
Log in - Grist
</title>
</head>
<body>
<div>
<div class="text-center loginbox">
<form method="post" style="width:100%;max-width:350px;padding:15px;margin:0 auto;">
<img alt="" class="mb-4" src="/images/grist_logo_m_black.png"/>
<h1 class="h3 mb-3 font-weight-normal">
Please sign in
</h1>
<div class="text-danger validation-summary-valid" data-valmsg-summary="true">
<ul>
<li style="display:none">
</li>
</ul>
</div>
<label class="sr-only" for="inputEmail">
Email address
</label>
<input autofocus="" class="form-control my-1" data-val="true" data-val-email="The Email field is not a valid e-mail address." data-val-required="The Email field is required." id="Input_Email" name="Input.Email" placeholder="Email address" required="" type="email" value=""/>
<label class="sr-only" for="inputPassword">
Password
</label>
<input class="form-control my-1" data-val="true" data-val-required="The Password field is required." id="Input_Password" name="Input.Password" placeholder="Password" required="" type="password"/>
<div class="checkbox my-3">
<label>
<input data-val="true" data-val-required="The Remember me? field is required." id="Input_RememberMe" name="Input.RememberMe" type="checkbox" value="true"/>
Remember me
</label>
<p>
<a href="/Account/ForgotPassword">
Forgot your password?
</a>
</p>
</div>
<button class="btn btn-lg btn-primary btn-block" type="submit">
Sign in
</button>
<p class="mt-5 mb-3 text-muted">
© 2018-2022
</p>
<input name="__RequestVerificationToken" type="hidden" value="CfDJ8CxpSY-tCd5Ou0L0wqhntPACCikaoFBOUQLV0RgCaVUJgt9wRSd3p9aVswNuSLU6OPRKsbIm-qvOyZyZErcEm-E__Q2tPauexh3z_T02Oh5TZCpeY12PsUsERY3INO5LUBBmWXeUR6nG5BFHnnNdW70">
<input name="Input.RememberMe" type="hidden" value="false"/>
</input>
</form>
</div>
</div>
<script src="/lib/jquery-validation/dist/Jquery.validate.min.js">
</script>
<script src="/lib/jquery-validation-unobtrusive/jquery.validate.unobtrusive.min.js">
</script>
</body>
</html>
At this point I believe that the field names that I want to deliver the payload are:
name="Input.Email" and name="Input.Password"
However I note that there is no action attribute in the HTML code, so I plan to send the payload to the original URL as you will see below.
payload = {
'Input.Email': 'MyEmail', #yes in practice this is my actual information instead of this placeholder
'Input.Password': 'MyPassword', #same here real password used instead
}
with requests.Session() as session:
post = session.post('https://app.gristanalytics.com/Account/Login', data=payload)
r = session.get('https://app.gristanalytics.com/Data/Brewhouse')
soup = BeautifulSoup(r.content, 'html.parser')
print(soup.prettify())
The output of this is:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/>
<link href="/lib/bootstrap/css/bootstrap.min.css" rel="stylesheet"/>
<link href="/lib/fontawesome/css/all.min.css" rel="stylesheet"/>
<link href="/lib/datetimepicker/bootstrap-datetimepicker.min.css" rel="stylesheet"/>
<link href="/lib/vue-multiselect/vue-multiselect.min.css" rel="stylesheet"/>
<link href="/css/site.css" rel="stylesheet"/>
<title>
Log in - Grist
</title>
</head>
<body>
<div>
<div class="text-center loginbox">
<form method="post" style="width:100%;max-width:350px;padding:15px;margin:0 auto;">
<img alt="" class="mb-4" src="/images/grist_logo_m_black.png"/>
<h1 class="h3 mb-3 font-weight-normal">
Please sign in
</h1>
<div class="text-danger validation-summary-valid" data-valmsg-summary="true">
<ul>
<li style="display:none">
</li>
</ul>
</div>
<label class="sr-only" for="inputEmail">
Email address
</label>
<input autofocus="" class="form-control my-1" data-val="true" data-val-email="The Email field is not a valid e-mail address." data-val-required="The Email field is required." id="Input_Email" name="Input.Email" placeholder="Email address" required="" type="email" value=""/>
<label class="sr-only" for="inputPassword">
Password
</label>
<input class="form-control my-1" data-val="true" data-val-required="The Password field is required." id="Input_Password" name="Input.Password" placeholder="Password" required="" type="password"/>
<div class="checkbox my-3">
<label>
<input data-val="true" data-val-required="The Remember me? field is required." id="Input_RememberMe" name="Input.RememberMe" type="checkbox" value="true"/>
Remember me
</label>
<p>
<a href="/Account/ForgotPassword">
Forgot your password?
</a>
</p>
</div>
<button class="btn btn-lg btn-primary btn-block" type="submit">
Sign in
</button>
<p class="mt-5 mb-3 text-muted">
© 2018-2022
</p>
<input name="__RequestVerificationToken" type="hidden" value="CfDJ8CxpSY-tCd5Ou0L0wqhntPAwaiYOz80Q50p5gOcDk9qSF-gR4JJpzNGOdSKiQOzcVPp8hBKgDaEwXOrbFnpgdYXkedfcnLQlXIJ1Z7HnIi5vKZybNd6VSKk_Xs5Az444e3Oug-u1UFcxq_OLX1Iu0wU">
<input name="Input.RememberMe" type="hidden" value="false"/>
</input>
</form>
</div>
</div>
<script src="/lib/jquery-validation/dist/Jquery.validate.min.js">
</script>
<script src="/lib/jquery-validation-unobtrusive/jquery.validate.unobtrusive.min.js">
</script>
</body>
</html>
Which is the same HTML as the first time, it is clear that I am not logged in, and as a result I can't get to the HTML code of the URL I want.
I have tried other variations for the payload field names including:
inputEmail (from for=)
Input_Email (from id=)
email (from type=)
sample code for variation 1 would be
payload = {
'inputEmail': 'MyEmail', #yes in practice this is my actual information instead of this placeholder
'inputPassword': 'MyPassword', #same here real password used instead
}
I get no error or warning messages when running this code so I'm a bit stuck as to what to do.
The following code helped me to login and get me where I wanted to go!
Big thank you to #bushcat69 for the help he provided, I probably wouldn't have looked seriously at the verification token without them.
As well as the following [1, 2] stack exchange posts for additional information that I used.
with requests.Session() as session:
read = session.get('https://app.gristanalytics.com/Account/Login')
soup = BeautifulSoup(read.content, 'html.parser')
token = soup.select_one('[name="__RequestVerificationToken"]').get('value')
payload = {
'Input.Email': 'MyEmail#email.com',
'Input.Password': 'MyPassword',
'__RequestVerificationToken': token,
'Input.RememberMe': 'false'
}
post = session.post('https://app.gristanalytics.com/Account/Login', data=payload)
r = session.get('https://app.gristanalytics.com/Data/Brewhouse')
tastySoup = BeautifulSoup(r.content, 'html.parser')
print(tastySoup.prettify())
I am now having issues where it seems that some of the content that I want to scrape is working through Ajax / javascript which I don't know how to get. If you're having similar issues look into my future questions, I will also leave a comment here with the stackexchange/whatever website if I find content that helps me figure it out.
first of all i wanted to say that im quite new to python and selenium in particular.
Me and a co-worker are trying to automate a web app to sort payments for us.
everything works up to the part where a pop window appears.This pop up basically its a credit card form.
The Issue is:
You can only interact with the fields, the two buttons and you can close it using the "X".
you cant inspect with developer tools, and i cant download any other programs for that because its
a work computer.
the things i found so far are the link as you can see in the address bar, and i was able to inspect it with the developer tools 'Network' tab.
here is the code that is in the window:
<html>
<head>
<title> ÷ìéèú ôøèé àùøàé </title>
</head>
<body>
<BR>
<iframe id="iFrame" width="100%" height="100%" src="" frameborder="0" ></iframe>
<script Language="JavaScript">
if (typeof(window.dialogArguments)!="undefined") document.getElementById('iFrame').src = window.dialogArguments;
</script>
</body>
</html>
the following is the code which sits inside the page itself (the input fields etc.):
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="cache-control" content="no-cache">
<meta http-equiv="pragma" content="no-cache">
<meta http-equiv="expires" content="-1">
<script src="merchantPages/WebSources/js/HE.js"></script>
<script src="merchantPages/leaseacar/js/main.js"></script>
<script src="merchantPages/leaseacar/js/tafnitTrack2.js"></script>
<link href="merchantPages/WebSources/css/main.rtl.css" rel="stylesheet" type="text/css" />
<title>דף תשלום מאובטח</title>
</head>
<body onload="resetForm();">
<!-- DISPLAY TRANSACTION INFORMATION -->
<div id="main_frame" align="center">
<div id="main">
<div id="page_title"><img id="ssl_logo" src="merchantPages/WebSources/images/ssl.png" alt="" title=""/><h1>דף תשלום מאובטח</h1></div>
<div id="logo"><img src="merchantPages/leaseacar/images/eTafnit.gif" alt="" title="" width="87" height="118"/></div>
<div class="sep"></div>
<div id="page_desc">
<em>עמוד זה הינו עמוד התשלום עבור העסקה שביצעת באתר.<br />תשלום בעבור ההזמנה יבוצע רק לאחר בחירה בכפתור "שליחה" שבתחתית המסך.<br />אנא אשר תחילה כי כל הפרטים בדף זה נכונים והזן את פרטי כרטיס האשראי כנדרש.</em>
</div>
<form id="creditForm" onsubmit="return formValidator(10);" method="POST" action="ProcessCreditCard">
<input type="hidden" name="txId" value="0ce8b10c-2550-4293-ac6b-fee0b61c5b80"/>
<input type="hidden" name="lang" value="HE"/>
<input type="hidden" name="last4d" value="" autocomplete="off" />
<input type="hidden" name="cavv" value="" autocomplete="off" />
<input type="hidden" name="eci" value="" autocomplete="off" />
<input type="hidden" name="transactionCode" value="Phone" autocomplete="off" />
<input type="hidden" name="expMonth" value="10" autocomplete="off" />
<input type="hidden" name="expYear" value="24" autocomplete="off" />
<input type="hidden" name="personalId" value="" autocomplete="off" />
<input type="hidden" name="userData1" value=""/>
<input type="hidden" name="userData2" value=""/>
<input type="hidden" name="userData3" value=""/>
<input type="hidden" name="userData4" value=""/>
<input type="hidden" name="userData5" value=""/>
<input type="hidden" name="userData6" value=""/>
<input type="hidden" name="userData7" value=""/>
<input type="hidden" name="userData8" value=""/>
<input type="hidden" name="userData9" value=""/>
<input type="hidden" name="userData10" value=""/>
<table class="data_tbl">
<thead>
<td colspan="4" class="td_style_1">הזנת פרטי כרטיס אשראי לתשלום</td>
</thead>
<tr>
<td class="td_style_fieldName">Card Number:</td>
<td><input type="text" id="cardNumber" name="cardNumber" maxlength="19"
style="width: 160px;" autocomplete="off" onchange="validateCardNumber();"/></td>
<td class="td_style_invalidField" id="invalidCardNumber"> </td>
</tr>
<tr>
<td class="td_style_fieldName">CVV:</td>
<td><input type="text" name="cvv" id="cvv" maxlength="4" style="width: 50px;" autocomplete="off" onchange="validateCvv();"/> <img src="merchantPages/WebSources/images/qm.png" onmouseover="showHideCVVhelp();" onmouseout="showHideCVVhelp();" style="cursor:pointer;"/><div id="CVVhelp" style="display: none; position: absolute; border: 1px #cccccc solid; padding: 10px; background: white;"><img src="merchantPages/WebSources/images/cvv.jpg" /></div></td>
<td class="td_style_invalidField" id="invalidCvv"> </td>
</tr>
<tr>
<td class="td_style_fieldName">Track2:</td>
<td><input type="password" name="track2" id="track2" value="" autocomplete="off" onchange="tafnitValidateTrack2();"/></td>
<td class="td_style_invalidField" id="invalidTrack2"> </td>
</tr>
</table>
<div id="form_buttons"><input type="submit" id="submitBtn" value="שליחה" /> | <input id="resetBtn" type="reset" value="ניקוי" /></div>
</form>
</div>
<br />
<img src="merchantPages/WebSources/images/rapidssl.gif" alt="" title=""/>
</div>
</body>
</html>
if someone could give me some advice on how could i target the window so i can close, or if its even possible.
this is what i got so far:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.alert import Alert
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
#IMPORTS!-------------------------------------
usr = "***"
pwd = "***"
#app.
driver = webdriver.Ie(executable_path='C:/Users/natbag/Desktop/proj/IEDriverServer.exe')
driver.get('http://111.10.10.10/MENU1/LOGIN7.CSP')
driver.implicitly_wait(10)
username_box = driver.find_element_by_id('Username')
username_box.clear()
username_box.send_keys(usr)
password_box = driver.find_element_by_id('Password')
password_box.send_keys(pwd)
login_btn = driver.find_element_by_xpath('//input[#type="submit"]').click()
#------------------------End of Login ------------------------------------
#update contract page ----------
upd_con = driver.find_element_by_partial_link_text('עדכון חוזה').click()
#input latest Contract (F7) ----
input_con = driver.find_element_by_id('HOZ').send_keys(Keys.F7)
#goto receipts --------
receipts_btn = driver.find_element_by_xpath('//input[#name="BKBL"]').click()
Alert(driver).accept()
#input employee name ----- F7 for now later can be changed for custom number -----
work_name = driver.find_element_by_xpath('//input[#name="OVED"]').send_keys(Keys.F7)
#Receipts after username --------------------
receipts_btkn = driver.find_element_by_xpath('//input[#name="BTKBL"]').click()
#2 Seconds SLEEP IS MANDATORY!!! ----
#select Credit Card by Last Value (F7) can be changed to custom value
cc = driver.find_element_by_id('KSM').send_keys(Keys.F7)
#-------------- CANT INTERACT WITH CC WINDOW -----------------
WebDriverWait(driver, 5).until(EC.frame_to_be_available_and_switch_to_it((driver.find_element_by_xpath('//iframe[#id"iFrame"]'))))
#WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#type='text'] and [#id='cvv']"))).click()
sorry for the long post,
thanks in advance
Well i figured it out using pyautogui.
tnx everyone
I want to click an frame source's radio button, but it doesn't work. I think, frame source doesn't have iframe id or name.
my code is this.
import time
from selenium import webdriver
Url='https://www.youtube.com/watch?v=eIStvhR347g'
driver = webdriver.Firefox()
driver.get('https://video-download.online')
driver.find_element_by_id("link").send_keys(Url)
driver.find_element_by_id("submit").click()
time.sleep(5)
#this part is problems... don't working
driver.switch_to_frame(driver.find_element_by_xpath('//iframe'))
driver.find_elements_by_xpath("//*[#type='radio']")[0].click()
driver.find_element_by_xpath(".//button[contains(text(),'Proceed')]").click()
html source is this.
<html>
<head>
<meta charset="utf-8"/>
...
<script>
if (top.location !== location) {
top.location = self.location;
}
if (location.host !== 'video-download.online' && location.host !== 'beta.video-download.online') {
eval("location.href=''+'//'+'video-download.online';");
}
</script>
<meta name="msapplication-tap-highlight" content="no"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no"/>
<meta name="author" content="Luca Steeb"/>
<meta name="theme-color" content="#008679"/>
<meta name="description" content="Download from this uploaded.net premium link generator with highspeed. For free."/>
<meta property="og:title" content="Video-Download.online"/>
<meta property="og:description" content="Download videos, mp3s and playlists from 1979 sites - for free."/>
<meta property="og:type" content="website"/>
<meta property="og:url" content="video-download.online"/>
<meta property="fb:page_id" content="1143492155680932"/>
<meta property="og:image" content="//video-download.online/img/logo.jpg"/>
<link rel="shortcut icon" href="/favicon.ico?1"/>
<link rel="icon" sizes="192x192" href="/img/logo_small.jpg"/>
<link rel="apple-touch-icon" href="/img/apple-touch-icon.png"/>
<link rel="search" type="application/opensearchdescription+xml" title="Video-Download.online" href="/opensearch.xml"/>
<title>
Online Video Download
</title>
<meta name="robots" content="noindex, nofollow"/>
<link rel="stylesheet" href="/css/bt.css"/>
<link rel="stylesheet" href="/css/style.css"/>
<noscript><iframe height=0 src="//www.googletagmanager.com/ns.html?id=GTM-TWMNRP"style=display:none;visibility:hidden width=0></iframe></noscript><script>!function(e,t,a,n,r){e[n]=e[n]||[],e[n].push({"gtm.start":(new Date).getTime(),event:"gtm.js"});var g=t.getElementsByTagName(a)[0],m=t.createElement(a),s="dataLayer"!=n?"&l="+n:"";m.async=!0,m.src="//www.googletagmanager.com/gtm.js?id="+r+s,g.parentNode.insertBefore(m,g)}(window,document,"script","dataLayer","GTM-TWMNRP")</script>
<script>(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');ga('create','UA-54289597-5','auto');ga('send','pageview');</script>
<script>function send(action) { ga('send', 'event', 'button', 'click', 'download-' + action) }</script>
<script async src="/js/jquery.js"></script>
<script>function _(n,i){i?window[n]?i():setTimeout(function(){_(n,i)},100):window.jQuery?window.jQuery(document).ready(function(){n(window.jQuery)}):setTimeout(function(){_(n)},100)}</script>
</head>
<body>
<header class="navbar navbar-info navbar-fixed-top">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle no-waves" data-toggle="collapse" data-target=".navbar-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<div class="nav-brand">
<a class="navbar-brand" href="/">
<img class="logo" src="/img/logo.svg" alt=""/>
<div class="parent">
Video-Download<small>.online</small>
<br/>
<span class="small-text">1979 sites officially supported</span>
<span id="changelog"></span>
</div>
<span class="clear"></span>
</a>
</div>
</div>
<div class="navbar-collapse collapse">
<ul class="nav navbar-nav navbar-right">
<div class="social">
<div class="fb-like fb-nav" data-href="https://www.facebook.com/VideoDownload.online"
data-layout="button_count" data-action="like" data-show-faces="false" data-share="true">
</div>
</div>
<div class="social-close" title="Hide all social buttons">×</div>
<li class=" ">
<a href="/">
Home
</a>
</li>
<li class=" ">
<a href="/sites">
Sites
</a>
</li>
<li class=" ">
<a href="/contact">
Contact
</a>
</li>
<!--html.navItem('/app', 'Mobile App')-->
</ul>
</div>
</div>
</header>
<noscript class="fixed">
<div class="container">Please enable javascript. Video Download and almost all other sites don't work properly
without it.
</div>
</noscript>
<div id="alert" class="alert alert-fixed alert-dismissible m-t-15 hidden">
<div class="container relative">
<span id="alertText"></span>
<span class="close-alert" onclick="_(function() { $('#alert').remove();$('body').removeClass('alert-showing') })" title="close">×</span>
</div>
</div>
<main class="container">
<div>
<h1>Legal</h1>
<div style="width: 50%;" class="center">
</div>
</main>
<script src="/sweetalert/sweetalert.js"></script>
<link rel="stylesheet" href="/sweetalert/sweetalert.css">
<script src="/waves/waves.min.js"></script>
<script src="/js/dropdown.js"></script>
<script src="/js/main.js"></script>
<script src="/js/bootstrap.js"></script>
<script type="text/javascript">/* <![CDATA[ */(function(d,s,a,i,j,r,l,m,t){try{l=d.getElementsByTagName('a');t=d.createElement('textarea');for(i=0;l.length-i;i++){try{a=l[i].href;s=a.indexOf('/cdn-cgi/l/email-protection');m=a.length;if(a&&s>-1&&m>28){j=28+s;s='';if(j<m){r='0x'+a.substr(j,2)|0;for(j+=2;j<m&&a.charAt(j)!='X';j+=2)s+='%'+('0'+('0x'+a.substr(j,2)^r).toString(16)).slice(-2);j++;s=decodeURIComponent(s)+a.substr(j,m-j)}t.innerHTML=s.replace(/</g,'<').replace(/>/g,'>');l[i].href='mailto:'+t.value}}catch(e){}}}catch(e){}})(document);/* ]]> */</script></body>
</html>
frame source is this.
<html>
<head>
<meta charset="utf-8"/>
<meta name="robots" content="noindex, nofollow"/>
...
<link rel="stylesheet" href="/css/bt.css"/>
<link rel="stylesheet" href="/css/style.css"/>
<script src="/js/jquery.js"></script>
<style>body{margin:0!important;}tr{clear:both;cursor:pointer;}td{padding:0 20px 0 0;white-space:nowrap;}.radio{margin-top:5px;margin-bottom:5px;}.small{padding:0;font-size:85%;}#media (max-width: 319px) {table{font-size:12px;}}#media (max-width: 480px) {td{padding:0 5px 0 0;}.small{font-size:50%;}}</style>
</head>
<body>
<form id="format" method="post">
<input name="id" type="hidden" value="isxklqy5blw9of7"/>
<table>
<tr>
<td class="small">
<div class="">
<div class="radio " id="undefined-wrapper">
<label class="text-capitalize" for="undefined">
<input id="undefined" name="format" type="radio" value="undefined" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>I don't care</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td class="small">
<div class="">
<div class="radio " id="160-wrapper">
<label class="text-capitalize" for="160">
<input id="160" name="format" type="radio" value="160" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>256x144</td>
<td></td>
<td>
30 FPS
</td>
<td>23.2 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="133-wrapper">
<label class="text-capitalize" for="133">
<input id="133" name="format" type="radio" value="133" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>426x240</td>
<td></td>
<td>
30 FPS
</td>
<td>51.2 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="134-wrapper">
<label class="text-capitalize" for="134">
<input id="134" name="format" type="radio" value="134" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>640x360</td>
<td></td>
<td>
30 FPS
</td>
<td>65.4 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="135-wrapper">
<label class="text-capitalize" for="135">
<input id="135" name="format" type="radio" value="135" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>854x480</td>
<td></td>
<td>
30 FPS
</td>
<td>138.0 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="136-wrapper">
<label class="text-capitalize" for="136">
<input id="136" name="format" type="radio" value="136" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>1280x720</td>
<td>HD</td>
<td>
30 FPS
</td>
<td>272.2 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="298-wrapper">
<label class="text-capitalize" for="298">
<input id="298" name="format" type="radio" value="298" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>1280x720</td>
<td>HD</td>
<td>
60 FPS
</td>
<td>385.0 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="137-wrapper">
<label class="text-capitalize" for="137">
<input id="137" name="format" type="radio" value="137" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>1920x1080</td>
<td>Full HD</td>
<td>
30 FPS
</td>
<td>535.0 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="299-wrapper">
<label class="text-capitalize" for="299">
<input id="299" name="format" type="radio" value="299" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>1920x1080</td>
<td>Full HD</td>
<td>
60 FPS
</td>
<td>707.0 MB</td>
</tr>
</table>
<button class="btn center" type="submit">Proceed »</button>
</form>
</body>
<script>
$('tr').click(function() {
$(this).find('input').prop('checked', true);
});
</script>
<script src="/waves/waves.min.js"></script>
<script>
Waves.attach('.btn', ['waves-light']);
Waves.init();
</script>
html and frame sources are very long code, you can be check this full code.
html link
frame link
and I tried these codes, but don't work.
driver.switch_to_frame(driver.find_element_by_xpath('//iframe[contains(#name, "frame")]'))
driver.switch_to_frame(driver.find_element_by_tag_name("iframe"))
driver.switch_to_frame(driver.find_element_by_xpath('//iframe'))
driver.switch_to_frame(0)
these iframe switching codes are any exception is nothing, but result is '[]'.
>>> driver.switch_to_frame(driver.find_element_by_xpath('//iframe'))
>>> driver.find_elements_by_xpath("//*[#type='radio']")
[]
>>> driver.find_elements_by_xpath("//*[#type='radio']")[0].click()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range
I needs your help. thanks you.
As I'm seeing on this url, there are multiple iframe present while, you are trying to switch to iframe with it's tagName only which will switch to first find iframe in order while your desired iframe is at 4th index which has no id and name attribute present, so you should try using CSS_SELECTOR with WebDriverWait using EC.frame_to_be_available_and_switch_to_it which will try to wait until desired iframe to be available and then switch to it as below working code :-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe:not([id]):not([name])")))
#now do your stuff to find element inside this iframe
#after doing all stuff inside this iframe switch back to default content for further steps
driver.switch_to_default_content()
Note:- You can also consider below one of these options to switch your desired iframe :-
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, ".//iframe[not(#name) and not(#id)]")))
or
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[src*='selectFormat']")))
or using index :-
wait.until(EC.frame_to_be_available_and_switch_to_it(3))
The iframe is inside <noscript> tag, witch means the driver will see it only when javascript is disabled, as human user would. You can try to disable javascript when creating the driver
fp = webdriver.FirefoxProfile()
fp.set_preference("javascript.enabled", False)
driver = webdriver.Firefox(firefox_profile=fp)
I have created a simple method for switching the frames :
**public static void fn_SwitchToFrame(WebDriver driver, String frame){
if(frame.contains(">")){
String[] List = frame.split(">");
int ListSize = List.length;
for (int i=0; i<=(ListSize-1); i++){
driver.switchTo().frame(List[i]);
System.out.println("Switched to Frame :"+List[i]);
}
} else {
driver.switchTo().frame(frame);
System.out.println("Switched to Frame :"+frame);
}
}**
It works like this :
className.fn_SwitchToFrame(driver, "searchView>searchContent");
This will first switch to frame 'searchView' and then to 'searchContent'.
Hope this helps you.
I want to click an iframe's radio button, but it doesn't work.
my code is this.
import time
from selenium import webdriver
Url='https://www.youtube.com/watch?v=eIStvhR347g'
driver = webdriver.Firefox()
driver.get('https://video-download.online')
driver.find_element_by_id("link").send_keys(Url)
driver.find_element_by_id("submit").click()
time.sleep(5)
#this part is problems... don't working
driver.switch_to_frame(driver.find_element_by_tag_name("iframe"))
driver.find_element_by_css_selector("136-wrapper").click()
driver.find_element_by_link_text("Proceed").click()
html source is this.
<html>
<head>
<meta charset="utf-8"/>
<script>
if (top.location !== location) {
top.location = self.location;
}
if (location.host !== 'video-download.online' && location.host !== 'beta.video-download.online') {
eval("location.href=''+'//'+'video-download.online';");
}
</script>
<meta name="msapplication-tap-highlight" content="no"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no"/>
<meta name="author" content="Luca Steeb"/>
<meta name="theme-color" content="#008679"/>
<meta name="description" content="Download from this uploaded.net premium link generator with highspeed. For free."/>
<meta property="og:title" content="Video-Download.online"/>
<meta property="og:description" content="Download videos, mp3s and playlists from 1979 sites - for free."/>
<meta property="og:type" content="website"/>
<meta property="og:url" content="video-download.online"/>
<meta property="fb:page_id" content="1143492155680932"/>
<meta property="og:image" content="//video-download.online/img/logo.jpg"/>
<link rel="shortcut icon" href="/favicon.ico?1"/>
<link rel="icon" sizes="192x192" href="/img/logo_small.jpg"/>
<link rel="apple-touch-icon" href="/img/apple-touch-icon.png"/>
<link rel="search" type="application/opensearchdescription+xml" title="Video-Download.online" href="/opensearch.xml"/>
<title>
Online Video Download
</title>
<meta name="robots" content="noindex, nofollow"/>
<link rel="stylesheet" href="/css/bt.css"/>
<link rel="stylesheet" href="/css/style.css"/>
<noscript><iframe height=0 src="//www.googletagmanager.com/ns.html?id=GTM-TWMNRP"style=display:none;visibility:hidden width=0></iframe></noscript><script>!function(e,t,a,n,r){e[n]=e[n]||[],e[n].push({"gtm.start":(new Date).getTime(),event:"gtm.js"});var g=t.getElementsByTagName(a)[0],m=t.createElement(a),s="dataLayer"!=n?"&l="+n:"";m.async=!0,m.src="//www.googletagmanager.com/gtm.js?id="+r+s,g.parentNode.insertBefore(m,g)}(window,document,"script","dataLayer","GTM-TWMNRP")</script>
<script>(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');ga('create','UA-54289597-5','auto');ga('send','pageview');</script>
<script>function send(action) { ga('send', 'event', 'button', 'click', 'download-' + action) }</script>
<script async src="/js/jquery.js"></script>
<script>function _(n,i){i?window[n]?i():setTimeout(function(){_(n,i)},100):window.jQuery?window.jQuery(document).ready(function(){n(window.jQuery)}):setTimeout(function(){_(n)},100)}</script>
</head>
<body>
<header class="navbar navbar-info navbar-fixed-top">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle no-waves" data-toggle="collapse" data-target=".navbar-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<div class="nav-brand">
<a class="navbar-brand" href="/">
<img class="logo" src="/img/logo.svg" alt=""/>
<div class="parent">
Video-Download<small>.online</small>
<br/>
<span class="small-text">1979 sites officially supported</span>
<span id="changelog"></span>
</div>
<span class="clear"></span>
</a>
</div>
</div>
<div class="navbar-collapse collapse">
<ul class="nav navbar-nav navbar-right">
<div class="social">
<div class="fb-like fb-nav" data-href="https://www.facebook.com/VideoDownload.online"
data-layout="button_count" data-action="like" data-show-faces="false" data-share="true">
</div>
</div>
<div class="social-close" title="Hide all social buttons">×</div>
<li class=" ">
<a href="/">
Home
</a>
</li>
<li class=" ">
<a href="/sites">
Sites
</a>
</li>
<li class=" ">
<a href="/contact">
Contact
</a>
</li>
<!--html.navItem('/app', 'Mobile App')-->
</ul>
</div>
</div>
</header>
<noscript class="fixed">
<div class="container">Please enable javascript. Video Download and almost all other sites don't work properly
without it.
</div>
</noscript>
<div id="alert" class="alert alert-fixed alert-dismissible m-t-15 hidden">
<div class="container relative">
<span id="alertText"></span>
<span class="close-alert" onclick="_(function() { $('#alert').remove();$('body').removeClass('alert-showing') })" title="close">×</span>
</div>
</div>
<main class="container">
<div>
<h1>Legal</h1>
<div style="width: 50%;" class="center">
</div>
</main>
<script src="/sweetalert/sweetalert.js"></script>
<link rel="stylesheet" href="/sweetalert/sweetalert.css">
<script src="/waves/waves.min.js"></script>
<script src="/js/dropdown.js"></script>
<script src="/js/main.js"></script>
<script src="/js/bootstrap.js"></script>
<script type="text/javascript">/* <![CDATA[ */(function(d,s,a,i,j,r,l,m,t){try{l=d.getElementsByTagName('a');t=d.createElement('textarea');for(i=0;l.length-i;i++){try{a=l[i].href;s=a.indexOf('/cdn-cgi/l/email-protection');m=a.length;if(a&&s>-1&&m>28){j=28+s;s='';if(j<m){r='0x'+a.substr(j,2)|0;for(j+=2;j<m&&a.charAt(j)!='X';j+=2)s+='%'+('0'+('0x'+a.substr(j,2)^r).toString(16)).slice(-2);j++;s=decodeURIComponent(s)+a.substr(j,m-j)}t.innerHTML=s.replace(/</g,'<').replace(/>/g,'>');l[i].href='mailto:'+t.value}}catch(e){}}}catch(e){}})(document);/* ]]> */</script></body>
</html>
frame source is this.
<html>
<head>
<meta charset="utf-8"/>
<meta name="robots" content="noindex, nofollow"/>
<link rel="stylesheet" href="/css/bt.css"/>
<link rel="stylesheet" href="/css/style.css"/>
<script src="/js/jquery.js"></script>
<style>body{margin:0!important;}tr{clear:both;cursor:pointer;}td{padding:0 20px 0 0;white-space:nowrap;}.radio{margin-top:5px;margin-bottom:5px;}.small{padding:0;font-size:85%;}#media (max-width: 319px) {table{font-size:12px;}}#media (max-width: 480px) {td{padding:0 5px 0 0;}.small{font-size:50%;}}</style>
</head>
<body>
<form id="format" method="post">
<input name="id" type="hidden" value="isxklqy5blw9of7"/>
<table>
<tr>
<td class="small">
<div class="">
<div class="radio " id="undefined-wrapper">
<label class="text-capitalize" for="undefined">
<input id="undefined" name="format" type="radio" value="undefined" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>I don't care</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td class="small">
<div class="">
<div class="radio " id="160-wrapper">
<label class="text-capitalize" for="160">
<input id="160" name="format" type="radio" value="160" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>256x144</td>
<td></td>
<td>
30 FPS
</td>
<td>23.2 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="133-wrapper">
<label class="text-capitalize" for="133">
<input id="133" name="format" type="radio" value="133" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>426x240</td>
<td></td>
<td>
30 FPS
</td>
<td>51.2 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="134-wrapper">
<label class="text-capitalize" for="134">
<input id="134" name="format" type="radio" value="134" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>640x360</td>
<td></td>
<td>
30 FPS
</td>
<td>65.4 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="135-wrapper">
<label class="text-capitalize" for="135">
<input id="135" name="format" type="radio" value="135" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>854x480</td>
<td></td>
<td>
30 FPS
</td>
<td>138.0 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="136-wrapper">
<label class="text-capitalize" for="136">
<input id="136" name="format" type="radio" value="136" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>1280x720</td>
<td>HD</td>
<td>
30 FPS
</td>
<td>272.2 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="298-wrapper">
<label class="text-capitalize" for="298">
<input id="298" name="format" type="radio" value="298" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>1280x720</td>
<td>HD</td>
<td>
60 FPS
</td>
<td>385.0 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="137-wrapper">
<label class="text-capitalize" for="137">
<input id="137" name="format" type="radio" value="137" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>1920x1080</td>
<td>Full HD</td>
<td>
30 FPS
</td>
<td>535.0 MB</td>
</tr><tr>
<td class="small">
<div class="">
<div class="radio " id="299-wrapper">
<label class="text-capitalize" for="299">
<input id="299" name="format" type="radio" value="299" onchange="" undefined required />
<span class=circle></span><span class=check></span>
<span class="grey-text"></span>
</label>
</div>
</div>
</td>
<td>1920x1080</td>
<td>Full HD</td>
<td>
60 FPS
</td>
<td>707.0 MB</td>
</tr>
</table>
<button class="btn center" type="submit">Proceed »</button>
</form>
</body>
<script>
$('tr').click(function() {
$(this).find('input').prop('checked', true);
});
</script>
<script src="/waves/waves.min.js"></script>
<script>
Waves.attach('.btn', ['waves-light']);
Waves.init();
</script>
I tried these codes, but don't work.
driver.switch_to_frame(driver.find_element_by_xpath('//iframe[contains(#name, "frame")]'))
driver.switch_to_frame(driver.find_element_by_tag_name("iframe"))
driver.switch_to_frame(driver.find_element_by_xpath('//iframe'))
driver.switch_to_frame(0)
these iframe switching codes are any exception is nothing, but result is '[]'.
>>> driver.switch_to_frame(driver.find_element_by_xpath('//iframe'))
>>> driver.find_elements_by_xpath("//*[#type='radio']")
[]
>>> driver.find_elements_by_xpath("//*[#type='radio']")[0].click()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range
thanks for your help
(Assuming provided HTML is correct) actually your both locator is wrong to locate radio button element as well as proceed button element.
driver.find_element_by_css_selector("136-wrapper").click()
Using css_selector id locator used with #, so correct css_selector with id should be #136-wrapper
But this locator will be locate to <div> element while you want <input id="136" name="format" type="radio" value="136" onchange="" undefined required /> element, so here you can locate radio button and click using find_element_by_id() as :-
driver.find_element_by_id("136").click()
driver.find_element_by_link_text("Proceed").click()
find_element_by_link_text() works to locate only <a> anchor element with text while you want to locate <button> element, so this locator wouldn't work.
To locate Proceed button you should try using find_element_by_xpath() as :-
driver.find_element_by_xpath(".//button[contains(text(),'Proceed')]").click()
I want to enter some text (login, password) inside a page with frames. The structure is something like:
<html xmlns="http://www.w3.org/1999/xhtml">
<head>Whatever</head>
<body id="Body">
<form id="frmM" onsubmit="javascript:return whatever();" action="Login.aspx" method="post">
<div id="alldata">
<div id="header">
<iframe id="Login_SSL" scrolling="no" frameborder="1" src="https://www.whatever.com/User/LoginFrame.aspx?redir=/User/Login.aspx">
<html xmlns="http://www.w3.org/1999/xhtml">
<body onload="refreshParent()">
<form id="form1" action="LoginFrame.aspx?redir=%2fUser%2fLogin.aspx" method="post">
<div id="loginRow" class="loginMenuRow" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'cmdLogin')">
<input id="cmdLogin" type="submit" tabindex="3" onclick="aspnetForm.target ='_top';" value="Login" name="cmdLogin">
<input id="Password" class="textbox" type="password" tabindex="2" name="Password">
<span id="lblPassword" class="loginMenu">Password:</span>
</div>
</form>
</body>
</html>
</iframe>
</div>
</div>
</form>
</body>
</html>
If I try to find the form, I see that only the top level one is available.
>>br.select_form("form1")
FormNotFoundError: no form matching name 'form1'
>>[f.attrs['id'] for f in br.forms()]
['frmM']
How do I go about logging in to this site?
2 options:
Access the URL within the iframe directly and then do your form login
Login to the site from your browser and use an extension like Firebug
to track the posted data. Then replicate this request to login
automatically.