I have been working on these codes for quiet some time (even tried to use chatgpt's help lol), but I still haven't completed it because of this last problem. These codes' main purpose is for user to be able to input a song name in the html5 page, then the name should be sent to my python code, which would generate similar song names and send them to my html5 code, which would display it for user. Unfortunately, I have been running into some obstacles. For example, right now, when I input the song name, instead of getting recommended song names, I get my C:/ files displayed instead. Do you have any ideas how to fix it? Here are my codes:
,,index.html":
<!DOCTYPE html>
<html>
<head>
<title>Song4u</title>
<meta charset="utf-8">
<link rel="stylesheet" type="text/css" href="style.css">
<script type="text/javascript" href="script.js"></script>
<link href="https://fonts.googleapis.com/css2?family=Roboto+Mono:wght#100&display=swap" rel="stylesheet">
</head>
<body>
<h3><header>Song4u</header></h3><br>
<br>
<h1>FIND YOUR NEW FAVOURITE SONG!</h1> <br>
<h2>Insert your youtube link and we will show you songs that suit your vibe the most!</h2>
<form action="/" method="post">
<label for="songname">Songname:</label>
<input type="text" id="songname" name="songname">
<input type="submit" value="Submit">
</form>
</body>
</html>
"gptcode.py":
from flask import Flask, request, render_template
import subprocess
import sys
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
app = Flask(__name__, template_folder='templates')
def install(package):
subprocess.check_call([sys.executable, "-m", "pip", "install", package])
install("selenium")
#app.route('/', methods=['GET', 'POST'])
def index():
if request.method == 'POST':
songname = request.form['songname']
options = Options()
options.headless = True
options.add_argument('--disable-extensions')
options.add_argument('--disable-gpu')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--no-sandbox')
driver = webdriver.Chrome(options=options)
driver.get("https://www.chosic.com/playlist-generator/")
sleep(1)
driver.find_element(By.XPATH, ('//*[text()="AGREE"]')).click()
driver.find_element(By.NAME, 'q').send_keys(songname)
sleep(3)
driver.find_element(By.XPATH, ('//*[#id="hh1"]')).click()
sleep(2)
driver.find_element(By.XPATH, ('//*[#id="generate-button"]')).click()
sleep(2)
siul1 = driver.find_element(By.XPATH, ('//*[#id="result"]/div/div[4]/div[1]/span')).get_attribute('textContent')
siul2 = driver.find_element(By.XPATH, ('//*[#id="result"]/div/div[7]/div[1]/span')).get_attribute('textContent')
siul3 = driver.find_element(By.XPATH, ('//*[#id="result"]/div/div[10]/div[1]/span')).get_attribute('textContent')
siul4 = driver.find_element(By.XPATH, ('//*[#id="result"]/div/div[13]/div[1]/span')).get_attribute('textContent')
siul5 = driver.find_element(By.XPATH, ('//*[#id="result"]/div/div[16]/div[1]/span')).get_attribute('textContent')
return render_template('index.html', siul1=siul1, siul2=siul2, siul3=siul3, siul4=siul4, siul5=siul5 )
if __name__ == '__main__':
app.run(debug=True)
(btw these codes are solely just for a school project)
The selenium part of the code was made by my friend, so I don't really have that much knowledge there, the problem might be hiding in there, I have no idea, but for what I have gathered till now, I think that it might be that the html5 code doesn't really understand it's purpose and the python code fails to send the answers to it?
I tried playing with "post" and "get" commands, with whole <form> code part, getting help from chatgpt, but the only change that I'd receive would be my python code displayed instead of C:/ files like in the one displayed :)
(Edit):
That's what I get when I insert a song's name in the tab:
Related
I did several hours of research and asked a bunch of people on fiverr who all couldn't solve a a specific problem I have.
I installed Selenium and tried to access a Website. Unfortunately the site won't allow a specific request and doesn't load the site at all. However, if I try to access the website with my "normal" Chrome Browser, it works fine.
I tried several things such as:
Different IP's
Deleting Cookies
Incognito Mode
Adding different UserAgents
Hiding features which might reveal that a Webdriver is being used
Nothing helped.
Here is a Screenshot of the Error I'm receiving:
And here is the very simple script I'm using:
# coding: utf8
from selenium import webdriver
url = 'https://registrierung.gmx.net/'
# Open ChromeDriver
driver = webdriver.Chrome();
# Open URL
driver.get(url)
If anyone has a solution for that I would highly appreciate it.
I'm also willing to give a big tip if someone could help me out here.
Thanks a lot!
Stay healthy everyone.
I took your code modified with a couple of arguments and executed the test. Here are the observations:
Code Block:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://registrierung.gmx.net/")
print(driver.page_source)
Console Output:
<html style="" class=" adownload applicationcache blobconstructor blob-constructor borderimage borderradius boxshadow boxsizing canvas canvastext checked classlist contenteditable no-contentsecuritypolicy no-contextmenu cors cssanimations csscalc csscolumns cssfilters cssgradients cssmask csspointerevents cssreflections cssremunit cssresize csstransforms3d csstransforms csstransitions cssvhunit cssvmaxunit cssvminunit cssvwunit dataset details deviceorientation displaytable display-table draganddrop fileinput filereader filesystem flexbox fullscreen geolocation getusermedia hashchange history hsla indexeddb inlinesvg json lastchild localstorage no-mathml mediaqueries meter multiplebgs notification objectfit object-fit opacity pagevisibility performance postmessage progressbar no-regions requestanimationframe raf rgba ruby scriptasync scriptdefer sharedworkers siblinggeneral smil no-strictmode no-stylescoped supports svg svgfilters textshadow no-time no-touchevents typedarrays userselect webaudio webgl websockets websqldatabase webworkers datalistelem video svgasimg datauri no-csshyphens"><head>
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="-1">
<meta http-equiv="CacheControl" content="no-cache">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<link rel="shortcut icon" href="data:;base64,iVBORw0KGgo=">
<script type="text/javascript">
(function(){
window["bobcmn"] = "10111111111010200000005200000005200000006200000001249d50ae8200000096200000000200000002300000000300000000300000006/TSPD/300000008TSPD_10130000000cTSPD_101_DID300000005https3000000b0082f871fb6ab200097a0a5b9e04f342a8fdfa6e9e63434256f3f63e9b3885e118fdacf66cc0a382208ea9dc3b70a28002d902f95eb5ac2e5d23ffe409bb24b4c57f9cb8e1a5db4bcad517230d966c75d327f561cc49e16f4300000002TS200000000200000000";
.
.
<script type="text/javascript" src="/TSPD/082f871fb6ab20009afc88ee053e87fea57bf47d9659e73d0ea3c46c77969984660358739f3d19d0?type=11"></script>
<script type="text/javascript">
(function(){
window["blobfp"] = "01010101b00400000100e803000000000d4200623938653464333234383463633839323030356632343563393735363433343663666464633135393536643461353031366131633362353762643466626238663337210068747470733a2f2f72652e73656375726974792e66356161732e636f6d2f72652f0700545350445f3734";window["slobfp"] = "08c3194e510b10009a08af8b7ee6860a22b5726420e697e4";
})();
</script>
<script type="text/javascript" src="/TSPD/082f871fb6ab20009afc88ee053e87fea57bf47d9659e73d0ea3c46c77969984660358739f3d19d0?type=12"></script>
<noscript>Please enable JavaScript to view the page content.<br/>Your support ID is: 11993951574422772310.</noscript>
</head><body>
<style>canvas {display:none;}</style><canvas width="800" height="600"></canvas></body></html>
Browser Snapshot:
Conclusion
From the Page Source it's quite clear that Selenium driven ChromeDriver initiated google-chrome Browsing Context gets detected and the navigation is blocked.
I could have dug deeper and provide some more insights but suprisingly now even manually I am unable to access the webpage. Possibly my IP is black-listed now. Once my IP gets whitelisted I will provide more details.
References
You can find a couple of relevant detailed discussions in:
Can a website detect when you are using selenium with chromedriver?
Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
I'm trying to scrape a site that needs login information, and after hours of trying to figure out why I keep getting "Login failed", I believe it is simply because the "Log in" or "Submit" button is not actually getting clicked. I realized this by saving a screenshot of the browser right when it "fails". My username and password are filled into the fields.
I've tried things like wait, elementScrollBehavior, nothing seems to work. I'd really appreciate some help with this! Code below.
def load(self):
global browser
DesiredCapabilities.PHANTOMJS["elementScrollBehavior"] = 1
#browser field
browser = webdriver.PhantomJS()
wait = WebDriverWait(browser, 10)
#browser = webdriver.Firefox()
#browser = webdriver.Chrome()
loginId = self.id
password = self.pw
browser.get('https://link.example.com')
browser.find_element_by_id('cf-login').send_keys(loginId)
browser.find_element_by_id('password').send_keys(password)
browser.find_element_by_name('submit').click()
#wait.until(EC.presence_of_element_located((By.ID, "crefli_HC_SSS_STUDENT_CENTER")))
try:
if browser.find_element_by_id('crefli_HC_SSS_STUDENT_CENTER'):
#return login status
return True
else:
return False
except:
print('element not found on page')
print(browser.current_url)
#browser.save_screenshot('~/Desktop/screen2.png')
HTML of form:
<form name="loginform" action="/oam/server/auth_cred_submit" method="post">
<div class="nonfloat-box">
Username:
<input type="text" id="cf-login" name="username" class="username inputbox" autocomplete="OFF">
</div>
<div class="float-box">
Password:
<input id="password" name="password" type="password" class="password inputbox" autocomplete="OFF">
</div>
<input type="image" src="https://www.cuny.edu/site/citizencuny/cunyfirst-login/loginbutton.jpg" onclick="javascript: return signon_validate()" alt="Submit" name="submit">
</form>
I believe I need to SOMEHOW get that bit of javascript to run. But HOW?
UPDATE: Selenium has a submit() method that automatically submits the <form> in HTML. Even using this, it does not work. As you can see in the HTML, it IS a form. At this point I do not know what else to try.
Please try this, hope it helps
from selenium.webdriver.common.keys import Keys
driver.find_element_by_name('submit').send_keys(Keys.RETURN)
(or)
driver.find_element_by_name('submit').send_keys(Keys.ENTER)
I have a feeling 'submit' is not being found by find_element_by_name. Try find_element_by_xpath(//*[#name='submit'])
HTML is needed to make a more accurate determination, but I usually use enter key to submit forms, sometimes javascript messes up the ability to submits by click and a simple enter usually does the trick
from selenium.webdriver.common.keys import Keys
def load(self):
(...)
browser.find_element_by_id('password').send_keys(password)
browser.find_element_by_id('password').send_keys(Keys.ENTER)
(...)
Otherwise, make sure you mean find_element_by_name and not find_element_by_tag_name.
Use xpath, Please try this
//input[#name='submit']
or
//input[contains(#name,'submit')AND contains(#alt,'Submit')]
You can use submit() to submit the form. It needs to be sent to the <form> tag
browser.find_element_by_id('cf-login').send_keys(loginId)
browser.find_element_by_id('password').send_keys(password)
browser.find_element_by_name('loginform').submit()
If this doesn't work you can use JavaScript click as a work around
submit = browser.find_element_by_name('submit')
browser.execute_script("arguments[0].click();", submit)
You can also try sending Enter
from selenium.webdriver.common.keys import Keys
browser.find_element_by_id('cf-login').send_keys(loginId)
password_field = browser.find_element_by_id('password')
password_field.send_keys(password)
password_field.send_keys(Keys.RETURN)
#OR
password_field.send_keys(Keys.ENTER)
When you use selenium do automatic testing or scraping ，I suggest you use the method
webdriver.find_element_by_xpath(xpathString)
because you can check the xpathwebbroser's console
Try this command on console:
$x('xpathString')
I wuold like to handle alerts with Python. What I wuold like to do is:
Open a url
Submit a form or click some links
Check if an alert occurs in the new page
I made this with Javascript using PhantomJS, but I would made even with Python.
Here is the javascript code:
file test.js:
var webPage = require('webpage');
var page = webPage.create();
var url = 'http://localhost:8001/index.html'
page.onConsoleMessage = function (msg) {
console.log(msg);
}
page.open(url, function (status) {
page.evaluate(function () {
document.getElementById('myButton').click()
});
page.onConsoleMessage = function (msg) {
console.log(msg);
}
page.onAlert = function (msg) {
console.log('ALERT: ' + msg);
};
setTimeout(function () {
page.evaluate(function () {
console.log(document.documentElement.innerHTML)
});
phantom.exit();
}, 1000);
});
file index.html
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title></title>
<meta charset="utf-8" />
</head>
<body>
<form>
<input id="username" name="username" />
<button id="myButton" type="button" value="Page2">Go to Page2</button>
</form>
</body>
</html>
<script>
document.getElementById("myButton").onclick = function () {
location.href = "page2.html";
};
</script>
file page2.html
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title></title>
<meta charset="utf-8" />
</head>
<body onload="alert('hello')">
</body>
</html>
This works; it detects the alert on page2.html.
Now I made this python script:
test.py
import requests
from test import BasicTest
from selenium import webdriver
from bs4 import BeautifulSoup
url = 'http://localhost:8001/index.html'
def main():
#browser = webdriver.Firefox()
browser = webdriver.PhantomJS()
browser.get(url)
html_source = browser.page_source
#browser.quit()
soup = BeautifulSoup(html_source, "html.parser")
soup.prettify()
request = requests.get('http://localhost:8001/page2.html')
print request.text
#Handle Alert
if __name__ == "__main__":
main();
Now, how can I check if an alert occur on page2.html with Python? First I open the page index.html, then page2.html.
I'm at the beginning, so any suggestions will be appreciate.
p.s.
I also tested webdriver.Firefox() but it is extremely slow.
Also i read this question : Check if any alert exists using selenium with python
but it doesn't work (below is the same previous script plus the solution suggested in the answer).
.....
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
....
def main():
.....
#Handle Alert
try:
WebDriverWait(browser, 3).until(EC.alert_is_present(),
'Timed out waiting for PA creation ' +
'confirmation popup to appear.')
alert = browser.switch_to.alert()
alert.accept()
print "alert accepted"
except TimeoutException:
print "no alert"
if __name__ == "__main__":
main();
I get the error :
"selenium.common.exceptions.WebDriverException: Message: Invalid
Command Method.."
PhantomJS uses GhostDriver to implement the WebDriver Wire Protocol, which is how it works as a headless browser within Selenium.
Unfortunately, GhostDriver does not currently support Alerts. Although it looks like they would like help to implement the features:
https://github.com/detro/ghostdriver/issues/20
You could possibly switch to the javascript version of PhantomJS or use the Firefox driver within Selenium.
from selenium import webdriver
from selenium.common.exceptions import NoAlertPresentException
if __name__ == '__main__':
# Switch to this driver and switch_to_alert will fail.
# driver = webdriver.PhantomJS('<Path to Phantom>')
driver = webdriver.Firefox()
driver.set_window_size(1400, 1000)
driver.get('http://localhost:8001/page2.html')
try:
driver.switch_to.alert.accept()
print('Alarm! ALARM!')
except NoAlertPresentException:
print('*crickets*')
Right now, my company has about 40 information boards scattered throughout the buildings with information relevant to each area. Each unit is a small linux based device that is programmed to launch an RDP session, log in with a user name and password, and pull up the appropriate powerpoint and start playing. The boards would go down every 4 hours for about 5 minutes, copy over a new version of a presentation (if applicable) and restart.
We now have "demands" for live data. Unfortunately I believe powerpoint will no longer be an option as we used the Powerpoint viewer, which does not support plugins. I wanted to use Google Slides, but also have the restriction that we cannot have a public facing service like Google Drive, so there goes that idea.
I was thinking of some kind of way to launch a web browser and have it rotate through a list of specified webpages (perhaps stored in a txt or csv file). I found a way to launch Firefox and have it autologin to OBIEE via python:
#source: http://obitool.blogspot.com/2012/12/automatic-login-script-for-obiee-11g_12.html
import unittest
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
# Hardcoding this information is usually not a good Idea!
user = '' # Put your user name here
password = '' # Put your password here
serverName = '' # Put Host name here
class OBIEE11G(unittest.TestCase):
def setUp(self):
# Create a new profile
self.fp = webdriver.FirefoxProfile()
self.fp.set_preference("browser.download.folderList",2)
self.fp.set_preference("browser.download.manager.showWhenStarting",False)
# Associate the profile with the Firefox selenium session
self.driver = webdriver.Firefox(firefox_profile=self.fp)
self.driver.implicitly_wait(2)
# Build the Analytics url and save it for the future
self.base_url = "http://" + serverName + ":9704/analytics"
def login(self):
# Retreive the driver variables created in setup
driver = self.driver
# Goto the loging page
driver.get(self.base_url + "/")
# The 11G login Page has following elements on it
driver.find_element_by_id("sawlogonuser").clear()
driver.find_element_by_id("sawlogonuser").send_keys(user)
driver.find_element_by_id("sawlogonpwd").clear()
driver.find_element_by_id("sawlogonpwd").send_keys(password)
driver.find_element_by_id("idlogon").click()
def test_OBIEE11G(self):
self.login()
#
if __name__ == "__main__":
unittest.main()
If I can use this, I would just need a way to rotate to a new webpage every 30 seconds. Any ideas / recommendations?
You could put a simple javascript snippet on each page that waits a specified time then redirects to the new page. This has the advantage of simple implementation, however it may be annoying to maintain this over many html files.
The other option is to write your copy in a markdown file, then have a single html page that rotates through a list of files and renders and displays the markdown. You would then update the data by rewriting the markdown files. It wouldn't be exactly live, but if 30 second resolution is ok you can get away with it. Something like this for the client code:
HTML
<!DOCTYPE html>
<html lang="en">
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Message Board</title>
<link rel="stylesheet" type="text/css" href="/css/style.css">
</head>
<body>
<div id="content" class="container"></div>
<!-- jQuery -->
<script src="//code.jquery.com/jquery-2.1.4.min.js" defer></script>
<!-- markdown compiler https://github.com/chjj/marked/ -->
<script src="/js/marked.min.js" defer></script>
<script src="/js/main.js" defer></script>
</body>
</html>
And the javascript
// main.js
// the markdown files
var sites = ["http://mysites.com/site1.md", "http://mysites.com/site2.md", "http://mysites.com/site3.md", "http://mysites.com/site4.md"]
function render(sites) {
window.scrollTo(0,0);
//get first element and rotate list of sites
var file = sites.shift();
sites.push(file);
$.ajax({
url:file,
success: function(data) { $("#content").html(marked(data)); },
cache: false
});
setTimeout(render(sites), 30000);
}
// start the loop
render(sites);
You can then use any method you would like to write out the markdown files.
I'm stuck writing a Selenium WebDriver script for Instagram web login. I think I switched to the appropriate iframe but WebDriver keeps timing out when it should locate the user input field.
Relevant source from Instagram site:
https://instagram.com/accounts/login/
<iframe class="hiFrame" data-reactid=".0.0.0.1.0.1.0.0.$frame" src="https://instagram.com/accounts/login/ajax/?targetOrigin=https%3A%2F%2Finstagram.com" scrolling="no" seamless="">
<!DOCTYPE html>
<html class="hl-en not-logged-in " lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<body class="LoginFormChrome ">
<div class="LoginFormPage" data-reactid=".0">
<form data-reactid=".0.0">
<p class="lfField" data-reactid=".0.0.0">
<label class="lfFieldLabel" data-reactid=".0.0.0.0">
<input class="lfFieldInput" type="text" data-reactid=".0.0.0.1" value="" autocorrect="false" autocapitalize="false" maxlength="30" name="username">
</p>
Source from Selenium script:
login_url = 'https://instagram.com/accounts/login/'
profile_url = '<path_firefix_profile>'
user = '<user_name>'
#login
my_profile = FirefoxProfile(profile_url)
self.driver = webdriver.Firefox(my_profile)
self.driver.get(login_url)
self.driver.implicitly_wait(10)
my_iframe = self.driver.find_element_by_css_selector("iframe.hiFrame")
#my_iframe = self.driver.find_element_by_css_selector("iframe:nth-of-type(1)")
#my_iframe = self.driver.find_element_by_tag_name("iframe")
self.driver.switch_to_frame(my_iframe)
try:
element = WebDriverWait(self.driver, 30).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input[name='username']")))
user_input = self.driver.find_element_by_css_selector("input[name='username']")
user_input.send_keys(user)
finally:
print('user name input appeared')
Results:
This error results from WebDriver:
File "instagram_firefox.py", line 51, in setUp
element = WebDriverWait(self.driver, 45).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input[name='username']")))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/support/wait.py", line 71, in until
raise TimeoutException(message)
I tried to verify that the css selector for the input field was correct. On the page, https://instagram.com/accounts/login/, FireFox FireFinder does not recognize the css selector that I used. But if I open another tab with the source of the iframe, https://instagram.com/accounts/login/ajax/?targetOrigin=https%3A%2F%2Finstagram.com, then Firefinder recognizes the css selector that I used. Does this mean I need to manually get the url of the iframe source or should that be done automatically when WebDriver switches to the iframe?
We should wait first for div spinner element to disappear, then we can retrieve that iframe you need:
user = "user"
self.driver.get("https://instagram.com/accounts/login/")
#Wait for spinner to disappear
WebDriverWait(self.driver, 10).until(EC.invisibility_of_element_located((By.CSS_SELECTOR, "div.liSpinnerLayer")))
#Get iframe and switch to it
my_iframe = self.driver.find_element_by_css_selector("iframe.hiFrame")
self.driver.switch_to_frame(my_iframe)
element = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input[name='username']")))
element.send_keys(user)