Currently, I have a code that uses remote selenium server to look for ads and save their images, or take screenshots in case their image isn't easily available. As of now I use:
iframes = list(driver.find_elements_by_tag_name('iframe'))
to obtain each iframe and then loop through them doing iframe.screenshot(filename). This works fine for the most part, but some pages take bad screenshots (they screenshot other aprts of the page or sometimes they have an element blocking the ad).
So what I'm trying to do is use the newly implemented chrome dev tools protocol on selenium to isolate each element, so it's the only visible element on the screen. This should be possible since I'm able enter chrome's dev tools, right click my desired element and "enter isolation mode". I just don't know where to begin with the code because the documentation is pretty messy and no one seems to have posted about this previously.
I've found these methods: https://chromedevtools.github.io/devtools-protocol/tot/Overlay/#method-setShowIsolatedElements
https://chromedevtools.github.io/devtools-protocol/tot/Page/#method-createIsolatedWorld
but I'm not sure how to even test them.
If anyone's got any experience working wiht this or can think of a better solution, I'd greatly apreciate your help.
Related
I'm trying to get a specific data from a website, but this is a little bit complicated to understand so here is some images.
So, first, I'm on this page,
Image1
then I click on the icon in the middle and something pop,
popup
then I have to click on this,
almost there
And finally I land here
arrival
And I want to get all the names of the people here
So, my question is, is there a way to get directly this list with a requests ?
If yes, how do i have do to ? I can't find the URL of this kind of pop up and I'm a complete beginner with requests and all this kind of things..
(To get the name, I have to be connected on my account by the way)
So, since I don't know how to access to the pop-up windows, this is the only code I got :
import requests
x = requests.get('https://www.tiktok.com/#programm___r?lang=en', headers={'User-Agent':'test'})
print(x.text)
I checked what it prints, and i didn't see a sign of the pop-up window
you can get some sort of network interception tool like Burpsuite and watch the network traffic that comes through each time you click on each link along the way to your final destination, this should give you an endpoint you may be able to send your request too. I think this network information should also be available in the browser tools but I'm not sure. A potential issue here is that usually tokens and other information has to be passed down the chain along the way, which might make scripting something like this too hard.
So aside from that, with browser automation software like selenium, you could automate the process of getting to that point on the page, and be able to pull out the list you want once you're there. I've used selenium myself and it's really usable and well documented!
url:https://www.zhihu.com/question/305744720/answer/557418746
use selenium can not reply answer,only human
options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
browser = webdriver.Chrome(executable_path='.\chromedriver.exe',chrome_options=options)
button_li = self.browser.find_elements_by_class_name('Button--blue')
if len(button_li) > 2:
print(len(button_li))
button_ele = button_li[4]
button_ele.click()
time.sleep(random.uniform(0.5, 3))
browser.find_element_by_css_selector('div.AnswerForm-editor').click()
time.sleep(random.uniform(0.5, 2))
js="""
var div=document.getElementsByClassName('public-DraftStyleDefault-block')
var text =document.createTextNode("君");
div[0].firstChild.appendChild(text)
"""
self.browser.execute_script(js)
browser.find_element_by_css_selector('Button.Button.AnswerForm-submit').click()
Summary of problem:
My problem is that I wrote the content to the answer box successfully, but I was identified as a machine. After that, my actions on the page seemed to stop working. How can I avoid being identified as a machine so that I can still use selenium to select my element?
I'm not sure where you are trying to select the submit button, but the following selector worked for me:
browser.find_element_by_css_selector('Button.Button.AnswerForm-submit').click()
With respect to being detected as a 'machine', it's not easy to avoid that.
There are several different ways they can detect you.
That said, here's one thing I found that can avoid some of the Selenium detection attempts. One thing they look for document variables called $cdc_ and $wdc_ that selenium uses, and for Chrome it would be $cdc_. Here is what I suggest you try:
Download a hex editor if you don't already have one. I used one from here.
open your chromedriver.exe in the hex editor.
Use the Search functionality to find any instance of $cdc_ or $wdc_ and replace basically any other string ending in an underscore. Myself, I found just one instance of $cdc_, and it looked like this:
'$cdc_asdjflasutopfhvcZLmcfl_'
I simply replaced it with 'Random_'
Hopefully this works and now you can traverse the site unimpeded. If not, try some of the following; the only problem with this is that it might break your chromedriver file so that the tests no longer work. But it could be worth a try, and if it does break you can easily download a fresh one.
Search the document for any usage of the words 'selenium', 'webdriver', or 'chromedriver' and delete them. These is another way that a site can tell you are using selenium.
Let me know if any of this helps or you have any questions. It's hard for me to come up with a concrete answer because I don't know how exactly the site is detecting selenium.
The problem: I want to write a Python script that takes a screenshot of a website I have opened in a browser each time it loads.
The thing is that I have a website where there are like 300 exam questions which I can get through, try each one of them and I will have the correction when I submit my answer. I will not have access to this questionnaire after a certain date, but I want to keep the questions (which I could write down, but laziness is strong in me, and want to learn Python).
The "attempt": I thought of doing a simple Python script with imgkit to take the screenshots. I'm opened to other suggestions, as imgkit was the first thing I saw while looking for this, and the code looks plain and simple to me:
import imgkit
imgkit.from_url('http://webpage.com', 'out.jpg')
But I have to provide the url for each webpage, and that will be more tedious than taking a screenshot with OS features, thus I want to automatize it.
The questions:
There is a way to make Python monitor a browser tab and take a screenshot each time it reloads (that will be when a new question appears)?
Or maybe get the tab's URL to pass it to imgkit and take the screenshot.
Another thing that I saw is that imgkit can generate a "screenshot" from a HTML file. Can Python download the HTML code from a tab I have open in my browser?
Selenium is your friend here. It is a framework designed for testing but it will make what you want really easy.
Selenium allows you to spin-up a web browser and control it. So you can instruct it to go to the web address you want and then do things. Normally you would instruct it to click here, write in a form, etc.
In your case you only want it to open a certain address, take a screenshot, go the the next address and repeat.
Here you have a tutorial on how to do exactly what you want.
The specific code is:
from selenium import webdriver
#1. Get the driver to manage the web-browser you choose
driver = webdriver.Chrome()
#2. Go the the webadress you want
driver.get('https://python.org')
#3. Take a screenshot
driver.save_screenshot("screenshot.png")
driver.close()
PS: In order for the tutorial to run you will need to have installed the web driver for Selenium to be able to spin-up and run Chrome. Here are the instructions for that.
I'm new to coding and trying to use Selenium with Python to click through a website and fill a shopping cart. I've got things working well except for the random ForeSee survey popup. When it appears (and it doesn't always appear in the same location), my code stops working at that point.
I read the ForeSee documentation and it says "...when the invitation is displayed, the fsr.r...cookie is dropped. This cookie prevents a user from being invited again for X days (default 90)."
Hoping for a quick fix, I created a separate Firefox profile and ran through the website and got the ForeSee pop up invitation--no more pop up when manually using that profile. But I still get the pop up when using Selenium.
I used this code:
fp = webdriver.FirefoxProfile('C:\path\to\profile')
browser = webdriver.Firefox(firefox_profile=fp)
EDIT: I got the cookie working. I was using the Local folder instead of the Roaming folder in C:\path\to\profile. Using the roaming folder solved the problem.
My question edited to delete the part about the cookie not working:
Can someone suggest code to permanently handle the ForeSee pop up that appears randomly and on random pages?
I'm using using Protractor with JS, so I can't give you actual code to handle the issue, but I can give you an idea how to approach this.
In a nutshell
When following script is executed in the browser's console -
window.FSR.setFSRVisibility(true);
it makes ForeSee popup appear behind the rest of HTML elements. And doesn't affect UI tests anymore
So my protractor script will look like so
await browser.executeScript(
`window.FSR.setFSRVisibility(true);`
);
Theory
So ForeSee is one of those services that can be integrated with any web app, and will be pulling js code from their API and changing HTML of your app, by executing the code on the scope of the website. Another example of such company is walkme
Obviously in modern world, if these guys can overlay a webpage, they should have a configuration to make it optional (at least for lower environments) and they actually do. What I mentioned as a solution came from this page. But assuming they didn't have such option, one could reach out their support and ask how to workaround their popups. Even if they didn't have such option they would gladly consider it as a feature for improvement.
Alright, I'm confused. So I want to scrape a page using Selenium Webdriver and Python. I've recorded a test case in the Selenium IDE. It has stuff like
Command Taget
click link=14
But I don't see how to run that in Python. The desirable end result is that I have the source of the final page.
Is there a run_test_case command? Or do I have to write individual command lines? I'm rather missing the link between the test case and the actual automation. Every site tells me how to load the initial page and how to get stuff from that page, but how do I enter values and click on stuff and get the source?
I've seen:
submitButton=driver.find_element_by_xpath("....")
submitButton.click()
Ok. And enter values? And get the source once I've submitted a page? I'm sorry that this is so general, but I really have looked around and haven't found a good tutorial that actually shows me how to do what I thought was the whole point of Selenium Webdriver.
I've never used the IDE. I just write my tests or site automation by hand.
from selenium import webdriver
browser = webdriver.Firefox()
browser.get("http://www.google.com")
print browser.page_source
You could put that in a script and just do python wd_script.py or you could open up a Python shell and type it in by hand, watch the browser open up, watch it get driven by each line. For this to work you will obviously need Firefox installed as well. Not all versions of Firefox work with all versions of Selenium. The current latest versions of each (Firefox 19, Selenium 2.31) do though.
An example showing logging into a form might look like this:
username_field = browser.find_element_by_css_selector("input[type=text]")
username_field.send_keys("my_username")
password_field = browser.find_element_by_css_selector("input[type=password]")
password_field.send_keys("sekretz")
browser.find_element_by_css_selector("input[type=submit]").click()
print browser.page_source
This kind of stuff is much easier to write if you know css well. Weird errors can be caused by trying to find elements that are being generated in JavaScript. You might be looking for them before they exist for instance. It's easy enough to tell if this is the case by putting in a time.sleep for a little while and seeing if that fixes the problem. More elegantly you can abstract some kind of general wait for element function.
If you want to run Webdriver sessions as part of a suite of integration tests then I would suggest using Python's unittest to create them. You drive the browser to the site under test, and make assertions that the actions you are taking leave the page in a state you expect. I can share some examples of how that might work as well if you are interested.