I was trying to automate some email testing, through selenium and python, and I came across the Temp-Mail website: 'https://temp-mail.org/en/'. I was trying to grab the email from it, with the code:
driver.find_element_by_xpath('//*[#id="mail"]').text
although this comes up empty. I was wondering what method I should be using for this, as the html text is
<input id="mail" type="text" onclick="select(this);" data-original-title="Your Temporary Email Address" data-placement="bottom" data-value="Loading" class="emailbox-input opentip" readonly="">
I managed to fix it with
driver.find_element_by_xpath('//*[#id="mail"]').get_attribute('value')
and a while loop to make sure it wasn't grabbing too early.
For multiple reasons, logging into sites like Gmail and Facebook using WebDriver is not recommended. Aside from being against the usage terms for these sites (where you risk having the account shut down), it is slow and unreliable.
The ideal practice is to use the APIs that email providers offer, or in the case of Facebook the developer tools service which exposes an API for creating test accounts, friends and so forth. Although using an API might seem like a bit of extra hard work, you will be paid back in speed, reliability, and stability. The API is also unlikely to change, whereas webpages and HTML locators change often and require you to update your test framework.
Logging in to third party sites using WebDriver at any point of your test increases the risk of your test failing because it makes your test longer. A general rule of thumb is that longer tests are more fragile and unreliable.
Related
I have a Python script that runs Selenium and makes a search for me on YouTube. After my .send_keys() and .submit() commands I attempt to get the current url of the search page with print(driver.current_url) but it only gives me the original url from my driver.get('https://www.youtube.com') command.
How can I get the full current url path of the search page once I'm there? For example https://www.youtube.com/results?search_query=election instead of https://www.youtube.com.
Thank you.
As you have not shared the code you have tried. I am guessing issue is with your page load. After clicking on submit you are not giving any time for page to load before you get your url. Please give some wait time. The simplest ( No so good) way is to use :
time.sleep(5)
print(driver.current_url)
Above will wait for 5 sec.
Why are you practicing social media to automation
For multiple reasons, logging into sites like Gmail and Facebook using WebDriver is not recommended. Aside from being against the usage terms for these sites (where you risk having the account shut down), it is slow and unreliable.
The ideal practice is to use the APIs that email providers offer, or in the case of Facebook the developer tools service which exposes an API for creating test accounts, friends, and so forth. Although using an API might seem like a bit of extra hard work, you will be paid back in speed, reliability, and stability. The API is also unlikely to change, whereas webpages and HTML locators change often and require you to update your test framework.
Logging in to third-party sites using WebDriver at any point of your test increases the risk of your test failing because it makes your test longer. A general rule of thumb is that longer tests are more fragile and unreliable.
WebDriver implementations that are W3C conformant also annotate the navigator object with a WebDriver property so that Denial of Service attacks can be mitigated.
You can simply wait for a period of time.
driver.implicitly_wait(5)
print(driver.current_url)
To get the current URL after clicking on random videos on particular search, current_url is the only way.
The reason because of which you are getting the previous URL may be the page is not loaded, you may check for the page load by comparing the title of the page
For eg:
expectedURL = "demo class"
actualURL=driver.title
assert expectedURL == actualURL
If the assert gives you true then you may have the command to get the current URL
driver.current_url
I am making a project where I want to click on the button but the button is hidden. I am working on python. I want to access this 3 dot button which appears next to comment when i hover over it
This is worst practice in selenium.
For multiple reasons, logging into sites like Gmail and Facebook using WebDriver is not recommended. Aside from being against the usage terms for these sites (where you risk having the account shut down), it is slow and unreliable.
The ideal practice is to use the APIs that email providers offer, or in the case of Facebook the developer tools service which exposes an API for creating test accounts, friends and so forth. Although using an API might seem like a bit of extra hard work, you will be paid back in speed, reliability, and stability. The API is also unlikely to change, whereas webpages and HTML locators change often and require you to update your test framework.
Logging in to third party sites using WebDriver at any point of your test increases the risk of your test failing because it makes your test longer. A general rule of thumb is that longer tests are more fragile and unreliable.
Please check selenium Hq web site
I was trying to parse tweets (let's say https://twitter.com/Tesla), but I ran into a problem that once I download the source code using html = browser.page_source it does not match what I see when inspecting the element (Ctrl+Shift+I). It shows some of the tweets, but not nearly all of them, moreover, when saving the code to file and opening it in Chrome, I get something incomprehensible. I had experience working with selenium before and have never ran into such a problem. Maybe there is some other function to get the source?
By the way, I know that Twitter provides an API, but they declined my request without giving any reasons even though I do not plan to do anything against their terms.
Hey this is one of worst practice in selenium
For multiple reasons, logging into sites like Gmail and Facebook using WebDriver is not recommended. Aside from being against the usage terms for these sites (where you risk having the account shut down), it is slow and unreliable.
The ideal practice is to use the APIs that email providers offer, or in the case of Facebook the developer tools service which exposes an API for creating test accounts, friends and so forth. Although using an API might seem like a bit of extra hard work, you will be paid back in speed, reliability, and stability. The API is also unlikely to change, whereas webpages and HTML locators change often and require you to update your test framework.
Logging in to third party sites using WebDriver at any point of your test increases the risk of your test failing because it makes your test longer. A general rule of thumb is that longer tests are more fragile and unreliable.
Is it possible to make a script that automatically:
Connects to the PayPal HTTPS website
Logs in with username + password
Goes to a specific URL
Modifies a few values in the dropdown lists
Clicks on a button
All of this from Python with standard methods like urllib.request?
import urllib.request; s = urllib.request.urlopen('https://example.com').read().decode()
Or because of the login involved, do we have to do it by "automating" or "faking" a real browser, using Selenium methods / or other methods à la PhantomJS? (I'd rather not have to do that, if possible)
Context: I've looked into various places and asked their support by email + phone, but it seems that PayPal doesn't offer access to an API to trigger the generation and download of a CSV report of all transactions between e.g. 01/01/2020 and 31/01/2020 ... except if you have > 20k sales per month and in this case you have access to their Reporting API + reports on a FTP server.
If this is not the case this Reporting API is not available, and you have to manually log in, go to the page https://business.paypal.com/merchantdata/reportHome?reportType=DLOG (see screenshot above), click in various lists, click Create report, wait a few minutes/hours, re log-in, go back on this page, and finally click on Download. This is a boring task to do each month (and their website is rather slow in general) that I would like to automate.
The correct link to SFTP reports is: https://developer.paypal.com/docs/reports/sftp-reports/
And those are all stricly daily report files, not monthly or any other time period. You'd have to read and parse in each daily file to cover the time period you desire.
It is theoretically possible to emulate the PayPal website UI with something like urllib or mechanize.Browser and automate requesting an activity log file and downloading it. But that's a fair amount of work for something you're only going to be doing once a month, and may break when PayPal changes their site. The login portion is particularly tricky to automate, since PayPal can require additional verification beyond just a password.
If you are going to automate some portion of this, I would recommend only doing so inside the browser (Tampermonkey, Selenium, etc; note that Tampermonkey blacklists paypal.tld by default, for your own protection against importing a malicious script)
Is there a way, using some library or method, to scrape a webpage in real time as a user navigates it manually? Most scrapers I know of such as python mechanize create a browser object that emulates a browser - of course this is not what I am looking for since if I have a browser open, it will be different than the one mechanize creates.
If there is no solution, my problem is I want to scrape elements from a HTML5 game to make an intelligent agent of sorts. I won't go into more detail, but I suspect if others are trying to do the same in the future (or any real time scraping with a real user), a solution to this could be useful for them as well.
Thanks in advance!
Depending on what your use-case is, you could set up a SOCKS proxy or some other form of proxy and configure it to log all traffic, then instruct your browser to use it. You'd then scrape that log somehow.
Similarly, if you have control over your router, you could configure capture and logging there, e.g. using tcpdump. This wouldn't decrypt encrypted traffic, of course.
If you are working with just one browser, there may be a way to instruct it to do something at each action via a custom browser plugin, but I'd have to guess you'd be running into security model issues a lot.
The problem with a HTML5 game is that typically most of its "navigation" is done using a lot of Javascript. The Javascript is typically doing a lot -- manipulating the DOM, triggering requests for new content to fit into the DOM, etc...
Because of this you might be better off looking into OS-level or browser-level scripting services that can "drive" keyboard and mouse events, take screenshots, or possibly even take a snapshot of the current page DOM and query it.
You might investigate browser automation and testing frameworks like Selenium for this.
I am not sure if this would work in your situation but it is possible to create a simple web browser using PyQt which will work with HTML5 and from this it might be possible to capture what is going on when a live user plays the game.
I have used PyQt for a simple browser window (for a completely different application) and it seems to handle simple, sample HTML5 games. How one would delve into the details of what is going on the game is a question for PyQt experts, not me.