I'm working on a python software with selenium. The problem is I want my script and selenium to save cookies after logging in. I save cookies using both "pickle" module and the below argument:
opts.add_argument("user-data-dir=cachedD")
But when I quit the browser and then launch it again and going to the same URL as it left off, the website again redirects to the login page. The website is using "moodle", and I guess it's cookies expire after quitting the browser. How can I save cookies and continue where it left off? I should say that there's just a maximum 15 seconds gap between two launches.
You're potentially not using the tag correctly.
With this tag you specify a folder path. If you review this page:
--user-data-dir
Directory where the browser stores the user profile. ↪
That link may not look right, but the chromium page say that's the right list.
Historically, I've had success with:
.add_argument("user-data-dir=C:\Temp")
If that is still not working as you expect, there are a few other things you can look at.
Review this page - cookies can be deleted when you close your browser. You'll want to verify the value of this option.
Another check is to open your chromedriver via selenium and goto chrome://version/ . From here you can review what you're running and you'll see there are a lot more tags that are enabled by default. You should check that these match up to how you want your browser to behave.
Related
I try reading the website cookies after logging in, but the following code cannot read it.
After the code runs, the driver_cookies I get is just an empty list. However, I can manually find the cookies in Chrome Development Tools.
driver.find_element(by=By.ID,value='login_id').send_keys(login_id)
driver.find_element(by=By.ID,value='password').send_keys(password)
driver.find_element(by=By.ID,value='login_btn').click()
while(driver.current_url!="https://www.theWebsiteThatWeGoTo.com"):
continue
time.sleep(3)
# debug, the cookies can be found in the development tools after the code run here,
# but the following code just cannot read it.
os.system("pause")
driver.switch_to.window(driver.window_handles[0])
driver_cookies = driver.get_cookies()
print(driver_cookies)
Something that I think I need to mention is the website, which the login website redirects to, uses a different protocol because it is used for an electron application and thus Chrome cannot directly load the page (will show ERR_SSL_VERSION_OR_CIPHER_MISMATCH). I don't know if it is the reason.
I am trying to make a python gui application.
What I want to do is to open a web browser by clicking a button. (Tkinter)
When the web browser is opened, I do login.
After logging it, it will redirect to the page.
And that page url will consist of code as a param I need to use later in code.
I used webbrowser.open_new('') to open a web browser.
But the limitation was it is only for opening.. there was no way to get the final redirected url I need.
Is there a way I can use to open a web browser and do something on that page and finally get that final url?
I am using python.
There are a few main approaches for automating interactions with browsers:
Telling a program how and what to click, like a human would, sometimes using desktop OS automation tools like Applescript
Parse files that contain browser data (will vary browser to browser, here is Firefox)
Use a tool or library that relies on the WebDriver protocol (e.g. selenium, puppeteer)
Access the local SQLite database of the browser and run queries against it
Sounds like 3 is what you need, assuming you're not against bringing in a new dependency.
I have the following setup:
Python 3.7
Selenium 3.141.0
Firefox 67.0.4
Geckodriver 0.24.0
I've written a program that scrapes hotel data off of a hotel operator website. Using the link as a query, the program supplies the site with city, check-in and check-out dates, number of persons, etc.
The program starts up firefox and makes the first query and it all goes well. The problem appears from the second query and on because no matter what is provided in the following links it will keep showing the results of the city in the first query, only changing the dates.
When the webdriver is restarted it's back to normal for the first query but beginning with the second one it all goes like before.
I tried using delete_all_cookies() and configuring the webdriver profile to no create any cache but it does not work. I tried using Python to delete all files inside the profile folder and it still does not work.
The strange thing is that if I go in the browser and manually delete "Cookies and other site data" it's ok, but I can't find a way of doing this programmatically. Tried it both in Firefox and Chrome.
Restarting the webdriver also works. I understand that it clears the profile and starts with a fresh one every time. But this is too costly from a time pov.
#First link, it all goes ok
URL = 'https://www.wyndhamhotels.com/en-us/hotels/beijing-china?brand_id=ALL&checkInDate=8/10/2019&checkOutDate=8/11/2019&useWRPoints=false&children=0&adults=2&rooms=1'
DRIVER.get(URL)
# From the second link on, no matter how many searches I d, I always get the results for Beijing
URL = 'https://www.wyndhamhotels.com/en-us/hotels/bremen-germany?brand_id=ALL&checkInDate=9/11/2019&checkOutDate=9/11/2019&useWRPoints=false&children=0&adults=2&rooms=1'
DRIVER.get(URL)
URL = 'https://www.wyndhamhotels.com/en-us/hotels/paris-france?brand_id=ALL&checkInDate=9/11/2020&checkOutDate=9/11/2020&useWRPoints=false&children=0&adults=2&rooms=1'
DRIVER.get(URL)
Is there any way to programmatically delete all cookies and other site data the way that this happens when you do it manually from the menu, while the webdriver is running?
Or, in another train of thoughts, what exactly happens when you manually delete cookies and other site data from the browser menu? What gets deleted and from where?
Please note, this question is Python 3.5.2, only Python answers will be accepted. Unless this can definitely be handled in Java? Automating a process as part of an internal project. Everything works just fine using the IE webdriver, but not phantomJS web driver (which is expected due to limited functionality). However, a work-around / solution is required.
When opening the internal site, a Windows Security login dialog box comes up prompting for a username, password and press 'Ok'. With the IE web driver, it is handled just fine with:
loginAlert = driver.switch_to_alert()
loginAlert.authenticate(username, password)
The javascript:
driver.execute_script("window.confirm = function(){return true;}")
Being run before loading the page that gives the prompt, doesn't seem to confirm the login alert, for either phantom or IE. Even if it did, this doesn't type in the login details. As mentioned, it's a Windows Security prompt from the browser, not an element.
Once logged in, the page is reloaded with an ASP.NET_SessionId Cookie which expires once the session is ended. I've tried logging in through IE, then adding the cookie into Phantom, but it doesn't seem to match up the domains.
I've tried using:
driver.save_screenshot(filename) to see what's happening in phantom
Which works with IE driver, but with PhantomJS, only a transparent image is saved. The whole http://username:pass#site.com thing doesn't work for either IE or phantom driver. It can't load / use the URL when this is done.
How can the Windows Security login dialog be handled, or worked around? I tried looking into alternatives, such as pyvirtualdisplay, but found no information on how to get this working with Python 3 on windows.
I have also tried setting phantomjs desired capabilities custom header authentication, but that doesn't seem to do anything for this either.
I have also tried using ActionChains, however they don't work when the Alert is there (in either IE or phantom driver). An UnexpectedAlertPresentException is thrown, even if this is caught and you try to perform the actions, once caught, the alert seems to close.
My bad!
Whilst the username:pass#domain.com didn't work in the IE webdriver - it did work in the PhantomJS web driver.
However, the website has limited browser compatibility - it doesn't load properly in either Chrome or Firefox, it is IE particular.
PhantomJS seems to handle the site the same way as Chrome / Firefox based on page source comparisons.
As such, I am trying to find a way to make the current IE driver invisible / hidden.
I have found:
headless-selenium-for-win using Python
However, despite the user here saying they got it to work, when I try to initialize the driver, it just hangs, the code doesn't proceed and no error messages are provided.
Asking another question regarding this.
I am currently automating a website and have a test which checks the functionality of the Remember Me option.
My test script logs in, entering a valid username and password and checks the Remember Me checkbox before clicking Log In button.
To test this functionality I save the cookies to file using pickle, close the browser and then reopen the browser (loading the cookies file).
def closeWebsite(self, saveCookies=False):
if saveCookies:
pickle.dump(self.driver.get_cookies(), open('cookies.pkl', 'wb'))
self.driver.close()
def openWebsite(self, loadCookies=False):
desired_caps = {}
desired_caps['browserName'] = 'firefox'
profile = webdriver.FirefoxProfile(firefoxProfile)
self.driver = webdriver.Firefox(profile)
self.driver.get(appUrl)
if loadCookies:
for cookie in pickle.load(open('cookies.pkl', 'rb')):
self.driver.add_cookie(cookie)
However, when I do this, the new browser is not logged in. I understand that everytime you call the open the browser a new Session is created and this session ID can be obtained using driver.session_id
Is it possible, in the openWebsite method to load a driver and specify the sessionID?
When I test this manually, the Remember Me option works as expected. I'm just having trouble understanding how Selenium handles this case.
For starters you're loading the page before adding the cookies. Although there is the potential for them to arrive before the page needs / queries them, this isn't correct let alone reliable.
Yet, if you try to set the cookies before any page has loaded you will get an error.
The solution seems to be this:
First of all, you need to be on the domain that the cookie will be valid for. If you are trying to preset cookies before you start interacting with
a site and your homepage is large / takes a while to load an
alternative is to find a smaller page on the site, [...]
In other words:
Navigate to your home page, or a small entry page on the same domain as appUrl (no need to wait until fully loaded).
Add your cookies.
Load appUrl. From then on you should be fine.