I'm writing a bunch of automated tests using selenium and one of the tests requires user authentication. After entering user's credentials (email and password) I need to wait for the process of authentication to complete. So basically it boils down to waiting for the server to respond with an Auth cookie.
But how do I get it?
I tried searching on the Internet but didn't find anything that answers that question. If I do driver.get_cookies() it returns a whole bunch of them, which one of them should I use then?
driver.manage().getCookies();
will provide Set of cookies used or available at that instance. You have to cross check which cookie is responsible for what and use it.
As knows driver.manage().addCookie(arg0) is used to add required cookies, if you what get required cookies then driver.manage().getCookieNamed(arg0)
Related
I am new to the SSO world, so perhaps I miss already answered question.
I run an app in python3 and I need to make some API calls.
To do so, I need to be identified using my corporate SSO.
I know the SSO is stored as a cookie but I do not know how to:
Open a browser
Connect
Close the browser
Get the cookie
Do stuff
If possible without external library.
Also, I can ask my user to define its credentials as variable if it eases the process.
Thank you for any help
I am a lot confused about cookies.
I am trying to work out scraping with many post/get requests chained
And I notice that every step needs 'Cookie' evaluated in the headers dictionary, because passing the right dictionary is the only way I don't get any access errors.
However, I look always at my cookie jar (via .cookies method) as it was at the previous step, but I cannot find what I need for the current step. And I know that by inspecting the network data in my browser.
So how shall I build up step by step a chain like 1) login, 2) botton interaction for changing dates, 3) file downloading?
My fault is that I am using requests instead of selenium?
I already use requests.Session().... but this means that I don't need to show up Cookie field when sending headers? Either way (showing or not showing Cookie in headers) I get server access error AFTER having correctly logged in...
Thanks,
David
If you cant access your cookies probably the login credentials are stored in an HTTP Only cookies. This is a secure place to store them to prevent CSS attacks.
You should try using a requests session to send this cookies along with your future requests.
First of all, I googled this question but found some generic explanations which didn't provide me with good understanding how to do things.
Second - I'm a valid system user (not admin) and have access to the data. I.e. I have valid user credentials and may download file manually but for small automation I would like to have it downloaded by python script from my PC.
The download itself is simple, the only thing - I need to provide a valid session id cookie with request. I.e. finally I need to get this cookie by easiest way.
If my understaning is right in terms of SAML I'm a User Agent and want to download a file from Sevice Provider which need to authenticate me with Identity Provider (Microsoft). Usually I do it via browser and now I'm able to emulate it with help of PySide6 (QWebEngineView). I load target URL first in QWebEngineView. Actually it is a small embedded web-browser, it redirects me to login.microsoft.com, asks credentials and then redirects me back to Service Provider site and sets session id cookie. Then I'm able to use this cookie with my requests. It works but I would like to get rid of GUI (PySide) if possible.
I decided to replicate a flow that browser does and failed almost at the begining. What happens:
I'm requesting a file from my Service Provider side with usual get request.
Service provider replies with HTML page (instead of target file) as I'm not authenticated.
This HTML page contains Java script triggered by onPageLoad event - this java script simply redirects browswer to login.microsoft.com (long URL with some parameters).
Next request with this long URL for login.microsoft.com ends with "302 Moved Temporarily" with the same URL in "Location" header. And when I go with this URL it again gives me 302 with the same URL.
With the same scenario browswer gets only two redirections and finally receives an URL of web page with login/password request from microsoft.com.
I understand that I should put some more headers/cookies when I go again with URL provided in "Location" header of 302 response. But... I have no idea what login.microsoft.com expects here.
So my question is - is there any source where this message flow is described? Or maybe someone did it already and may give me advice how to proceed?
I found some SAML-related libraries for python but I see there quite complex configuration with x509 certificates and more stuff - it looks like they are more targeted for implementation on Service Provider side, not for external login.
I am having trouble authenticating against a web service that has Oauth provided by google.
Basically, I want to login with my google account to a web page to do some scraping on it.
As the web service is not mine, I don't have the app secret_key, only the clientID, redirect_URL and scope that I could recover from seeing the parameters of request method used while being logged in.
Once authenticated, the web page only requieres a cookie named SID (Session ID I would guess) to answer back as an authenticated user. There is no Bearer token, just the SID cookie.
Is it possible to automate this type of authentication? I've read many topics related but they all need the secret_key which I don't have because I'm not the owner of the app.
(Cannot comment due to rep)
Yes, what you're asking is possible. You could theoretically follow and match all the requests to authenticate yourself successfully to get the SID and perform scraping, albeit this would be a very difficult task for some basic web-scraping, it's like programming a full-blown scientific calculator to do 5 + 5. What you are asking is a really difficult task, you're going to run into all sorts of security issues and be asked for phone/authenticator app/email verification when attempting to login to your account with Python requests and then you'd need to keep track of those security cookies and keeping them updated, it's a real mess and would be extremely difficult for anyone.
I think the better method would be to manually authenticate yourself and get the SID cookie and hard-code that into your scraper within the cookie HTTP header.
I understand this brings up the concern of what to do when the SID cookie expires. Since you haven't said the site, It would be hard for me to imagine a site that makes you authenticate yourself with Google often rather than having their own internal SID/JWT refreshing system to keep you logged in.
My recommendations would be:
Check the expiration of the SID cookie, if it's viable to manually copy-and-paste it after authenticating yourself, do that.
If the SIDs expire soon, check if there's an API request anywhere to get yourself a new SID (Without going through the OAuth again), in your Network panel look for the set-cookie response header setting a new SID, you might need to change and keep track of these inside your program but it'll be much easier than writing a program to login to Google.
If there's no way to refresh the SID and they expire often and you need to do long-term web scraping and sitting there getting a new cookie manually every 30 minutes isn't enough, I'd recommend looking into doing this with Puppeteer/Chromium as it'll be much easier than doing it via Python HTTP requests.
I know this is an old one to be asking about, but I am trying to figure out a way, through my python login script, to set the CookieCrumbler cookie(s) that are automatically sent to the user upon successful login and visiting a restricted URL.
My goal is to allow my python script to call a zsql method and redirect the user based on their account_type (a column in my db). I have this all working right now, but across two different python scripts. The first logs in the user and redirects to the second script which makes the call to the zsql method and redirects the user accordingly. I would love to be able to do this all in one script, but if the cookies aren't sent the username of that user isn't stored in a cookie and thus cannot be accessed by the zsql method.
I know how to expire the cookies upon logout, but when I change it to setCookie it wants me to define the entire cookie, which of course I need CookieCrumbler to do.
One of my thoughts was to somehow get python to invisibly hit a secure URL and deliver the cookies to the user's browser, sleep for 2 seconds to ensure they got through, and then proceed to the sql stuff.
Another option is what I already mentioned, which is to get CookieCrumbler to send the cookies to the user's browser directly instead of having to try and access a secure URL.
Here's the code I have now:
import time
request = container.REQUEST
response = request.response
context.REQUEST.RESPONSE.setCookie('__ac', path='/')
context.REQUEST.RESPONSE.setCookie('__ac_name', path='/')
time.sleep(3)
for user in context.account_data_select():
if user.account_type == 12 :
response.redirect("https://secure.mydomain.com/secure/blah")
else:
response.redirect("https://secure.mydomain.com/secure")
Of course this doesn't work because setCookie requires 3 variables (name, value, options). I do not want to set these manually. I want CookieCrumbler to do its thing and provide the appropriate cookies to the user's browser.
Btw, I am using Zope 2.13.29 and am not using Plone. Just the standard old Zope with Python 2.7.2