I'm writing tests for SeleniumRC in Python. The tests run against a Firefox instance in Ubuntu. I've got a webapplication that uses BasicAuth.
In the setup of my testclass I set up a page (foo.html) in my webapplication.
I've got a lot of tests in this class that go to this page and authenticate themselves using a URL like:
self.selenium.open('http://user:pass#%s:%s/foo.html' % (host, port))
I've got one test (let's call it BAR) that creates another page (bar.html) and then opens that page:
self.selenium.open('http://user:pass#%s:%s/bar.html' % (host, port))
And here's the center of my problem:
If I run just this one test (BAR) the login works.
If I run another test before this one that does BasicAuth and goes to another URL (like foo.html, which is common in my case) I get a BasicAuth screen.
If I change the call for opening bar.html and have:
self.selenium.open('bar.html')
If I run just this one test (BAR) the login fails (I get a BasicAuth screen)
If I run another test before this one that does BasicAuth and goes to another URL I can succesfully go to the URL.
So the problem basically is: I can't always do a BasicAuth because then the BAR test will fail most of the time. Never doing a BasicAuth is also not an option because I can't count on the BAR test coming after another testing doing a BasicAuth.
The problem seems to look like the one described here in the SeleniumRC docs, but I can't figure out why this re-authenticating should give an error in this case...
For Firefox.
Try changing the following flag by typing in url about:config and search for below filter:
browser.safebrowsing.malware.enabled
You can change the state of this flag in the default Firefox profile using about:config service page (double click on the flag to change it's state) From now on the Firefox should let you go through http authentication using name and password in the url. (Note that if you are using Selenium WebDriver 2.6 or higher this flag should be disabled by default)
Related
For some unknown reasons ,my browser open test pages of my remote server very slowly. So I am thinking if I can reconnect to the browser after quitting the script but don't execute webdriver.quit() this will leave the browser opened. It is probably kind of HOOK or webdriver handle.
I have looked up the selenium API doc but didn't find any function.
I'm using Chrome 62,x64,windows 7,selenium 3.8.0.
I'll be very appreciated whether the question can be solved or not.
No, you can't reconnect to the previous Web Browsing Session after you quit the script. Even if you are able to extract the Session ID, Cookies and other session attributes from the previous Browsing Context still you won't be able to pass those attributes as a HOOK to the WebDriver.
A cleaner way would be to call webdriver.quit() and then span a new Browsing Context.
Deep Dive
There had been a lot of discussions and attempts around to reconnect WebDriver to an existing running Browsing Context. In the discussion Allow webdriver to attach to a running browser Simon Stewart [Creator WebDriver] clearly mentioned:
Reconnecting to an existing Browsing Context is a browser specific feature, hence can't be implemented in a generic way.
With internet-explorer, it's possible to iterate over the open windows in the OS and find the right IE process to attach to.
firefox and google-chrome needs to be started in a specific mode and configuration, which effectively means that just
attaching to a running instance isn't technically possible.
tl; dr
webdriver.firefox.useExisting not implemented
Yes, that's actually quite easy to do.
A selenium <-> webdriver session is represented by a connection url and session_id, you just reconnect to an existing one.
Disclaimer - the approach is using selenium internal properties ("private", in a way), which may change in new releases; you'd better not use it for production code; it's better not to be used against remote SE (yours hub, or provider like BrowserStack/Sauce Labs), because of a caveat/resource drainage explained at the end.
When a webdriver instance is initiated, you need to get the before-mentioned properties; sample:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.google.com/')
# now Google is opened, the browser is fully functional; print the two properties
# command_executor._url (it's "private", not for a direct usage), and session_id
print(f'driver.command_executor._url: {driver.command_executor._url}')
print(f'driver.session_id: {driver.session_id}')
With those two properties now known, another instance can connect; the "trick" is to initiate a Remote driver, and provide the _url above - thus it will connect to that running selenium process:
driver2 = webdriver.Remote(command_executor=the_known_url)
# when the started selenium is a local one, the url is in the form 'http://127.0.0.1:62526'
When that is ran, you'll see a new browser window being opened.
That's because upon initiating the driver, the selenium library automatically starts a new session for it - and now you have 1 webdriver process with 2 sessions (browsers instances).
If you navigate to an url, you'll see it is executed on that new browser instance, not the one that's left from the previous start - which is not the desired behavior.
At this point, two things need to be done - a) close the current SE session ("the new one"), and b) switch this instance to the previous session:
if driver2.session_id != the_known_session_id: # this is pretty much guaranteed to be the case
driver2.close() # this closes the session's window - it is currently the only one, thus the session itself will be auto-killed, yet:
driver2.quit() # for remote connections (like ours), this deletes the session, but does not stop the SE server
# take the session that's already running
driver2.session_id = the_known_session_id
# do something with the now hijacked session:
driver.get('https://www.bing.com/')
And, that is it - you're now connected to the previous/already existing session, with all its properties (cookies, LocalStorage, etc).
By the way, you do not have to provide desired_capabilities when initiating the new remote driver - those are stored and inherited from the existing session you took over.
Caveat - having a SE process running can lead to some resource drainage in the system.
Whenever one is started and then not closed - like in the first piece of the code - it will stay there until you manually kill it. By this I mean - in Windows for example - you'll see a "chromedriver.exe" process, that you have to terminate manually once you are done with it. It cannot be closed by a driver that has connected to it as to a remote selenium process.
The reason - whenever you initiate a local browser instance, and then call its quit() method, it has 2 parts in it - the first one is to delete the session from the Selenium instance (what's done in the second code piece up there), and the other is to stop the local service (the chrome/geckodriver) - which generally works ok.
The thing is, for Remote sessions the second piece is missing - your local machine cannot control a remote process, that's the work of that remote's hub. So that 2nd part is literally a pass python statement - a no-op.
If you start too many selenium services on a remote hub, and don't have a control over it - that'll lead to resource drainage from that server. Cloud providers like BrowserStack take measures against this - they are closing services with no activity for the last 60s, etc, yet - this is something you don't want to do.
And as for local SE services - just don't forget to occasionally clean up the OS from orphaned selenium drivers you forgot about :)
OK after mixing various solutions shared on here and tweaking I have this working now as below. Script will use previously left open chrome window if present - the remote connection is perfectly able to kill the browser if needed and code functions just fine.
I would love a way to automate the getting of session_id and url for previous active session without having to write them out to a file during hte previous session for pick up...
This is my first post on here so apologies for breaking any norms
#Set manually - read/write from a file for automation
session_id = "e0137cd71ab49b111f0151c756625d31"
executor_url = "http://localhost:50491"
def attach_to_session(executor_url, session_id):
original_execute = WebDriver.execute
def new_command_execute(self, command, params=None):
if command == "newSession":
# Mock the response
return {'success': 0, 'value': None, 'sessionId': session_id}
else:
return original_execute(self, command, params)
# Patch the function before creating the driver object
WebDriver.execute = new_command_execute
driver = webdriver.Remote(command_executor=executor_url, desired_capabilities={})
driver.session_id = session_id
# Replace the patched function with original function
WebDriver.execute = original_execute
return driver
remote_session = 0
#Try to connect to the last opened session - if failing open new window
try:
driver = attach_to_session(executor_url,session_id)
driver.current_url
print(" Driver has an active window we have connected to it and running here now : ")
print(" Chrome session ID ",session_id)
print(" executor_url",executor_url)
except:
print("No Driver window open - make a new one")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=myoptions)
session_id = driver.session_id
executor_url = driver.command_executor._url
Without getting into why do you think that leaving an open browser windows will solve the problem of being slow, you don't really need a handle to do that. Just keep running the tests without closing the session or, in other words, without calling driver.quit() as you have mentioned yourself. The question here though framework that comes with its own runner? Like Cucumber?
In any case, you must have some "setup" and "cleanup" code. So what you need to do is to ensure during the "cleanup" phase that the browser is back to its initial state. That means:
Blank page is displayed
Cookies are erased for the session
I am wondering if there is a way to obtain the hostname of a Django application when running tests. That is, I would like the tests to pass both locally and when run at the staging server. Hence a need to know http://localhost:<port> vs. http://staging.example.com is needed because some tests query particular URLs.
I found answers on how to do it inside templates, but that does not help since there is no response object to check the hostname.
How can one find out the hostname outside the views/templates? Is it stored in Django settings somewhere?
Why do you need to know the hostname? Tests can run just fine without it, if you use the test client. You do not need to know anything about the system they're running on.
You can also mark tests with a tag and then have the CI system run the tests including that tag.
And finally there is the LiveServerTestCase:
LiveServerTestCase does basically the same as TransactionTestCase with one extra feature: it launches a live Django server in the background on setup, and shuts it down on teardown. This allows the use of automated test clients other than the Django dummy client such as, for example, the Selenium client, to execute a series of functional tests inside a browser and simulate a real user’s actions.
The live server listens on localhost and binds to port 0 which uses a free port assigned by the operating system. The server’s URL can be accessed with self.live_server_url during the tests.
Additional information from comments:
You can test if the URL of an image file is present in your response by testing for the MEDIA_URL:
self.assertContains(response, f'{settings.MEDIA_URL}/default-avatar.svg')
You can test for the existence of an upload in various ways, but the easiest one is to check if there's a file object associated with the FileField. It will throw ValueError if there is not.
I am trying to record HTTP GET/POST requests sent by my browser using the library scotch.
I am using their sample code: http://darcs.idyll.org/~t/projects/scotch/doc/recipes.html#id2
import scotch.proxy
app = scotch.proxy.ProxyApp()
import scotch.recorder
recorder = scotch.recorder.Recorder(app, verbosity=1)
try:
from wsgiref.simple_server import WSGIServer, WSGIRequestHandler
server_address = ('', 8000)
httpd = WSGIServer(server_address, WSGIRequestHandler)
httpd.set_app(app)
while 1:
httpd.handle_request()
finally:
from cPickle import dump
outfp = open('recording.pickle', 'w')
dump(recorder.record_holder, outfp)
outfp.close()
print 'saved %d records' % (len(recorder.record_holder))
So I ran above code, went over to google chrome, and visited a few sites to see if that would get recorded.
However, I do not see how the code should terminate. It seems that there has to be an error in httpd.handle_request() for the code to terminate.
I tried a variation of the code where I removed the try and finally syntax, and changed the while condition so that the loop ran for 30 seconds. However, that seems to be running forever as well.
Any ideas on how to get this working? I am also open to using other python libraries available for what I am trying to do: record my browser's GET/POST requests, including logons, and replay this within python.
Thanks.
Correct me if I'm wrong, but you're trying to log the activity of your local browser by setting a local proxy. If this is the case your browser needs to go through your proxy in order for your proxy server to log the activity.
The code that you've provided sets a proxy server at localhost:8000, so you need to tell your browser about this. The actual setting depends on the browser, I'm sure you'd be able to google it easily.
When I've asked to check if the code is running I actually mean whether your local proxy accepts some kind of request from the browser. Do you see the 'saved records' print out of your code at some point?
I'm running Selenium 2.0b4dev on Selenium Grid in Ubuntu 10.04, using Python code to write test cases. I've been having trouble with getting basic HTTP authentication to a specific site working, and with a quick google search found that my problem could be solved with the addition of the line self.selenium.add_custom_request_header("Authorization", "Basic %s" % _encoded) (with a proper line break in the middle to conform to PEP 8, of course.)
Unfortunately, apparently also through my search I found in order for that line of code to work I need to configure my browser (whichever one I'm using to run the test cases on the grid) to treat Selenium's (automatically running, apparently?) proxy server as a proxy for that browser to use. But apparently I need to modify the profile of Firefox (or IE)'s launcher to automatically use that proxy, since the whole point of these Selenium Grid test cases is that they aren't supposed to require user intervention, and I have little to no idea how to do that. I've just been using the "ant launch-hub" and "ant launch-remote-control" and then running python programs on the hub that import selenium and unittest.
If anyone could help, that would be just fantastic.
I wrote up an article on how to do this in Ruby. It links to a complementary article on testing self-signed certificates and gives you the set of flags you need to launch Selenium with.
http://mogotest.com/blog/2010/06/23/how-to-perform-basic-auth-in-selenium
To pass args through from grid to the underlying RC server, you need to use something like:
ant -DseleniumArgs="-trustAllSSLCertificates" launch-remote-control
Re: browsers . . . firefox will auto-enable the proxy mode stuff if you pass trustAllSSLCertificates now. Otherwise you need to use *firefoxproxy. IE requires the use of *iexploreproxy or a custom HTA launcher that configures the proxy (the article links to one we open-sourced but would need to be updated to work with 2.0 beta 4).
I am writing an application in Pylons that relies on the output of some system commands such as traceroute. I would like to display the output of the command as it is generated rather than wait for it to complete and then display all at once.
I found how to access the output of the command in Python with the answer to this question:
How can I perform a ping or traceroute in python, accessing the output as it is produced?
Now I need to find a way to get this information to the browser as it is being generated. I was planning on using jQuery's loadContent() to load the output of a script into a . The problem is that Pylons controllers use return so the output has to be complete before Pylons renders the page and the web server responds to the client with the content.
Is there any way to have a page display content as it is generated within Pylons or will this have to be done with scripting outside of Pylons?
Basically, I'm trying to do something like this:
http://network-tools.com/default.asp?prog=trace&host=www.bbc.co.uk
pexpect will let you get the output as it comes, with no buffering.
To update info promptly on the user's browser, you need javascript on that browser sending appropriate AJAX requests to your server (dojo or jquery will make that easier, though they're not strictly required) and updating the page as new responses come -- without client-side cooperation (and JS + AJAX is the simplest way to get that cooperation), there's no sensible way to do it on the server side alone.
So the general approach is: send AJAX query from browser, have server respond as soon as it has one more line, JS on the browser updates contents then immediately sends another query, repeat until server responds with an "I'm all done" marker (e.g. an "empty" response may work for that purpose).
You may want to look at this faq entry. Then with JS, you always clear the screen before writing new stuff.
I haven't tried it with pylons, but you could try to show the output of the slow component in an iframe on the page (using mime type text/plain) and yield each chunk to the iframe as it is generated. For fun I just put this together as a
WHIFF demo. Here is the slowly generated web content wsgi application:
import time
def slow(env, start_response):
start_response("200 OK", [('Content-Type', 'text/plain')])
return slow_generator()
def slow_generator():
yield "slowly generating 20 timestamps\n"
for i in range(20):
yield "%s: %s\n" % (i, time.ctime())
time.sleep(1)
yield "done!"
__wsgi__ = slow
This file is deployed on my laptop at: http://aaron.oirt.rutgers.edu/myapp/root/misc/slow.
Here is the WHIFF configuration template which includes the slow page in an
iframe:
{{env whiff.content_type: "text/html"/}}
Here is an iframe with slowly generated content:
<hr>
<iframe frameborder="1" height="300px" width="300px" scrolling="yes"
style="background-color:#99dddd;"
src="slow"
></iframe>
<hr>
Isn't that cool?
This is deployed on my laptop at http://aaron.oirt.rutgers.edu/myapp/root/misc/showSlowly.
hmmm. I just tried the above link in safari and it didn't work right... apparently there are some browser differences... Seems to work on Firefox at least...