We Keep Coding

Properly starting/stopping Selenium standalone server

I am using the Selenium standalone server for a remote web driver. One thing I am trying to figure out is how to start/stop it effectively. On their documentation, it says
"the caller is expected to terminate each session properly, calling either Selenium#stop() or WebDriver#quit."
What I am trying to do is figure out how to programmatically close the server, but is that even necessary? In other words, would it be okay to have the server up and running at all times, but to just close the session after each use with something like driver.quit()? Therefore when I'm not using it the server would be up but there would be no sessions.

While using Selenium standalone server as a Remote WebDriver you need to invoke quit() method at the end to terminate each session properly.
As per best practices, you should invoke the quit() method within the tearDown() {}. Invoking quit() DELETEs the current browsing session through sending "quit" command with {"flags":["eForceQuit"]} and finally sends the GET request on /shutdown EndPoint. Here is an example below :
1503397488598 webdriver::server DEBUG -> DELETE /session/8e457516-3335-4d3b-9140-53fb52aa8b74
1503397488607 geckodriver::marionette TRACE -> 37:[0,4,"quit",{"flags":["eForceQuit"]}]
1503397488821 webdriver::server DEBUG -> GET /shutdown
So on invoking quit() method the Web Browser session and the WebDriver instance gets killed completely.
References
You can find a couple of relevant detailed discussions in:
Selenium : How to stop geckodriver process impacting PC memory, without calling driver.quit()?
PhantomJS web driver stays in memory

you were right. Use seleniums driver.quit() as it properly closes all browser windows and ends driver's session/process. Especially the latter is what you want, because you most certainly run the script headless.
I have a selenium script running on as Raspberry Pi (hourly cron job) headless. That script calls driver.quit() at the end of each iteration. When i do a -ps A (to list al active processes under unix), no active selenium/python processes are shown anymore.
Hope that satisfies your question!

Multiple Chrome instances with Selenium: "Closing stream with result -2" Error in Instance 1 when running driver.close/quit in Instance 2

I'm running a combination of BeautifulSoup and Selenium for the following scenario:
Open Webdriver (Chrome) with Selenium, waiting for commands.
Infinite loop of BeautifulSoup in the background to check website for changes every few seconds
If a certain change is detected, use Selenium to open/load website and click a few buttons to render content I want to scrape.
I am tracking changes on two different websites. I could just test the two websites in the same script, but I need to track the changes as soon as possible. Therefore, I'm also opening Selenium at the beginning of the scripts to save time before changes are detected so that I don't have to wait for Chrome to first boot up as soon as a change is detected.
I need the cookies of my Chrome default profile to access the website (Two-factor login), so I copied my User Data folder to load two sessions of Chrome. If I don't do that, I get an error in the second instance, saying that the profile is being in use, can't edit the files yada yada...
Opening one instance/list/website is fine, my script runs without any issues.
When I open a second instance of the script, checking a different website in a new window of Chrome, the first script gives me the following error whenever the second script initiates driver.close() or driver.quit().
[22384:16640:0909/170836.278:ERROR:socket_stream.cc(219)] Closing stream with result -2
This doesn't shut down my first script, it seems to keep working fine. So I'm not even thinking this has any detrimental impact on what I am trying to do, but I wonder what is happening here and why are the two instances seemingly interacting with each other when they shouldn't really?
I'm using two different copies of chromedriver.exe for each script as well, not even sure if that is necessary.
Anyway, would appreciate if someone could enlighten me about what is going on.

This error message...
ERROR:socket_stream.cc(219)] Closing stream with result -2
...implies that the ChromeDriver was an error while invoking CloseStream().
This error is coming from void SocketInputStream::CloseStream() function defined in socket_stream.cc which is defined as:
void SocketInputStream::CloseStream(net::Error error,
const base::Closure& callback) {
DCHECK_LT(error, net::ERR_IO_PENDING);
ResetInternal();
last_error_ = error;
LOG(ERROR) << "Closing stream with result " << error;
if (!callback.is_null())
callback.Run();
}

(Selenium) Running many firefox browser with less performance [duplicate]

I am using selenium with Firefox to automate some tasks on Instagram. It basically goes back and forth between user profiles and notifications page and does tasks based on what it finds.
It has one infinite loop that makes sure that the task keeps on going. I have sleep() function every few steps but the memory usage keeps increasing. I have something like this in Python:
while(True):
expected_conditions()
...doTask()
driver.back()
expected_conditions()
...doAnotherTask()
driver.forward()
expected_conditions()
I never close the driver because that will slow down the program by a lot as it has a lot of queries to process. Is there any way to keep the memory usage from increasing overtime without closing or quitting the driver?
EDIT: Added explicit conditions but that did not help either. I am using headless mode of Firefox.

Well, This the serious problem I've been going through for some days. But I have found the solution. You can add some flags to optimize your memory usage.
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument('--no-sandbox')
options.add_argument('--disable-application-cache')
options.add_argument('--disable-gpu')
options.add_argument("--disable-dev-shm-usage")
These are the flags I added. Before I added the flags RAM usage kept increasing after it crosses 4GB (8GB my machine) my machine stuck. after I added these flags memory usage didn't cross 500MB. And as DebanjanB answers, if you running for loop or while loop tries to put some seconds sleep after each execution it will give some time to kill the unused thread.

To start with Selenium have very little control over the amount of RAM used by Firefox. As you mentioned the Browser Client i.e. Mozilla goes back and forth between user profiles and notifications page on Instagram and does tasks based on what it finds is too broad as a single usecase. So, the first and foremost task would be to break up the infinite loop pertaining to your usecase into smaller Tests.
time.sleep()
Inducing time.sleep() virtually puts a blanket over the underlying issue. However while using Selenium and WebDriver to execute tests through your Automation Framework, using time.sleep() without any specific condition defeats the purpose of automation and should be avoided at any cost. As per the documentation:
time.sleep(secs) suspends the execution of the current thread for the given number of seconds. The argument may be a floating point number to indicate a more precise sleep time. The actual suspension time may be less than that requested because any caught signal will terminate the sleep() following execution of that signal’s catching routine. Also, the suspension time may be longer than requested by an arbitrary amount because of the scheduling of other activity in the system.
You can find a detailed discussion in How to sleep webdriver in python for milliseconds
Analysis
There were previous instances when Firefox consumed about 80% of the RAM.
However as per this discussion some of the users feels that the more memory is used the better because it means you don't have RAM wasted. Firefox uses RAM to make its processes faster since application data is transferred much faster in RAM.
Solution
You can implement either/all of the generic/specific steps as follows:
Upgrade Selenium to current levels Version 3.141.59.
Upgrade GeckoDriver to GeckoDriver v0.24.0 level.
Upgrade Firefox version to Firefox v65.0.2 levels.
Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
If your base Web Client version is too old, then uninstall it and install a recent GA and released version of Web Client.
Some extensions allow you to block such unnecessary content, as an example:
uBlock Origin allows you to hide ads on websites.
NoScript allows you to selectively enable and disable all scripts running on websites.
To open the Firefox client with an extension you can download the extension i.e. the XPI file from https://addons.mozilla.org and use the add_extension(extension='webdriver.xpi') method to add the extension in a FirefoxProfile as follows:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.add_extension(extension='extension_name.xpi')
driver = webdriver.Firefox(firefox_profile=profile, executable_path=r'C:\path\to\geckodriver.exe')
If your Tests doesn't requires the CSS you can disable the CSS following the this discussion.

Use Explicit Waits or Implicit Waits.
Use driver.quit() to close all
the browser windows and terminate the WebDriver session because if
you do not use quit() at the end of the program, the WebDriver
session will not be closed properly and the files will not be cleared
off memory. And this may result in memory leak errors.

Creating new firefox profile and use it every time while running test cases in Firefox shall eventually increase the performance of execution as without doing so always new profile would be created and caching information would be done there and if driver.quit does not get called somehow before failure then in this case, every time we end up having new profiles created with some cached information which would be consuming memory.
// ------------ Creating a new firefox profile -------------------
1. If Firefox is open, close Firefox.
2. Press Windows +R on the keyboard. A Run dialog will open.
3. In the Run dialog box, type in firefox.exe -P
Note: You can use -P or -ProfileManager(either one should work).
4. Click OK.
5. Create a new profile and sets its location to the RAM Drive.
// ----------- Associating Firefox profile -------------------
ProfilesIni profile = new ProfilesIni();
FirefoxProfile myprofile = profile.getProfile("automation_profile");
WebDriver driver = new FirefoxDriver(myprofile);
Please share execution performance with community if you plan to implement this way.

There is no fix for that as of now.
I suggest you use driver.close() approach.
I was also struggling with the RAM issue and what i did was i counted the number of loops and when the loop count reached to a certain number( for me it was 200) i called driver.close() and then start the driver back again and also reset the count.
This way i did not need to close the driver every time the loop is executed and has less effect on the performance too.
Try this. Maybe it will help in your case too.

How can I debug an iOS Selenium test in Python

I'm trying to run an iOS Selenium test in debug mode. I'm using Appium, an iOS simulator (Xcode), and writing the tests in Python.
Once the code reach my breakpoint I can see all the variables, but few seconds later, instead of seeing their values I get the following exception:
A session is either terminated or not started
This is happening even though I can see the simulator is still running.
I've tried looking online but couldn't find a solution, Can you please help?
Thanks!

You might want to increase newCommandTimeout Desired Capability value to something which will allow you to inspect the elements values. The relevant code line to increase the timeout to 5 minutes would be:
desired_caps['newCommandTimeout'] = '300'
Full initialization routine just in case:
from appium import webdriver
desired_caps = {}
desired_caps['platformName'] = 'iOS'
desired_caps['platformVersion'] = '12.3'
desired_caps['automationName'] = 'xcuitest'
desired_caps['deviceName'] = 'iPhone SE'
desired_caps['newCommandTimeout'] = '300'
driver = webdriver.Remote('http://localhost:4723/wd/hub', desired_caps)
This way Appium will wait for a new command from the client (your code) for 5 minutes prior to considering the client idle and terminating the session, it should be enough to enable debugging, feel free to increase more if needed.
You can also consider switching to Appium Studio which makes your life easier when it comes to inspecting the mobile layout, managing iOS devices/provisioning profiles, generating unique XPath locators for elements automatically and having an extra set of Desired Capabilities allowing you to faster deal with edge cases

python: Selenium webdriver and hanging proxy issue

I am trying to understand how to handle cases when calls which are perfomed through proxies are hanging. For example I have this code:
def call_with_proxy(ip, port):
profile = FirefoxProfile()
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', ip)
profile.set_preference('network.proxy.socks_port', port)
profile.update_preferences()
driver= webdriver.Firefox(profile)
driver.get("http://somewebsite.com")
The proxy is taken from free proxies list here https://hidemyass.com/proxy-list/
Some times everythig works and I am getting the page I am requesting. But sometimes I am getting a blank firefox page (where I can see some elements of the website is being loaded, e.g. css), and this process lasts for a very long time. E.g. session is not being closed even after 10 minutes of such waiting time. I want to ask if there is a way, to automatically close browser if for example page is not loading for some time, or for example the test I am performing stopped execution (due to some reason related to proxies)

In java we have:
webDriver.manage().timeouts().pageLoadTimeout(30, TimeUnit.SECONDS);
From the doc:
pageLoadTimeout
WebDriver.Timeouts pageLoadTimeout(long time,
java.util.concurrent.TimeUnit unit)
Sets the amount of time to wait for a page load to complete before
throwing an error. If the timeout is negative, page loads can be
indefinite.
Parameters:
time - The timeout value.
unit - The unit of time. Returns:
A Timeouts interface.
Quick Googling shows:
webDriver.set_page_load_timeout(30)
for Python. Try this in try-catch (or try-except in your case)

Implement a heartbeat system using queues or other active runtime objects (ie/ weblistener). If you know the maximum runtime for the site script as a whole, you can use SE-Grid like functionality.
If you have variable times on the site, and are only worried about the initial load time, a heartbeat system is the only way I can think of.