We Keep Coding

how to avoid bot detection on websites using selenium python

We are trying to automate process using selenium python for a website but as we proceed with the process the bot gets detected every time and a captcha comes up. Even though a human solve that captcha the website does not allow to move forward and continuously keeps detecting the bot and again and again shows the captcha to solve. So far we have tried all methods whichever we have explored to overcome it but none of method worked. Can someone help on this issue ?
Some of the methods tried:
1.) using spoofing user agents
2.) using proxies
3.) turn-off useAutomationExtension
4.) changing the property value of navigator for web driver to undefined
5.) disable-blink-features
6.) exclude the collection of enable-automation switches

There is something called "Undetected ChromeDriver" you can check out!
Optimized Selenium Chromedriver patch which does not trigger anti-bot services like Distill Network / Imperva / DataDome / Botprotect.io Automatically downloads the driver binary and patches it.
here is the link
Here is another useful website you can check out, this website shows if a site will detect you using selenium or not or anything like that:
LINK
Also for future reference on Stack Overflow you should steer away from opinion-based questions. Read this to learn more about asking a good question.

Programmatically access and modify.aspx page

I am working on a project which needs to programmatically access and update a .aspx (ASP.NET) page. Specifically, I need to automatically access this page, use several html and JavaScript elements (click checkboxes, enter text in form fields, "click" buttons), and reload the page. Also, during the time the page is accessed, there is information being sent back and forth between the client and server.
What is the most efficient way to go about this? I am most likely thinking about writing something in bash + python to do this but I am not sure it is the best tool for the job.
Thanks

The optimal solution for your problem is using Selenium with python.
The selenium package is used to automate web browser interaction from Python.
pip install -U selenium
You can read the documentation to get familiar with the Selenium Webdriver API.
You cannot edit the pages that are hosted by others, but you can mimic the requests using selenium.

How to check for webpages' popups?

Is there a possibility if I code a program in python that allows to automatically browse a given website using mechanize to detect if there are popup windows (suggesting advertisements or downloading actions ...) using Python ?. I would appreciate any hint (for example, if you give me a library that fulfills this task I would be very happy)

Mechanize cannot handle javascript and popup windows:
How do I use Mechanize to process JavaScript?
Mechanize and Javascript
To accomplish the goal, you need to utilize a real browser, headless or not. This is where selenium would help. It has a built-in support for popup dialogs:
Selenium WebDriver has built-in support for handling popup dialog
boxes. After you’ve triggerd and action that would open a popup, you
can access the alert with the following:
alert = driver.switch_to_alert()
Example (using this jsfiddle):
from selenium import webdriver
url = "http://fiddle.jshell.net/ebkXh/show/"
driver = webdriver.Firefox()
driver.get(url)
button = driver.find_element_by_xpath('//button[#type="submit"]')
# dismiss
button.click()
driver.switch_to.alert.dismiss()
# accept
button.click()
driver.switch_to.alert.accept()
See also:
Handle Popup Windows
Click the javascript popup through webdriver

Unfortunately, Mechanize's browser seems to skip the pop-ups so the title, URL, and HTML are identical for both pop-ups and normal pages.
Frankly, Python is not the right tool for this job and is lagging behind in this respect IMHO. Having spent months doing web crawling, for sites that use Javascript extensively (the number of which is greatly increasing nowadays), I find that using Javascript-Based environments like PhantomJS or SlimerJS are simply better for what you're trying to do.
If you have the luxury to use Javascript-Based environments, I'd say go right ahead. However, you can still use python. PhantomJS embeds Ghost Driver. You can use Ghost.py to utilize the power of PhantomJS. Or you can use Selenium with Python as illustrated here.

Selenium: What functions would fire request?

I am new to Selenium and web applications. Please bear with me for a second if my question seems way too obvious. Here is my story.
I have written a scraper in Python that uses Selenium2.0 Webdriver to crawl AJAX web pages. One of the biggest challenge (and ethics) is that I do not want to burn down the website's server. Therefore I need a way to monitor the number of requests my webdriver is firing on each page parsed.
I have done some google-searches. It seems like only selenium-RC provides such a functionality. However, I do not want to rewrite my code just for this reason. As a compromise, I decided to limit the rate of method calls that potentially lead to the headless browser firing requests to the server.
In the script, I have the following kind of method calls:
driver.find_element_by_XXXX()
driver.execute_script()
webElement.get_attribute()
webElement.text
I use the second function to scroll to the bottom of the window and get the AJAX content, like the following:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
Based on my intuition, only the second function will trigger request firing, since others seem like parsing existing html content.
Is my intuition wrong?
Many many thanks
Perhaps I should elaborate more. I am automating a process of crawling on a website in Python. There is a subtantial amount of work done, and the script is running without large bugs.
My colleagues, however, reminded me that if in the process of crawling a page I made too many requests for the AJAX list within a short time, I may get banned by the server. This is why I started looking for a way to monitor the number of requests I am firing from my headless PhantomJS browswer in script.
Since I cannot find a way to monitor the number of requests in script, I made the compromise I mentioned above.

Therefore I need a way to monitor the number of requests my webdriver
is firing on each page parsed
As far as I know, the number of requests is depending on the webpage's design, i.e. the resources used by the webpage and the requests made by Javascript/AJAX. Webdriver will open a browser and load the webpage just like a normal user.
In Chrome, you can check the requests and responses using Developer Tools panel. You can refer to this post. The current UI design of Developer Tools is different but the basic functions are still the same. Alternatively, you can also use the Firebug plugin in Firefox.
Updated:
Another method to check the requests and responses is by using Wireshark. Please refer to these Wireshark filters.

Selenium Webdriver for Python: get page, enter values, click submit, get source

Alright, I'm confused. So I want to scrape a page using Selenium Webdriver and Python. I've recorded a test case in the Selenium IDE. It has stuff like
Command Taget
click link=14
But I don't see how to run that in Python. The desirable end result is that I have the source of the final page.
Is there a run_test_case command? Or do I have to write individual command lines? I'm rather missing the link between the test case and the actual automation. Every site tells me how to load the initial page and how to get stuff from that page, but how do I enter values and click on stuff and get the source?
I've seen:
submitButton=driver.find_element_by_xpath("....")
submitButton.click()
Ok. And enter values? And get the source once I've submitted a page? I'm sorry that this is so general, but I really have looked around and haven't found a good tutorial that actually shows me how to do what I thought was the whole point of Selenium Webdriver.

I've never used the IDE. I just write my tests or site automation by hand.
from selenium import webdriver
browser = webdriver.Firefox()
browser.get("http://www.google.com")
print browser.page_source
You could put that in a script and just do python wd_script.py or you could open up a Python shell and type it in by hand, watch the browser open up, watch it get driven by each line. For this to work you will obviously need Firefox installed as well. Not all versions of Firefox work with all versions of Selenium. The current latest versions of each (Firefox 19, Selenium 2.31) do though.
An example showing logging into a form might look like this:
username_field = browser.find_element_by_css_selector("input[type=text]")
username_field.send_keys("my_username")
password_field = browser.find_element_by_css_selector("input[type=password]")
password_field.send_keys("sekretz")
browser.find_element_by_css_selector("input[type=submit]").click()
print browser.page_source
This kind of stuff is much easier to write if you know css well. Weird errors can be caused by trying to find elements that are being generated in JavaScript. You might be looking for them before they exist for instance. It's easy enough to tell if this is the case by putting in a time.sleep for a little while and seeing if that fixes the problem. More elegantly you can abstract some kind of general wait for element function.
If you want to run Webdriver sessions as part of a suite of integration tests then I would suggest using Python's unittest to create them. You drive the browser to the site under test, and make assertions that the actions you are taking leave the page in a state you expect. I can share some examples of how that might work as well if you are interested.