Sort of like a bot, i have already checked out some sites such as pyjamas and scrapy, I know how to print data of websites, but still dont know how to interact with buttons. Can somebody help me with some demonstarative code?
Lets say i have a form
<form name="input" action="html_form_action.asp" method="get">
Username: <input type="text" name="user" />
<input type="submit" value="Submit" />
</form>
how do i identify the button to be clicked, so that python can click it for me?
If anybody knows any sites with demonstarative code, i would be very pleased.
You can use mechanize for that. It provides an easy way for interacting with websites.
If you’re looking to really simulate a browser, you might want to look at Selenium, which allows you to control a real web browser.
If the website you’re looking to interface with uses a lot of JavaScript (e.g. onclick handlers), it can be very handy.
Related
Im very new to python and web scraping etc but Im trying to learn while I read but Im stuck now. I have managed to use python and BeautifulSoup to grab a kinda web form from a page where theres alot of checkboxes. What Im trying to do is change some of these checkboxes from checked to unchecked or the other way around. But dont know where to really go from here.
The output from a checkbox thats checked looks like this:
<div class="checkbox">
<label>
<input checked="checked" name="Permissions" type="checkbox" value="SeeDetailedInformation"/>
</label>
</div>
The output from a checkbox thats not checked looks like this:
<div class="checkbox">
<label>
<input name="Permissions" type="checkbox" value="AdjustCounters"/>
</label>
</div>
Question is how do I change the checkbox to not beeing check or checked if thats what I want using requests.post or any other good method of changing this.
Any help with either code I can try or pointers to where I should read up on this is much appriciated. I have read abit about selenium and webdriver but dont think this will do it for me as I have 500+ pages/forms on different urls to change. (Going from url to url isnt a problem, I just need some input on have to change the checkbox)
You should look into selenium and its use as a driver for your browser. Beautifulsoup is great for parsing documents, and I'm not familiar with it's usage in form completion, but selenium can certainly do what you're looking for. I would check out this snippet of code too:
checkboxes = webdriver.find_elements_by_class('checkbox')
for i in range(len(checkboxes)):
checkbox = checkboxes[i]
if ((values[i] and not checkbox.is_selected()) or (not values[i] and checkbox.is_selected())):
checkbox.click()
where values is a list of booleans representing whether each checkbox should be selected or not.
So there is a page with the following contents;
<input type="text" name="input1">
<input type="text" name="input2">
<button onclick="dosomething()" id="check">Check</button>
This page gets loaded in with javascript, so you need to wait until it's fully loaded. After that the text files has to be filled in, and click the submit button. If it succeeds, it will set a cookie and redirect you. If it doesn't succeeds, it doesn't do anything.
How can I make this work in python? I couldn't really find a answer that would work for me. This python code will be runned in Windows.
Most likely you could just send direct request with form's field. Based on form's method it'll be GET or POST request. Later you can catch and save coockies and head to redirected page.
If it doesn't work you could also try Selenium to emulate user's behaviour
I'm working on a voting application where people can vote as many times as they want, and i would like to have a reCaptcha checkbox in it, but rather than making people have to fill check the images thingy after a few votes, i'd rather have them just have the fill the checkbox every time.
I tried changing the security level to the lowest setting but it keeps showing the test after about two or three votes.
I know it's not a very good idea security wise, but it just doesn't feel good to have people check several images every time rather than just clicking it once which would already to some fine help holding back spambots.
I'm working on a Django application and implemented reCaptcha with Django-reCaptcha 2 .
No where in the documentation for Google's reCAPTCHA 2.0 does it state that you can do this. I would assume, based some some research into how the CAPTCHA works, that it's impossible for a user to set a checkbox only CAPTCHA.
Google's reCAPTCHA uses a variety of tests, including request tracking, page interaction tracking, scroll tracking and keystroke tracking to attempt to detect if the user on the site is human. You can read more about this here:
http://scraping.pro/no-captcha-recaptcha-challenge/
If Google can't assume with a high enough certainty that the user is human then it will display challenge to attempt to verify this.
As you stated you can lower the Security Level in the admin settings and this will lower the threshold that Google requires to verify a user as human, but if this is still not met a challenge will be presented.
In your case where it works a few times and then presents you with a challenge, this is probably because Google has detected your request count as abnormal and thus can't guarantee you reach the minimum threshold to verify you're a human. You therefore receive a challenge.
So in summary it doesn't seem like you can change this to check box only as Google still needs to be able to obtain enough data to prove the user is human.
However on the lowest Security Level setting you'd imagine most real users will only receive the checkbox.
You can also check how many challenges are being generated in the reCAPTCHA admin panel, as documented here:
https://developers.google.com/recaptcha/docs/faq
Index.html
<html>
<head>
<title>Google recapcha demo - Codeforgeek</title>
<script src='https://www.google.com/recaptcha/api.js'></script>
</head>
<body>
<h1>Google reCAPTHA Demo</h1>
<form id="comment_form" action="form.php" method="post">
<input type="email" placeholder="Type your email" size="40"><br><br>
<textarea name="comment" rows="8" cols="39"></textarea><br><br>
<input type="submit" name="submit" value="Post comment"><br><br>
<div class="g-recaptcha" data-sitekey="=== Your site key ==="></div>
</form>
</body>
</html>
verify.php
<?php
$email;$comment;$captcha;
if(isset($_POST['email']))
$email=$_POST['email'];
if(isset($_POST['comment']))
$comment=$_POST['comment'];
if(isset($_POST['g-recaptcha-response']))
$captcha=$_POST['g-recaptcha-response'];
if(!$captcha){
echo '<h2>Please check the the captcha form.</h2>';
exit;
}
$response=json_decode(file_get_contents("https://www.google.com/recaptcha/api/siteverify?secret=YOUR SECRET KEY&response=".$captcha."&remoteip=".$_SERVER['REMOTE_ADDR']), true);
if($response['success'] == false)
{
echo '<h2>You are spammer ! Get the #$%K out</h2>';
}
else
{
echo '<h2>Thanks for posting comment.</h2>';
}
?>
I have this little website i want to fill in a form with the requests library. The problem is i cant get to the next site when filling the form data and hitting the button(Enter does not work).
The important thing is I can't do it via a clicking bot of some kind. This needs to be done so I can run in without graphics.
info = {'name':'JohnJohn',
'message':'XXX',
'sign':"XXX",
'step':'1'}
First three entries name, message, sign are the text areas and step is I think the button.
r = requests.get(url)
r = requests.post(url, data=info)
print(r.text)
The Form Data looks like this when i send a request via chrome manually:
name:JohnJohn
message:XXX
sign:XXX
step:1
The button element looks like this:
<td colspan="2" style="text-align: center;">
<input name="step" type="hidden" value="1">
<button id="button" type="button" onclick="myClick();"
style="background-color: #ef4023; width: 80px; font-face: times; font-size: 14pt;">
Wyślij
</button>
</td>
The next site if i do this manually has the same adres.
As you might see from the snipped you posted, clicking the button is triggering some JavaScript code, namely a method called myClick().
It is not straightforward to click on this thing using pythons requests library. You might have more luck trying to find out what happens inside myClick(). My guess would be that at some point, a POST request will be made to a HTTP endpoint. If you can figure this out, you can translate it into your python code.
If that does not work, another option would be to use something like Selenium/PhantomJS, which give you the ability to have a real, headless and scriptable browser. Using such a tool, you can actually have it fill out forms and click buttons. You can have a look at this so answer, as it shows you how to use Selenium+PhantomJS from python.
Please make sure not to abuse such methods by spamming forums or [insert illegal or otherwise abusive activity here].
In such a situation when you need to forge scripted button's request, it may be easier not to guess the logic of JS but instead perform a physical click and look into chrome devtools' network sniffer which gives you a plain request made which, in turn, can be easily forged in Python
I have seen many separate answers and they just seem difficult to follow. I'm hoping someone can give me a simple explanation or at least show me a simple tutorial.
My objective is to build a web page (just on my machine for now) that will have three things: A title, a series of 5-10 dropdown menus/buttons, and a Matplotlib Plot (generated from python code). I want to be able to browse this page:
<html>
<head>
<title>My Example Web Page</title>
</head>
<body bgcolor="#ccffcc">
<form action="my .py file that I want to run when user presses submit" method="post" target="_blank">
<select name="dropdown">
<option value="Opt1" selected>Opt1</option>
<option value="Opt2">Opt2</option>
</select>
<input type="submit" value="Submit"/>
</form>
<p align="center"><img src="MY MATPLOTLIB IMAGE WOULD GO HERE???" width="100" height="100">
</body>
</html>
As seen above, I have the heading no problem. I have one dropdown box -- but when I push submit -- nothing happens. I point to "C:\Python27\myfile.py" but nothing seems to happen. I want to take in the dropdown info (which option was selected), run my code, and then return a MATPLOTLIB plot to the page. I don't understand how to do these two steps. Does anyone have any advice?
Again:
Receive information from user "submit" form
Run the .py file on "submit"
Plot resulting plot on the webpage in the appropriate spot.
FYI - I am not bad with Python programming but have never done HTML (or other web languages).
Thanks!