I have seen many separate answers and they just seem difficult to follow. I'm hoping someone can give me a simple explanation or at least show me a simple tutorial.
My objective is to build a web page (just on my machine for now) that will have three things: A title, a series of 5-10 dropdown menus/buttons, and a Matplotlib Plot (generated from python code). I want to be able to browse this page:
<html>
<head>
<title>My Example Web Page</title>
</head>
<body bgcolor="#ccffcc">
<form action="my .py file that I want to run when user presses submit" method="post" target="_blank">
<select name="dropdown">
<option value="Opt1" selected>Opt1</option>
<option value="Opt2">Opt2</option>
</select>
<input type="submit" value="Submit"/>
</form>
<p align="center"><img src="MY MATPLOTLIB IMAGE WOULD GO HERE???" width="100" height="100">
</body>
</html>
As seen above, I have the heading no problem. I have one dropdown box -- but when I push submit -- nothing happens. I point to "C:\Python27\myfile.py" but nothing seems to happen. I want to take in the dropdown info (which option was selected), run my code, and then return a MATPLOTLIB plot to the page. I don't understand how to do these two steps. Does anyone have any advice?
Again:
Receive information from user "submit" form
Run the .py file on "submit"
Plot resulting plot on the webpage in the appropriate spot.
FYI - I am not bad with Python programming but have never done HTML (or other web languages).
Thanks!
Related
Im very new to python and web scraping etc but Im trying to learn while I read but Im stuck now. I have managed to use python and BeautifulSoup to grab a kinda web form from a page where theres alot of checkboxes. What Im trying to do is change some of these checkboxes from checked to unchecked or the other way around. But dont know where to really go from here.
The output from a checkbox thats checked looks like this:
<div class="checkbox">
<label>
<input checked="checked" name="Permissions" type="checkbox" value="SeeDetailedInformation"/>
</label>
</div>
The output from a checkbox thats not checked looks like this:
<div class="checkbox">
<label>
<input name="Permissions" type="checkbox" value="AdjustCounters"/>
</label>
</div>
Question is how do I change the checkbox to not beeing check or checked if thats what I want using requests.post or any other good method of changing this.
Any help with either code I can try or pointers to where I should read up on this is much appriciated. I have read abit about selenium and webdriver but dont think this will do it for me as I have 500+ pages/forms on different urls to change. (Going from url to url isnt a problem, I just need some input on have to change the checkbox)
You should look into selenium and its use as a driver for your browser. Beautifulsoup is great for parsing documents, and I'm not familiar with it's usage in form completion, but selenium can certainly do what you're looking for. I would check out this snippet of code too:
checkboxes = webdriver.find_elements_by_class('checkbox')
for i in range(len(checkboxes)):
checkbox = checkboxes[i]
if ((values[i] and not checkbox.is_selected()) or (not values[i] and checkbox.is_selected())):
checkbox.click()
where values is a list of booleans representing whether each checkbox should be selected or not.
This has been pissing me off from yesterday, and I'm just out of ideas.
I'm trying to write a PDF with a subclassed pdfkit.PDFKit (let's call it MyPDFKit): it works well (I just subclassed it to add the possibility of using xvfb-run in the args). I specify that is not a problem of the class.
I was trying to convert some HTML to PDF. The template looks like this:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<!-- Simplified for reading. -->
<style type="text/css">..</style>
</head>
<body>
<!-- Simplified for reading. -->
{% for obj in objs %}
<div>
<div>
<p>{{ obj.name }}</p>
</div>
<p>{{ obj.age }}</p>
</div>
{% endfor %}
</body>
</html>
With these template, and objs having near 400 instances, the output of the HTML is near 5k lines.
The problem comes when trying to splash that into the file. It could be in one of this two places:
MyPDFKit.to_pdf(..) (called from MyPDFKit.from_string(..))'s stdout has a limit size, and truncates part of the string (source code of the function is here).
f.write(..) is the one that truncates the string you pass in.
Cannot be a problem of the template or of the objects' data, because I can create PDFs correctly when getting only a certain range of then (more than 350 items in the same rendering starts leading to the problem due to HTML number of lines). For example, objs[:315] works well, but objs[:350] not.
I've tried setting the buffer size to -1, which is unlimited, but also don't work. Anyone had this issue before?
Ok, so finally, with the help of another programmer, I found the issue.
It looks like PDFKit, when processing a large amount of HTML (in number of PDF pages we're talking more than 349 more or less), sends progress bars comments to the buffer to see how it goes. Then, when it finish, also sends a done comment message.
This comments (I say comments to give they a type of data, cause I don't know really how PDF files handle comments), in programs like Adobe Reader, cannot be handle, so it detects that the file is corrupted/damage, while in programs like SumatraPDF/Edge, it just ignores then and shows the PDF nicely.
Now, how to prevent this behaviour? Passing the --quiet argument. But, for that, you'll need to subclass PDFKit (as I did with MyPDFKit), and add the args manually (line of code).
Problem solved.
EDIT
Seems that I can pass --quiet in the options kwargs, so no need of subclassing if that's only the problem (although it would be nice to have it active by default...)
I'm working on a voting application where people can vote as many times as they want, and i would like to have a reCaptcha checkbox in it, but rather than making people have to fill check the images thingy after a few votes, i'd rather have them just have the fill the checkbox every time.
I tried changing the security level to the lowest setting but it keeps showing the test after about two or three votes.
I know it's not a very good idea security wise, but it just doesn't feel good to have people check several images every time rather than just clicking it once which would already to some fine help holding back spambots.
I'm working on a Django application and implemented reCaptcha with Django-reCaptcha 2 .
No where in the documentation for Google's reCAPTCHA 2.0 does it state that you can do this. I would assume, based some some research into how the CAPTCHA works, that it's impossible for a user to set a checkbox only CAPTCHA.
Google's reCAPTCHA uses a variety of tests, including request tracking, page interaction tracking, scroll tracking and keystroke tracking to attempt to detect if the user on the site is human. You can read more about this here:
http://scraping.pro/no-captcha-recaptcha-challenge/
If Google can't assume with a high enough certainty that the user is human then it will display challenge to attempt to verify this.
As you stated you can lower the Security Level in the admin settings and this will lower the threshold that Google requires to verify a user as human, but if this is still not met a challenge will be presented.
In your case where it works a few times and then presents you with a challenge, this is probably because Google has detected your request count as abnormal and thus can't guarantee you reach the minimum threshold to verify you're a human. You therefore receive a challenge.
So in summary it doesn't seem like you can change this to check box only as Google still needs to be able to obtain enough data to prove the user is human.
However on the lowest Security Level setting you'd imagine most real users will only receive the checkbox.
You can also check how many challenges are being generated in the reCAPTCHA admin panel, as documented here:
https://developers.google.com/recaptcha/docs/faq
Index.html
<html>
<head>
<title>Google recapcha demo - Codeforgeek</title>
<script src='https://www.google.com/recaptcha/api.js'></script>
</head>
<body>
<h1>Google reCAPTHA Demo</h1>
<form id="comment_form" action="form.php" method="post">
<input type="email" placeholder="Type your email" size="40"><br><br>
<textarea name="comment" rows="8" cols="39"></textarea><br><br>
<input type="submit" name="submit" value="Post comment"><br><br>
<div class="g-recaptcha" data-sitekey="=== Your site key ==="></div>
</form>
</body>
</html>
verify.php
<?php
$email;$comment;$captcha;
if(isset($_POST['email']))
$email=$_POST['email'];
if(isset($_POST['comment']))
$comment=$_POST['comment'];
if(isset($_POST['g-recaptcha-response']))
$captcha=$_POST['g-recaptcha-response'];
if(!$captcha){
echo '<h2>Please check the the captcha form.</h2>';
exit;
}
$response=json_decode(file_get_contents("https://www.google.com/recaptcha/api/siteverify?secret=YOUR SECRET KEY&response=".$captcha."&remoteip=".$_SERVER['REMOTE_ADDR']), true);
if($response['success'] == false)
{
echo '<h2>You are spammer ! Get the #$%K out</h2>';
}
else
{
echo '<h2>Thanks for posting comment.</h2>';
}
?>
Sort of like a bot, i have already checked out some sites such as pyjamas and scrapy, I know how to print data of websites, but still dont know how to interact with buttons. Can somebody help me with some demonstarative code?
Lets say i have a form
<form name="input" action="html_form_action.asp" method="get">
Username: <input type="text" name="user" />
<input type="submit" value="Submit" />
</form>
how do i identify the button to be clicked, so that python can click it for me?
If anybody knows any sites with demonstarative code, i would be very pleased.
You can use mechanize for that. It provides an easy way for interacting with websites.
If you’re looking to really simulate a browser, you might want to look at Selenium, which allows you to control a real web browser.
If the website you’re looking to interface with uses a lot of JavaScript (e.g. onclick handlers), it can be very handy.
I'm using django-uni-form to display forms. I've included all the css and javascript (notably jquery) in the page. But now I get some weird looking lines. The image below show how it looks:
http://i243.photobucket.com/albums/ff176/cwalkrox/uni-form1.jpg
You can notice that for username and email address, the lines are aligned with the upper side of text inputs while for two passwords, the lines are below the password inputs. In uni-form's official website, I can't see any line in the 3 examples. Even if it gives me some lines, should it be consistent?
So the strange story doesn't stop here. The jquery can highlight the selected inputs. But the ways it highlights username, email and password are still inconsistent. The following images prove it:
i243.photobucket.com/albums/ff176/cwalkrox/uni-form2.jpg
i243.photobucket.com/albums/ff176/cwalkrox/uni-form3.jpg
So every problems seem to stem from the mysterious lines. So how this happens?
BTW, the page I show you is rendered with the form of django-registration. The rendering snippet is:
<form action="" method="post" class="uniForm">
<fieldset>
{{ form|as_uni_form }}
</fieldset>
</form>
Those lines are due to the css files included in django-uniform: uni-form.css, uni-form-generic.css and uni-form.jquery.css.
It seems weird but at least in my case (a pinax project) the forms look better without the provided css.
my 2 cents