Filling an input in any website and pressing enter - python

I'm trying to write a small python script that will get a list of URLs (which are mostly just a search bar with some other elements), "write" something in the search bar and "press" enter.
My goal is to get to the next page, where all the search results are, and look at the new URL
The sites are different, so I can't just get the query parameter since I don't know it, and every site is different.
I was thinking about searching for the "input" part of the page (since the search bar is supposed to be the only input there), and send something to it. Then wait for the new URL.
Is that possible? Is there a smarter way?
I'm trying to avoid using something else but python for now (selenium, etc..)
I was searching every possible answer here and on the web, but nothing was possible so I was thinking about using the input part somehow...

Without selenium and similar softwares, you'll have to understand what is actually happening when you click such a button.
I'll take an example on a famous site (hint: if you read this you'll know which site I mean). In the HTML source I can see that (truncated) piece of code:
<form id="search" role="search" action=/search method="get" class="grid--cell fl-grow1 searchbar px12 js-searchbar " autocomplete="off">
<div class="ps-relative">
<input name="q"
type="text"
...
What it means is that when you submit the form, the browser would hit the URL /search (which here means https://stackoverflow.com/search), with a get request, wrapping all of the form's fields in the URL, after a ?. If I searched for the term python it would lead to http://stackoverflow.com/search?q=python.
Note that according to their content, the parameters would need to be URL-encoded.
If the form contains more input fields, you'll have to wrap them too, separating them by & signs, in such a way: param1=value1&param2=value2&...
To sum up, searching only inputs won't be sufficient, you'll have to parse the forms.
Not knowing more about your data, I cannot elaborate, but I think you might be able to do something with that.

Related

Django - turn field into tags with dropdown

In my form on django, I have a field called package_includes and a field called price. Right now, the content in package_includes it is simply text. So if I input "paper, glue, glitter" it will display exactly as i typed it, unable to change anything. However, I want the text to work like individual tags ->How I want tags to look
So when you click on one of the individual items (EX: "paper", "glue", or "glitter") i want it to display a drop down that allows you the option to put extra glitter for an additional $2.00 to the total.
Also, at the end of the text I want a tag that displays "Add" and this function would allow you to add items not included in this specific package. For example, you should be able to add crayons for $5, highlighter for $2.50, etc.
I am new to programming and don't fully understand how to add a function to text like this in a field in django. Thank you in advance for your help!
The answer to your question is unfortunately not a step-by-step instruction of how to go from 0 to a complete working example - only a pointer in the right direction.
What you need is achieved through CSS (producing the visual effect you want) and JavaScript (handling the drop-down functionality; it most probably has to be your own script based on jQuery). All of what you want needs to live (execute) within the Internet browser, and Django can only help there by serving the static CSS and script files, and referencing them in the actual page, when you put them in the correct template.
Here is an example of Wagtail Admin doing something similar:
If you open the page source in your browser, all you will see as HTML code for this is the following:
<div class="field-content">
<div class="input ">
<input id="id_tags" name="tags" type="text" value=""hello world", blog" /><script>initTagField("id_tags", "/admin/tag-autocomplete/");</script>
<span></span>
</div>
</div>
But if you open the page with FireBug, your browser's Development Tools, or the Web Developer extension, with visual styles applied it will decompose to something like this:
If you notice, the applied style to the form input element is display: none;. What you see is actually the styled unordered list elements right after. In order to be able to use them in your script, you need to be able to get to them/their contents from within your JS function. One strategy would be to assign an id attribute to all of the visible <li> elements. It can be something counter-based (e.g. id="shopping-cart-item-0", id="shopping-cart-item-1", etc.). Another way would be to assign an id to the <div class="input"> element, and within your function get all its DOM descendants of type <li>. Whatever works better for you. Then you could parse the label (inner text) of the list element in your script to get the type of item you are dealing with, and from then on find the discount price and apply it...
I would suggest that you start with a working example - find some app that uses taggit or similar package for Django and install it in a test application. Then you can use your browser's or preferred web development extension to play around with the CSS on them and see how different options affect the visual aspect of what you need. Once you are comfortable with that, see how you can build drop-down menus with jQuery. I believe the jQuery site has enough tutorials on the topic.

How to change automatically html <input> tags using python?

Problem.
I have a real html page, which could be parsed by beautifulsoup (or something else) if needed.
I have some identical input tags on this page. Say, a 1000 input tags that look like:
<input class=​"text_class​" type=​"text" size=​"3" value=​"20">​
And on the end of the page I have submit-type input tag that looks like this:
<input class=​"submit_class" type=​"submit" value=​"Submit">​
The question is how could I change the values of my 1000 tags to be, lets say, 30, not 20 and submit all the changes using final submit-tag.
All I need is a human-like behavior on the page on a big amount of data performed by python script.
I suspect that maybe some special python library would be helpful in this case.
Thanks in advance for any help or pointing directions to look at!
You have several options to change the input values and submit a form:
mechanize
selenium
Ghost.py
Note that Selenium and Ghost.py use a real browser. Mechanize is the easiest way to go in your case, just note that it may not work for you if there is a lot of javascript involved in the form initializing or submitting process.

Web Automation - Dealing with .aspx

I'm trying to accomplish a little bit of automation which includes submitting a form on a webpage. The values for the form are already coded per item in the list.
I've tried many different modules with Python and nothing seems to give me an answer. I don't have access to Visual Basic and I've personally never dealt with .aspx pages before.
This is the Form name
And I thought I was set and ready to go when I found the parameters for the form:
function ShowEditForm(id, param1, param2, param3, param4) #actual parameter names removed for security
And this is the part that's the major headache:
<INPUT id=__EVENTTARGET type=hidden name=__EVENTTARGET> <INPUT id=__EVENTARGUMENT type=hidden name=__EVENTARGUMENT> <INPUT id=__VIEWSTATE type=hidden value=/wEPDw... #This continues for 800+ characters
I believe this is the cause of my failure of code, am I on a witchhunt trying to post to an .aspx form in python?
Thanks
you would need to parse/parameterize your post headers and contents. this can be non-trivial.
check out mechanize for access at the HTTP level, with some form handling convenience.
check out selenium, for driving a real browser in Python.
I don't think aspx has anything to do with it.
Have you tried http://pypi.python.org/pypi/selenium ?
Indeed, the server-side handling of the POST request won't work if those hidden values aren't present. ASP.NET uses that stuff to track statefulness across multiple requests. Reverse-engineering ASP.NET Web Forms HTTP requests isn't a fun endeavor.
You'll probably need to request the page, scrape the hidden values it gives you, and include those in the POST.
Stepping through a manual interaction with the page and capturing requests/responses in something like FireBug will also give you a good idea of the values being sent back and forth between the client and the server. It wouldn't surprise me if there's some JavaScript emitted to the response which dynamically modifies some hidden fields in server-pre-determined ways as well, helping to indicate which button was pressed or which control was in some way modified.
Asp.net has a feature called viewstate (encrypted page state settings) which you can't fake, and which the page may be using by default and will expect to see on post to the form when submitting back to itself (called post back).
If you control the .aspx code it likely has an associated .cs or .vb file with the code to do the form processing. You can change the code to get values from posted form or URL parameters instead of (or in addition to) controls on the original form. If the site is compiled and you don't see any .vb or .cs files to change you would need to locate the original source files for the solution.

Plone : Check if a form was submitted

I'm working on a Plone template. Currently I'm implementing a search on the template. There is a form on the template which submits to the same template i.e. the same page.
I need to enable certain parts of the page to be viewed i.e. the results DIV, only when the form in the page has been submitted. I am not sure of how to check if a page has been submitted and display certain portions of the page.
I've looked at this link on using form.submitted, but it is really not helpful. I'm guessing whatever I have to do will be done in tal:condition tag.
Any help will be appreciated.
You can add an hidden field, name for example "submitedform" with value="1" and then, make a condition on :
request.get('submitedform', 0)
I found a way to make it work.
I used a tal:condition and checked for data that is inside the request object upon form submission.
To examine what is in the request object, simply put the following on your page:
<div tal:replace="structure request" />
Note: when the request object is called, it renders a readable, HTML version of the data. We use "structure" to prevent escaping the HTML.
Now, for my case, I needed to check whether my search form was submitted. So inside the DIV tag, I checked if the query string was empty or not. Below is a sample of the code:
<!-- Form has been submitted -->
<div tal:condition="python:request.environ.get('QUERY_STRING') != ''"></div>
<!-- Form has not been submitted -->
<div tal:condition="python:request.environ.get('QUERY_STRING') == ''"></div>
Note that you can use any variable you want. I simply chose to use the QUERY_STRING from the request object.
This might not be perfect, but I hope it will give direction to someone else.

html form submission

I'm looking at the html form of an external website (not maintained by myself) in the following format :
<form onsubmit="return check(this)" method=post action=address.aspx target=ttPOST>
....
</form>
I wish to post data to this external website without having to go through the form and pressing submit.
Can I simply go to the url 'address.aspx' and append it with the required input parameters ?
My goal is to automate the periodic posting of information, chosen from a list of frequently changing values on the website. (Using Python and AutoIt)
You can use JQuery.post()
<form action="#" class="myForm" method="post">
<input type="text" id="field1" name="field1" value="" runat="server" />
</form>
// Submit
<div onclick="submit();return false;">Submit</div>
Then the submit() function looks like
function submit() {
$.post("address.aspx", $("form.myForm").serialize(), function(data, textStatus)
{
// Handle success or error
});
}
In codebehind, you can access the post variables
Request.Form["field1"]
I should note that I'm unclear if you were wanting to automate the posting of data from outside a web browser or not. Others have answered doing it with script and such like from the web page so I thought I'd cover how it works when you are doing it from a standalone program.
For most languages you can get things that will help you simulate web requests. Most of these would probably allow you to make a post request and supply post data. I dont' know python and autoit but in teh general sense you'd just need to get the name value pairs by looking at the HTML of the form (or by examining a request being made to the server is probably better) and then make a post request.
As others have said if you want to just append the values to teh url then the server will need to be happy to accept a GET request instead of a post. Sometimes you will find that servers will do this happily (they don't care how form submission is done, as long as they get data), other times it will be specifically looking for a post and will ignore parameters passed as part of the querystring.
Change the method attribute from POST to GET.
Read about the differences here.
You can get away with changing the URL to go to the external site using the same syntax of GET parameter (?param1=val1&param2=val2...), but to achieve this you will need to write this logic yourself in javascript.
Why do you want to get around the form submit? Submit the form and use the values in HTTP_POST. You can use HTTP_GET by changing the method to GET in the above html but the form will still submit.
One way to submit the params to address.aspx would be to use javascript....build a string of all the params and submit that to address.aspx when the user clicks on the submit button. You'll need to disable the submit buttons default behaviour.
You could make AJAX GET requests (appending the data to the URL) and execute them with a certain time interval.
I would recommend jQuery, but it may not be ideal in a .NET environment.
jQuery.ajax() - http://api.jquery.com/jQuery.ajax/
jQuery.get() - http://api.jquery.com/jQuery.get/
NOTE:
I wish to post data to this external website without having to go through the form and pressing submit.
I took this as meaning you didn't want to actually submit the form, do you want to submit the form but simply not by pressing the submit button? If you actually want the form to submit, then you should indeed just change the form's method attribute to get instead of post.
If however you do wish to stay on the form page and need to post the data to some other resource with the data in the URL string (there are reasons for and aganist doing this that you should look into -- scroll down to 9.3 GET), then just make an AJAX GET request, otherwise just POST the data using AJAX. Either way, you'll have to make some sort of an asynchronous call.
POST is not the same as GET -- if you append the data that you want to the URL, the page will not see your variables. That is to say GETTING (a.k.a going to) http://www.the-other-site.com/address.aspx?variable1=a&variable2=b will not work.
You need to build POST request and submit it to address.aspx.
If you want to do it yourself then you'll need Python's urllib2 module (More specifically, the Request object). If you want to use a pre-built solution, you'll need something like mechanize to POST your variables.

Categories