I'm trying to accomplish a little bit of automation which includes submitting a form on a webpage. The values for the form are already coded per item in the list.
I've tried many different modules with Python and nothing seems to give me an answer. I don't have access to Visual Basic and I've personally never dealt with .aspx pages before.
This is the Form name
And I thought I was set and ready to go when I found the parameters for the form:
function ShowEditForm(id, param1, param2, param3, param4) #actual parameter names removed for security
And this is the part that's the major headache:
<INPUT id=__EVENTTARGET type=hidden name=__EVENTTARGET> <INPUT id=__EVENTARGUMENT type=hidden name=__EVENTARGUMENT> <INPUT id=__VIEWSTATE type=hidden value=/wEPDw... #This continues for 800+ characters
I believe this is the cause of my failure of code, am I on a witchhunt trying to post to an .aspx form in python?
Thanks
you would need to parse/parameterize your post headers and contents. this can be non-trivial.
check out mechanize for access at the HTTP level, with some form handling convenience.
check out selenium, for driving a real browser in Python.
I don't think aspx has anything to do with it.
Have you tried http://pypi.python.org/pypi/selenium ?
Indeed, the server-side handling of the POST request won't work if those hidden values aren't present. ASP.NET uses that stuff to track statefulness across multiple requests. Reverse-engineering ASP.NET Web Forms HTTP requests isn't a fun endeavor.
You'll probably need to request the page, scrape the hidden values it gives you, and include those in the POST.
Stepping through a manual interaction with the page and capturing requests/responses in something like FireBug will also give you a good idea of the values being sent back and forth between the client and the server. It wouldn't surprise me if there's some JavaScript emitted to the response which dynamically modifies some hidden fields in server-pre-determined ways as well, helping to indicate which button was pressed or which control was in some way modified.
Asp.net has a feature called viewstate (encrypted page state settings) which you can't fake, and which the page may be using by default and will expect to see on post to the form when submitting back to itself (called post back).
If you control the .aspx code it likely has an associated .cs or .vb file with the code to do the form processing. You can change the code to get values from posted form or URL parameters instead of (or in addition to) controls on the original form. If the site is compiled and you don't see any .vb or .cs files to change you would need to locate the original source files for the solution.
Related
So I had a number of amino acid sequence strings that I wanted to use as input into a tool that studies its interactions with certain components of the human immune system (http://www.cbs.dtu.dk/services/NetMHCcons/).
I wanted to ask what, if any, would be a way of accessing, inputting data and getting the output, via a script (R or python preferably). My main issue was I had a lot of sequences that need to be queried separately so wanted to automate the whole thing. The website has one field that reads "Submission" which takes in the string input. There is another field "select species/loci" which gives a drop down menu from which an option needs to be selected. Lastly there's a "submit" button. The output simply loads on the page after hitting submit.
I've tentatively poked around with RSelenium and Rcurl but wanted to ask if there was a more efficient method.
I took a look at what it'd take to send a POST request to this service from Python, and it looks possible:
this form takes in "multipart/form-data" (see: How to send a "multipart/form-data" with requests in python?), you'll need to send your data in this format. You could inspect a request from the browser (using the dev tools) and copy the fields from there as a starting point.
once the form is submitted, it doesn't give you the result right away. You'd need to get your job ID from the response, and then poll the URL: http://www.cbs.dtu.dk/cgi-bin/webface2.fcgi?jobid={your_job_id}&wait=20 until it gives you the result
the result will then need to be downloaded and parsed
This tool is however available as a portable version for linux/mac: https://services.healthtech.dtu.dk/software.php
Perhaps downloading this version would make it easier?
Try this :
Submitting to a web form using python
This link is an answer to how to send web forms in python, using urllib. Check your source code and extract the necessary data using re module from the source code of the link you have put up, and send the request.
save the HTML source code of http://www.cbs.dtu.dk/services/NetMHCcons/ in the python file as
source_code = '''...'''
The HTML can be found by using CTRL+U in firefox.
So I am attempting to use bottle.py and twitter bootstrap together to make a small website. I need to be able to insert a reasonable amount of data at various points in the HTML using Python but I am not really sure I understand how the HTML and python are communicating.
Here is an example of twitter and bottle working together.
He mentions linking a couple of .js files in the html and I can see where he does that but I am not really sure how that affects how the python interacts with the html. Is there a callback from the Javascript that the python catches using request.GET.get().strip():?
Also one of the last lines is:
return template('templates/gpio.tpl', colour1=colour1, colour2=colour2, colour3=colour3)
I am not sure how the templates/gpio.tpl is connected to the html he mentions below. I understand that the colour# variables are referenced in the html (I assume this happens with the {{}} syntax) but I am not sure how the html gets called at all.
From what I understand (which so far isnt a whole lot) this is how it goes:
User enters "server:port/gpio" into a webbrowser
The python decorator gets called by bottle and the function is run returning the template at the bottom.
This is where I get confused.
A) How does the python script know to call the html?
B) How does the gpio.tpl template code get sent to the html?
C) Is it safe to assume that the python arguements sent to the template function can be referenced using the {{}} syntax or is there more to it?
D) How does the html call back to the python to update the buttons he shows at the bottom?
D.1) does the JS linked at the top have something to do with this?
Lastly: If anyone has a another/better example of linking bootstrap and bottle I would be very happy to see it.
This is quite a loaded post. Thank you for your patience. :D
You'd really need to first learn how the HTTP protocol works... But let's try to quickly answer your main question:
how the HTML and python are communicating
Quite simply: they don't. What happens is:
your client (usually your browser) send an HTTP request to your site
the front web server (Apache, Nginx, whatever) sends this request to the bottle.py application
the bottle.py app dispatch the request to the right controller function matching the url's path portion and request's method against the defined routes)
the controller function does what it has to do and returns an HTTP response to the front web server
the front web server send this response to your client
Usually - but not necessarily - the HTTP response contains HTML content, generated by the controller using a template. IOW : bottle.py uses the template to generate html that is sent back to the client.
Once the response is sent, there's no more "communication" until the client sends another request.
So if I wanted to make an html button that changed something on the page how would I send that response back to bottle.py regenerate the page with the change?
It depends on what you want to change...
For example, pushing the button could trigger a new HTTP request, you should then define a new feature in your code.
Let's take the previous example, and imagine you want to add a switchOff button.
You have to add the following button somewhere in the gpio.tpl :
<input type="submit" class="btn" name="LedsOff" value="Turn off the leds!">
Then, modify the function gpio() to add a new condition with the following :
elif request.GET.get('LedsOff','').strip():
from quick2wire.gpio import Pin, exported
with exported(Pin(12, Pin.Out)) as out1, exported(Pin(13, Pin.Out)) as out2:
out1.value = 0
out2.value = 0
I am currently pulling data from a public series data from https://www3.bcb.gov.br/expectativas/publico/en/serieestatisticas
This is a public page that uses apache wicket I believe.
I usually am ok with scraping, whether GET or POST. Here I and my colleagues are stuck. Can anyone help understand what URL needs to be used to actually make the request. Here's what I've got so far:
The form with inputs:
The Fiddler capture manually executed:
Text View:
form19_hf_0=&indicador=0&calculo=0&linhaPeriodicidade%3Aperiodicidade=0&tfDataInicial=11%2F10%2F2015&tfDataFinal=11%2F24%2F2015&divPeriodoRefereEstatisticas%3AgrupoAnoReferencia%3AanoReferenciaInicial=16&divPeriodoRefereEstatisticas%3AgrupoAnoReferencia%3AanoReferenciaFinal=16&btnCSV=Generate+CSV
Form data I'm passing in the request:
Summary:
I need some help, I can't seem to get the POST working correctly, it takes me to a different page, and I'm not sure of how to work through this one.
NB: I'm trying to grab back a CSV.
The libraries I'm using are primarily Requests (I was going to use LXML but I don't think its going to be applicable here).
I've been trying to figure out the right form with Postman and Fiddler to understand what the request needs to be.
So,
The solution to this was somewhat indirect. We were not able to do a straight POST because the the page incremented the actual POST url in a way that was generally impossible to predict.
The solution that we used was installing Selenium web driver and using that to simulate the dropdown visible values and button clicks.
This worked out very cleanly.
Thanks and HTH anyone else who might have a similar problem.
I'm looking at the html form of an external website (not maintained by myself) in the following format :
<form onsubmit="return check(this)" method=post action=address.aspx target=ttPOST>
....
</form>
I wish to post data to this external website without having to go through the form and pressing submit.
Can I simply go to the url 'address.aspx' and append it with the required input parameters ?
My goal is to automate the periodic posting of information, chosen from a list of frequently changing values on the website. (Using Python and AutoIt)
You can use JQuery.post()
<form action="#" class="myForm" method="post">
<input type="text" id="field1" name="field1" value="" runat="server" />
</form>
// Submit
<div onclick="submit();return false;">Submit</div>
Then the submit() function looks like
function submit() {
$.post("address.aspx", $("form.myForm").serialize(), function(data, textStatus)
{
// Handle success or error
});
}
In codebehind, you can access the post variables
Request.Form["field1"]
I should note that I'm unclear if you were wanting to automate the posting of data from outside a web browser or not. Others have answered doing it with script and such like from the web page so I thought I'd cover how it works when you are doing it from a standalone program.
For most languages you can get things that will help you simulate web requests. Most of these would probably allow you to make a post request and supply post data. I dont' know python and autoit but in teh general sense you'd just need to get the name value pairs by looking at the HTML of the form (or by examining a request being made to the server is probably better) and then make a post request.
As others have said if you want to just append the values to teh url then the server will need to be happy to accept a GET request instead of a post. Sometimes you will find that servers will do this happily (they don't care how form submission is done, as long as they get data), other times it will be specifically looking for a post and will ignore parameters passed as part of the querystring.
Change the method attribute from POST to GET.
Read about the differences here.
You can get away with changing the URL to go to the external site using the same syntax of GET parameter (?param1=val1¶m2=val2...), but to achieve this you will need to write this logic yourself in javascript.
Why do you want to get around the form submit? Submit the form and use the values in HTTP_POST. You can use HTTP_GET by changing the method to GET in the above html but the form will still submit.
One way to submit the params to address.aspx would be to use javascript....build a string of all the params and submit that to address.aspx when the user clicks on the submit button. You'll need to disable the submit buttons default behaviour.
You could make AJAX GET requests (appending the data to the URL) and execute them with a certain time interval.
I would recommend jQuery, but it may not be ideal in a .NET environment.
jQuery.ajax() - http://api.jquery.com/jQuery.ajax/
jQuery.get() - http://api.jquery.com/jQuery.get/
NOTE:
I wish to post data to this external website without having to go through the form and pressing submit.
I took this as meaning you didn't want to actually submit the form, do you want to submit the form but simply not by pressing the submit button? If you actually want the form to submit, then you should indeed just change the form's method attribute to get instead of post.
If however you do wish to stay on the form page and need to post the data to some other resource with the data in the URL string (there are reasons for and aganist doing this that you should look into -- scroll down to 9.3 GET), then just make an AJAX GET request, otherwise just POST the data using AJAX. Either way, you'll have to make some sort of an asynchronous call.
POST is not the same as GET -- if you append the data that you want to the URL, the page will not see your variables. That is to say GETTING (a.k.a going to) http://www.the-other-site.com/address.aspx?variable1=a&variable2=b will not work.
You need to build POST request and submit it to address.aspx.
If you want to do it yourself then you'll need Python's urllib2 module (More specifically, the Request object). If you want to use a pre-built solution, you'll need something like mechanize to POST your variables.
I would like to know if it is possible to submit a flash form from python and, if it is, how?
I have done form submitting from python before, but the forms were HTML not flash. I really have no idea on how to do this. In my research about this I kept getting 'Ming'. However, Ming is only to create .swf files and that's not what I intend to do.
Any help on this is greatly appreciated.
You can set the url attribute (I think it's url, please correct me if I'm wrong) on a Flash form control to a Python script - then it will pass it through HTTP POST like any normal HTML form.
You've got nothing to be afraid of, it uses the same protocol to communicate, it's just a different submission process.
For your flash app, there's no difference if the backend is python, php or anything, so you can follow a normal "php + flash contact form" guide and then build the backend using django or any other python web framework, receive the information from the http request (GET or POST, probably the last one) and do whatever you wanted to do with them.
Notice the response from python to flash works the same as with php, it's just http content, so you can use XML or even better, JSON.