We Keep Coding

How to "log in" to zalando.de using Python's Requests module?

Regardless of Zalando usually blocking any requests traffic (I already know how to get around this), how can I detect the Post method that Zalando uses to login me in using their form zalando.de/myaccount/? With DevTools I don't seem to find the specific post method.
As far as I could see: with having the data that is needed, I could then perform something like this: How to "log in" to a website using Python's Requests module?
Can anyone show me how such a request would look like? Thanks.

Firstly, please respect their wishes to not send direct requests to their site.
For any other site that allows it:
Make sure you set up your dev tools to not clear the log on redirect
Check the response header (or your browser-storage) for which cookies have been set, since those are used to identify your session (e.g. Service-Client-Id, ...)
Make a request with those cookies from your script to pretend to have a logged in user
NEVER SHARE OR POST THOSE COOKIES ANYWHERE and read up on session-hijacking

Python request get data through HTTPS tunnel

I am currently pulling data from a public series data from https://www3.bcb.gov.br/expectativas/publico/en/serieestatisticas
This is a public page that uses apache wicket I believe.
I usually am ok with scraping, whether GET or POST. Here I and my colleagues are stuck. Can anyone help understand what URL needs to be used to actually make the request. Here's what I've got so far:
The form with inputs:
The Fiddler capture manually executed:
Text View:
form19_hf_0=&indicador=0&calculo=0&linhaPeriodicidade%3Aperiodicidade=0&tfDataInicial=11%2F10%2F2015&tfDataFinal=11%2F24%2F2015&divPeriodoRefereEstatisticas%3AgrupoAnoReferencia%3AanoReferenciaInicial=16&divPeriodoRefereEstatisticas%3AgrupoAnoReferencia%3AanoReferenciaFinal=16&btnCSV=Generate+CSV
Form data I'm passing in the request:
Summary:
I need some help, I can't seem to get the POST working correctly, it takes me to a different page, and I'm not sure of how to work through this one.
NB: I'm trying to grab back a CSV.
The libraries I'm using are primarily Requests (I was going to use LXML but I don't think its going to be applicable here).
I've been trying to figure out the right form with Postman and Fiddler to understand what the request needs to be.

So,
The solution to this was somewhat indirect. We were not able to do a straight POST because the the page incremented the actual POST url in a way that was generally impossible to predict.
The solution that we used was installing Selenium web driver and using that to simulate the dropdown visible values and button clicks.
This worked out very cleanly.
Thanks and HTH anyone else who might have a similar problem.

Web Automation - Dealing with .aspx

I'm trying to accomplish a little bit of automation which includes submitting a form on a webpage. The values for the form are already coded per item in the list.
I've tried many different modules with Python and nothing seems to give me an answer. I don't have access to Visual Basic and I've personally never dealt with .aspx pages before.
This is the Form name
And I thought I was set and ready to go when I found the parameters for the form:
function ShowEditForm(id, param1, param2, param3, param4) #actual parameter names removed for security
And this is the part that's the major headache:
<INPUT id=__EVENTTARGET type=hidden name=__EVENTTARGET> <INPUT id=__EVENTARGUMENT type=hidden name=__EVENTARGUMENT> <INPUT id=__VIEWSTATE type=hidden value=/wEPDw... #This continues for 800+ characters
I believe this is the cause of my failure of code, am I on a witchhunt trying to post to an .aspx form in python?
Thanks

you would need to parse/parameterize your post headers and contents. this can be non-trivial.
check out mechanize for access at the HTTP level, with some form handling convenience.
check out selenium, for driving a real browser in Python.

I don't think aspx has anything to do with it.
Have you tried http://pypi.python.org/pypi/selenium ?

Indeed, the server-side handling of the POST request won't work if those hidden values aren't present. ASP.NET uses that stuff to track statefulness across multiple requests. Reverse-engineering ASP.NET Web Forms HTTP requests isn't a fun endeavor.
You'll probably need to request the page, scrape the hidden values it gives you, and include those in the POST.
Stepping through a manual interaction with the page and capturing requests/responses in something like FireBug will also give you a good idea of the values being sent back and forth between the client and the server. It wouldn't surprise me if there's some JavaScript emitted to the response which dynamically modifies some hidden fields in server-pre-determined ways as well, helping to indicate which button was pressed or which control was in some way modified.

Asp.net has a feature called viewstate (encrypted page state settings) which you can't fake, and which the page may be using by default and will expect to see on post to the form when submitting back to itself (called post back).
If you control the .aspx code it likely has an associated .cs or .vb file with the code to do the form processing. You can change the code to get values from posted form or URL parameters instead of (or in addition to) controls on the original form. If the site is compiled and you don't see any .vb or .cs files to change you would need to locate the original source files for the solution.

Python Online Form Submision

I am using Python 2.7.1 to access an online website. I need to load a URL, then submit a POST request to that URL that causes the website to redirect to a new URL. I would then like to POST some data to the new URL. This would be easy to do, except that the website in question does not allow the user to use browser navigation. (As in, you cannot just type in the URL of the new page or press the back button, you must arrive there by clicking the "Next" button on the website). Therefore, when I try this:
import urllib, urllib2, cookielib
url = "http://www.example.com/"
jar = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar))
form_data_login = "POSTDATA"
form_data_try = "POSTDATA2"
resp = opener.open(url, form_data_login)
resp2 = opener.open(resp.geturl(), form_data_try)
print resp2.read()
I get a "Do not use the back button on your browser" message from the website in resp2. Is there any way to POST data to the website resp gives me? Thanks in advance!
EDIT: I'll look into Mechanize, so thanks for that pointer. For now, though, is there a way to do it with just Python?

Have you taken a look at mechanize? I believe it has the functionality you need.

You're probably getting to that page by posting something via that Next button. You'll have to take a look at the POST parameters sent when pressing that button and add all of these post parameters to your call.
The website could though be set up in such a way that it only accepts a particular POST parameter that ensures that you'll have to go through the website itself (e.g. by hashing a timestamp in a certain way or something like that) but it's not very likely.

html form submission

I'm looking at the html form of an external website (not maintained by myself) in the following format :
<form onsubmit="return check(this)" method=post action=address.aspx target=ttPOST>
....
</form>
I wish to post data to this external website without having to go through the form and pressing submit.
Can I simply go to the url 'address.aspx' and append it with the required input parameters ?
My goal is to automate the periodic posting of information, chosen from a list of frequently changing values on the website. (Using Python and AutoIt)

You can use JQuery.post()
<form action="#" class="myForm" method="post">
<input type="text" id="field1" name="field1" value="" runat="server" />
</form>
// Submit
<div onclick="submit();return false;">Submit</div>
Then the submit() function looks like
function submit() {
$.post("address.aspx", $("form.myForm").serialize(), function(data, textStatus)
{
// Handle success or error
});
}
In codebehind, you can access the post variables
Request.Form["field1"]

I should note that I'm unclear if you were wanting to automate the posting of data from outside a web browser or not. Others have answered doing it with script and such like from the web page so I thought I'd cover how it works when you are doing it from a standalone program.
For most languages you can get things that will help you simulate web requests. Most of these would probably allow you to make a post request and supply post data. I dont' know python and autoit but in teh general sense you'd just need to get the name value pairs by looking at the HTML of the form (or by examining a request being made to the server is probably better) and then make a post request.
As others have said if you want to just append the values to teh url then the server will need to be happy to accept a GET request instead of a post. Sometimes you will find that servers will do this happily (they don't care how form submission is done, as long as they get data), other times it will be specifically looking for a post and will ignore parameters passed as part of the querystring.

Change the method attribute from POST to GET.
Read about the differences here.
You can get away with changing the URL to go to the external site using the same syntax of GET parameter (?param1=val1&param2=val2...), but to achieve this you will need to write this logic yourself in javascript.

Why do you want to get around the form submit? Submit the form and use the values in HTTP_POST. You can use HTTP_GET by changing the method to GET in the above html but the form will still submit.
One way to submit the params to address.aspx would be to use javascript....build a string of all the params and submit that to address.aspx when the user clicks on the submit button. You'll need to disable the submit buttons default behaviour.

You could make AJAX GET requests (appending the data to the URL) and execute them with a certain time interval.
I would recommend jQuery, but it may not be ideal in a .NET environment.
jQuery.ajax() - http://api.jquery.com/jQuery.ajax/
jQuery.get() - http://api.jquery.com/jQuery.get/
NOTE:
I wish to post data to this external website without having to go through the form and pressing submit.
I took this as meaning you didn't want to actually submit the form, do you want to submit the form but simply not by pressing the submit button? If you actually want the form to submit, then you should indeed just change the form's method attribute to get instead of post.
If however you do wish to stay on the form page and need to post the data to some other resource with the data in the URL string (there are reasons for and aganist doing this that you should look into -- scroll down to 9.3 GET), then just make an AJAX GET request, otherwise just POST the data using AJAX. Either way, you'll have to make some sort of an asynchronous call.

POST is not the same as GET -- if you append the data that you want to the URL, the page will not see your variables. That is to say GETTING (a.k.a going to) http://www.the-other-site.com/address.aspx?variable1=a&variable2=b will not work.
You need to build POST request and submit it to address.aspx.
If you want to do it yourself then you'll need Python's urllib2 module (More specifically, the Request object). If you want to use a pre-built solution, you'll need something like mechanize to POST your variables.