We Keep Coding

Scraping ASPX after login with Python but every login gives you a different URL

I'm trying to get the exam result data from my college website for every Roll No. in my class.
Normally you can POST url (www.example.com/login.aspx)with login information, and GET a fixed url after login(www.example.com/home.aspx).
But the page I'm trying to get has a different URL for every Roll no. entered. The URL of login page look like this: "www.example.com/View.aspx". After login, the URL of the result page looks like: "www.example.com/ovengine.aspx?enc=BunchOfNumbersandAlphabets". And those numbers and alphabets are different for each roll number.
So I can't put a URL in my code to get the final result. I don't know how to get the page that comes automatically after the login, without mentioning it's URL.

But the page I'm trying to get has a different URL for every Roll no. entered
No, it is the same URL, and the URL has a parameter. You see this in URL's all the time.
So, for a temperature site it might look like
www.TheWeatherSite.com/?City=Rome
So, the above URL is always the same, but the web site "city" parameter is for the City of Rome. The web code behind can thus use/get/grab/consume that parameter in the code behind. That way we don't create a web page for EACH weather for each city.
so you create ONE page, and then and then PASS the web page a city value that the code behind can consume and use. (say query temperature data from a database for city = above value).
And thus you have to know ahead of time what city you want the weather for. Of course this approach is great since you don't have to create a new web site page to just show/display the weather in a given city.
You are in effect passing a value to some code behind that will run, and use that passed value.
The same goes for your example URL. You note there is ONE parameter called "enc".
So, the web site code behind would:
Grab, get, set the users ID. However, the users ID would be from the security system and the authentication provider. Unless you logged in as that particular user, then you not get that user id.
So, both a user ID (limited to the internal code).
And the "enc" value as the parameter in the URL you have would be required.
So, note in the above sql, we VERY likely need both a studentID and ALSO the "enc" value that some OTHER code from another page gets/grabs from the database.
Now that funny "GUID" (please do google what a GUID is), from a programmers point of view WOULD be sufficient to pull this one row of data from the database, but by ALSO using in the query the users logged on internal id?
Well, then only a given logged on user would be able to see their own set of values that belong to them.
In other words?
Only a drunken un-employed Rodeo clown would JUST require that GUID for pulling out that data. Since if that was the case, then any user could type in that GUID and see others peoples marks. However, there is "some" security by using a GUID, since a user could never guess that value.
If they used "city" like my first URL and parameter example? Then yes, you could guess and know the city value to type in. Or they could have used say student name, or even student number - those you COULD guess with relative ease.
But, for such data, no doubt the user adopted something MUCH more difficult then a starting number like a row number or PK id from a database. So, when the code added the results to that table? They also added a GUID of some type and saved that as a row in the database also.
So you NOT only need JUST the GUID, but that URL will ONLY work for a given pair of values. (the student ID - which is ONLY internal to the code and pulled FROM the authenticated provider. That was this line of code:
= Membership.GetUser.ProviderUserKey
So that above value is going to be the users logon internal ID.
The enc (external) exposed value in the web URL as a parameter, and ALSO the internal logged on value. So the code behind (asp.net) would look something like this:
Dim strSQL As String
strSQL = "SELECT * from tblStudentMarks where StudentID = #pID " &
" AND TestResultsGID = #GID"
Dim cmdSQL As New SqlCommand(strSQL, GetCon)
cmdSQL.Parameters.Add("#pID", SqlDbType.Int).Value = Membership.GetUser.ProviderUserKey
cmdSQL.Parameters.Add("#GID", SqlDbType.VarChar).Value = Request.QueryString("enc")
Dim dReader As New SqlDataAdapter(cmdSQL)
Dim rstData As DataTable
dReader.Fill(rstData)
Note the code:
Request.QueryString("enc")
That allows the code behind to get/grab the parameter (enc) from the URL. But, as I stated, it is high unlikely that JUST the "enc" number is required here. It is possible that ONLY this value is required to pull the data from the row, but then that would be a security hole the size of a open barn door.
Think of your on-line banking.
www.mybank.com/?CustomerNumber=1234
Well, if we JUST use the above CustomerNumber as the means to pull bank data, then I could go to the site and type in YOUR number, or someone's else's number.
So, for this to work?
You will need to obtain a list of enc values (that messy funny long string). Without that parameter then you not be able to set the parameter in the URL.
However, as I stated, you ALSO very likely need some internal "user" logon id that is NOT included in the public exposed URL to ALSO grab that one row of data from the database.
And, even more important? Such web pages usually cannot be hit UNLESS you are a logged in as an authenticated user. In other words that web page will ONLY be dished out to logged in users - if you not logged in, then the server security will automatic NOT dish out the web page unless you are logged in user.
So, for this to work, you need to contact the web site developers, and obtain that list of "enc" values. Once you have that list, then you can generate some code to process that list and insert the correct parameter in the URL. However, you also need to ask if that URL and parameter value will work for JUST you the logged in user, or if that this URL and parameter ONLY works for a give logged in user. Without these values, and without knowing if the URL and parameter will work for any user? (which I doubt it would), then just using a URL to get these values will not work.
It would be even BETTER to have the web site folks create a web service that you can call and in one command it would return all of the data you need anyway, as opposed to over and over having to send the "enc" value, which you don't have anyway.

filling a Django form from a separate requests.post python script

while reading about the Requests library, it occurred to me that I could try to fill a form by using it. So, since I have Django and the server, I first checked that I got the URL well and got a 200 answer code.
like this:
import requests
r = requests.get('127.0.0.1/myform')
print(r.status_code)
and yes, it was a 200
so the next step was going to be entering a value in the textfield, actually the form, for this example is just one field.
I tried this:
r = requests.post('127.0.0.1/myform', data ={'name' : 'Mexico'})
and nothing was entered. Here I have like 3 intriguing questions:
When I inspect the element, because the form was created using the ModelForm of Django, I could not actually give the field a name, (which would have been name) and 2. Django instead fills me the name with the csrf token value and 3. actually, I don't see how the requests.post presses the submit button.
What would be what is missing here in order to succeed posting the value through the form?
thank you

In Django web applications, at execution time the front end and the back end are separated and communicate over http/https requests and responses. The front end displays elements to the user, collects input, etc, but then to communicate with the back end it builds a POST request from that data and sends it when the user clicks submit. The form itself is not sent, just a collection of key value pairs corresponding to the fields of the form and the user input in each of those fields. On the back end that data is handed to the form so it can validate/save/etc.
So to answer your questions:
1) I'm not entirely sure what you are asking, but the important thing is that the post payload has a key that matches the name of the field in the corresponding Django form.
2) CSRF tokens are a security measure, and you'll have to manually account for this in your requests.post call. It may be easier to start by disabling them until you get your post working and then work on getting them working.
3) The requests.post doesn't actually press the submit button, you are just recreating the same http request that gets generated by the browser when a user presses the submit button. Does this make sense? This is a key point to understand.
Finally, in order to help you with a working solution we would need to see more. What does your form look like? What does your view look like?

Batch Editing in Flask-Admin

I'm using Flask-Admin and I want to be able to update many fields at once from the list view. It seemed like what I'm looking for is a custom action.
I was able to make it work, but I suspect not in the best way. I'm wondering if it could be done more "Flask"-ily.
What I do now, for example if I was updating all rows in table cars to have tires = 4:
A custom action in the CarView class collects the ids of the rows to be modified, a callback url from request.referrer, and the tablename cars, and returns render_template(mass_update_info.html) with these as parameters.
mass_update_info.html is an HTML form where the user specifies 1) the field they would like to change and 2) the value to change it to. On submit, the form makes a POST to a a certain view (do_mass_update) with this data (everything else is passed as hidden fields in this form).
do_mass_update uses the data sent to it to construct a SQL query string -- in its entirety, "UPDATE {} SET {}='{}' WHERE id IN ({})".format(table, column, value, ids) -- which is run via db.engine.execute().
The user is redirected to the callback url.
It bothers me that I don't seem to be using any of SQLAlchemy, but (from a newbie's perspective) it all seems to be based on the model objects e.g. User.query(...), while I only have access to the model/table name as a string. Can I get some kind of identifier from the model, pass that through, and do a lookup to retrieve the on the other side?

Tastypie - get user groups resource_uri

I'm trying to update the groups assigned to users via an API (via rest) with Tastypie.
I tried passing the group id's directly in, however it says that the URL provided is not a valid resource. I then tried passing in a URL such as '/api/v1/groups/1/' but that is saying that's not a link to a valid resource.
Any hints? I'm creating user records just fine from a standard django view/form, but I would like to do this as a REST action.

Do you need to specify the format, such as: /api/v1/groups/1/?format=json ?

Sending objects from Jinja Templates to Python

I started using Jinja Templating with Python to develop web apps. With Jinja, I am able to send objects from my Python code to my index.html, but is it possible to receive objects from my index.html to my Python code? For example, passing a list back and forth. If so, do you have any examples?
Thank You!

Why do this? Any logic that you implement in the template is accessible to you in the controller of your app, including any variables that you place in the template context.
If the data has been changed due to interaction with the user, then the best way to retrieve data, in my opinion, is to set up a form and use the normal POST method to send the request and the required data, correctly encoded and escaped, back to your program. In this way, you are protected from XSS issues, among other inconveniences. I would never do any processing in a template, and only use any local logic to modify the presentation itself.
EDIT Taking into account your scenario, I suggest the following:
User presses a button on a page and invokes a Get handler
Get handler queries a database and receives a list of images the list is cached, maybe in a memcache and the key is sent with the list of images encoded as a parameter in the GET URL displayed by the template
List of images get passed to the template engine for display
Another button is pressed and a different Get handler is invoked using the key received encoded in the GET URL, after sanitising and validation, to retrieve the cached list
If you don't want the intermediate step of caching a key-value pair, you may want to encode the whole list in the GET URL, and the step of sanitising and validation should be as easy on the whole list as on a key to the list. Both methods avoid a round trip to the database, protect you from malicious use, and respect the separation of data, presentation, and logic.

Just a thought.. Have you tried accessing the variables in the dict you passed to jinja after processing the template?