We Keep Coding

Unable to access link except by clicking button on website

So there is a link from a website I have been trying to access using python requests library. Normally on clicking the button, it redirects to another website but copying and pasting the referrer link either in the browser directly or using requests.get() only returns the referrer page.
The link to the referrer page is: "https://www.thenetnaija.net/videos/kdrama/16426-alchemy-of-souls/season-1/episode-7"
Here's the html with the button
<a
href="https://www.thenetnaija.net/videos/kdrama/16426-alchemy-of-souls/season-1/episode-7/download"
class="btn"
type="submit"
title="Download Video"
>
<i class="fas fa-download"></i> Download <i class="fas fa-file-video"></i>
<span class="small-text">(Video)</span>
</a>
if I try to copy and paste the link ("https://www.thenetnaija.net/videos/kdrama/16426-alchemy-of-souls/season-1/episode-7/download") directly in browser, it redirects to this link ("https://www.thenetnaija.net/videos/kdrama/16426-alchemy-of-souls/season-1/episode-7") instead of this ("https://www.sabishare.com/file/mHxiMiZHW15-alchemy-of-souls-s01e07-netnaija-com-mp4")
so the only way to get to this url ("https://www.sabishare.com/file/mHxiMiZHW15-alchemy-of-souls-s01e07-netnaija-com-mp4") is by clicking the button in this page ("https://www.thenetnaija.net/videos/kdrama/16426-alchemy-of-souls/season-1/episode-7").
Also, this is my python code:
def gen_link(url):
headers = {
'Authorization': 'Bearer {token}',
'Content-Type':'application/json',
}
print(dUrl)
resp = requests.get(dUrl, headers=headers, allow_redirects=True)
print(resp.url)
how is it that the destination url is somewhat blocked and can only be accessed if i click the button from the referrer webpage?

The issues with your script is the lack of the http request referer header
Here is a YouTube video to further explain this Unable to access link except by clicking button on websit
Here is the code snippet
import requests
url='https://www.thenetnaija.net/videos/kdrama/16426-alchemy-of-souls/season-1/episode-7/download'
headers={'Referer':'https://www.thenetnaija.net/videos/kdrama/16426-alchemy-of-souls/season-1/episode-7'}
resp = requests.get(url,headers=headers)
print(resp.text)

Flask - href to anchor on a different page (navigation bar)

I am using Flask to develop a web app. On the home page (index.html), the navigation bar navigates one to specific sections on the page using anchors:
<a class='text' href='#body2'>calculate</a>
<a id="body2"></a>
On the home page, there is a form which links you to a new page (output.html). I want the same navigation bar to navigate a user to the previous page (index.html) and the specific sections. I have written the navigation links on the second page as shown below:
<a class='text' href="{{ url_for('index') }}#body2">calculate</a>
When I click the navigation links, the new page does not load. However, this is the strange thing, when I inspect the navigation link element in my browser and click the link through the inspect client, it does take me to the correct page/section.
If I remove '#body2' from the above line, it successfully navigates me to the previous page, but not to the specific section.
(If you want to physically try out the navigation links on the web app, use the following link:
http://yourgreenhome.appspot.com/ - Enter some random values into the blank form entries and it will take you to the second page. It is running through Google's App Engine but this is definitely not causing the problem because the problem still occurs when I run the site on local host).

You have an error in smoothscroll.js
$(document).ready(function(){
$("a").on('click', function(event) {
if (this.hash !== "") {
event.preventDefault();
var hash = this.hash;
$('html, body').animate({
scrollTop: $(hash).offset().top
}, 800, function(){
window.location.hash = hash;
});
}
});
});
In advpy page, $(hash).offset() is undefined, thus top is undefined. Because you are preventing the default event (event.preventDefault();) the click on the link doesn't occur.

Using a Flask page inside of the HTML-iframe block

I've created a Flask. It works fine if I use a direct link to Flask pages HTTP://flask_host:flask_port/.
If I'm using the same link inside of HTML iframe tags it returns an empty block (page is not showing)
<html>
<body>
<iframe src="HTTP://flask_host:flask_port/" name="iframe_a"></iframe>
</body>
</html>
What am I doing wrong?

The problem was that I used to HTTP instead of HTTPS. I had seen the error inside of the browser developer console. Flask was configured by me to use the simplest ad-hoc HTTPS.
After that, I'm able to see the page inside of the iframe. content of dynamically created iframe is empty

How to call a postback in ASP.Net with Python

I am trying to web-scrape some elements and their values off a page with Python; However, to get more elements, I need to simulate a click on the next button. There is a post back tied to these buttons, so I am trying to call it. Unfortunately, Python is only printing the same values over and over again [meaning the post back for the next button isn't being called]. I am using requests to do my POST/GET.
import re
import time
import requests
TARGET_GROUP_ID = 778092
SESSION = requests.Session()
REQUEST_HEADERS = {"Accept-Encoding": "gzip,deflate"}
GROUP_URL = "http://roblox.com/groups/group.aspx?gid=%d"%(TARGET_GROUP_ID)
POST_BUTTON_HTML = 'pagerbtns next'
EVENTVALIDATION_REGEX = re.compile(r'id="__EVENTVALIDATION" value="(.+)"').search
VIEWSTATE_REGEX = re.compile(r'id="__VIEWSTATE" value="(.+)"').search
VIEWSTATEGENERATOR_REGEX = re.compile(r'id="__VIEWSTATEGENERATOR" value="(.+)"').search
TITLE_REGEX = re.compile(r'<a id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_ctrl\d+_hlAvatar".*?title="(\w+)".*?ID=(\d+)"')
page = SESSION.get(GROUP_URL, headers = REQUEST_HEADERS).text
while 1:
if POST_BUTTON_HTML in page:
for (ids,names) in re.findall(TITLE_REGEX, page):
print ids,names
postData = {
"__EVENTVALIDATION": EVENTVALIDATION_REGEX(page).group(1),
"__VIEWSTATE": VIEWSTATE_REGEX(page).group(1),
"__VIEWSTATEGENERATOR": VIEWSTATEGENERATOR_REGEX(page).group(1),
"__ASYNCPOST": True,
"ct1000_cphRoblox_rbxGroupRoleSetMembersPane_currentRoleSetID": "4725789",
"ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl02$ctl00": "",
"ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl01$HiddenInputButton": "",
"ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl01$PageTextBox": "3"
}
page=SESSION.post(GROUP_URL, data = postData, stream = True).text
time.sleep(2)
How can I properly call the post back in ASP.NET from Python to fix this issue? As stated before, it's only printing out the same values each time.
This is the HTML Element of the button
<a class="pagerbtns next" href="javascript:__doPostBack('ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl02$ctl00','')"> </a>
And this is the div it is in:
<div id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_MembersPagerPanel" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_HiddenInputButton')">
<div id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_Div1" class="paging_wrapper">
Page <input name="ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl01$PageTextBox" type="text" value="1" id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_PageTextBox" class="paging_input"> of
<div class="paging_pagenums_container">125</div>
<input type="submit" name="ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl01$HiddenInputButton" value="" onclick="loading('members');" id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_HiddenInputButton" class="pagerbtns translate" style="display:none;">
</div>
</div>
I was thinking of using a JS library and executing the JS __postback method, however, I would like to first see if this can be achieved in pure Python.

Yes it should be achievable you just have to submit correct values on correct fields. But i assume web page you are trying parse uses asp.net web forms so it should be really time consuming to find values and such. I suggest you to look into selenium with that you can easily call click and events on a webpage without writing so much code.
driver = webdriver.Firefox()
driver.get("http://site you are trying to parse")
driver.find_element_by_id("button").click()
//then get the data you want

ajax response in web2py

i am working on adding ajax to comments in my webpage.
following are the functions and the view file
def normalajax():
news=db(db.newsfeed.id>0).select(orderby=~db.newsfeed.created_on)
return dict(news=news)
def new_post():
form=SQLFORM(db.newsfeed)
if form.accepts(request.vars, formname=None):
news=db(db.newsfeed.created_by==auth.user_id).select(orderby=~db.newsfeed.created_on)
return DIV(news)
elif form.errors:
return TABLE(*[TR(k, v) for k, v in form.errors.items()])
{{extend 'layout.html'}}
<form id="myform">
<input name="body" id="body" />
<input type="submit" />
</form>
<script>$('textarea').css('width','600px').css('height','50px');</script>
<script>
jQuery('#myform').submit(function() {
ajax('{{=URL('new_post')}}',
['body'], 'target');
return false;
});
</script>
<div id="target">
{{for post in news:}}
<div style="background: #ffffff; margin-bottom: 5px; padding: 8px;">
<h3>{{=db.auth_users[post.created_by].first_name}}</h3> On {{=post.created_on}}:
{{=MARKMIN(post.body)}}
</div>
{{pass}}
</div>
The problem is that, when i post a new comment, the entire div is replaced by the new content,without all the styling which i have given in the for loop
i have given the links of screenshots:
this is before posting comment:
https://skydrive.live.com/redir?resid=AFF5DF0EB4A5BCD5!122&authkey=!AFg6utSsGLyYRG4&v=3&ithint=photo%2c.png
this is after posting comment:
https://skydrive.live.com/redir?resid=AFF5DF0EB4A5BCD5!123&authkey=!AJyroKLQLp5ssPs&v=3&ithint=photo%2c.png
after posting comment, the ajax response replaces contents of div target, how do i access the response and display it like it is dislayed before posting...

Once the page is in the browser, you cannot execute Python code on it. The Python code in your original template does not exist once the page is in the browser -- that has all been executed to generate pure HTML source code. When you submit the form, the new_post function returns DIV(news), which gets converted to HTML. That HTML replaces the content of the "target" div on the page. When you generate an Ajax response, you have to generate the final HTML on the server and send that to the browser -- the browser does not do any template processing.
In any case, a better approach would probably be to use Ajax components. Not tested, but something like:
def news():
return dict()
def news_list():
if request.args(0) == 'user':
query = db.newsfeed.created_by == auth.user_id
else:
query = db.newsfeed.id > 0
news = db(query).select(orderby=~db.newsfeed.created_on)
return dict(news=news)
def new_post():
form = SQLFORM(db.newsfeed)
if form.process().accepted:
url = URL('default', 'news_list.load', args='user')
response.js = "$.web2py.component('%s', target='news');" % url
return dict(form=form)
In /views/default/news.html:
{{extend 'layout.html'}}
{{=LOAD('default', 'new_post.load', ajax=True)}}
{{=LOAD('default', 'news_list.load', ajax=True, target='news')}}
In /views/default/news_list.load:
{{for post in news:}}
<div style="background: #ffffff; margin-bottom: 5px; padding: 8px;">
<h3>{{=post.created_by.first_name}}</h3> On {{=post.created_on}}:
{{=MARKMIN(post.body)}}
{{pass}}
In /views/default/new_post.load:
{{=form}}
In the above code, news.html is the main page, and it contains two Ajax components -- one for the form, and one for the news list. Initially, the news list component loads all the news posts (you might consider limiting that or paginating if there are many posts). When the form is submitted, it returns some Javascript via response.js, which gets executed after a successful submission. The Javascript calls $.web2py.component() in the browser, which reloads the news list component in the "news" div by calling the news_list() function with "user" as a URL arg (the "user" arg is a flag that tells the news_list() function to return only the posts of the current user).

apply same css to your Ajax response to get the same view
this css is missing from ajax reponse.
view this example
$.ajax({
url: "test.html",
cache: false
})
.done(function( result) {
$( "#results" ).append( result); // or replace your "result" coming from server
});