i am working on adding ajax to comments in my webpage.
following are the functions and the view file
def normalajax():
news=db(db.newsfeed.id>0).select(orderby=~db.newsfeed.created_on)
return dict(news=news)
def new_post():
form=SQLFORM(db.newsfeed)
if form.accepts(request.vars, formname=None):
news=db(db.newsfeed.created_by==auth.user_id).select(orderby=~db.newsfeed.created_on)
return DIV(news)
elif form.errors:
return TABLE(*[TR(k, v) for k, v in form.errors.items()])
{{extend 'layout.html'}}
<form id="myform">
<input name="body" id="body" />
<input type="submit" />
</form>
<script>$('textarea').css('width','600px').css('height','50px');</script>
<script>
jQuery('#myform').submit(function() {
ajax('{{=URL('new_post')}}',
['body'], 'target');
return false;
});
</script>
<div id="target">
{{for post in news:}}
<div style="background: #ffffff; margin-bottom: 5px; padding: 8px;">
<h3>{{=db.auth_users[post.created_by].first_name}}</h3> On {{=post.created_on}}:
{{=MARKMIN(post.body)}}
</div>
{{pass}}
</div>
The problem is that, when i post a new comment, the entire div is replaced by the new content,without all the styling which i have given in the for loop
i have given the links of screenshots:
this is before posting comment:
https://skydrive.live.com/redir?resid=AFF5DF0EB4A5BCD5!122&authkey=!AFg6utSsGLyYRG4&v=3&ithint=photo%2c.png
this is after posting comment:
https://skydrive.live.com/redir?resid=AFF5DF0EB4A5BCD5!123&authkey=!AJyroKLQLp5ssPs&v=3&ithint=photo%2c.png
after posting comment, the ajax response replaces contents of div target, how do i access the response and display it like it is dislayed before posting...
Once the page is in the browser, you cannot execute Python code on it. The Python code in your original template does not exist once the page is in the browser -- that has all been executed to generate pure HTML source code. When you submit the form, the new_post function returns DIV(news), which gets converted to HTML. That HTML replaces the content of the "target" div on the page. When you generate an Ajax response, you have to generate the final HTML on the server and send that to the browser -- the browser does not do any template processing.
In any case, a better approach would probably be to use Ajax components. Not tested, but something like:
def news():
return dict()
def news_list():
if request.args(0) == 'user':
query = db.newsfeed.created_by == auth.user_id
else:
query = db.newsfeed.id > 0
news = db(query).select(orderby=~db.newsfeed.created_on)
return dict(news=news)
def new_post():
form = SQLFORM(db.newsfeed)
if form.process().accepted:
url = URL('default', 'news_list.load', args='user')
response.js = "$.web2py.component('%s', target='news');" % url
return dict(form=form)
In /views/default/news.html:
{{extend 'layout.html'}}
{{=LOAD('default', 'new_post.load', ajax=True)}}
{{=LOAD('default', 'news_list.load', ajax=True, target='news')}}
In /views/default/news_list.load:
{{for post in news:}}
<div style="background: #ffffff; margin-bottom: 5px; padding: 8px;">
<h3>{{=post.created_by.first_name}}</h3> On {{=post.created_on}}:
{{=MARKMIN(post.body)}}
{{pass}}
In /views/default/new_post.load:
{{=form}}
In the above code, news.html is the main page, and it contains two Ajax components -- one for the form, and one for the news list. Initially, the news list component loads all the news posts (you might consider limiting that or paginating if there are many posts). When the form is submitted, it returns some Javascript via response.js, which gets executed after a successful submission. The Javascript calls $.web2py.component() in the browser, which reloads the news list component in the "news" div by calling the news_list() function with "user" as a URL arg (the "user" arg is a flag that tells the news_list() function to return only the posts of the current user).
apply same css to your Ajax response to get the same view
this css is missing from ajax reponse.
view this example
$.ajax({
url: "test.html",
cache: false
})
.done(function( result) {
$( "#results" ).append( result); // or replace your "result" coming from server
});
Related
I have a flask application with a few custom built tools. I'm trying to bring in some other tools into that flask application to have a single place for everything. One of those tools is MicroStrategy. I'm rendering a template and the MicroStrategy login page is working, but when I log in, it just kicks me back to the login page. When I look at the request, there are two Set-Cookie's in the header with errors.
Is it possible to do what I'm trying to do? A way to read the headers from the MicroStrategy page in the iframe and modify SameSite=None?
Here is my flask app:
#dash_app.server.route("/mstr")
def mstr():
resp = make_response(render_template("mstr.html"))
return resp
mstr.html:
<div style="position:fixed; width:100%; top:50px; left:0px; right:0px; bottom:0px; z-index:1;">
<iframe src="https://webserver.com/MicroStrategy/asp/Main.aspx" title="MicroStrategy" style="width:100%; height:100%; border:none; margin:0; padding:0; overflow:hidden;"></iframe>
</div>
I'm trying to get the HTML content of a password protected site using Ghost.py.
The web server I have to access, has the following HTML code (I cut it just to the important parts):
URL: http://192.168.1.60/PAGE.htm
<html>
<head>
<script language="JavaScript">
function DoHash()
{
var psw = document.getElementById('psw_id');
var hpsw = document.getElementById('hpsw_id');
var nonce = hpsw.value;
hpsw.value = MD5(nonce.concat(psw.value));
psw.value = '';
return true;
}
</script>
</head>
<body>
<form action="PAGE.HTM" name="" method="post" onsubmit="DoHash();">
Access code <input id="psw_id" type="password" maxlength="15" size="20" name="q" value="">
<br>
<input type="submit" value="" name="q" class="w_bok">
<br>
<input id="hpsw_id" type="hidden" name="pA" value="180864D635AD2347">
</form>
</body>
</html>
The value of "#hpsw_id" changes every time you load the page.
On a normal browser, once you type the correct password and press enter or click the "submit" button, you land on the same page but now with the real contents.
URL: http://192.168.1.60/PAGE.htm
<html>
<head>
<!–– javascript is gone ––>
</head>
<body>
Welcome to PAGE.htm content
</body>
</html>
First I tried with mechanize but failed, as I need javascript. So now I´m trying to solve it using Ghost.py
My code so far:
import ghost
g = ghost.Ghost()
with g.start(wait_timeout=20) as session:
page, extra_resources = session.open("http://192.168.1.60/PAGE.htm")
if page.http_status == 200:
print("Good!")
session.evaluate("document.getElementById('psw_id').value='MySecretPassword';")
session.evaluate("document.getElementsByClassName('w_bok')[0].click();", expect_loading=True)
print session.content
This code is not loading the contents correctly, in the console I get:
Traceback (most recent call last): File "", line 8, in
File
"/usr/local/lib/python2.7/dist-packages/ghost/ghost.py", line 181, in
wrapper
timeout=kwargs.pop('timeout', None)) File "/usr/local/lib/python2.7/dist-packages/ghost/ghost.py", line 1196, in
wait_for_page_loaded
'Unable to load requested page', timeout) File "/usr/local/lib/python2.7/dist-packages/ghost/ghost.py", line 1174, in
wait_for
raise TimeoutError(timeout_message) ghost.ghost.TimeoutError: Unable to load requested page
Two questions...
1) How can I successfully login to the password protected site and get the real content of PAGE.htm?
2) Is this direction the best way to go? Or I'm missing something completely which will make things work more efficiently?
I'm using Ubuntu Mate.
This is not the answer I was looking for, just a work-around to make it work (in case someone else has a similar issue in the future).
To skip the javascript part (which was stopping me to use python's request), I decided to do the expected hash on python (and not on web) and send the hash as the normal web form would do.
So the Javascript basically concatenates the hidden hpsw_id value and the password, and makes a md5 from it.
The python now looks like this:
import requests
from hashlib import md5
from re import search
url = "http://192.168.1.60/PAGE.htm"
with requests.Session() as s:
# Get hpsw_id number from website
r = s.get(url)
hpsw_id = search('name="pA" value="([A-Z0-9]*)"', r.text)
hpsw_id = hpsw_id.group(1)
# Make hash of ID and password
m = md5()
m.update(hpsw_id + 'MySecretPassword')
pA = m.hexdigest()
# Post to website to login
r = s.post(url, data=[('q', ''), ('q', ''), ('pA', pA)])
print r.content
Note: the q, q and pA are the elements that the form (q=&q=&pA=f08b97e5e3f472fdde4280a9aa408aaa) is sending when I login normally using internet browser.
If someone however knows the answer of my original question I would be very appreciated if you post it here.
I am scraping a page, using both Scrapy and Splash. The page contains a dropdown box (technically, a select HTML element). Each time an element is selected in the dropdown box, a new page is loaded using AJAX.
The HTML segment below, is a simplified version of the page I'm processing:
<html>
<head><title>Title goes here ...</title></head>
<body>
<select class="foo">
<option value=100 data-reactid=1>One</option>
<option value=200 data-reactid=2>Two</option>
<!-- ... -->
<option value=900 data-reactid=9>Nine</option>
</select>
</body>
</html>
Snippet of my scrapy/splash code:
# Fetch the options ... now what ?
options = response.css("select[class=foo] option[data-reactid]")
How do I programatically use Splash to 'click' and receive the reloaded AJAX page in my response object?
You might try to use Splash's execute endpoint with LUA script that will fill the select with each option's value and return the result. Something like:
...
script = """
function main(splash)
splash.resource_timeout = 10
splash:go(splash.args.url)
splash:wait(1)
splash:runjs('document.getElementsByClassName("foo")[0].value = "' .. splash.args.value .. '"')
splash:wait(1)
return {
html = splash:html(),
}
end
"""
# base_url refers to page with the select
values = response.xpath('//select[#class="foo"]/option/#value').extract()
for value in values:
yield scrapy_splash.SplashRequest(
base_url, self.parse_result, endpoint='execute',
args={'lua_source': script, 'value': value, 'timeout': 3600})
Of course, this isn't tested, but you might start there and play with it.
I am trying to web-scrape some elements and their values off a page with Python; However, to get more elements, I need to simulate a click on the next button. There is a post back tied to these buttons, so I am trying to call it. Unfortunately, Python is only printing the same values over and over again [meaning the post back for the next button isn't being called]. I am using requests to do my POST/GET.
import re
import time
import requests
TARGET_GROUP_ID = 778092
SESSION = requests.Session()
REQUEST_HEADERS = {"Accept-Encoding": "gzip,deflate"}
GROUP_URL = "http://roblox.com/groups/group.aspx?gid=%d"%(TARGET_GROUP_ID)
POST_BUTTON_HTML = 'pagerbtns next'
EVENTVALIDATION_REGEX = re.compile(r'id="__EVENTVALIDATION" value="(.+)"').search
VIEWSTATE_REGEX = re.compile(r'id="__VIEWSTATE" value="(.+)"').search
VIEWSTATEGENERATOR_REGEX = re.compile(r'id="__VIEWSTATEGENERATOR" value="(.+)"').search
TITLE_REGEX = re.compile(r'<a id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_ctrl\d+_hlAvatar".*?title="(\w+)".*?ID=(\d+)"')
page = SESSION.get(GROUP_URL, headers = REQUEST_HEADERS).text
while 1:
if POST_BUTTON_HTML in page:
for (ids,names) in re.findall(TITLE_REGEX, page):
print ids,names
postData = {
"__EVENTVALIDATION": EVENTVALIDATION_REGEX(page).group(1),
"__VIEWSTATE": VIEWSTATE_REGEX(page).group(1),
"__VIEWSTATEGENERATOR": VIEWSTATEGENERATOR_REGEX(page).group(1),
"__ASYNCPOST": True,
"ct1000_cphRoblox_rbxGroupRoleSetMembersPane_currentRoleSetID": "4725789",
"ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl02$ctl00": "",
"ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl01$HiddenInputButton": "",
"ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl01$PageTextBox": "3"
}
page=SESSION.post(GROUP_URL, data = postData, stream = True).text
time.sleep(2)
How can I properly call the post back in ASP.NET from Python to fix this issue? As stated before, it's only printing out the same values each time.
This is the HTML Element of the button
<a class="pagerbtns next" href="javascript:__doPostBack('ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl02$ctl00','')"> </a>
And this is the div it is in:
<div id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_MembersPagerPanel" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_HiddenInputButton')">
<div id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_Div1" class="paging_wrapper">
Page <input name="ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl01$PageTextBox" type="text" value="1" id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_PageTextBox" class="paging_input"> of
<div class="paging_pagenums_container">125</div>
<input type="submit" name="ctl00$cphRoblox$rbxGroupRoleSetMembersPane$dlUsers_Footer$ctl01$HiddenInputButton" value="" onclick="loading('members');" id="ctl00_cphRoblox_rbxGroupRoleSetMembersPane_dlUsers_Footer_ctl01_HiddenInputButton" class="pagerbtns translate" style="display:none;">
</div>
</div>
I was thinking of using a JS library and executing the JS __postback method, however, I would like to first see if this can be achieved in pure Python.
Yes it should be achievable you just have to submit correct values on correct fields. But i assume web page you are trying parse uses asp.net web forms so it should be really time consuming to find values and such. I suggest you to look into selenium with that you can easily call click and events on a webpage without writing so much code.
driver = webdriver.Firefox()
driver.get("http://site you are trying to parse")
driver.find_element_by_id("button").click()
//then get the data you want
Using Scrapy, how to navigate to "nextpage" link, from any results page generate by sciencedirect.com?
The nextpage link is the input element:
<div class="paginationBar">
<span style="color:#A4A4A4;" aria-disabled="true" alt="Previous Page" title="Previous Page"><< Previous</span>
<span class="pageText">Page 1 of 20462</span>
<input class="nextPrev" type="submit" title="Next Page" alt="Next Page" name="bottomNext" onmouseout="this. className='nextPrev'" onmouseover="this.className='nextPrevHov'" value="Next >>">
</div>
And exists some javascript but I dont know how to take it :(
The answer is simple: there is no JavaScript involved.
If you look at the site you can see, that the link Next >> is an input field which submits the form.
When looking at the form itself, you can see, that it sends a get request to a site. The input fields to this request you can gather together and then yield a new Request with Scrapy to scrape the next site.
An example would be:
form = response.xpath('//form[#name="Tag"]')[0]
url = 'http://www.sciencedirect.com/science/?'
for inp in form.xpath('.//input[#type="hidden"]'):
url += inp.xpath('./#name').extract()[0]+'='+inp.xpath('./#value').extract()[0]+'&'
url += 'bottomNext=Next+%3E%3E&resultsPerPage=25'
yield Request(url)
Naturally some error handling is needed (for example after 1000 results you cannot view more so you will get an error site which does not have the form).