A program is written to communicate between HTML and Python Server. Connection is established but communication message that I wrote in HTML (is string format) is not getting displayed on server side.
I have tried Python-server programs, websockets and html programming
'''Server side program using python'''
def RcvDatafromUI(self):
while True:
json_data = self.conn.recv(1024)
try:
if json_data == b'\r\n' or b'':
print("Looping Again")
continue
else:
print("RcvdDAta : ", json_data)
json_string = json_data.decode('utf-8')
print("DecodedDAta : ", json_string)
#print('type decode=',type(json_string))
json_dict = json.loads(ast.literal_eval(json_string))
print('json_dict=', json_dict)
y = parsejsondata.ParseJson(json_dict)
if y == "Success":
PloverMain.SendDAtatoUI('{"Command":"Start
","Status":"Success/Fail"}')
PloverMain.RcvDatafromUI()
'''html code'''
<!DOCTYPE HTML>
<html>
<head>
<script type = "text/javascript">
function WebSocketTest() {
if ("WebSocket" in window) {
alert("WebSocket is supported by your Browser!");
// Let us open a web socket
var exampleSocket = new WebSocket("ws://localhost:12345/echo");
alert("website opened")
exampleSocket.onopen = function(event) {
// Web Socket is connected, send data using send()
exampleSocket.send('{"Command":"Start","Status":"Check"}');
alert("Message is sent...");
};
exampleSocket.onmessage = function (event) {
var received_msg = event.data;
alert("Message is received...");
};
exampleSocket.onclose = function() {
// websocket is closed.
alert("Connection is closed...");
};
} else {
// The browser doesn't support WebSocket
alert("WebSocket NOT supported by your Browser!");
}
}
</script>
</head>
<body>
<div id = "sse">
Run WebSocket
</div>
</body>
</html>
Expected:(on server side)
RcvdDAta : '{"Command":"Start","Status":"Check"}'
Acutal:
RcvdDAta :
b'GET /echo HTTP/1.1\r\nHost: localhost:12345\r\nConnection: Upgrade\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nUpgrade: websocket\r\nOrigin: file://\r\nSec-WebSocket-Version: 13\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36\r\nAccept-Encoding: gzip, deflate, br\r\nAccept-Language: en-US,en;q=0.9\r\nSec-WebSocket-Key: s1qAa96xxm9IZqU21C0TQA==\r\nSec-WebSocket-Extensions: permessage-deflate; client_max_window_bits\r\n\r\n'
Related
Faced this problem 2 days ago and I try to solve it to this day. The bottom line is that I wanted to use the requests library to send requests, but the answer to it does not come at all the one that should come. When trying to use another library, the correct answer came up, but this library does not support sessions. How to make it so that the correct response to the request comes through the requests library (urllib same problem)
import http.client
conn = http.client.HTTPSConnection("external-api.mediabilling.yandex.ru")
payload = ''
headers = {'Accept': '*/*','Referer': 'https://payment-widget.ott.yandex.ru/','Host': 'external-api.mediabilling.yandex.ru','sec-ch-ua-platform': '"Windows"','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36','Origin': 'https://payment-widget.ott.yandex.ru','Content-Type': 'application/json'}
conn.request("POST", "/v12/promo-code/info?code=7VBKMNRPYV&language=ru&platform=web", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
True answer:
{
"timestamp": 1674771545,
"status": 403,
"error": {
"name": "security",
"message": "Access denied"
},
"path": "/api/v12/promo-code/info"
}
import httpx
with httpx.Client() as session:
url = 'https://external-api.mediabilling.yandex.ru/v12/promo-code/info?code=7VBKMNRPYV&language=ru&platform=web'
response = session.post(url, headers=headers)
print(response.read())
False answer:
<!DOCTYPE html>
<html>
<head>
<title>403</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
</head>
</html>
Headers are the same!
1 request json answer
2 request text answer WTF
all the solutions i was looking for
I am trying to send a get request with python like this:
import requests
url = "internal_url" # I replaced all internal urls
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0", "Accept": "*/*", "Accept-Language": "en-GB,en;q=0.5", "Accept-Encoding": "gzip, deflate", "X-Requested-With": "XMLHttpRequest", "Connection": "close", "Referer": "internal url"}
r = requests.get(url , headers=header)
print(r.text)
As reponse I am expecting json data. But instead I get this:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta http-equiv="Content-Script-Type" content="text/javascript">
<script type="text/javascript">
function getCookie(c_name) { // Local function for getting a cookie value
if (document.cookie.length > 0) {
c_start = document.cookie.indexOf(c_name + "=");
if (c_start!=-1) {
c_start=c_start + c_name.length + 1;
c_end=document.cookie.indexOf(";", c_start);
if (c_end==-1)
c_end = document.cookie.length;
return unescape(document.cookie.substring(c_start,c_end));
}
}
return "";
}
function setCookie(c_name, value, expiredays) { // Local function for setting a value of a cookie
var exdate = new Date();
exdate.setDate(exdate.getDate()+expiredays);
document.cookie = c_name + "=" + escape(value) + ((expiredays==null) ? "" : ";expires=" + exdate.toGMTString()) + ";path=/";
}
function getHostUri() {
var loc = document.location;
return loc.toString();
}
setCookie('STRING redacted', IP-ADDRESS redacted, 10);
try {
location.reload(false);
} catch (err1) {
try {
location.reload();
} catch (err2) {
location.href = getHostUri();
}
}
</script>
</head>
<body>
<noscript>This site requires JavaScript and Cookies to be enabled. Please change your browser settings or upgrade your browser.</noscript>
</body>
</html>
When I changed the request to use the burp suite proxy so I can see the request, it suddenly works and I get the correct response:
proxies = {"http": "127.0.0.1:8080", "https": "http://127.0.0.1:8080"}
r = requests.get(url, headers=headers, verify=False, proxies=proxies)
My browser displays the correct results as text when I visit the link itself. Burp suite proxy not needed.
I think its possible that it has to do with the company proxy.
But even when I tried to run the request with company proxies supplied it still does not work.
Is there something I am missing?
EDIT:
After some more searching it seems like I get redirected when I dont use any proxies in python. That doesnt happen when I go over the burp suite proxy.
After a few days and some outside help I finally found the solution. Posting it here for the future.
My problem was that I was using a partially qualified domain name instead of a fully qualified domain name
So for example: myhost instead of myhost.example.com
Burp suite or the browser were handling the translation for me but in python I had to do it myself.
I would like to write a function which takes an archive.is (or archive.fo, archive.li, or archive.today) link as input and gives the URL of the original site as output.
For example, if the input was 'http://archive.is/9mIro', then I would want the output to be 'http://www.dailytelegraph.com.au/news/nsw/australian-army-bans-male-recruits-to-get-female-numbers-up/news-story/69ee9dc1d4f8836e9cca7ca2e3e5680a'.
How can I do this in python?
Yes, your approach could work for another site, but archive.is seems to protect their data from automatic queries, when I try curl, python (urllib2) I get error Empty reply from server. You need something like phantomjs that mimic real browser. And I believe it will only work for few queries and then will show captcha or give errors. Also they seem to log ip addresses and even phantomjs get errors from same machine where curl or python was tried.
Here's phantomjs code that works:
var webPage = require('webpage');
var page = webPage.create();
var system = require('system');
var args = system.args;
page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36';
function getOriginalUrl(shortUrl, cb) {
page.open(shortUrl, function(status) {
//console.log(status);
var url = page.evaluate(function(){
return document.querySelector('form input').value;
});
cb(url);
});
}
if (args.length > 1) {
getOriginalUrl(args[1],function(url){
console.log(url);
phantom.exit();
});
} else {
console.log('Pass url');
phantom.exit();
}
I am trying to crawl the data from a website. There is a url which redirects to another url when we hit it from a browser but when I try to do the same with python scripts it does not redirects as it supposed to be.
The first url is -
http://www.marriott.com/reservation/availabilitySearch.mi?propertyCode=ABZAP&isSearch=true&isRateCalendar=false&flexibleDateSearchRateDisplay=false&flexibleDateLowestRateMonth=&flexibleDateLowestRateDate=&fromDate=08/10/17&toDate=08/11/17&clusterCode=&corporateCode=&groupCode=&numberOfRooms=1&numberOfGuests=2&incentiveType_Number=&incentiveType=false&marriottRewardsNumber=&useRewardsPoints=false&numberOfChildren=0&childrenAges=&numberOfAdults=2
And it redirects to -
http://www.marriott.com/reservation/rateListMenu.mi
I am also maintaining the session for cookies and also included the required headers.
Please look at the below code -
session.headers = ({
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0)Gecko/20100101 Firefox/45.0',
'Host': 'www.marriott.com',
'Connection': 'keep-alive',
'Referer': 'http://www.marriott.com/search/findHotels.mi',
'Accept': '*/*'})
second_page ='http://www.marriott.com/reservation/availabilitySearch.mi?propertyCode=ABZAP&isSearch=true&isRateCalendar=false&flexibleDateSearchRateDisplay=false&flexibleDateLowestRateMonth=&flexibleDateLowestRateDate=&fromDate=08/10/17&toDate=08/11/17&clusterCode=&corporateCode=&groupCode=&numberOfRooms=1&numberOfGuests=2&incentiveType_Number=&incentiveType=false&marriottRewardsNumber=&useRewardsPoints=false&numberOfChildren=0&childrenAges=&numberOfAdults=2'
response = session.get(second_page, headers=session.headers)
print(response.url)
When I am printing the url, it prints the original url and never gets redirected and when I try to print the first page's content it print this -
<!doctype html>
<html>
<head><script type="text/javascript" src="/common/js/marriottCommon.js"> </script>
<meta charset="utf-8">
</head>
<body>
<script>
var xhttp = new XMLHttpRequest();
xhttp.addEventListener("load", function(a,b,c){
window.location.reload()
});
xhttp.open('GET', '/reservation/availabilitySearch.mi?istl_enable=true&istl_data', true);
xhttp.send();
</script>
</body>
Please tell me what am I missing here.
<canvas class="word-cloud-canvas" id="word-cloud-canvas-1892" height="270" width="320"></canvas>
How to get the URL source of an image from HTML5 canvas using Selenium Python?
I tried to use
driver.execute_script("return arguments[0].toDataURL('image/png');", canvasElement)
But it only return the binary? of the image.
I don't want to save the image, but get the URL of the image. Is it possible?
I faced a similar issue and the only alternative I could find is to use subprocess and phantomjs
Here is the Python code
import json, subprocess
output = check_output(['phantomjs', 'getResources.js', main_url])
urls = json.loads(output)
for url in urls:
#filter and process URLs
and the Javascript file content
// getResources.js
// Usage:
// phantomjs getResources.js your_url
var page = require('webpage').create();
var system = require('system');
var urls = Array();
page.onResourceRequested = function(request, networkRequest) {
urls.push(request.url)
};
page.onLoadFinished = function(status) {
setTimeout(function() {
console.log(JSON.stringify(urls));
phantom.exit();
}, 16000);
};
page.onResourceError = function() {
return false;
}
page.onError = function() {
return false;
}
page.open(system.args[1]);
PhantomJS supports various options as well; for example to change the user agent you can use something like this:
page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1; WOW64) ...';
This is a simplified version of this answer which I used for my issue.