Indexnow protocol POST request with JSON on PYTHON - python

I am trying to make a POST request for Indexnow protocol, which allows the user to send link massive to search engine (BING) for instant indexing. As I read all documentation, I found that this protocol suggests such JSON request construction:
POST /indexnow HTTP/1.1
Content-Type: application/json; charset=utf-8
Host: <searchengine>
{
"host": "www.example.com",
"key": "265f7fb3bd6c41118f6bf05568d9c825",
"urlList": [
"https://www.example.com/url1",
"https://www.example.com/folder/url2",
"https://www.example.com/url3"
]
}
I chose to use Python, and here is my code for this POST request:
import advertools as adv
import pandas as pd
import requests
import json
import time
def submit_url_set(set_:list, key, location, host="https://www.bing.com", headers={"Content-type":"application/json", "charset":"utf-8"}):
key = "91ca7c564b334f38b0b1ed90eec8g8b2"
data = {
"host": "www.bing.com",
"key": key,
"keyLocation": "https://uchet.kz/91ca7c564b334f38b0b1ed90eec8g8b2.txt",
"urlList": [
'https://uchet.kz/news/formirovanie-obshchestva-chistogo-ot-korruptsii-dobivaetsya-tokaev/',
'https://uchet.kz/news/pravila-polucheniya-iin-inostrantsu-v-rk-izmeneny/',
'https://uchet.kz/news/zabolevaemost-koronavirusom-sredi-shkolnikov-vyrosla-v-13-raza/',
'https://uchet.kz/news/izmeneny-pravila-provedeniya-tamozhennoy-ekspertizy/'
]
}
r = requests.post(host, data=data, headers=headers)
return r.status_code
After script runs, nothing returns.
I expected the script to return server response code HTTP-code 200 OK
What was I really expected
Getting the server response code for each indexing url.

I am the author of the code.
IndexNow API doesn't return a response, if you don't take an error message it means that the request has been taken.
You can check your log files to validate that.
Also, https://www.searchenginejournal.com/indexnow-api-python/429726/ in the article, this is stated.

Go to the Post Requests Online
Type your search engine URL
Put your code in the window below in this format:
{
"host": "your_site",
"key": "your_key",
"urlList": [
"https://your_site/1",
"https://your_site/2",
"https://your_site/3"
]
}
Press SEND.
Done
SCREENSHOT

Related

how to get access tokens from refresh token? does refresh token expire?

I'm trying to create a python script which takes a (.csv with access tokens and a file) as input and uploads that file to multiple google drives whose access tokens are in that csv
but after sometime access tokens get expired and I have to get them again...just saw there's something called refresh and it refreshes access token
Is it possible to do this from python script, please explain.
Do refresh token expire?
import json
import requests
import pandas as pd
headers = {}
para = {
"name": "update",
}
files = {
'data': ('metadata', json.dumps(para), 'application/json; charset=UTF-8'),
'file': open("./update.txt", "rb")
}
tokens = pd.read_csv('tokens.csv')
for i in tokens.token:
headers={"Authorization": i}
r = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers=headers,
files=files
)
print(r.text)
In order to be able to get a new access_token programmatically using a refresh_token, you must have set access_type to offline when redirecting the user to Google's OAuth 2.0 server.
If you did that, you can get a new access_token if you do the following POST request to https://oauth2.googleapis.com/token:
POST /token HTTP/1.1
Host: oauth2.googleapis.com
Content-Type: application/x-www-form-urlencoded
client_id=your_client_id&
client_secret=your_client_secret&
refresh_token=refresh_token&
grant_type=refresh_token
The corresponding response would be something like:
{
"access_token": "1/fFAGRNJru1FTz70BzhT3Zg",
"expires_in": 3920,
"scope": "https://www.googleapis.com/auth/drive",
"token_type": "Bearer"
}
Note:
You can find code snippets for several languages in the reference I provide below, including Python, but considering you are not using the Python library, I think the HTTP/REST snippet I provided might be more useful in your situation.
Reference:
Refreshing an access token (offline access)

Basic authentication not working in node-rest-client node

I'm using node-rest-client library to call my rest API running a python flask app in a test server. The Node setup and request code is below.
If I add a user token to the request header it words fine, but in the call to obtain the user token using basic auth my python system is successfully authenticating and returning the token with a 200 status, but the flask server is then changing this to a 400 Bad Request. It works fine making the call using Postman.
Is there something missing from my two config objects for node-rest-client?
Cheers.
var options_auth = {
user: "bob",
password: "password",
mimetypes: {
json: ["application/json", "application/json;charset=utf-8"]
}
};
var Client = require('node-rest-client').Client;
var client = new Client(options_auth);
var method = "/authtoken";
var args = {
headers: {
"Content-Type": "application/json",
"api-key": "asf89a7assa98d7sd98d98ds",
//"token": "dsf98s7dsf98dsf7sd98f7dsf",
"Accept": "application/json"
},
responseConfig: {
timeout: 1000 //response timeout
}
};
client.get(Api.url+method, args, function (data, response) {
// parsed response body as js object
// raw response
//console.log(response);
if(Buffer.isBuffer(data)){
data = data.toString('utf8');
}
console.log(data);
var stringData = data.toString('utf8');
console.log("String data = "+stringData);
}).on('error', function (err) {
console.error('Something went wrong with the http client', err);
});
Also, spotted these differences between the request headers received by the server:
// Node Request fails: 400
'headers': EnvironHeaders([
('Authorization', u'Basic XXXXXXXXXXXXXX=='),
('Vga-Api-Key', u'xxxxxxxxxxxxxxxxxxxxxxxx'),
('Content-Length', u'0'),
('Connection', u'close'),
('Host', u'127.0.0.1:8080'),
('Accept', u'*/*'),
('Content-Type', u'application/json')]),
// Postman Request works: 200
'headers': EnvironHeaders([
('Authorization', u'Basic XXXXXXXXXXXXXX=='),
('Vga-Api-Key', u'xxxxxxxxxxxxxxxxxxxxxxxx'),
* ('Content-Length', u''),
* ('Connection', u'keep-alive'),
('Host', u'127.0.0.1:8080'),
* ('Cache-Control', u'no-cache'),
('Accept', u'*/*'),
('Content-Type', u''),
* ('Accept-Encoding', u'gzip, deflate')]),
The problem is your setting of the header Content-Type: application/json and the probable calling in the server of request.get_json() directly, or indirectly via the (deprecated) request.json property.
When get_json() is called Flask will check to see that a JSON payload has been sent in the body of the request and then parse it if present. That's OK if the request actually contains JSON in the body, but, being a GET request, yours doesn't. In this case, its JSON expectation being unfulfilled, the server raises a BadRequest error and returns a HTTP 400 error response.
From what you've shown your request doesn't need to be JSON because the authorisation username and password are passed in the Authorization: Basic xxxxxxxx header.
The easiest and best way to fix the problem is to simply omit the content type header.
Alternatively you can tell Flask not to complain if there is no JSON data to parse by passing silent=True to get_json, but this just papers over the problem and is not a good idea.

Translate this request from Bash to Python?

I was given a request in Bash and I have to translate it to Python 2.7. I did this kind of translations several times, but now I am not able to make it work and I do not understand why.
First of all, I was given this Bash request:
curl -X POST -v -u user#domain:password --data "#file.json" -H "Content-Type:application/json" http://destination_url_a
With the file file.json, whose content is the following one:
{
"username":"user#domain",
"password":"password",
"shortName":"a-short-name",
"visibility":"PRIVATE",
"sitePreset":"site-dashboard",
"title":"A Title",
"description":"A description."
}
If I execute the Bash line in my computer, the result is succesful.
As always, I tried to use requests library in Python to make it work. What I did is:
import requests
from requests.auth import HTTPBasicAuth
import json
data = {
"username": "user#domain",
"password": "password",
"shortName": "a-short-name",
"visibility": "PRIVATE",
"sitePreset": "site-dashboard",
"title": "A Title",
"description": "A description.",
}
headers = {'Content-Type': 'application/json'}
data_json = json.dumps(data)
r = requests.post(
url='http://destination_url_a',
data=data_json,
headers=headers,
auth=HTTPBasicAuth('user#domain', 'password'),
verify=False,
)
Unfortunately, the response, stored in r variable, is an error, despite the status code is 200.
What could be happening? Does anyone find a problem in my code or has any idea?
EDIT
However, this is another example very similar which worked perfectly:
Bash:
curl -v -H "Content-Type:application/json" -X POST --data "#file.json" -u user#domain:password http://destination_url_b
My Python code
import requests
from requests.auth import HTTPBasicAuth
import json
data = {
"userName": "user#domain",
"password": "password",
"firstName": "Firstname",
"lastName": "Lastname",
"email": "email#domain.com",
"disableAccount": "False",
"quota": -1,
"groups": ["a_group",],
}
headers = {'Content-Type': 'application/json'}
data_json = json.dumps(data)
r = requests.post(
url='http://destination_url_b',
data=data_json,
headers=headers,
auth=HTTPBasicAuth('user#domain', 'password'),
verify=False,
)
It seems to be almost the same to the other request, but this works. Different data is sent, and to a different subdomain (both are sent to the same domain). Will these modifications be important if we are talking about the User-Agent you mentioned?
Sometimes, webservices filter on user-agent. User agent of curl and python are different. That may explain.
Try to forge a "legitimate" user-agent by modifying the request header.
Finally, there was no error in the code neither in the User-Agent.
The problem was that the destination application did not accept sites with the same short-name value.
What I was doing is creating the site from Bash, which worked, then removing it from the interface of the app and trying to create it from Python with the same data. I was getting error when doing that because in spite of having removed the site, I had to remove some residual data of it from the trashcan of the app too. If not, app returned an error since it considered that the site I was trying to create already existed.
So if I had introduced different short-name in each attempt, I would not have had any error.

Facebook Messenger with Flask

I'm trying to get the FB messenger API working using Python's Flask, adapting the following instructions: https://developers.facebook.com/docs/messenger-platform/quickstart
So far, things have been going pretty well. I have verified my callback and am able to receive the messages I send using Messenger on my page, as in the logs in my heroku server indicate the appropriate packets of data are being received by my server. Right now I'm struggling a bit to send responses to the client messenging my app. In particular, I am not sure how to perform the following segment from the tutorial in Flask:
var token = "<page_access_token>";
function sendTextMessage(sender, text) {
messageData = {
text:text
}
request({
url: 'https://graph.facebook.com/v2.6/me/messages',
qs: {access_token:token},
method: 'POST',
json: {
recipient: {id:sender},
message: messageData,
}
}, function(error, response, body) {
if (error) {
console.log('Error sending message: ', error);
} else if (response.body.error) {
console.log('Error: ', response.body.error);
}
});
}
So far, I have this bit in my server-side Flask module:
#app.route('/', methods=["GET", "POST"])
def chatbot_response():
data = json.loads(req_data)
sender_id = data["entry"][0]["messaging"][0]["sender"]["id"]
url = "https://graph.facebook.com/v2.6/me/messages"
qs_value = {"access_token": TOKEN_OMITTED}
json_response = {"recipient": {"id": sender_id}, "message": "this is a test response message"}
response = ("my response text", 200, {"url": url, "qs": qs_value, "method": "POST", "json": json_response})
return response
However, running this, I find that while I can process what someone send my Page, it does not send a response back (i.e. nothing shows up in the messenger chat box). I'm new to Flask so any help would be greatly appreciated in doing the equivalent of the Javascript bit above in Flask.
Thanks!
This is the code that works for me:
data = json.loads(request.data)['entry'][0]['messaging']
for m in data:
resp_id = m['sender']['id']
resp_mess = {
'recipient': {
'id': resp_id,
},
'message': {
'text': m['message']['text'],
}
}
fb_response = requests.post(FB_MESSAGES_ENDPOINT,
params={"access_token": FB_TOKEN},
data=json.dumps(resp_mess),
headers = {'content-type': 'application/json'})
key differences:
message needs a text key for the actual response message, and you need to add the application/json content-type header.
Without the content-type header you get the The parameter recipient is required error response, and without the text key under message you get the param message must be non-empty error response.
This is the Flask example using fbmq library that works for me:
echo example :
from flask import Flask, request
from fbmq import Page
page = fbmq.Page(PAGE_ACCESS_TOKEN)
#app.route('/webhook', methods=['POST'])
def webhook():
page.handle_webhook(request.get_data(as_text=True))
return "ok"
#page.handle_message
def message_handler(event):
page.send(event.sender_id, event.message_text)
In that scenario in your tutorial, the node.js application is sending an HTTP POST request back to Facebook's servers, which then forwards the content on to the client.
So far, sounds like your Flask app is only receiving (AKA serving) HTTP requests. The reason is that that's what the Flask library is all about, and it's the only thing that Flask does.
To send an HTTP request back to Facebook, you can use any Python HTTP client library you like. There is one called urllib in the standard library, but it's a bit clunky to use... try the Requests library.
Since your request handler is delegating to an outgoing HTTP call, you need to look at the response to this sub-request also, to make sure everything went as planned.
Your handler may end up looking something like
import json
import os
from flask import app, request
# confusingly similar name, keep these straight in your head
import requests
FB_MESSAGES_ENDPOINT = "https://graph.facebook.com/v2.6/me/messages"
# good practice: don't keep secrets in files, one day you'll accidentally
# commit it and push it to github and then you'll be sad. in bash:
# $ export FB_ACCESS_TOKEN=my-secret-fb-token
FB_TOKEN = os.environ['FB_ACCESS_TOKEN']
#app.route('/', method="POST")
def chatbot_response():
data = request.json() # flasks's request object
sender_id = data["entry"][0]["messaging"][0]["sender"]["id"]
send_back_to_fb = {
"recipient": {
"id": sender_id,
},
"message": "this is a test response message"
}
# the big change: use another library to send an HTTP request back to FB
fb_response = requests.post(FB_MESSAGES_ENDPOINT,
params={"access_token": FB_TOKEN},
data=json.dumps(send_back_to_fb))
# handle the response to the subrequest you made
if not fb_response.ok:
# log some useful info for yourself, for debugging
print 'jeepers. %s: %s' % (fb_response.status_code, fb_response.text)
# always return 200 to Facebook's original POST request so they know you
# handled their request
return "OK", 200
When doing responses in Flask, you have to be careful. Simply doing a return statement won't return anything to the requester.
In your case, you might want to look at jsonify(). It will take a Python dictionary and return it to your browser as a JSON object.
from flask import jsonify
return jsonify({"url": url, "qs": qs_value, "method": "POST", "json": json_response})
If you want more control over the responses, like setting codes, take a look at make_response()

Bad request response from Google cloud messaging server, using urllib2

I have been trying to send messages to the GCM server using python urllib2. But it has been returning Error 400 (suggesting Malformed JSON). This is a very similar question to Google Cloud Messaging HTTP Error 400: Bad Request but I have already tried the solution provided there, which did not work for me.
Please find my code below
idarr = []
for obj in objects:
idarr.append(str(obj["gcid"]))
key = "AIzaxxxxxxL9xxxxxxxxxxxxxxxxxxxWbk"
data = { "registration_ids": idarr , "data": "on" }
data = json.dumps(data)
headers = {"Content-Type":"application/json","Authorization": "key="+key}
request = urllib2.Request('https://android.googleapis.com/gcm/send',data,headers )
f=urllib2.urlopen(request)
Also I am presenting here the json I am posting in the data field
{
"data": "on",
"registration_ids": [
"APA91bEPvyQuqRndnwZ94YxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxnmFVPy6nv4mPU5UyhDEZ7Jha9HnrV9nXQf25ojs8Hc_-Bw8-ElJ320koeL0PGL6hnWj0xuyltozoND_x",
"APA91bEPvyQxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxVPy6nv4mPU5UyhDEZ7Jha9HnrV9nXQf25ojs8Hc_-Bw8-ElJ320koeL0PGL6hnWj0xuyltozoND_x"
]
}

Categories