We Keep Coding

How to connect to Splunk API via Python, receiving javascript error

I am trying to connect to Splunk via API using python. I can connect, and get a 200 status code but when I read the content, it doesn't read the content of the page. View below:
Here is my code:
import json
import requests
import re
baseurl = 'https://my_splunk_url:8888'
username = 'my_username'
password = 'my_password'
headers={"Content-Type": "application/json"}
s = requests.Session()
s.proxies = {"http": "my_proxy"}
r = s.get(baseurl, auth=(username, password), verify=False, headers=None, data=None)
print(r.status_code)
print(r.text)
I am new to Splunk and python so any ideas or suggestions as to why this is happening would help.

You need to authenticate first to get a token, then you'll be able to hit the rest of REST endpoints. The auth endpoint it at /servicesNS/admin/search/auth/login, which will give you the session_key, which you then provide to subsequent requests.
Here is some code that uses requests to authenticate to a Splunk instance, then start a search. It then checks to see if the search is complete, if not, wait a second and then check again. Keep checking and sleeping until the search is done, then print out the results.
import time # need for sleep
from xml.dom import minidom
import json, pprint
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
base_url = 'https://localhost:8089'
username = 'admin'
password = 'changeme'
search_query = "search=search index=*"
r = requests.get(base_url+"/servicesNS/admin/search/auth/login",
data={'username':username,'password':password}, verify=False)
session_key = minidom.parseString(r.text).getElementsByTagName('sessionKey')[0].firstChild.nodeValue
print ("Session Key:", session_key)
r = requests.post(base_url + '/services/search/jobs/', data=search_query,
headers = { 'Authorization': ('Splunk %s' %session_key)},
verify = False)
sid = minidom.parseString(r.text).getElementsByTagName('sid')[0].firstChild.nodeValue
print ("Search ID", sid)
done = False
while not done:
r = requests.get(base_url + '/services/search/jobs/' + sid,
headers = { 'Authorization': ('Splunk %s' %session_key)},
verify = False)
response = minidom.parseString(r.text)
for node in response.getElementsByTagName("s:key"):
if node.hasAttribute("name") and node.getAttribute("name") == "dispatchState":
dispatchState = node.firstChild.nodeValue
print ("Search Status: ", dispatchState)
if dispatchState == "DONE":
done = True
else:
time.sleep(1)
r = requests.get(base_url + '/services/search/jobs/' + sid + '/results/',
headers = { 'Authorization': ('Splunk %s' %session_key)},
data={'output_mode': 'json'},
verify = False)
pprint.pprint(json.loads(r.text))
Many of the request calls thare used include the flag, verify = False to avoid issues with the default self-signed SSL certs, but you can drop that if you have legit certificates.
Published a while ago at https://gist.github.com/sduff/aca550a8df636fdc07326225de380a91

Nice piece of coding. One of the wonderful aspects of Python is the ability to use other people's well written packages. In this case, why not use Splunk's Python packages to do all of that work, with a lot less coding around it.
pip install splunklib.
Then add the following to your import block
import splunklib.client as client
import splunklib.results as results
pypi.org has documentation on some of the usage, Splunk has an excellent set of how-to documents. Remember, be lazy, use someone else's work to make your work look better.

Issue with AppDynamics REST call with Python

I tried to call AppDynamics API using python requests but face an issue.
I wrote a sample code using the python client as follows...
from appd.request import AppDynamicsClient
c = AppDynamicsClient('URL','group','appd#123')
for app in c.get_applications():
print app.id, app.name
It works fine.
But if I do a simple call like the following
import requests
usr =<uid>
pwd =<pwd>
url ='http://10.201.51.40:8090/controller/rest/applications?output=JSON'
response = requests.get(url,auth=(usr,pwd))
print 'response',response
I get the following response:
response <Response [401]>
Am I doing anything wrong here ?

Couple of things:
I think the general URL format for app dynamics applications are (notice the '#'):
url ='http://10.201.51.40:8090/controller/#/rest/applications?output=JSON'
Also, I think the requests.get method needs an additional parameter for the 'account'. For instance, my auth format looks like:
auth = (_username + '#' + _account, _password)
I am able to get a right response code back with this config. Let me know if this works for you.

You could also use native python code for more control:
example:
import os
import sys
import urllib2
import base64
# if you have a proxy else comment out this line
proxy = urllib2.ProxyHandler({'https': 'proxy:port'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
username = "YOUR APPD REST API USER NAME"
password = "YOUR APPD REST API PASSWORD"
#Enter your request
request = urllib2.Request("https://yourappdendpoint/controller/rest/applications/141/events?time-range-type=BEFORE_NOW&duration-in-mins=5&event-types=ERROR,APPLICATION_ERROR,DIAGNOSTIC_SESSION&severities=ERROR")
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
response = urllib2.urlopen(request)
html = response.read()
This will get you the response and you can parse the XML as needed.
If you prefer it in JSON simply specify it in the request.

Script to Mobile-Friendly test

I wanted to write a shell/python script which will check if a website is mobile friendly or not. Using browser this can be easily done by visiting-
https://www.google.com/webmasters/tools/mobile-friendly/?url=<website_addr>
For eg.-
https://www.google.com/webmasters/tools/mobile-friendly/?url=http://facebook.com
I tried fetching the content through curl, wget, lynx commands but it did not worked.
How can I do so?

Sanchit,
I suggest you look at the requests library for retrieving the url. Also, as has already been said (I don't have experience with this api) you need to call 'https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?url=http://facebook.com' instead of the url you posted.
Here's an example:
import requests
r = requests.get('https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?url=http://facebook.com')
data = r.json()
That would give you a json file with all the data that the website you posted uses.

The page uses a JSONP request to an as-yet unpublished Google PageSpeed API. Google publishes PageSpeeds Insights API v2, but the page appears to be using a v3beta1 endpoint.
When you go to the https://www.google.com/webmasters/tools/mobile-friendly/?url=http://facebook.com page for example and look at the network tab of your browser developer tools, you'll see a request for:
https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?key=AIzaSyDkEX-f1JNLQLC164SZaobALqFv4PHV-kA&screenshot=true&snapshots=true&locale=en_US&url=http%3A%2F%2Ffacebook.com%2F&strategy=mobile&filter_third_party_resources=false&callback=_callbacks_._Ce2bYp0wchLY
The url parameter is directly taken from the url parameter passed to the page, the callback parameter is there for the JSONP request to provide a callback wrapper.
There is a chance Google will swap out the API key used there, but in the meantime you can use Python code to validate the mobile friendliness of a site with:
import requests
url_to_test = 'http://facebook.com'
params = {
'key': 'AIzaSyDkEX-f1JNLQLC164SZaobALqFv4PHV-kA',
'url': url_to_test,
}
api_url = 'https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady'
response = requests.get(api_url, params=params)
data = response.json()
passed = all(rule['pass'] for rule in data['ruleGroups'].values())
print('{} is {}'.format(url_to_test, 'mobile friendly' if passed else 'not mobile friendly'))

Solved it myself, with help of #TimberlakeCoding & #MartijnPieters. Here it is-
$ wget -q -O - https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?url=http://facebo‌​ok.com | grep "\"pass\": true"
If the exit status code is 0, that means website is mobile friendly else not.
Hope it helps someone!
Thanks

I wrote a simple python script for this similar task to send multiple network requests to google Mobile-Friendly Test api and save "pass" and some other fields to mysql db. It's very fast and efficient.
# download mysql connector for python
# from: https://dev.mysql.com/downloads/connector/odbc/
# select your Platform from drop-down and install it
from twisted.internet import reactor, threads
from urlparse import urlparse
import httplib
import itertools
import json
import mysql.connector
GOOGLE_API_KEY = 'YOUR GOOGLE API KEY HERE'
db = mysql.connector.connect(user='root', password='root',
host='127.0.0.1',
database='mobiletracker', autocommit=True)
cursor = db.cursor()
concurrent = 10
finished=itertools.count(1)
reactor.suggestThreadPoolSize(concurrent)
def getData(ourl):
googleapiUrl = 'https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?url=' + ourl + '&key=' + GOOGLE_API_KEY
print googleapiUrl
url = urlparse(googleapiUrl)
conn = httplib.HTTPSConnection(url.netloc)
conn.request("GET", url.path + '?' + url.query)
res = conn.getresponse()
return res.read()
def processResponse(response,url):
jsonData = json.loads(response)
try:
score = str(jsonData['ruleGroups']['USABILITY']['score'])
except Exception, e:
score = '0'
try:
pass_ = jsonData['ruleGroups']['USABILITY']['pass'] #Boolean
if pass_:
pass_ = '1'
else:
pass_ = '0'
except Exception, e:
pass_ = '0'
try:
cms = str(jsonData['pageStats']['cms'])
except Exception, e:
cms = ''
cursor.execute("SELECT id FROM mobile WHERE url='" + url + "'")
result = cursor.fetchone()
try:
id_ = str(result[0])
query = "UPDATE mobile SET score='" + score + "', pass='" + pass_ + "', cms='" + cms + "' WHERE id = '" + id_ + "'"
print query
cursor.execute(query)
except Exception, e:
query = "INSERT INTO mobile SET url='" + url + "', score='" + score + "', pass='" + pass_ + "', cms='" + cms + "'"
print query
cursor.execute(query)
processedOne()
def processError(error,url):
print "error", url, error
processedOne()
def processedOne():
if finished.next()==added:
reactor.stop()
def addTask(url):
req = threads.deferToThread(getData, url)
req.addCallback(processResponse, url)
req.addErrback(processError, url)
added=0
for url in open('urllist.csv'):
added+=1
addTask(url.strip())
try:
reactor.run()
except KeyboardInterrupt:
reactor.stop()
Also available at https://github.com/abm-adnan/multiple-requests

Anyone coming to this page like I did, searching for an answer, the API is no longer "beta". Here's an example:
curl -H 'Content-Type: application/json' --data '{url: "https://URL_OF_WEBSITE.COM/"}' 'https://searchconsole.googleapis.com/v1/urlTestingTools/mobileFriendlyTest:run?key=YOUR_API_KEY'
Then, it will return JSON like this:
{
"testStatus": {
"status": "COMPLETE"
},
"mobileFriendliness": "MOBILE_FRIENDLY",
"resourceIssues": [
{
"blockedResource": {
"url": "https://assist.zoho.com/login/embed-remote-support.jsp"
}
}
]
}

urllib2.HTTPError: HTTP Error 401 while querying using the new Bing API ( in azure marketplace )

So, I ve made corrections based on most of the answers under the same roof in stack overflow, I'm still unable to resolve this problem.
queryBingFor = "Google Fibre"
quoted_query = urllib.quote(queryBingFor)
account_key = "dslfkslkdfhsehwekhrwkj2187iwekjfkwej3"
rootURL = "https://api.datamarket.azure.com/Bing/Search/v1/"
searchURL = rootURL + "Image?format=json&Query=" + quoted_query
cred = base64.encodestring(accountKey)
reqBing = urllib2.Request(url=searchURL)
author = "Basic %s" % cred
reqBing.add_header('Authorization',author)
readURL = urllib2.urlopen(reqBing)
I know I'm missing out something in the above code, that gives me a:
urllib2.HTTPError: HTTP Error 401: The authorization type you provided is not supported. Only Basic and OAuth are supported
Any clue on what the problem could be?
Thanks!

So, here's the working code. The problem I was creating is the query keywords' format.
queryBingFor = "'google fibre'" # the apostrophe's required as that is the format the API Url expects.
quoted_query = urllib.quote(queryBingFor)
rootURL = "https://api.datamarket.azure.com/Bing/Search/"
searchURL = rootURL + "Image?$format=json&Query=" + quoted_query
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, searchURL,username,accountKey)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
readURL = urllib2.urlopen(searchURL).read()
This should give the results in the respective JSON format. As I'm using urllib2's httpbasicauthhandler, the password's converted into base64 implicitly, I presume.

VCloud Director Org user authentication for RestAPI in python

I have VMware setup for testing. I create one user abc/abc123 to access the Org url "http://localhost/cloud/org/MyOrg". I want to access the RestAPI of the VCloud. I tried with RestClient plugin in firefox. Its working fine.
Now I tried with python code.
url = 'https://localhost/api/sessions/'
req = urllib2.Request(url)
base64string = base64.encodestring('%s:%s' % ('abc#MyOrg', 'abc123'))[:-1]
authheader = "Basic %s" % base64string
req.add_header("Authorization", authheader)
req.add_header("Accept", 'application/*+xml;version=1.5')
f = urllib2.urlopen(req)
data = f.read()
print(data)
This is the code i get from stackoverflow. But for my example its give "urllib2.HTTPError: HTTP Error 403: Forbidden" Error.
I also tried HTTP authentication for the same.

After doing some googling I found the solution from the post https://stackoverflow.com/a/6348729/243031. I change the code for my usability. I am posting the answer because if some one has same error then he will get the answer directly.
My change code is:
import urllib2
import base64
# make a string with the request type in it:
method = "POST"
# create a handler. you can specify different handlers here (file uploads etc)
# but we go for the default
handler = urllib2.HTTPSHandler()
# create an openerdirector instance
opener = urllib2.build_opener(handler)
# build a request
url = 'https://localhost/api/sessions'
request = urllib2.Request(url)
# add any other information you want
base64string = base64.encodestring('%s:%s' % ('abc#MyOrg', 'abc123'))[:-1]
authheader = "Basic %s" % base64string
request.add_header("Authorization", authheader)
request.add_header("Accept",'application/*+xml;version=1.5')
# overload the get method function with a small anonymous function...
request.get_method = lambda: method
# try it; don't forget to catch the result
try:
connection = opener.open(request)
except urllib2.HTTPError,e:
connection = e
# check. Substitute with appropriate HTTP code.
if connection.code == 200:
data = connection.read()
print "Data :", data
else:
print "ERRROR", connection.code
Hope this will help some one who want to send POST request without the data.