We Keep Coding

How to connect to Splunk API via Python, receiving javascript error

I am trying to connect to Splunk via API using python. I can connect, and get a 200 status code but when I read the content, it doesn't read the content of the page. View below:
Here is my code:
import json
import requests
import re
baseurl = 'https://my_splunk_url:8888'
username = 'my_username'
password = 'my_password'
headers={"Content-Type": "application/json"}
s = requests.Session()
s.proxies = {"http": "my_proxy"}
r = s.get(baseurl, auth=(username, password), verify=False, headers=None, data=None)
print(r.status_code)
print(r.text)
I am new to Splunk and python so any ideas or suggestions as to why this is happening would help.

You need to authenticate first to get a token, then you'll be able to hit the rest of REST endpoints. The auth endpoint it at /servicesNS/admin/search/auth/login, which will give you the session_key, which you then provide to subsequent requests.
Here is some code that uses requests to authenticate to a Splunk instance, then start a search. It then checks to see if the search is complete, if not, wait a second and then check again. Keep checking and sleeping until the search is done, then print out the results.
import time # need for sleep
from xml.dom import minidom
import json, pprint
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
base_url = 'https://localhost:8089'
username = 'admin'
password = 'changeme'
search_query = "search=search index=*"
r = requests.get(base_url+"/servicesNS/admin/search/auth/login",
data={'username':username,'password':password}, verify=False)
session_key = minidom.parseString(r.text).getElementsByTagName('sessionKey')[0].firstChild.nodeValue
print ("Session Key:", session_key)
r = requests.post(base_url + '/services/search/jobs/', data=search_query,
headers = { 'Authorization': ('Splunk %s' %session_key)},
verify = False)
sid = minidom.parseString(r.text).getElementsByTagName('sid')[0].firstChild.nodeValue
print ("Search ID", sid)
done = False
while not done:
r = requests.get(base_url + '/services/search/jobs/' + sid,
headers = { 'Authorization': ('Splunk %s' %session_key)},
verify = False)
response = minidom.parseString(r.text)
for node in response.getElementsByTagName("s:key"):
if node.hasAttribute("name") and node.getAttribute("name") == "dispatchState":
dispatchState = node.firstChild.nodeValue
print ("Search Status: ", dispatchState)
if dispatchState == "DONE":
done = True
else:
time.sleep(1)
r = requests.get(base_url + '/services/search/jobs/' + sid + '/results/',
headers = { 'Authorization': ('Splunk %s' %session_key)},
data={'output_mode': 'json'},
verify = False)
pprint.pprint(json.loads(r.text))
Many of the request calls thare used include the flag, verify = False to avoid issues with the default self-signed SSL certs, but you can drop that if you have legit certificates.
Published a while ago at https://gist.github.com/sduff/aca550a8df636fdc07326225de380a91

Nice piece of coding. One of the wonderful aspects of Python is the ability to use other people's well written packages. In this case, why not use Splunk's Python packages to do all of that work, with a lot less coding around it.
pip install splunklib.
Then add the following to your import block
import splunklib.client as client
import splunklib.results as results
pypi.org has documentation on some of the usage, Splunk has an excellent set of how-to documents. Remember, be lazy, use someone else's work to make your work look better.

Authorization header in Requests package in python

I am trying write a python script using requests package to use an online mongodb query service API hosted within the organization. The API expects the authorization header in the format 'websitename/username:Password' and using the basic authentication base64 encoding. I tried to create the GET request using the requests package which has the authorization header in the following format:
import requests
headers = {'Authorization': 'Basic %s' % 'Base64encoded
websitename/username:Password string here'}
content_res = requests.get(get_url, headers = headers).json()
But I am getting a parse error here for the string as I think the expected string for header in requests package is in form of 'username:password' here and not the desired format i.e. 'websitename/username:password'.
Is there a way in which I could use the base64 encoded sting in the format which the service is expecting i.e. 'websitename/username:password' in requests package?
Any help is highly appreciated.
Thanks.

It sounds to me like you are getting a HTTP response error because the authorization header value you are passing is not base64 encoded. To correct this you can simply encode the string using the base64 python module:
Python 2.7 https://docs.python.org/2/library/base64.html
Python 3.5 https://docs.python.org/3.5/library/base64.html
An example would be something like this:
import base64
import requests
website = 'example.com'
username = 'root'
password = '1234'
auth_str = '%s/%s:%s' % (website, username, password)
b64_auth_str = base64.b64encode(auth_str)
headers = {'Authorization': 'Basic %s' % b64_auth_str}
content_res = requests.get(get_url, headers=headers).json()

import base64
import requests
website = 'example.com'
username = 'root'
password = '1234'
auth_str = '%s/%s:%s' % (website, username, password)
b64_auth_str = base64.b64encode(auth_str.encode('ascii'))
headers = {'Authorization': 'Basic %s' % b64_auth_str}
content_res = requests.get(get_url, headers=headers).json()

Querying Jira with Python and Rest

I want to pull a list of users in the jira-users group. as i understand it, it can be done with Python using restkit.
Does anyone have any examples or links that give an example of this?
thanks.

If somebody still need a solution, you can install JIRA rest api lib https://pypi.python.org/pypi/jira/.
Just a simple example for your question:
from jira.client import JIRA
jira_server = "http://yourjiraserver.com"
jira_user = "login"
jira_password = "pass"
jira_server = {'server': jira_server}
jira = JIRA(options=jira_server, basic_auth=(jira_user, jira_password))
group = jira.group_members("jira-users")
for users in group:
print users

Jira has a REST API for external queries, it's using HTTP protocol for request and responses and the response content is formed as JSON. So you can use python's urllib and json packages to run request and then parse results.
This is Atlassian's document for Jira REST API: http://docs.atlassian.com/jira/REST/latest/ and for example check the users API: http://docs.atlassian.com/jira/REST/latest/#id120322
Consider that you should do authentication before send your request, you can find necessary information in the document.

import urllib2, base64
import requests
import ssl
import json
import os
from pprint import pprint
import getpass
UserName = raw_input("Ener UserName: ")
pswd = getpass.getpass('Password:')
# Total number of users or licenses used in JIRA. REST api of jira can take values of 50 incremental
ListStartAt = [0,50,100,150,200,250,300]
counter = 0
for i in ListStartAt:
request = urllib2.Request("https://jiraserver.com/rest/api/2/group/member?groupname=GROUPNAME&startAt=%s" %i)
base64string = base64.encodestring('%s:%s' % (UserName, pswd)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
gcontext = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
result = urllib2.urlopen(request, context=gcontext)
JsonGroupdata = result.read()
jsonToPython = json.loads(JsonGroupdata)
try:
for i in range (0,50):
print jsonToPython["values"][i]["key"]
counter = counter+1
except Exception as e:
pass
print counter

VCloud Director Org user authentication for RestAPI in python

I have VMware setup for testing. I create one user abc/abc123 to access the Org url "http://localhost/cloud/org/MyOrg". I want to access the RestAPI of the VCloud. I tried with RestClient plugin in firefox. Its working fine.
Now I tried with python code.
url = 'https://localhost/api/sessions/'
req = urllib2.Request(url)
base64string = base64.encodestring('%s:%s' % ('abc#MyOrg', 'abc123'))[:-1]
authheader = "Basic %s" % base64string
req.add_header("Authorization", authheader)
req.add_header("Accept", 'application/*+xml;version=1.5')
f = urllib2.urlopen(req)
data = f.read()
print(data)
This is the code i get from stackoverflow. But for my example its give "urllib2.HTTPError: HTTP Error 403: Forbidden" Error.
I also tried HTTP authentication for the same.

After doing some googling I found the solution from the post https://stackoverflow.com/a/6348729/243031. I change the code for my usability. I am posting the answer because if some one has same error then he will get the answer directly.
My change code is:
import urllib2
import base64
# make a string with the request type in it:
method = "POST"
# create a handler. you can specify different handlers here (file uploads etc)
# but we go for the default
handler = urllib2.HTTPSHandler()
# create an openerdirector instance
opener = urllib2.build_opener(handler)
# build a request
url = 'https://localhost/api/sessions'
request = urllib2.Request(url)
# add any other information you want
base64string = base64.encodestring('%s:%s' % ('abc#MyOrg', 'abc123'))[:-1]
authheader = "Basic %s" % base64string
request.add_header("Authorization", authheader)
request.add_header("Accept",'application/*+xml;version=1.5')
# overload the get method function with a small anonymous function...
request.get_method = lambda: method
# try it; don't forget to catch the result
try:
connection = opener.open(request)
except urllib2.HTTPError,e:
connection = e
# check. Substitute with appropriate HTTP code.
if connection.code == 200:
data = connection.read()
print "Data :", data
else:
print "ERRROR", connection.code
Hope this will help some one who want to send POST request without the data.

Read https url from Python with basic access authentication

How do you open https url in Python?
import urllib2
url = "https://user:password#domain.com/path/
f = urllib2.urlopen(url)
print f.read()
gives:
httplib.InvalidURL: nonnumeric port: 'password#domain.com'

This has never failed me
import urllib2, base64
username = 'foo'
password = 'bar'
auth_encoded = base64.encodestring('%s:%s' % (username, password))[:-1]
req = urllib2.Request('https://somewebsite.com')
req.add_header('Authorization', 'Basic %s' % auth_encoded)
try:
response = urllib2.urlopen(req)
except urllib2.HTTPError, http_e:
# etc...
pass

Please read about the urllib2 password manager and the basic authentication handler as well as the digest authentication handler.
http://docs.python.org/library/urllib2.html#abstractbasicauthhandler-objects
http://docs.python.org/library/urllib2.html#httpdigestauthhandler-objects
Your urllib2 script must actually provide enough information to do HTTP authentication. Usernames, Passwords, Domains, etc.

If you want to pass username and password information to urllib2 you'll need to use an HTTPBasicAuthHandler.
Here's a tutorial showing you how to do it.

You cannot pass credentials to urllib2.open like that. In your case, user is interpreted as the domain name, while password#domain.com is interpreted as the port number.