write log entry to splunk via HTTP in python - python

We have a python program that needs to send logs to splunk. Our splunk admins have created a service collector HTTP endpoint to publish logs to with the following:
index
token
hostname
URI
We can't find where to input the URI in the splunk python SDK client. For example:
import splunklib.client as client
import splunklib.results as results_util
HOST="splunkcollector.hostname.com"
URI="services/collector/raw"
TOKEN="ABCDEFG-8A55-4ABB-HIJK-1A7E6637LMNO"
PORT=443
# Create a Service instance and log in
service = client.connect(
host=HOST,
port=PORT,
token=TOKEN)
# Retrieve the index for the data
myindex = service.indexes["cloud_custodian"]
# Submit an event over HTTP
myindex.submit("Dummy test python client log")
As you can see I never use the URI variable. The above code results in:
Traceback (most recent call last):
File "splunk_log.py", line 15, in <module>
myindex = service.indexes["cloud_custodian"]
File "/usr/local/lib/python2.7/site-packages/splunklib/client.py", line 1230, in __getitem__
raise KeyError(key)
KeyError: UrlEncoded('cloud_custodian')

Ended up performing a stock POST with requests. I'm not sure if the splunk client is even intended to support the HTTP Event Collector.
import requests
url='https://splunkcollector.hostname.com:443/services/collector/event'
authHeader = {'Authorization': 'Splunk {}'.format('ABCDEFG-8A55-4ABB-HIJK-1A7E6637LMNO')}
jsonDict = {"index":"cloud_custodian", "event": { 'message' : "Dummy test python client log" } }
r = requests.post(url, headers=authHeader, json=jsonDict, verify=False)
print r.text

You should look into the HTTP Event Collector in Splunk. It's as simple as enabling it, generating a token, and making the call.
If you wanted to send data to Splunk HEC, it would look like this
<protocol>://<host>:<port>/<endpoint>
https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Data/UsetheHTTPEventCollector

Related

I am trying to send a request to OVH'S API, but when trying to do this i get an error

I am trying to send a request to OVH's API using their python API wrapper to check if my IP address is in mitigation, when trying to do this I get the following error:
result = client.get(f'/ip/{quote(ipblock)}/mitigation/{ipOnMitigation}', _need_auth=False)
File "/usr/local/lib/python3.8/dist-packages/ovh/client.py", line 347, in get
return self.call('GET', _target, None, _need_auth)
File "/usr/local/lib/python3.8/dist-packages/ovh/client.py", line 442, in call
raise ResourceNotFoundError(json_result.get('message'),
ovh.exceptions.ResourceNotFoundError: Got an invalid (or empty) URL
Here is my code
import json
import ovh
from urllib.parse import quote
client = ovh.Client(
endpoint='ovh-ca',
application_key='xxxxxxx',
application_secret='xxxxxxx',
consumer_key='xxxxxxxxx'
)
ipblock = "xxxx/28"
ipOnMitigation = "xxx/32"
result = client.get(f'/ip/{quote(ipblock)}/mitigation/{ipOnMitigation}', _need_auth=False)
# Pretty print
print(json.dumps(result))
Maybe the endpoint needs to look like a path (ie /ovh-ca)?
Edit: the exception indicates that something is wrong with the request. If it’s not the endpoint, it must be some other parameter your passing to the client (or not passing). I see you have a app key/secret pair. Does this endpoint require a consumer secret (to build a request signature for instance) also?

Got 503 error code while retrieveing mediaid in wechat?

API http://admin.wechat.com/wiki/index.php?title=Transferring_Multimedia_Files
import requests
r='https://api.wechat.com/cgi-bin/token?grant_type=client_credential&appid=wx82c6ebdb6e33ad33&secret=c2861ec348b3c94087c4b64cbe166fbb' #credentials sharing no problem
a=(requests.get(r).json()['access_token'])
print(a)
params = (
('access_token', a),
('type', 'image'),
)
import os
files = {
'media': ('1.jpg', open('static/1.jpg', 'rb'),'image/jpg',),
}
print()
response = requests.post('http://file.api.wechat.com/cgi-bin/media/upload', params=params, files=files)
def uprint(x,file=None):
try:
pass
if x:
print(x.encode('cp65001',errors='backslashreplace').decode('cp1252'),file=file)
except Exception as e:
return f'{x}\n{e}'
def prin(*a):print(ascii(*a))
print(response.text,file=open('z.html','a',encoding="utf-8"))
print(response.headers)
Looks like you are using http for the upload call. I have seen this error returned from sites for this reason before.
The HTTP error 503 means "Service Unavailable". It is usually returned by a server when it is unable to handle the request due to a temporary overloading or maintenance of the server.
After checking the API documentation for wechat, I notided this:
This API must be used via HTTPS.
And then I noticed this in the Q&A:
Q: Which server should I send API requests to?
A: If you have an International Official Account, use api.wechat.com.
If you have a China Official Account, use api.weixin.qq.com.
So, in your case I think you need to use https, and the domain api.wechat.com, like this:
response = requests.post('https://api.wechat.com/cgi-bin/media/upload', params=params, files=files)

Splunk Python Connection Lost

I'm using python to execute a splunk search query and return the results. I connect with the following string:
service = client.connect(
host=HOST,
port=PORT,
username=USERNAME,
password=PASSWORD
)
The variables have been tested to work, and it connects to splunk, but sometimes, when I run these lines of code:
print "Installed App Names \n"
for app in service.apps:
print app.name
It returns this error:
Request Failed: Session is not logged in
About 50% of the time, the code works, and it executes. Is this inconsistency in code results do to the service = lines of code not actually connecting to the splunk server? Can these connections time out?
connect can take an autologin=True argument to allow the bindings to try to re-connect when authentication fails, instead of raising that error immediately.
Probably you should get the token and session id of splunk using your python code. Please find the below code if this could help you.
import json,os,sys,requests
BASE_URL = "https://SPLUNKLB / SPLUNK WEB URL"
def getToken():
# body for token request
payload = {'username': "",'password': ""}
TOKEN_URL = "/services/auth/login?output_mode=json"
# post token request
res = requests.post(BASE_URL+TOKEN_URL, data=payload, verify=False)
if (res.status_code == 200):
# Get token out of response
resJson = json.loads(res.content)
return resJson.get('sessionKey')
else:
print res.status_code, res.content

Insert post Blogger API failing with Python

This code now works.
I'm having an issue inserting a new blog post on to google's Blogger site via python2.7 calling an API.
I have all the oauth2client modules from google to handle the authentication.
I have permission to use the Blogger V3 api - this is activated on the google developer console.
I have run simple api requests with the same credentials.dat that have worked:
this worked (full code not included)
service = build('blogger','v3', http=http)
try:
request = service.blogs().get(blogId="6814573853229626501")
response = request.execute()
print response
The google api discovery service leads me to believe this is what the code should look like to insert a post
https://developers.google.com/apis-explorer/#p/blogger/v3/blogger.posts.insert
service = build('blogger','v3', http=http)
try:
body = {
"kind": "blogger#post",
"id": "6814573853229626501",
"title": "posted via python",
"content":"<div>hello world test</div>"
}
request = service.posts().insert(blogId="6814573853229626501",body=body)
response = request.execute()
print response
I'm sure it's the body=body part that I'm messing up? Any clues?
here is the error that I get:
Traceback (most recent call last):
File "blogger.py", line 104, in <module>
main()
File "blogger.py", line 93, in main
response = request.execute()
File "/usr/local/lib/python2.7/dist-packages/google_api_python_client-1.0c2-py2.7.egg/apiclient/http.py", line 654, in execute
raise HttpError(resp, content, self.uri)
apiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/blogger/v3/blogs/6814573853229626501/posts?alt=json returned "Invalid Value">
If you're interested I'm experimenting with posting charts from my google fusion tables generated by eBay data that i'm interested in at the time.
You can post on any blogger somehow
__author__ = 'spandey2405#gmail.com (Saurabh Pandey)'
import sys
from oauth2client import client
from googleapiclient import sample_tools
# Authenticate and construct service.
service, flags = sample_tools.init(
argv, 'blogger', 'v3', __doc__, __file__,
scope='https://www.googleapis.com/auth/blogger')
try:
users = service.users()
# Retrieve this user's profile information
thisuser = users.get(userId='self').execute()
print('This user\'s display name is: %s' % thisuser['displayName'])
blogs = service.blogs()
# Retrieve the list of Blogs this user has write privileges on
thisusersblogs = blogs.listByUser(userId='self').execute()
for blog in thisusersblogs['items']:
print('The blog named \'%s\' is at: %s' % (blog['name'], blog['url']))
posts = service.posts()
body = {
"kind": "blogger#post",
"id": "6701167141462934671",
"title": "posted via python",
"content":"<div>hello world test</div>"
}
insert = posts.insert(blogId='6701167141462934671', body=body)
posts_doc = insert.execute()
print posts_doc
except client.AccessTokenRefreshError:
print ('The credentials have been revoked or expired, please re-run'
'the application to re-authorize')
You do not need the "data": object wrapper around your data, the client library will add that if the server needs it. This documentation shows the form of the object to use in the insert call:
https://google-api-client-libraries.appspot.com/documentation/blogger/v3/python/latest/blogger_v3.posts.html#insert

django/python the works of views and bringing in an API

I'm just beginning to learn about python/django. I know PHP, but I wanted to get to know about this framework. I'm trying to work with yelp's API. I'm trying to figure out what to do when someone brings in a new file into the project.
In their business.py they have this:
import json
import oauth2
import optparse
import urllib
import urllib2
parser = optparse.OptionParser()
parser.add_option('-c', '--consumer_key', dest='consumer_key', help='OAuth consumer key (REQUIRED)')
parser.add_option('-s', '--consumer_secret', dest='consumer_secret', help='OAuth consumer secret (REQUIRED)')
parser.add_option('-t', '--token', dest='token', help='OAuth token (REQUIRED)')
parser.add_option('-e', '--token_secret', dest='token_secret', help='OAuth token secret (REQUIRED)')
parser.add_option('-a', '--host', dest='host', help='Host', default='api.yelp.com')
parser.add_option('-i', '--id', dest='id', help='Business')
parser.add_option('-u', '--cc', dest='cc', help='Country code')
parser.add_option('-n', '--lang', dest='lang', help='Language code')
options, args = parser.parse_args()
# Required options
if not options.consumer_key:
parser.error('--consumer_key required')
if not options.consumer_secret:
parser.error('--consumer_secret required')
if not options.token:
parser.error('--token required')
if not options.token_secret:
parser.error('--token_secret required')
if not options.id:
parser.error('--id required')
url_params = {}
if options.cc:
url_params['cc'] = options.cc
if options.lang:
url_params['lang'] = options.lang
path = '/v2/business/%s' % (options.id,)
def request(host, path, url_params, consumer_key, consumer_secret, token, token_secret):
"""Returns response for API request."""
# Unsigned URL
encoded_params = ''
if url_params:
encoded_params = urllib.urlencode(url_params)
url = 'http://%s%s?%s' % (host, path, encoded_params)
print 'URL: %s' % (url,)
# Sign the URL
consumer = oauth2.Consumer(consumer_key, consumer_secret)
oauth_request = oauth2.Request('GET', url, {})
oauth_request.update({'oauth_nonce': oauth2.generate_nonce(),
'oauth_timestamp': oauth2.generate_timestamp(),
'oauth_token': token,
'oauth_consumer_key': consumer_key})
token = oauth2.Token(token, token_secret)
oauth_request.sign_request(oauth2.SignatureMethod_HMAC_SHA1(), consumer, token)
signed_url = oauth_request.to_url()
print 'Signed URL: %s\n' % (signed_url,)
# Connect
try:
conn = urllib2.urlopen(signed_url, None)
try:
response = json.loads(conn.read())
finally:
conn.close()
except urllib2.HTTPError, error:
response = json.loads(error.read())
return response
response = request(options.host, path, url_params, options.consumer_key, options.consumer_secret, options.token, options.token_secret)
print json.dumps(response, sort_keys=True, indent=2)
Its very lengthy, I appologize for that. But my concern is what do I do with this? They've set up a def request() in here, and I'm assuming that I have to import this into my views?
I've been following the django documentation of creating a new app. In the documentation they've set up a bunch of def inside the views.py file. I'm just confused as to how am I supposed to make this work with my project? If I wanted to search for a business in the URL, how would it send the data out?
Thanks for your help.
This is a command line script that makes http requests to the yelp api. You probably don't want to make such an external request within the context of a main request handler. Well, you could call a request handler that makes this call to yelp. Let's see ...
You could import the request function and instead of invoking it with command line options, call it yourself.
from yelp.business import request as yelp_req
def my_request_handler(request):
json_from_yelp = yelp_req(...
# do stuff and return a response
Making this kind of external call inside a request handler is pretty meh though (that is, making an http request to an external service within a request handler). If the call is in ajax, it may be ok for the ux.
This business.py is just an example showing you how to create a signed request with oauth2. You may be able to just import the request function and use it. OTOH, you may prefer to write your own (perhaps using the requests library). You probably want to use celery or some other async means to make the calls outside of your request handlers and/or cache the responses to avoid costly external http io with every request.

Categories