I'm trying to post the data from a DataFrame file to the Watson Personality Insights API using Object storage in IBM DataScienceExperience..
I've loaded the txt file into ObjectStorage and created a DataFrame. Works fine. Don't understand how to post the data in the dataframe to the API. The provided documentation does not point me into the right direction.
This is what I've done
from io import StringIO
import requests
import json
import pandas as pd
def get_object_storage_file_with_credentials(container, filename):
"""This functions returns a StringIO object containing
the file content from Bluemix Object Storage."""
url1 = ''.join(['https://identity.open.softlayer.com', '/v3/auth/tokens'])
data = {
'auth': {
'identity': {
'methods': ['password'],
'password': {
'user': {
'name': 'UID UID UID',
'domain': {
'id': 'ID ID ID'
},
'password': 'PASS PASS'
}
}
}
}
}
headers1 = {'Content-Type': 'application/json'}
resp1 = requests.post(url=url1, data=json.dumps(data), headers=headers1)
resp1_body = resp1.json()
for e1 in resp1_body['token']['catalog']:
if(e1['type']=='object-store'):
for e2 in e1['endpoints']:
if(e2['interface']=='public'and e2['region']=='dallas'):
url2 = ''.join([e2['url'],'/', container, '/', filename])
s_subject_token = resp1.headers['x-subject-token']
headers2 = {'X-Auth-Token': s_subject_token, 'accept': 'application/json'}
resp2 = requests.get(url=url2, headers=headers2)
return StringIO(resp2.text)
PI_text = get_object_storage_file_with_credentials('MyDSXProjects', 'myPI.txt')
Next I want to post the DataFrame content to the API
I would like to know how, hope someone can provide a tip...
My Python knowledge is lacking here.
According to the Watson Personality Insights API reference, you can provide text, HTML or JSON input. You dataset is available as a pandas dataframe. Try converting the relevant column in the DataFrame to text format. For example by:
pi_api_text = PI_text['<TEXT_COLUMN>'].str.cat(sep='. ').encode('ascii', 'ignore')
Make sure you have the Python package installed:
pip install --upgrade watson-developer-cloud
Once you have the relevant data in text format make a call to the Watson Personality Insights API. For example as:
personality_insights = PersonalityInsightsV3(
version='xxxxxxxxx',
username='xxxxxxxxxx',
password='xxxxxxxxxx')
profile = personality_insights.profile(
pi_api_text, content_type='text/plain',
raw_scores=True, consumption_preferences=True)
The response will be a JSON object containing the personality traits, which you can re transform to a pandas dataframe.
Related
I'm trying to pull a complete user list from our private gitlab server instance and input it into an S3 Bucket to reference whenever we need. Eventually I will have some form of Lambda/cfn deleting it and running it again every week to update it. I'm not so great with Python and this is what I have so far..
import json
import boto3
import re
import os
import sys
import botocore
import urllib3
from pprint import pprint
sess = boto3.Session(profile_name="sso-profile-here")
s3_client = sess.client("s3")
bucket_name = "user-statistics"
http = urllib3.PoolManager()
baseuri = "https://git.tools.dev.mycompany.net/api/v4/"
access_token = "access-token-code"
def get_gitlab_users(access_token=access_token, baseuri=baseuri):
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer {}".format(access_token),
}
url = "{}/users/?per_page=100&active=true&without_project_bots=true&next_page=x-next-page".format(
baseuri
)
req = http.request(method="GET", url=url, headers=headers)
result = json.loads(req.data)
s3_client.put_object(
Bucket=bucket_name, Key="get_users_gitlab.json", Body=json.dumps(result)
)
if __name__ == "__main__":
get_gitlab_users(access_token=access_token, baseuri=baseuri)
What I would like to be able to do is pull all the users on each page and also format it a bit neater in the S3 bucket, When I download it from the bucket the format is really unreadable and I'm not sure if I can improve it, can anyone suggest anything I can do?
Please also ignore the fact my access token is directly in the code here, it's for testing at this stage and I will make sure it's not stored directly in code.
Thanks in advance for any suggestions.
You can try to use python-gitlab package instead of requests. It should be a lot easier to get user infos :
import gitlab
baseuri = "https://git.tools.dev.mycompany.net"
access_token = "access-token-code"
gl = gitlab.Gitlab(baseuri , private_token=access_token)
users = [user.asdict() for user in gl.users.list()]
users
# [{'id': 1,
# 'username': 'username1',
# 'name': 'name1',
# 'state': 'active',
# 'avatar_url': 'https://avatar.com/1',
# 'web_url': 'https://git.tools.dev.mycompany.net/username1'},
# ...]
So, I'm new to python and am struggling, self taught, and still learning to code. So be easy on me :)
I am using a script to get data from one source (Jira's API) and trying to use those results to post to another (PowerBi).
I've managed to successfully get the data, I just don't know how to pass the data to this other API.
I know how to use the GET and POST calls, it's just using the data from one to another than I can't seem to find anything about. Assuming since what I'm asking for is very specific?
edit: I also want to mention that while my get is asking for specific data, I'm getting more than I actually need. So I need a way to specify (hopefully) what data is actually being sent to PowerBi's API
import json
import requests
url = 'https://mydomain.atlassian.net/rest/api/2/search'
headers = { 'Content-Type' : 'application/json',
'Authorization' : 'Basic 123456789' }
params = {
'jql' : 'project IN (, PY, CH, NW, RP, DP, KR, DA, RE, SS, CR, CD, AB) AND issueType=incident AND statusCategory!=Done',
'startAt': 0,
'maxResults' : 50,
}
requestget = requests.get(url, headers=headers, params=params)
if requestget.status_code == 200:
print(json.dumps(json.loads(requestget.text), sort_keys=True, indent=4, separators=(",", ": ")))
else:
print("None")
Apologies if I miss understood what you were asking help on, but you could use this to send a POST request as json.
request = urllib.request.Request()#Put the powerbi api here
request.add_header('Content-Type', 'application/json; charset=utf-8')
jsondata = #your json data
jsonBytes = jsondata.encode('utf-8')
#Has to be bytes
request.add_header('Content-Length', len(jsonBytes))
response = urllib.request.urlopen(request, jsonBytes)
You could go with a requests.post instead.
jsondata = #put json data here
headers = {'content-type': 'application/json'}
response = requests.post(url, data=json.dumps(jsondata), headers=headers)
Requests documentation
I am trying to import a csv of responses into Qualtrics using the API shown here: https://api.qualtrics.com/docs/import-responses. But, since I'm a noob at Python and (by extension) at Requests, I'm having trouble figuring out why I keep getting a 413. I've gotten this far:
formTest = {
'surveyId': 'my_id',
'file': {
'value': open('dataFiles/myFile.csv', 'rb'),
'options': {
'contentType': 'text/csv'
}
}
}
headersTest = {
"X-API-TOKEN": "my_token",
'content-type': "multipart/form-data"
}
r = requests.request("POST", url, data=formTest, headers=headersTest)
print(r.text)
The format for the formTest variable is something I found when looking through other code bases for an angular implementation of this, which may not apply to a python version of the code. I can successfully use cUrl, but Python Requests, in my current situation is the way to go (for various reasons).
In a fit of desperation, I tried directly translating the cUrl request to python requests, but that didn't seem to help much either.
Has anyone done something like this before? I took a look at posts for importing contacts and the like, but there was no luck there either (since the data that needs to be sent is formatted differently). Is there something I am missing?
It's best not to mix post data and files but use two separate dictionaries. For the files you should use the files= parameter, because it encodes the POST data as a Multipart Form data and creates the required Content-Type headers.
import requests
url = 'Qualtrics API'
file_path = 'path/to/file'
file_name = 'file.name'
data = {'surveyId':'my_id'}
files = {'file' : (file_name, open(file_path, 'rb'), 'text/csv')}
headers = {'X-API-TOKEN': 'my_token'}
r = requests.post(url, data=data, files=files, headers=headers)
print(r.text)
The first value in files['file'] is the file name (optional), followed by the file object, followed by the file content type (optional).
You will find more info in the docs: Requests, POST a Multipart-Encoded File.
I'm trying to upload a file to the Folder object in salesforce using python and simple_salesforce. The file is uploaded but is empty. can anyone tell me why and how to fix the problem? Thanks.
import base64
import json
from simple_salesforce import Salesforce
userName = 'username'
passWord = 'password'
securitytoken = 'securitytoken'
sf=Salesforce(username='userName', password='passWord', security_token='securitytoken', sandbox = True)
sessionId = sf.session_id
body = ""
with open("Info.txt", "r") as f:
body = base64.b64encode(f.read())
response = requests.post('https://cs17.my.salesforce.com/services/data/v23.0/sobjects/Document/',
headers = { 'Content-type': 'application/json', 'Authorization':'Bearer %s' % sessionId},
data = json.dumps({
'Description':'Information',
'Keywords':'Information',
'FolderId': '00lg0000000MQykAAG',
'Name': 'Info',
'Type':'txt'
})
)
print response.text
Based on the code above, your request is incomplete. You create the body value as the b64encoding iof the file you wish to upload, but it isn't included in your data associated with a 'body' key.
Depending on your version of json, you may experience some problems getting the output to play nice with Salesforce. I switched from json to simplejson and that worked better for me.
Original Post:
I'm trying to create defects programmatically. I am getting a couple errors and having trouble getting any further. Here, essentially, is the code:
import requests, json
rally_auth = ('my_user', 'my_pw')
rally_auth_url = 'https://rally1.rallydev.com/slm/webservice/v2.0/security/authorize'
rally_defect = 'https://rally1.rallydev.com/slm/webservice/v2.0/defect/defect'
workspace_ref = 'https://rally1.rallydev.com/slm/webservice/v2.0/workspace/12345'
fe_project_ref = 'https://rally1.rallydev.com/slm/webservice/v2.0/project/7890'
current_fe_release_ref = "https://rally1.rallydev.com/slm/webservice/v2.0/release/45678"
r = requests.get(rally_auth_url, auth=rally_auth)
token = r.json()['OperationResult']['SecurityToken']
url = rally_defect + '/create?key=' + token
payload = {
'Name': 'My defect',
'State': 'Open',
'Project': fe_project_ref,
'Rank': 120,
'Release': current_fe_release_ref,
'key': token
}
headers = {'content-type': 'application/json'}
r = requests.post(url, data=json.dumps(payload), auth=rally_auth, headers=headers)
You'll notice that i've put the token in both the POST's URL and data. The API docs say that I should have the key in the URL, but if I do not include the key in the POST data I get:
{"CreateResult": {"_rallyAPIMajor": "2", "_rallyAPIMinor": "0", "Errors": ["Not authorized to perform action: Invalid key"], "Warnings": []}}
If I do include the key, the API request falls over differently. It will fail on the first comma.
{"CreateResult": {"_rallyAPIMajor": "2", "_rallyAPIMinor": "0", "Errors": ["Cannot parse input stream due to I/O error as JSON document: Parse error: expected '}' but saw ',' [ chars read = >>>{\"Name\": \"My defect\",<<< ]"], "Warnings": []}}
I am baffled.
Fixed code
Thanks to #nickm
import requests, json
rally_auth = ('my_user', 'my_pw')
rally_auth_url = 'https://rally1.rallydev.com/slm/webservice/v2.0/security/authorize'
rally_defect = 'https://rally1.rallydev.com/slm/webservice/v2.0/defect/defect'
workspace_ref = 'https://rally1.rallydev.com/slm/webservice/v2.0/workspace/12345'
fe_project_ref = 'https://rally1.rallydev.com/slm/webservice/v2.0/project/7890'
current_fe_release_ref = "https://rally1.rallydev.com/slm/webservice/v2.0/release/45678"
s = requests.Session()
r = s.get(rally_auth_url, auth=rally_auth)
token = r.json()['OperationResult']['SecurityToken']
url = rally_defect + '/create?key=' + token
payload = {
'Name': 'My defect',
'State': 'Open',
'Project': fe_project_ref,
'Rank': 120,
'Release': current_fe_release_ref,
}
headers = {'content-type': 'application/json'}
r = s.post(url, data=json.dumps(payload), headers=headers)
If you are using v2.0 of WS API, the token is required for update and create requests, so you are correct by including it in your post request url.
The invalid key error will come up if a token is not appended to the request or if the token is invalid for a specific session. When hitting the endpoints directly we have to maintain an http session with a cookie, otherwise the post happens in the context of a new session - different from the one in which we got the token.
Please see this post. It is not specific to Python, but conceptually it's the same.
I noticed Rank in the payload. Do you have a Rank custom field? There is no such built-in field in v2.0 in Rally. There is a DragAndDropRank, which is not numeric, and setting it by supplying a value of 120 will not work.Also, did you try double quotes instead of single quotes in the payload?
There is a pyral - Rally Python toolkit that provide convenience methods so you do not have to hit endpoints directly. Currently it works with 1.43 of WS API. The toolkit is not officially supported, but I expect that it will be updated to work with v2.0 of WS API before June 2014, when 1.43 is no longer supported (per this schedule).
Security token was introduced in v2.0. This extra authentication layer for post requests does not exist in 1.43 of WS API, and you do not have to deal with the token if using pyral.