I would like to access Analytics View data from Azure DevOps to have access to registered project activities.
Would anyone have any examples of how they would do it using the azure-devops python library?
I found no example involving extracting data from Analytics View. Basically, I need a Python script that shows all the Analytics View fields of my projects.
After some research, I managed to partially solve my problem because this solution does not bring everything from Analytics View, in addition there is a limitation of 20k records in the query result:
from azure.devops.connection import Connection
from msrest.authentication import BasicAuthentication
from azure.devops.v5_1.work_item_tracking.models import Wiql
token = 'xxx'
team_instance = 'https://dev.azure.com/xxx'
credentials = BasicAuthentication("", token)
connection = Connection(base_url=team_instance, creds=credentials)
def print_work_items(work_items):
for work_item in work_items:
print(
"{0} {1}: {2}".format(
work_item.fields["System.WorkItemType"],
work_item.id,
work_item.fields["System.Title"],
)
)
wit_client = connection.clients.get_work_item_tracking_client()
def get_TC_from_query(query):
query_wiql = Wiql(query=query)
results = wit_client.query_by_wiql(query_wiql).work_items
# WIQL query gives a WorkItemReference => we get the corresponding WorkItem from id
work_items = (wit_client.get_work_item(int(result.id)) for result in results)
print_work_items(work_items)
get_TC_from_query(
"""\
SELECT
[System.Id],
[System.WorkItemType],
[System.Title],
[System.State],
[System.AreaPath],
[System.IterationPath]
FROM workitems
WHERE
[System.TeamProject] = 'Project'
and [System.WorkItemType] = 'Product Backlog Item'
and [System.State] = 'Done'
ORDER BY [System.ChangedDate] DESC
"""
)
Related
i have a work item created in azure devops
from azure.devops.connection import Connection
from msrest.authentication import BasicAuthentication
from azure.devops.v6_0.py_pi_api import JsonPatchOperation
import pprint
# Fill in with your personal access token and org URL
personal_access_token = '<pat>'
organization_url = 'https://dev.azure.com/<org>'
# Create a connection to the org
credentials = BasicAuthentication('', personal_access_token)
connection = Connection(base_url=organization_url, creds=credentials)
# Get a client (the "core" client provides access to projects, teams, etc)
wi_client = connection.clients.get_work_item_tracking_client()
ads_id= 2129
parent_work_item = wi_client.get_work_item(ads_id);
# how to get child/parent work items?
if i run the get_work_item function, it returns data, but has 'relations': None as if there is no parent/child links, even though i can see those links online. is there someway I can get the parent/child links with a function call?
I know the reason for your issue, your must expand the results. I guess the difficulty of this issue is you can't find a ready-made python example(You may don't know how to expand the results in python). Then I will write you one now:
from azure.devops.connection import Connection
from msrest.authentication import BasicAuthentication
import pprint
#get all the work items linked to a work item
def get_work_items_parents_childs(wi_id):
#get a connection to Azure DevOps
organization_url = 'https://dev.azure.com/xxx'
personal_access_token = 'xxx'
credentials = BasicAuthentication('', personal_access_token)
connection = Connection(base_url=organization_url, creds=credentials)
work_item_tracking_client = connection.clients.get_work_item_tracking_client()
work_item = work_item_tracking_client.get_work_item(wi_id, expand="relations")
#create a list
work_items_parents_childs = []
#get the work item links
for item in work_item.relations:
#get parent and child work items
if item.attributes['name'] == 'Parent' or item.attributes['name'] == 'Child':
#get the work item id
work_item_id = item.url.split('/')[-1]
#get the work item
linked_work_item = work_item_tracking_client.get_work_item(work_item_id)
#add the work item to the list
work_items_parents_childs.append(linked_work_item)
return work_items_parents_childs
items = get_work_items_parents_childs(120)
for item in items:
print(item.fields['System.Title'])
I can successfully get them:
I would like to create a managed Compute Engine group, with some startup scripts, including one that creates a subscription to an Object Change Notification. Following these tutorials, these subscriptions all require a "subscription_name".
I can't set it to a static name since it'll cause clashes when >1 VMs are spun up.
Is there a way to automatically 'increment' the name, i.e. 'VM_sub_1', 'VM_sub_2', 'VM_sub_3'... etc.?
Or is leaving it blank and accepting randomly-generated names the only way to avoid clashes?
Code sample for creating subscription:
def create_push_subscription(project_id, topic_name, subscription_name, endpoint):
"""Create a new push subscription on the given topic."""
# [START pubsub_create_push_subscription]
from google.cloud
import pubsub_v1
project_id = "bucketcfpubsub"
topic_name = "bucketcfpubsub"
subscription_name = "VM_sub_1"
endpoint = "https://bucketcfpubsub.appspot.com/push"
subscriber = pubsub_v1.SubscriberClient()
topic_path = subscriber.topic_path(project_id, topic_name)
subscription_path = subscriber.subscription_path(project_id, subscription_name)
push_config = pubsub_v1.types.PushConfig(push_endpoint = endpoint)
subscription = subscriber.create_subscription(subscription_path, topic_path, push_config)
print('Push subscription created: {}'.format(subscription))
print('Endpoint for subscription is: {}'.format(endpoint))#[END pubsub_create_push_subscription]
Attempt at fetching metadata:
def check_instance():
import time
import requests
METADATA_URL = 'http://metadata.google.internal/computeMetadata/v1/instance/id'
METADATA_HEADERS = {'Metadata-Flavor': 'Google'}
instance_id = requests.get(METADATA_URL, headers=METADATA_HEADERS)
print(instance_id)
check_instance()
returns
Response [200]
meanning
Success! A value was changed, or you reached your specified timeout_sec and the request returned successfully.
I have a trial account with Azure and have uploaded some JSON files into CosmosDB. I am creating a python program to review the data but I am having trouble doing so. This is the code I have so far:
import pydocumentdb.documents as documents
import pydocumentdb.document_client as document_client
import pydocumentdb.errors as errors
url = 'https://ronyazrak.documents.azure.com:443/'
key = '' # primary key
# Initialize the Python DocumentDB client
client = document_client.DocumentClient(url, {'masterKey': key})
collection_link = '/dbs/test1/colls/test1'
collection = client.ReadCollection(collection_link)
result_iterable = client.QueryDocuments(collection)
query = { 'query': 'SELECT * FROM server s' }
I read somewhere that the key would be my primary key that I can find in my Azure account Keys. I have filled the key string with my primary key shown in the image but key here is empty just for privacy purposes.
I also read somewhere that the collection_link should be '/dbs/test1/colls/test1' if my data is in collection 'test1' Collections.
My code gets an error at the function client.ReadCollection().
That's the error I have "pydocumentdb.errors.HTTPFailure: Status code: 401
{"code":"Unauthorized","message":"The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'get\ncolls\ndbs/test1/colls/test1\nmon, 29 may 2017 19:47:28 gmt\n\n'\r\nActivityId: 03e13e74-8db4-4661-837a-f8d81a2804cc"}"
Once this error is fixed, what is there left to do? I want to get the JSON files as a big dictionary so that I can review the data.
Am I in the right path? Am I approaching this the wrong way? How can I read documents that are in my database? Thanks.
According to your error information, it seems to be caused by the authentication failed with your key as the offical explaination said below from here.
So please check your key, but I think the key point is using pydocumentdb incorrectly. These id of Database, Collection & Document are different from their links. These APIs ReadCollection, QueryDocuments need to be pass related link. You need to retrieve all resource in Azure CosmosDB via resource link, not resource id.
According to your description, I think you want to list all documents under the collection id path /dbs/test1/colls/test1. As reference, here is my sample code as below.
from pydocumentdb import document_client
uri = 'https://ronyazrak.documents.azure.com:443/'
key = '<your-primary-key>'
client = document_client.DocumentClient(uri, {'masterKey': key})
db_id = 'test1'
db_query = "select * from r where r.id = '{0}'".format(db_id)
db = list(client.QueryDatabases(db_query))[0]
db_link = db['_self']
coll_id = 'test1'
coll_query = "select * from r where r.id = '{0}'".format(coll_id)
coll = list(client.QueryCollections(db_link, coll_query))[0]
coll_link = coll['_self']
docs = client.ReadDocuments(coll_link)
print list(docs)
Please see the details of DocumentDB Python SDK from here.
For those using azure-cosmos, the current library (2019) I opened a doc bug and provided a sample in GitHub
Sample
from azure.cosmos import cosmos_client
import json
CONFIG = {
"ENDPOINT": "ENDPOINT_FROM_YOUR_COSMOS_ACCOUNT",
"PRIMARYKEY": "KEY_FROM_YOUR_COSMOS_ACCOUNT",
"DATABASE": "DATABASE_ID", # Prolly looks more like a name to you
"CONTAINER": "YOUR_CONTAINER_ID" # Prolly looks more like a name to you
}
CONTAINER_LINK = f"dbs/{CONFIG['DATABASE']}/colls/{CONFIG['CONTAINER']}"
FEEDOPTIONS = {}
FEEDOPTIONS["enableCrossPartitionQuery"] = True
# There is also a partitionKey Feed Option, but I was unable to figure out how to us it.
QUERY = {
"query": f"SELECT * from c"
}
# Initialize the Cosmos client
client = cosmos_client.CosmosClient(
url_connection=CONFIG["ENDPOINT"], auth={"masterKey": CONFIG["PRIMARYKEY"]}
)
# Query for some data
results = client.QueryItems(CONTAINER_LINK, QUERY, FEEDOPTIONS)
# Look at your data
print(list(results))
# You can also use the list as JSON
json.dumps(list(results), indent=4)
I am using a Python (2.7) script to download via API Google Search Console data. I would like to get rid of the property and dates arguments when launching the script:
>python script. py ´http://www.example.com´ ´01-01-2000´ ´01-02-2000´
For the latter I managed to do it importing timedelta and commenting out the lines referring to that argument:
argparser = argparse.ArgumentParser(add_help=False)
argparser.add_argument('property_uri', type=str,
help=('Site or app URI to query data for (including '
'trailing slash).'))
# Start and end dates are commented out as timeframe is dynamically set
'''argparser.add_argument('start_date', type=str,
help=('Start date of the requested date range in '
'YYYY-MM-DD format.'))
argparser.add_argument('end_date', type=str,
help=('End date of the requested date range in '
'YYYY-MM-DD format.'))'''
now = datetime.datetime.now()
StartDate = datetime.datetime.now()- timedelta(days=14)
EndDate = datetime.datetime.now()- timedelta(days=7)
From = StartDate.strftime('%Y-%m-%d' )
To = EndDate.strftime('%Y-%m-%d' )
request = {
'startDate': StartDate.strftime('%Y-%m-%d' ),
'endDate': EndDate.strftime('%Y-%m-%d' ),
'dimensions': ['query'],
Now I would like get rid also of the property argument, so that I can simply launch the script and have the property specified in the script itself. My final goal is to get data from several properties using only one script.
I tried to repeat the same procedure used for the dates but no luck. Needless to say I am a total beginner at coding.
I think I can help as I had the same problem when working from the sample script given by google as guidance. Which is what I think you gotten your code from?
The problem is that the script uses the sample_tools.py script in the googleapiclient library, which is meant to abstract away all the authentication bits so you can make a quick query easily. If you want to modify the code, I would recommend writing it from scratch.
These are my functions that I've pieced together from various bits of documentation that you might find useful.
Stage 1: Authentication
def authenticate_http():
"""Executes a searchAnalytics.query request.
Args:
service: The webmasters service to use when executing the query.
property_uri: The site or app URI to request data for.
request: The request to be executed.
Returns:
An array of response rows.
"""
# create flow object
flow = flow_from_clientsecrets('path to client_secrets.json',
scope='https://www.googleapis.com/auth/webmasters.readonly',
redirect_uri='urn:ietf:wg:oauth:2.0:oob')
storage = Storage('credentials_file')
credentials = storage.get()
if credentials:
# print "have auth code"
http_auth = credentials.authorize(Http())
else:
print "need auth code"
# get authorization server uri
auth_uri = flow.step1_get_authorize_url()
print auth_uri
# get credentials object
code_input = raw_input("Code: ")
credentials = flow.step2_exchange(code_input)
storage.put(credentials)
# apply credential headers to all requests
http_auth = credentials.authorize(Http())
return http_auth
Stage 2: Build the Service Object
def build_service(api_name, version):
# use authenticate_http to return the http object
http_auth = authenticate_http()
# build gsc service object
service = build(api_name, version, http=http_auth)
return service
Stage 3: Execute Request
def execute_request(service, property_uri, request):
"""Executes a searchAnalytics.query request.
Args:
service: The webmasters service to use when executing the query.
property_uri: The site or app URI to request data for.
request: The request to be executed.
Returns:
An array of response rows.
"""
return service.searchanalytics().query(
siteUrl=property_uri, body=request).execute()
Stage 4: Main()
def main():
# define service object for the api service you want to use
gsc_service = build_service('webmasters', 'v3')
# define request
request = {'request goes here'}
# set your property set string you want to query
url = 'url or property set string goes here'
# response from executing request
response = execute_request(gsc_service, url, request)
print response
For multiple property sets you can just create a list of property sets, then create a loop and pass each property set into the 'url' argument of the 'execute_request' function
Hope this helps!
I wrote a web service in python to accept the DatasetName and the TableName as inputs in url which will be passed to big query and the fields will be returned as output.
I have used python client for bigquery and accessing the schema information like this, but am not able to get the result as i expect.
it returns "Invalid dataset ID "publicdata:samples". Dataset IDs must be alphanumeric (plus underscores, dashes, and colons) and must be at most 1024 characters long.">"
import web
from bigquery import get_client
urls = (
'/GetFields(.*)', 'get_Fields'
)
app = web.application(urls, globals())
class get_Fields:
def GET(self,r):
datasetname = web.input().datasetName
tablename = web.input().tableName
# BigQuery project id as listed in the Google Developers Console.
project_id = 'din-1085'
# Service account email address as listed in the Google Developers Console.
service_account = '101363222700-epqo6lmkl67j6u1qafha9dke0pmcck3#developer.gserviceaccount.com'
# PKCS12 or PEM key provided by Google.
key = 'Digin-d2421e7da9.p12'
client = get_client(project_id, service_account=service_account,
private_key_file=key, readonly=True)
# Submit an async query.
job_id, _results = client.get_table_schema(datasetname,tablename)
# Retrieve the results.
return results
if __name__ == "__main__":
app.run()
This is the data that i pass:
http://localhost:8080/GetFields?datasetName=publicdata:samples&tableName=shakespeare
Dataset name :publicdata:samples
TableName : shakespeare
Expected Output:
word
word_count
corpus
corpus_date
Finally made it work by changing this line
From:
# Submit an async query.
job_id, _results = client.get_table_schema(datasetname,tablename)
To:
# Submit an async query.
results = client.get_table_schema(datasetname,tablename)