ATDD Google App Engine with Python - Trouble with Keys - python

My approach to this question might be completely wrong, so please don't hesitate to correct it. I also added ATDD in the question title, because I am trying to test the output of my web api which is bigger than just a unit test.
I am using:
Google App Engine 1.7.1
GAE High Replication Datastore
Python 2.7.3
I setup my tests using the boilerplate code:
self.testbed = testbed.Testbed()
self.testbed.activate()
self.policy = datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=0)
self.testbed.init_datastore_v3_stub()
Then I call a mock object setup to setup my entities and persist them to the testbed datastore. I use a counter to add 5 artists with artist_id 1 - 5:
def add_mock_artist_with(self, artist_id, link_id, name):
new_artist = dto.Artist(key_name=str(artist_id),
name=name,
link_id= str(link_id))
new_artist.put()
My test is that my web api returns:
{
"artists": [
{
"artist": {
"name": "Artist 1",
"links": {
"self": {
"href": "https://my_server.com/artist/1/"
}
}
}
},
.
.
.
{
"artist": {
"name": "Artist 5",
"links": {
"self": {
"href": "https://my_server.com/artist/5/"
}
}
}
}
],
"links": {
"self": {
"href": "https://my_server.com/artists/"
}
}
}
Initially I thought that if I were to startup a new test bed every time I run the tests, I could count on my artists being entered sequentially into the data store and therefore would get ids 1 - 5. My tests passed initially, but over time started to fail because of ids not matching (I would get a link like: "href": "https://my_server.com/artist/78/"). I felt a little guilty relying on ids being generated sequentially, so I resolved to fix it. I stumbled upon the concept of a key being either a name or a generated id. I updated my templates for the returned JSON to use:
artist.key().id_or_name()
In the case of a mock object, I supplied the key name on construction with:
key_name=str(artist_id)
For non-mock construction, I did not include that line of code and let GAE assign the id.
Since my template used key().id_or_name() to output the property, all went well and the tests passed.
However, now when I test the return of an individual artist which would be available by following http://my_server.com/artist/5/, my test fails. To grab the artist out of the datastore, I use the following code:
def retrieve_artist_by(id):
artist = dto.Artist.get_by_id()
In production, this will work fine because it will all be id based. However, in my test, it is not finding anything because I have used the key_name=str(artist_id) in my mock construction, and the name is not the same as the id.
I was hoping for something similar to:
artist = dto.Artist.get_by_id_or_name()
Any ideas?

Perhaps not the answer you are looking for, but it's possible to find out if you are running on the production or deployment servers and execute a different code path.
In Python, how can I test if I'm in Google App Engine SDK?
http://code.google.com/appengine/docs/python/runtime.html#The%5FEnvironment
os.environ['SERVER_SOFTWARE'].startswith('Development')

Here is what I am using right now. I don't like it, but it should be performant in prod as I will be using generated ids. In test, if it doesn't find it by id, it will attempt to look up by name.
def retrieve_artist_by(id):
artist = dto.Artist.get_by_id(id)
if artist is None:
artist = dto.Artist.get_by_key_name(str(id))
return artist

Related

Change status of issue in Jira using Rest call from Python

I am trying to update the status of issue from 'Request Info' state to 'Submitted' via rest API in python.
after digging a lot in the documentation , I ran a Rest call to get the allowed status for the issue ID and i can see that the status 'Submitted' is allows:
"expand": "transitions",
"transitions": [{
"id": "381",
"name": "Resubmit",
"to": {
"self": "https://ies-valor-jira.ies.mentorg.com/rest/api/2/status/10000",
"description": "",
"iconUrl": "https://ies-valor-jira.ies.mentorg.com/",
"name": "Submitted",
"id": "10000",
"statusCategory": {
"self": "https://ies-valor-jira.ies.mentorg.com/rest/api/2/statuscategory/2",
"id": 2,
"key": "new",
"colorName": "blue-gray",
"name": "To Do"
So now i would like to change the status with the following code:
from jira import JIRA
JIRA_SERVER = "https://ies-valor-jira.ies.mentorg.com/"
jira_user_name = "myuser"
jira_password = "mypassword!"
jira_connection = JIRA(basic_auth=(jira_user_name, jira_password),
server=JIRA_SERVER)
jira.Issue='SF-6831'
jira_connection.add_comment(jira.Issue, body="Resubmit issues")
jira_connection.transition_issue("SF-6831", "Resubmit")
But i get an error message that indicate :customfield_10509":"You must define "Resubmit Note: before you moving to "CCB Review" state"}
this error is expected because this field is mandatory and it must be filled with a reason for the status change so i need to know how to update this custom field in the Rest call to allow the issue to change status.
I tried to use add.command function but i don't know where i should specify the field name.
or is there a different way to do it.
Thanks.
I do most of this work in straight Python using the API, I have a lot of what you need in this repo - https://github.com/dren79/JiraScripting_public
I'll add a transition function in the helpers file now.
Let me know if it helps!

how to restart instance group via python google cloud library

I am not able to find any code sample or relevant documentation on python library for google cloud
Want to restart managed instance groups all vms via cloud function.
To list instances I am using something like this
import googleapiclient.discovery
def list_instances(compute, project, zone):
result = compute.instances().list(project=project, zone=zone).execute()
return result['items'] if 'items' in result else None
in requirement file I have
google-api-python-client==2.31.0
google-auth==2.3.3
google-auth-httplib2==0.1.0
From command line this is possible via SDK ->
https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/managed/rolling-action/restart
gcloud compute instance-groups managed rolling-action restart NAME [--max-unavailable=MAX_UNAVAILABLE] [--region=REGION | --zone=ZONE] [GCLOUD_WIDE_FLAG …]
But in python I am not able to write any code.
This is an incomplete answer since the python docs are pretty unreadable to me.
Looking at the gcloud cli code (which I couldn't find an official repo for so I looked here),
the restart command is triggered by something called a "minimal action".
minimal_action = (client.messages.InstanceGroupManagerUpdatePolicy.
MinimalActionValueValuesEnum.RESTART)
In the Python docs, there's references to these fields in the applyUpdatesToInstances method.
So I think the relevant code is something similar to:
compute.instanceGroupManagers().applyUpdatesToInstances(
project=project,
zone=zone,
instanceGroupManager='NAME',
body={"allInstances": True, "minimalAction": "RESTART"},
)
There may or may not be a proper Python object for the body, the docs aren't clear.
And the result seems to be an Operation object of some kind, but I don't know if there's execute() method or not.
This is confusing, because gcloud compute instance-groups managed rolling-action is syntactic sugar that does two things:
It turns on Proactive updater, by setting appropriate UpdatePolicy on the InstanceGroupManager resource
And it changes version name on the same resource to trigger an update.
It is covered in the docs in https://cloud.google.com/compute/docs/instance-groups/rolling-out-updates-to-managed-instance-groups#performing_a_rolling_replace_or_restart
Compare the gcloud and API tabs to get the idea.
Unfortunately I am illiterate in Python, so I am not able to translate it into Python code :(.
Using the documentation that #Grzenio provided, use patch() method to restart the instance group. See patch documentation to check its parameters.
This could be written in python using the code below. I provided the required parameters project,zone,instanceGroupManager and body. The value of body is from the example in the documentation.
import googleapiclient.discovery
import json
project = 'your-project-id'
zone = 'us-central1-a' # the zone of your instance group
instanceGroupManager = 'instance-group-1' # instance group name
body = {
"updatePolicy": {
"minimalAction": "RESTART",
"type": "PROACTIVE"
},
"versions": [{
"instanceTemplate": "global/instanceTemplates/instance-template-1",
"name": "v2"
}]
}
compute = googleapiclient.discovery.build('compute', 'v1')
rolling_restart = compute.instanceGroupManagers().patch(
project=project,
zone=zone,
instanceGroupManager=instanceGroupManager,
body=body
)
restart_operation = rolling_restart.execute() # execute the request
print(json.dumps(restart_operation,indent=2))
This will return an operation object and the instance group should restart in the rolling fashion:
{
"id": "3206367254887659944",
"name": "operation-1638418246759-5d221f9977443-33811aed-eed3ee88",
"zone": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a",
"operationType": "patch",
"targetLink": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a/instanceGroupManagers/instance-group-1",
"targetId": "810482163278776898",
"status": "RUNNING",
"user": "serviceaccountused#your-project-id.iam.gserviceaccount.com",
"progress": 0,
"insertTime": "2021-12-01T20:10:47.654-08:00",
"startTime": "2021-12-01T20:10:47.670-08:00",
"selfLink": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a/operations/operation-1638418246759-5d221f9977443-33811aed-eed3ee88",
"kind": "compute#operation"
}

How to improve spacing in a document when adding data in key value pair using google docs API and python

How should I modify my request payload to add data as key value pair in a google docx using google docs API in python.
When I use the following payload, the alignment gets ruined.
requests = [{
"insertText":{
"text":"\nName of the Organization\t\t\t\tStackOverflow\nIndustry\t\t\t\tSo
ftware\nBusiness_Id\t\t\t123\n",
"location":{
"index":4
}
}
}]
Output:
How can I align it properly so that the output is something like
Name Of the Organization StackOverflow
Industry Software
Business_Id 123
or can we put this in a table without showing the table borders?
In your situation, I would like to propose using a table for achieving your goal. When Docs API is used, the table can be created without borders. But unfortunately, in the current stage, I had thought that it is difficult to directly create a table using Docs API. So I had created a library for managing the table on Google Document using Docs API. In this answer, I would like to propose to achieve your goal using this library.
Usage:
1. Install library.
$ pip install gdoctableapppy
2. Sample script:
docs = build('docs', 'v1', credentials=creds) # Please use your script here.
documentId = "###" # Please set Google Document ID.
values = [['Name Of the Organization', 'StackOverflow'], ['Industry', 'Software'], ['Business_Id', '123']]
resource = {
"oauth2": creds,
"documentId": documentId,
"rows": len(values),
"columns": len(values[0]),
"append": True,
"values": values,
}
gdoctableapp.CreateTable(resource)
resource = {
"oauth2": creds,
"documentId": documentId,
}
res = gdoctableapp.GetTables(resource)
obj = {"color": {"color": {}}, "dashStyle": "SOLID", "width": {"magnitude": 0, "unit": "PT"}}
requests = [{
"updateTableCellStyle": {
"tableCellStyle": {
"borderBottom": obj,
"borderTop": obj,
"borderLeft": obj,
"borderRight": obj,
},
"tableStartLocation": {
"index": res['tables'][-1]['tablePosition']['startIndex']
},
"fields": "borderBottom,borderTop,borderLeft,borderRight"
}
}]
docs.documents().batchUpdate(documentId=documentId, body={'requests': requests}).execute()
3. Testing.
When the above script is run, the following result can be obtained.
References:
gdoctableapppy
This is a python library to manage the tables on Google Document using Google Docs API.
Method: documents.batchUpdate

How to resolve the error: "columns[11]: custom variable cannot be found" when running an SA360 API call?

Heyy!!
Hope everyone is doing well,
I'm pulling data from SA360 (or DS3, Doubleclick Search), although I receive this error when I try to download the report:
"columns[11]: A custom variable named 'DDA Product Sign Ups' with
platform source 'FLOODLIGHT' cannot be found.">
I know:
That conversion exists on the platform UI (second result)
That my script works because when I take off the conversion field I can deploy my function with no prob.
My Script (more or less):
conversion_name = "DDA Product Sign Ups"
request = ds3_manager.reports().request(body =
{
"reportScope": {
"agencyId": agency_id,
"advertiserId" : advertiser_id },
"reportType": "adGroup",
"columns": [
{ "columnName": "date"},
{ "columnName": "accountType"},
{ "columnName": "account" },
{ "columnName": "cost" },
{ "columnName": "impr" },
{ "columnName": "clicks" },
{
"customMetricName" : conversion_name,
"platformSource": "floodlight"
}
],
"timeRange": {
"startDate": start_date,
"endDate": end_date
},
"downloadFormat": "csv",
"maxRowsPerFile": 6000000,
"statisticsCurrency": "agency"
}
)
When I google the issue I land on this web result: Set up custom Floodlight metrics and dimensions but I don't understand, to me, it is already set up, as I can add to it to my reports on the UI or on my webqueries already... So I'm not why it is not picked up by the script..
If anyone has an idea that would be greatly appreciated :D.
Best,
Alex
In order to include custom columns in a report request, you first need to find out how that custom column is defined (check this link). You can then include the custom column in the request (check this link).
You will find useful information on this topic in the "Optional: Request for Specific User-defined Column" section of this article.

Is there a way to obtain Facebook Insights for specific posts trough the Graph API?

Sorry to bother with maybe a stupid question, but I'm still a beginner with the Graph API. A little background to better understand my question: I need to run an analysis on a Facebook page (of which I'm not the owner but administrator, small size, ~4000 likes and ~150 posts, more or less one per day). What I intended to do was the following:
Obtain the data trough Graph API. Namely, I was most interested in retrieving the messages, number of likes and reach of every post
Import the data in R and identify the outliers (I mean, the posts whose likes and reach are not in line with the mean)
Look for correlations between those messages (since the page, for its nature, needs to talk about a wide range of topics, I want to understand which of them generate the most reactions and plan accordingly)
I've already done an analysis "by hand", but I want to test if it is possible to make the same conclusions without involving a human operator.
I've looked on the web for tutorials on how to use the graph API in python, but I've not been able to find something comprehensive. I've set up my API and obtained the permanent page token with manage_pages and read_insights permissions.
Here an idea of what I'm doing:`
def get_facebook_page_data(page_id, access_token):
website = "https://graph.facebook.com/v3.1/"
location = "%s/posts/" % page_id
fields = "?fields=message,id" + \
"reactions.type(LIKE).limit(0).summary(total_count).as(reactions_like)"
authentication = "&limit=100&access_token=%s" % (access_token)
request_url = website + location + fields + authentication
data = json.loads(request_data_from_url(request_url))
return data`
So, with this function I'm able to obtain the id, message and number of likes of all the posts stored inside data, and with another function I write everything on a csv file.
First question: am I doing something wrong?
Second question: I cannot retrieve a lot of information. For example, when adding type to the fields, it says that this is deprecated (I'm running python 3.7.3)
Third questions: how do I retrieve the reach for every post? I'm assuming this is obtained by scraping the insights, by I don't seem to get it right... How do I query the Graph API for those data?
In general, I'm finding a lot of trouble in just getting the right keywords while building the links. I've installed facebook-sdk but I don't know how to use it (as I said, I'm a beginner). Do you have suggestions on this?
Thanks very much to everyone answering, and greetigs from Italy!
First of all I suggest to use the latest version of the API available, currently the 5.0, regarding your question:
Second question: I cannot retrieve a lot of information. For example,
when adding type to the fields, it says that this is deprecated (I'm
running python 3.7.3)
Regarding to the doc of the Page Feed see the attachments field, as example, adding this to the request:
attachments.fields(media_type)
Third questions: how do I retrieve the reach for every post? I'm
assuming this is obtained by scraping the insights, by I don't seem to
get it right... How do I query the Graph API for those data?
Regarding to the doc of the Page Insights see the page_impressions field, as example in order to return the page_impressions field for a lifetime period:
insights.period(lifetime).metric(post_impressions_unique)
A complete example:
https://graph.facebook.com/v3.1/<PAGE-ID>/posts?fields=message,id,reactions.type(LIKE).limit(0).summary(total_count).as(reactions_like),insights.period(lifetime).metric(post_impressions_unique),attachments.fields(media_type)
Will return:
{
"data": [{
"message": "Hello",
"id": "269816000129666_780829305694997",
"reactions_like": {
"data": [],
"summary": {
"total_count": 0
}
},
"insights": {
"data": [{
"name": "post_impressions_unique",
"period": "lifetime",
"values": [{
"value": 15
}],
"title": "Lifetime Post Total Reach",
"description": "Lifetime: The number of people who had your Page's post enter their screen. Posts include statuses, photos, links, videos and more. (Unique Users)",
"id": "269816000129666_780829305694997/insights/post_impressions_unique/lifetime"
}],
"paging": {
"previous": "https://graph.facebook.com/v3.1/269816000129666_780829305694997/insights?access_token=EAAAAKq6xRNcBAOMKY3StjWXPgL1REATIfPFsyZCY21KDAnZAZB7MpKgNGCHRlKVt9bZBoVZAHpV0jqxZAAVZCOKDIh96YxvpxPaavR1AYK5EQCEEOSMKqz4ZAItcX9WvVfEEN5FzqgyoQWi8oKZBQmQB4Nf80SgicaesluNbI0hDMw2QAxfV9rAFpRc10Pop1d1vtVeziPEjEKwZDZD&metric=post_impressions_unique&period=lifetime&since=1573891200&until=1574064000",
"next": "https://graph.facebook.com/v3.1/269816000129666_780829305694997/insights?access_token=EAAAAKq6xRNcBAOMKY3StjWXPgL1REATIfPFsyZCY21KDAnZAZB7MpKgNGCHRlKVt9bZBoVZAHpV0jqxZAAVZCOKDIh96YxvpxPaavR1AYK5EQCEEOSMKqz4ZAItcX9WvVfEEN5FzqgyoQWi8oKZBQmQB4Nf80SgicaesluNbI0hDMw2QAxfV9rAFpRc10Pop1d1vtVeziPEjEKwZDZD&metric=post_impressions_unique&period=lifetime&since=1574236800&until=1574409600"
}
},
"attachments": {
"data": [{
"media_type": "photo"
}]
}
},
{
"message": "Say hello!",
"id": "269816000129666_780826782361916",
"reactions_like": {
"data": [],
"summary": {
"total_count": 0
}
},
"insights": {
"data": [{
"name": "post_impressions_unique",
"period": "lifetime",
"values": [{
"value": 14
}],
"title": "Lifetime Post Total Reach",
"description": "Lifetime: The number of people who had your Page's post enter their screen. Posts include statuses, photos, links, videos and more. (Unique Users)",
"id": "269816000129666_780826782361916/insights/post_impressions_unique/lifetime"
}],
"paging": {
"previous": "https://graph.facebook.com/v3.1/269816000129666_780826782361916/insights?access_token=EAAAAKq6xRNcBAOMKY3StjWXPgL1REATIfPFsyZCY21KDAnZAZB7MpKgNGCHRlKVt9bZBoVZAHpV0jqxZAAVZCOKDIh96YxvpxPaavR1AYK5EQCEEOSMKqz4ZAItcX9WvVfEEN5FzqgyoQWi8oKZBQmQB4Nf80SgicaesluNbI0hDMw2QAxfV9rAFpRc10Pop1d1vtVeziPEjEKwZDZD&metric=post_impressions_unique&period=lifetime&since=1573891200&until=1574064000",
"next": "https://graph.facebook.com/v3.1/269816000129666_780826782361916/insights?access_token=EAAAAKq6xRNcBAOMKY3StjWXPgL1REATIfPFsyZCY21KDAnZAZB7MpKgNGCHRlKVt9bZBoVZAHpV0jqxZAAVZCOKDIh96YxvpxPaavR1AYK5EQCEEOSMKqz4ZAItcX9WvVfEEN5FzqgyoQWi8oKZBQmQB4Nf80SgicaesluNbI0hDMw2QAxfV9rAFpRc10Pop1d1vtVeziPEjEKwZDZD&metric=post_impressions_unique&period=lifetime&since=1574236800&until=1574409600"
}
},
"attachments": {
"data": [{
"media_type": "photo"
}]
}
},

Categories