We Keep Coding

BigQuery.jobs.create pemission

I am trying to run a python code to create feature store. When I am running I am getting Bigquery.jobs.create permission error. I checked the permissions for my account with gcloud iam roles describe roles/viewer and Bigquery permissions are there.
Now, what mistake I am making and how can I solve this error.

It seems that you need to create BigQuery job. At least the account you are using should have "BigQuery Job User" role.

All the required permissions and roles are mentioned here.
roles/bigquery.user
roles/bigquery.jobUser
roles/bigquery.admin
Click here to go to official documentation page mentioning the above for revised roles/permissions.

How to check whether a user exists when setting IAM policy in google Big-query?

I'm trying to set IAM policy in google Bigquery using Python. I'm doing it in the following way:
Get the current policy
Create a list of new members e.g. ['someone#gmail.com', 'someother#gmail.com', ...]
Modify the policy by adding the members from the above-created list
Set policy by calling setIamPolicy method.
The problem with the above approach is that it throws an error HttpError 400 User ... doesn't exist when anyone from the members list doesn't exist.
How can I avoid this issue? Is there any API to check whether a user exists or not?
I can also do it using a loop so that if one setIamPolicy call fails(because of user doesn't exits), other memberes still get added but that ends up calling the API multiple times.

If you have a mapping from emails to users, you can add the following simple python code using boto3 to check if the user exists in aws or not.
import boto3
iam = boto3.resource('iam')
user = iam.User('name')
user.load()
This will throw a NoSuchEntityException exception if the user doesn't exist.
You can catch the exception and continue if the user doesn't exist. If successful loaded you'll be able to access the users attributes, e.g. user.arn.

How to delete a particular resource from resource group in Azure using python

I am using azure.mgmt.resource for deleting a resource group from azure but can i delete a particular resource without deleting a resource group?

If you can't use the specific client of the resource type as explained in another answer, you can use one of the two generic delete in the azure-mgmt-resource package and the ResourceManagementClient:
delete_by_id
delete
Note that the ApiVersion asked is the one from the resource you want to delete. Please look at the RestAPI documentation to get an ApiVersion depending of the resource type you want to delete. For instance, you can get the ApiVersion of Storage in that page:
https://learn.microsoft.com/rest/api/storagerp/

For example, If you are deleting a VM in a resource group
Delete VM
print('\nDelete VM')
async_vm_delete = compute_client.virtual_machines.delete(GROUP_NAME, VM_NAME)
async_vm_delete.wait()
Azure-Samples/resource-manager-python-resources-and-groups
Manage Azure resources and resource groups with Python:
Additional information: Click here

Azure Batch Pool: How do I use a custom VM Image via Python?

I want to create my Pool using Python. I can do this when using an image (Ubuntu Server 16.04) from the marketplace, but I want to use a custom image (but also Ubuntu Server 16.04) -- one which I have prepared with the desired libraries and setup.
This is how I am creating my pool:
new_pool = batch.models.PoolAddParameter(
id=pool_id,
virtual_machine_configuration=batchmodels.VirtualMachineConfiguration(
image_reference=image_ref_to_use, # ??
node_agent_sku_id=sku_to_use),
vm_size=_POOL_VM_SIZE,
target_dedicated_nodes=_POOL_NODE_COUNT,
start_task=start_task,
max_tasks_per_node=_CORES_PER_NODE
)
I imaging that I need to use batch.models.ImageReference() to create my image reference... but I do not know how to use it.
Yes, I checked the documentation, which says the following:
A reference to an Azure Virtual Machines Marketplace image or a custom
Azure Virtual Machine image.
It lists the parameters as:
publisher (str)
offer (str)
sku (str)
version (str)
virtual_machine_image_id (str)
However, the parameter virtual_machine_image_id does not exists... In other words, batch.models.ImageReference(virtual_machine_image_id) is not allowed.
How can I use a custom image for my Pool?
UPDATE
So I figured out how to use a custom image... it turns out that no matter how many times I uninstall the azure python libraries and re-install them, the virtual_machine_image_id is never available.
I then went here downloaded the zip. Opened it up, checked the ImageReference class and low-and-behold, the virtual_machine_image_id was available in the __init__ function of the ImageReference class. I then downloaded the python wheel and used pip to install it. Boom it worked.
Or so I thought.
I then had to fight though trying to figure out what the node_agent_sku_id is... only by manually creating a Pool and seeing the Batch Node Agent SKU ID field did I manage to find it.
Now I am struggling with the Authentication...
The error I am getting is:
Server failed to authenticate the request. Make sure the value of
Authorization header is formed correctly including the signature.
AuthenticationErrorDetail: The specified type of authentication
SharedKey is not allowed when external resources of type Compute are
linked.
azure.batch.models.batch_error.BatchErrorException: {'lang':
'en-US', 'value': 'Server failed to authenticate the request. Make
sure the value of Authorization header is formed correctly including
the
signature.\nRequestId:f8c1a3b3-65c4-4efd-9c4f-75c5c253f992\nTime:2017-10-15T20:36:06.7898187Z'}
From the error, I understand that I am not allowed to use SharedKeyCredentials:
credentials = batchauth.SharedKeyCredentials(_BATCH_ACCOUNT_NAME,
_BATCH_ACCOUNT_KEY)
batch_client = batch.BatchServiceClient(
credentials,
base_url=_BATCH_ACCOUNT_URL)
What must I do?
UPDATE 2
OK. User fpark has informed me that I need to use:
from azure.batch import BatchServiceClient
from azure.common.credentials import ServicePrincipalCredentials
credentials = ServicePrincipalCredentials(
client_id=CLIENT_ID,
secret=SECRET,
tenant=TENANT_ID,
resource="https://batch.core.windows.net/"
)
batch_client = BatchServiceClient(
credentials,
base_url=BATCH_ACCOUNT_URL
)
to authenticate. Unfortunately, that the code above is described here and makes no reference to what CLIENT_ID et. al are.
I then managed to find another piece of documentation which appears to be the same thing: https://azure-sdk-for-python.readthedocs.io/en/v2.0.0rc3/resourcemanagementauthentication.html
That page pointed me to another webpage: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal
I followed that tutorial and managed to finally authenticate my application...
NOTE
When creating your application, the tutorial will tell you:
Provide a name and URL for the application. Select either Web app /
API or Native for the type of application you want to create. After
setting the values, select Create.
DO NOT select Native as you will not have the option to get an application key...

Required Minimum Azure Batch SDK
The azure-batch Python SDK v4.0.0 or higher is required. Typically with pip install --upgrade azure-batch you should just get the newest version. If that doesn't work you can add the --force-reinstall option to pip to force it (with --upgrade).
Node Agent Sku Id
Regarding the proper value for node_agent_sku_id, you need to use the list_node_agent_skus operation to see the mapping between operating systems and the node agent skus supported.
Azure Active Directory Authentication Required
Regarding the auth issue, you must use Azure Active Directory authentication to use this feature. It will not work with shared key auth.
Documentation
More information can be found in this guide, including all pre-requisites needed to enable custom images.

I am using azure-batch==9.0.0, and it turns out the docs are not updated as per the package itself. Using id instead of virtual_machine_image_id fixes the problem for me.

add worklog to JIRA (with user and activity metadata) using the API

I am using jira-python and request API to log work to JIRA tickets with the user and activity info but can't figure out a way to do it . I have the following code :
jira.add_worklog("issue number", timeSpent="2h", user="username")
but it seems to ignore the user keyword. I am also looking up JIRA API with request library but can't seem to find an API to log work with activity and user info.

It is not needed to add inside the arguments the user. you can use the following code:
jira.add_worklog("issue number", timeSpent="2h")
Yo can add other arguments like:
adjustEstimate – (optional) allows the user to provide specific instructions to update the remaining time estimate of the issue. The value can either be new, leave, manual or auto (default).
newEstimate – the new value for the remaining estimate field. e.g. “2d”
reduceBy – the amount to reduce the remaining estimate by e.g. “2d”
started – Moment when the work is logged, if not specified will default to now
comment – optional worklog comment
You can find more info about it in jira python api

There's no direct support for this action, see this discussed in
https://answers.atlassian.com/questions/29951977
and
https://jira.atlassian.com/browse/JRA-30197
It's possible to impersonate another user, either by writing an add-on with the ACT_AS_USER scope or by registering your application with Oauth2 authentication in your Jira configuration, for example as discussed here:
https://answers.atlassian.com/questions/247528/how-do-you-impersonate-a-user-with-jira-oauth
You'd probably be better off discussing implantation suggestions on answers.atlassian.com.