I'm a beginner at Python and I've been working to geocode a database using Pandas and Geocoder on Jupyter.
Since the df is a little long (around 3000 rows), I'd like to use Google's Geocoding API.
I've already created a free key, but I have no idea what I'm supposed to do with it. Help?
By the way, my code looks like this:
import geocoder
import pandas as pd
geo = geocoder
df=pd.read_excel('hsp2.xlsx')
df['Coordinates']=df['Address'].apply(geo.google).apply(lambda x: x.latlng if x != None else None)
df.to_csv('output.csv', sep='|', encoding='iso8859_15')
You need to set the environment variable before you import geocoder:
import os
os.environ["GOOGLE_API_KEY"] = "api_key_from_google_cloud_platform"
import geocoder
geo = geocoder.google(address)
latlong = geo.latlng
Note:
As Murmel mentioned in the comments, environment variables containing keys (and in general) should not be set inside of your code.
If you are deploying this somewhere then set up enviroment variables in your configuration file. Or even better, as a secret in something like Kubernetes.
Else set the environment variable in bash with
export GOOGLE_API_KEY=api_key_from_google_cloud_platform
Basically there are 2 options:
passing the API KEY as environment variable:
GOOGLE_API_KEY=YOUR-API-KEY-HERE python your_program.py
passing the API KEY as argument:
geocoder.google('some address', key='YOUR-API-KEY-HERE')
Details
You are using the python library called geocoder, which itself is a wrapper around multiple geocoding services.
If you look at the pypi page of geocoder, you can (ignoring the rendering problems) find the docs for geocoder. In your case you probably want to have a look at the Google related part of the docs.
For basic usage this seams to work even without an API KEY, but you can specify one using 2 variants:
Environment variable: Like Roman already showed. This approach is meant to be used to not have the API KEY in code - for security reasons. (Probably you want to upload your code into a public repository, but without exposing your API KEY to everyone.)
"Key" parameter: You can also provide your API KEY by specifying it using the key parameter, like:
geocoder.google('some address', key='YOUR-API-KEY-HERE')
I am agree with Roman answer. You can use that and it is working. I am bit afraid if I use geocoder in loop then google will definately block my ip address ,so I go through git hub code and found that geocoder get google api key from os.environ.get('GOOGLE_API_KEY'). You can see that in the picture:
Related
I am attempting to retrieve and add function/host keys for an Azure Government function app via Python. I am currently working with the information from this question and the corresponding API page. While these are not specific to Azure Government, I would think the process would be similar after updating the URLs to the Azure Government versions. However, I am receiving the error "No route registered for '/api/functions/admin/token'" when running the jwt part of the given code. Is this approach feasible for what I am trying to do?
I also found somewhere that I instead might want to try a GET request like this:
resp = requests.get("https://management.usgovcloudapi.net/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Web/sites/<function-app-name>/functions/admin/masterkey?api-version=20XX-XX-XX", headers={"Authorization": f"Bearer {something}"})
This gives me the error "{"error":{"code":"InvalidAuthenticationToken","message":"The access token is invalid."}}", though. If this is indeed the correct approach, then what format should the Bearer token take?
Bit late answering but it may be useful for someone else in the future, it took me a while to find out how to do this.
If you want to retrieve the keys of a specific function within a function app then you can use list_function_keys() function from the Python SDK
Working with the Az management API directly may be a bit annoying and since the Azure CLI is written in Python whatever operation you do with the CLI you can do it directly in a Python script.
Here's an example of how you can retrieve the keys
from azure.identity import DefaultAzureCredential
from azure.mgmt.web import WebSiteManagementClient
# Your subscription ID
SUB_ID = "00000000-0000-0000-0000-000000000000"
fn_name = "some_function" # Name of your function
app_name = "some_app" # Name of your site/function app
rg_name = "some_rg" # Resource group name to which the function belongs
web_client = WebSiteManagementClient(subscription_id=SUB_ID, credential=DefaultAzureCredential())
keys = web_client.web_apps.list_function_keys(rg_name, app_name, fn_name)
# Your keys will be accessible in the additional_properties param
print(keys.additional_properties)
Hope it helps! I'm new on Azure so if I'm doing something wrong, please don't hesitate to point out my mistake and share your correction
I'd like to create virtual networks in every location in Azure that can support them, using Azure python SDK.
In the code below I'm limiting only to location germanynorth, but that is just to help reproduce the issue.
from azure.common.client_factory import get_client_from_auth_file
from azure.mgmt.compute import ComputeManagementClient
from azure.mgmt.subscription import SubscriptionClient
from azure.mgmt.network import NetworkManagementClient
get_client_from_auth_file(ComputeManagementClient)
for location in get_client_from_auth_file(SubscriptionClient).subscriptions.list_locations(get_subscription_id()):
if location.name == 'germanynorth':
get_client_from_auth_file(NetworkManagementClient).virtual_networks.create_or_update(
resource_group_name=RESOURCE_GROUP_NAME,
virtual_network_name='test-network',
parameters={'location': location.name, 'address_space': {'address_prefixes': ['10.0.0.0/16']}, }
)
When running this I get the error:
msrestazure.azure_exceptions.CloudError: Azure Error: LocationNotAvailableForResourceType
Message: The provided location 'germanynorth' is not available for resource type 'Microsoft.Network/virtualNetworks'. List of available regions for the resource type is 'westus,eastus,northeurope,westeurope,eastasia,southeastasia,northcentralus,southcentralus,centralus,eastus2,japaneast,japanwest,brazilsouth,australiaeast,australiasoutheast,centralindia,southindia,westindia,canadacentral,canadaeast,westcentralus,westus2,ukwest,uksouth,koreacentral,koreasouth,francecentral,australiacentral,southafricanorth,uaenorth,switzerlandnorth,germanywestcentral,norwayeast'.
Very helpfully, the error includes a list of all the regions where virtualNetworks could be created, but of course this list will change over time.
What API in Azure can I use to figure out what locations (regions?) support virtual networks?
Thanks!
You can use Azure resource providers and types. You can refer the Microsoft article on the resource provider.
PowerShell script to get all supported azure regions to create Azure key Vault.
$locations = (((Get-AzResourceProvider -ProviderNamespace Microsoft.KeyVault)| Where-Object RegistrationState -eq "Registered").ResourceTypes | Where-Object ResourceTypeName -eq vaults).Locations
I actually figured out my own bounty.
https://learn.microsoft.com/en-us/rest/api/resources/providers/get
"Gets the specified resource provider."
I don't think this describes what it actually does, which is why I didn't find it. I had to just basically test a bunch of APIs to see what returned what.
This API will return a list of available locations for the provided resource type (in your subscription).
I just wish it didn't only return a list (East US) but also with the with a short code (code:location), for example (eastus:East US).
So to answer the actual question, if you can't use your python library for this, an option would be to use this REST API:
GET https://management.azure.com/subscriptions/{subscriptionId}/providers/Microsoft.Network?api-version=2021-04-01
Your list of locations will be under the json path: {response}.resourceTypes[0].locations, where resourceType eq "virtualNetworks"
To actually get the locationCode (short location code) you can query this API:
https://learn.microsoft.com/en-us/rest/api/resources/subscriptions/list-locations
then map your location from above with this response to get the short code ('East US' -> 'eastus'), which can be used in other rest APIs to create for example a virtual network.
There is an API with which we can list out all the available location under a given subscription id (I am not sure if there is an equivalent version of this API that can filter by resource type) -
API -
GET https://management.azure.com/subscriptions/{subscriptionId}/locations?api-version=2020-01-01
Quoting the documentation
This operation provides all the locations that are available for resource providers; however, each resource provider may support a subset of this list.
Perhaps, you can iterate through the list of available locations in your subscription and put it in a try/except block to create Vnet against all available regions in your subscription?
EDIT: Apologies, I realized it late, you are already iterating through the list of locations under your subscription id. I guess it's a matter of writing the code beneath the for loop in a try/except block, wherein you can except that particular error type and continue with your loop?
I'm attempting to programmatically register Service Principals in Azure AD using Python. It looks like I should be using the ServicePrincipalsOperations class, however the documentation does not clearly outline what is needed for the parameters. It appears that it is expecting data from other classes within azure.graphrbac, but the documentation is unclear.
I think I should be using the azure.graphrbac.GraphRbacManagementClient to generate the client parameter for ServicePrincipalsOperations, but that's just a guess at this point.
Similarly, I suspect that I would need to use azure.graphrbac.models.ServicePrincipalCreateParameters for the config parameter.
Has anyone successfully registered a Service Principal using Python that may be able to shed more light on these parameters?
Thanks in advance!
So you can use this test as a reference, but the documents do specify what you need to pass in to the method to create a service principal.
Sample code:
self.graphrbac_client.service_principals.create({
'app_id': app.app_id, # Do NOT use app.object_id
'account_enabled': False
})
More reading: Create service principal programmatically in Azure Python API
I am a newbie who is exploring Google BigQuery. I would like to insert a row into the BigQuery table from a python list which contains the row values.
To be more specific my list looks like this: [100.453, 108.75, 50.7773]
I found a couple of hints from BigQuery-Python library insert
and also looked in to pandas bigquery writer but not sure whether they are perfect for my usecase.
What would be the better solution?
Thanks in advance.
Lot's of resources but I usually find code examples to be the most informative for beginning.
Here's an excellent collection of bigquery python code samples: https://github.com/googleapis/python-bigquery/tree/master/samples.
One straight forward way to insert rows:
from google.cloud import bigquery
bq_client = bigquery.Client()
table = bq_client.get_table("{}.{}.{}".format(PROJECT, DATASET, TABLE))
rows_to_insert = [{u"COL1": 100.453, u"COL2": 108.75, u"COL3": 50.7773}, {u"COL1": 200.348, u"COL2": 208.29, u"COL3": 60.7773}]
errors = bq_client.insert_rows_json(table, rows_to_insert)
if errors == []:
print("success")
Lastly to verify if it's inserted successfully use:
bq query --nouse_legacy_sql 'SELECT * FROM `PROJECT.DATASET.TABLE`'
Hope that helps everyone!
To work with Google Cloud Platform services using Python, I would recommend using python google-cloud and for BigQuery specifically the submodule google-cloud-bigquery(this was also recommended by #polleyg. This is an open-source Python idiomatic client maintained by the Google. This will allow you to easily use all the google cloud services in a simple and consistent way.
More specifically, the example under Insert rows into a table’s data in the documentation shows how to insert Python tuples/lists into a BigQuery table.
However depending on your needs, you might need other options, my ordering of options:
If the you use code that has a native interface with Google Services (e.g. BigQuery) and this suits your needs, use this. In your case test if Pandas-BigQuery works for you.
If your current code/modules don't have a native interface, try the Google maintained idiomatic client google-cloud.
If that doesn't suit your needs, use an external idiomatic client like tylertreat/BigQuery-Python. The problem is that you will have different inconsistent clients for the different services. The benefit can be that it adds some functionalities not provided in the google-cloud module.
Finally, if you work with very new alpha/beta features, use the APIs directly with the Google API module, this is will always give you access to the latest APIs, but is a bit harder to work with. So only use this if the previous options don't give you what you need.
The Google BigQuery docs show you how:
https://cloud.google.com/bigquery/streaming-data-into-bigquery#bigquery_stream_data_python
https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/bigquery/cloud-client/stream_data.py
I'm using Azure and the python SDK.
I'm using Azure's table service API for DB interaction.
I've created a table which contains data in unicode (hebrew for example). Creating tables and setting the data in unicode seems to work fine. I'm able to view the data in the database using Azure Storage Explorer and the data is correct.
The problem is when retrieving the data.. Whenever I retrieve specific row, data retrieval works fine for unicoded data:
table_service.get_entity("some_table", "partition_key", "row_key")
However, when trying to get a number of records using a filter, an encode exception is thrown for any row that has non-ascii chars in it:
tasks = table_service.query_entities('some_table', "PartitionKey eq 'partition_key'")
Is this a bug on the azure python SDK? Is there a way to set the encoding beforehand so that it won't crash? (azure doesn't give access to sys.setdefaultencoding and using DEFAULT_CHARSET on settings.py doesn't work as well)
I'm using https://www.windowsazure.com/en-us/develop/python/how-to-guides/table-service/ as reference to the table service API
Any idea would be greatly appreciated.
This looks like a bug in the Python library to me. I whipped up a quick fix and submitted a pull request on GitHub: https://github.com/WindowsAzure/azure-sdk-for-python/pull/59.
As a workaround for now, feel free to clone my repo (remembering to checkout the dev branch) and install it via pip install <path-to-repo>/src.
Caveat: I haven't tested my fix very thoroughly, so you may want to wait for the Microsoft folks to take a look at it.