We Keep Coding

EDIT: Airflow GCSToS3Operator: keep_directory_structure=True is not being passed in the rendered template, prefix is being added in the dest_s3_key

I am running the task in composer with Composer version: 2.0.18 and Airflow version: 2.2.5
I am sending data to the AWS S3 from Google GCS. For which I am using GCSToS3Operator with the parameters as follow(with example) I have stored AWS credentials in Airflow Connections with connection id "S3-action-outbound"
gcs_to_s3 = GCSToS3Operator(
task_id="gcs_to_s3",
bucket="gcs_outbound",
prefix="legacy/action/20220629",
delimiter=".csv",
dest_aws_conn_id="S3-action-outbound",
dest_s3_key="s3a://action/daily/",
replace=False,
keep_directory_structure=True,
)
But in the end result it's copying the prefix as well. it's writing data at location: s3a://action/daily/legacy/action/20220629/test1.csv
I just want to add the data to the location which I have added s3a://action/daily/test1.csv
according to the documentation if keep_directory_structure= False only then it's suppose to copy the directory path. I tried making it false and it copied the path twice for example it did like this: s3a://action/daily/legacy/action/20220629/legacy/action/20220629/test1.csv
EDIT:
I just realized that there is an issue with the airflow not taking the variable from template. find the attached screenshot of the rendered template
It did not took the variable replace and keep_directory_structure

There is a discussion about it in the PR when keep_directory_structure was added.
https://github.com/apache/airflow/pull/22071/files
it was not implement as in gcs_to_sftp.py

It there any way to get GoogleDriveAPI supported export MIME types list in python or ruby?

Before upload a file to GooogleDrive via GoogleDriveAPI, I want to check whether the file's mime type is supported by GoogleDriveAPI.
https://developers.google.com/drive/api/v3/ref-export-formats
We can do that by using an array which contains all mime types inputted manually, but if there are any smart ways, I want to use that.

I believe your goal as follows.
You want to retrieve the values of export formats of Google Drive.
You want to achieve this using googleapis for ruby and python.
In this case, I would like to use the method of "About: get" in Drive API. The sample scripts for ruby and python are as follows.
Sample script: Ruby
service = Google::Apis::DriveV3::DriveService.new
service.authorization = authorize
res = service.get_about(fields: '*') # or service.get_about(fields: 'exportFormats')
export_formats = res.export_formats
puts export_formats
Sample script: Python
service = build('drive', 'v3', credentials=creds)
res = service.about().get(fields='*').execute() # or service.about().get(fields='exportFormats').execute()
export_formats = res['exportFormats']
print(export_formats)
Note:
This sample scripts use Drive API v3.
In this case, you can also use the value of exportFormats instead of * as the fields.
In this answer, it supposes that you have already been able to get the values from Google Drive using Drive API. Please be careful this.
By the way, for example, when you want to retrieve the values of importFormats, please modify export_formats = res.export_formats and export_formats = res['exportFormats'] to import_formats = res.import_formats and import_formats = res['importFormats'], respectively.
Reference:
About: get

Firebase .get function : get() takes 2 positional arguments but 3 were given

anyone knows whats gg on? I followed this online tutorial and actually used this exact same fucntion a year ago. Did updates happens? not sure why I can't run it anymore
this was from a youtube tutorial:
firebase = firebase.FirebaseApplication('https://capstone-eb408.firebaseio.com/',None)
result=firebase.get("/capstone-eb408/picdetails", '')
print(result)
I also tried this and I got the same error ( this was from my prev ears code which ran fine):
firebase = firebase.FirebaseApplication('https://capstone-eb408.firebaseio.com/',None)
result=firebase.get('picsdetails', 'picname')
print(result)
TypeError: get() takes 2 positional arguments but 3 were given
And when I tried this.... they said they needed a string for an argument? but its a string am I right? am confused ehehe:
firebase = firebase.FirebaseApplication('https://capstone-eb408.firebaseio.com/',None)
result=firebase.get('\capstone-eb408\picdetails')
print(result)
error: request = urllib.request.Request(json_url+'.json?auth='+self.firebaseToken)
TypeError: must be str, not NoneType
Send help hahah :) thanku!!!

FOUND THE ANS!! AND HOPE THE HELPS OTHERS!!!
realize that we need to get credentials and set up that part too now! so here is what to do!
In firebase, go to setting >project settings> service accounts> Firebase admin SDK
you will be able to generate your token! then paste the code you see given to help with authentification into whatever other code you have.
Remember to save the token in a folder and paste the path into the code!!
My code is now this
from firebase import firebase
import firebase_admin
from firebase_admin import credentials
''' Database Set Up '''
cred = credentials.Certificate("/Users/Kee Ga Mun/Documents/academics/Term 8/Capstone/firebasekey.json")
firebase_admin.initialize_app(cred)
firebase = firebase.FirebaseApplication("https://capstone-eb408.firebaseio.com/",None)
data={'Name':'Sembcorp','Number':123543}
result= firebase.post("/capstonert-d9746/Customer",data)
print(result)
And we can replace with other get.() etc functions too but this one post data :)

Provide blob type to read an Azure append blob from PySpark

The ultimate goal is to able to read the data in my Azure container into a PySpark dataframe.
Steps until now
The steps I have followed till now:
Written this code
spark = SparkSession(SparkContext())
spark.conf.set(
"fs.azure.account.key.%s.blob.core.windows.net" % AZURE_ACCOUNT_NAME,
AZURE_ACCOUNT_KEY
)
spark.conf.set(
"fs.wasbs.impl",
"org.apache.hadoop.fs.azure.NativeAzureFileSystem"
)
container_path = "wasbs://%s#%s.blob.core.windows.net" % (
AZURE_CONTAINER_NAME, AZURE_ACCOUNT_NAME
)
blob_folder = "%s/%s" % (container_path, AZURE_BLOB_NAME)
df = spark.read.format("text").load(blob_folder)
print(df.count())
Set public access and anonymous access to my Azure container.
Added two jars hadoop-azure-2.7.3.jar and azure-storage-2.2.0.jar to the path.
Problem
But now I am stuck with this error: Caused by: com.microsoft.azure.storage.StorageException: Incorrect Blob type, please use the correct Blob type to access a blob on the server. Expected BLOCK_BLOB, actual UNSPECIFIED..
I have not been able to find anything which talks about / resolves this issue. The closest I have found is this which does not work / is outdated.
EDIT
I found that the azure-storage-2.2.0.jar did not support APPEND_BLOB. I upgraded to azure-storage-4.0.0.jar and it changed the error from Expected BLOCK_BLOB, actual UNSPECIFIED. to Expected BLOCK_BLOB, actual APPEND_BLOB.. Does anyone know how to pass the correct type to expect?
Can someone please help me with resolving this.
I have minimal expertise in working with Azure but I don't think it should be this difficult to read and create a Spark dataframe from it. What am I doing wrong?

Unable to apply projection query over google datastore. Error : No matching index found

I am new to Google Datastore and following this google documentation.
https://cloud.google.com/datastore/docs/datastore-api-tutorial
I have done authorization for calling Google APIs by reading this doc.
https://developers.google.com/identity/protocols/application-default-credentials#callingpython
I have only two files:
1. client_secret.json
2. datastoreConnection.py
and this is my datastoreConnection.py till now:
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/path/to/client_secret.json'
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
from google.cloud import datastore
client = datastore.Client(project ="xyz",namespace = "staging")
def list_tasks(client):
query = client.query(kind='testData')
query.projection = ['patientName']
return list(query.fetch(limit=10))
print(list_tasks(client))
This code is working fine and returning me the required data.
Problem is arising when I apply Projection with multiple properties.
e.g
query.projection = ['patientName','age']
code is giving me the error:
google.cloud.exceptions.PreconditionFailed: 412 no matching index found. recommended index is:<br/>- kind: testData<br/> properties:<br/> - name: age<br/> - name: patientName<br/>
For making projection query I read this.
https://cloud.google.com/datastore/docs/concepts/queries#datastore-projection-query-python
I have cross checked the property names but still having the same error.
How can I solve this ?
I have seen index.yaml file in other questions related to this kind of problem. Is this necessary to use and what are its benefit?

Any time you need to retrieve more than one property, but not all properties (*), you need to have a matching composite index. In this case you need a composite index on patient name and age properties. You can create a composite index using index.yaml file or gcloud command line tool. The documentation below should help you:
https://cloud.google.com/datastore/docs/concepts/indexes.