I have a requirement of getting yesterdays date in my GCP Cloud Function which is written Python 3.9
This is the snippet I am using
from datetime import datetime, timedelta
cur_dt = datetime.today()
str_dt = str((datetime.today() - timedelta(1)).strftime('%Y-%m-%d'))
print(cur_dt)
print(str_dt)
This is working fine in Jupyter notebook. But if I place this same code in my cloud function, it fails to compile.
This is the error message I am getting: - Function failed on loading user code. This is likely due to a bug in the user code.
Would be of great help if anyone can help me fixing this error. This is strange and I dont understand why CLoud function isnt accepting something that is working fine in Jupyter notebook
Many Thanks in Advance.
As mentioned you need to follow the Cloud Functions schema which requires an entrypoint function that takes in the request parameter. Here is a codelab that walks through setting up a Cloud Function.
Your code should be updated to the following:
# imports
from datetime import datetime, timedelta
# entrypoint function with request param
def get_time(request):
cur_dt = datetime.today()
str_dt = str((datetime.today() - timedelta(1)).strftime('%Y-%m-%d'))
# can still print here if you want
return str_dt
If you are going through the UI make sure to update the entrypoint field to be get_time or whatever you name your entrypoint function.
Related
I have this Flask/Dash application that I deployed as a service running in the background and it works fine. I use a datepicker (calendar) in the application to choose the date for which the data will be fetched and processed. However, the date of today is being grayed (deactivated) every day until I restart the service. I am using this :
from dash import dcc
dcc.DatePickerSingle(id='previ_date',
min_date_allowed=datetime.date(2022, 5, 10),
max_date_allowed=datetime.date.today(),
initial_visible_month=datetime.date.today(),
date=datetime.date.today())
Normally, the max_date_allowed argument is set to today's date, however, it doesn't behave as expected. Any help on how to overcome this issue is appreciated.
Eventually, I solved the issue as recommended by #coralvanda by setting the initial value as None, then doing the checking and updating inside a callback function.
dcc.DatePickerSingle(id='previ_date',
min_date_allowed=datetime.date(2022, 5, 10),
max_date_allowed=None,
initial_visible_month=None,
date=datetime.date.today())
#app.callback([Output("previ_date", "max_date_allowed"), Output("previ_date", "initial_visible_month")],
[Input("previ_date", "max_date_allowed"), Input("previ_date", "initial_visible_month")])
def update_date(max_date, current_month):
if max_date==None:
max_date=datetime.date.today()
if current_month==None:
current_month=datetime.date.today()
return max_date, current_month
I´ve made a simple pipeline in Python to read from kafka, the thing is that the kafka cluster is on confluent cloud and I am having some trouble conecting to it.
Im getting the following log on the dataflow job:
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:820)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:631)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:612)
at org.apache.beam.sdk.io.kafka.KafkaIO$Read$GenerateKafkaSourceDescriptor.processElement(KafkaIO.java:1495)
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
So I think Im missing something while passing the config since it mentions something related to it, Im really new to all of this and I know nothing about java so I dont know how to proceed even reading the JAAS documentation.
The code of the pipeline is the following:
import apache_beam as beam
from apache_beam.io.kafka import ReadFromKafka
from apache_beam.options.pipeline_options import PipelineOptions
import os
import json
import logging
os.environ['GOOGLE_APPLICATION_CREDENTIALS']='credentialsOld.json'
with open('cluster.configuration.json') as cluster:
data=json.load(cluster)
cluster.close()
def logger(element):
logging.INFO('Something was found')
def main():
config={
"bootstrap.servers":data["bootstrap.servers"],
"security.protocol":data["security.protocol"],
"sasl.mechanisms":data["sasl.mechanisms"],
"sasl.username":data["sasl.username"],
"sasl.password":data["sasl.password"],
"session.timeout.ms":data["session.timeout.ms"],
"auto.offset.reset":"earliest"
}
print('======================================================')
beam_options = PipelineOptions(runner='DataflowRunner',project='project',experiments=['use_runner_v2'],streaming=True,save_main_session=True,job_name='kafka-stream-test')
with beam.Pipeline(options=beam_options) as p:
msgs = p | 'ReadKafka' >> ReadFromKafka(consumer_config=config,topics=['users'],expansion_service="localhost:8088")
msgs | beam.FlatMap(logger)
if __name__ == '__main__':
main()
I read something about passing a property java.security.auth.login.config in the config dictionary but since that example is with java and I´am using python Im really lost at what I have to pass or even if that´s the property I have to pass etc.
btw Im getting the api key and secret from here and this is what I am passing to sasl.username and sasl.password
I faced the same error the first time I tried the beam's expansion service. The key sasl.mechanisms that you are supplying is incorrect, try with sasl.mechanism also you do not need to supply the username and password since you are connection is authenticated by jasl basically the consumer_config like below worked for me:
config={
"bootstrap.servers":data["bootstrap.servers"],
"security.protocol":data["security.protocol"],
"sasl.mechanism":data["sasl.mechanisms"],
"session.timeout.ms":data["session.timeout.ms"],
"group.id":"tto",
"sasl.jaas.config":f'org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="Kafka" username=\"{data["sasl.username"]}\" password=\"{data["sasl.password"]}\";',
"auto.offset.reset":"earliest"
}
I got a partial answer to this question since I fixed this problem but got into another one:
config={
"bootstrap.servers":data["bootstrap.servers"],
"security.protocol":data["security.protocol"],
"sasl.mechanisms":data["sasl.mechanisms"],
"sasl.username":data["sasl.username"],
"sasl.password":data["sasl.password"],
"session.timeout.ms":data["session.timeout.ms"],
"group.id":"tto",
"sasl.jaas.config":f'org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="Kafka" username=\"{data["sasl.username"]}\" password=\"{data["sasl.password"]}\";',
"auto.offset.reset":"earliest"
}
I needed to provide the sasl.jaas.config porpertie with the api key and secret of my cluster and also the service name, however, now Im facing a different error whe running the pipeline on dataflow:
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
This error shows after 4-5 mins of trying to run the job on dataflow, actually I have no idea how to fix this but I think is related to my broker on confluent rejecting the connection, I think this could be related to the zone execution since the cluster is in a different zone than job region.
UPDATE:
I tested the code on linux/ubuntu and I dont know why but the expansión service gets downloaded automatically so you wont get unsoported signal error, still having some issues trying to autenticate to confluent kafka tho.
In prefect workflow, I'm trying to persist data of every schedule run. I need to compare data of every previous and current result. I tried Localresult and checkpoint=true but its not working. For example,
from prefect import Flow, task
from prefect.engine.results import LocalResult
from prefect.schedules import IntervalSchedule
from datetime import timedelta, datetime
import os
import prefect
#task("func_task_target.txt", checkpoint=True, result=LocalResult(dir="~/.prefect"))
def file_scan():
files = os.listdir(test)
#prefect.context.a = files
return files
schedule = IntervalSchedule(interval=timedelta(seconds=61))
with Flow("Test persist data", schedule) as flow:
a = file_scan()
flow.run()
My flow scheduled for every 61 seconds/a minute. On the first run I might get empty result but for the 2nd scheduled run I should get previous flow result to compare. can anyone help me to achieve this? Thanks!
Update (15 November 2021) :
Not sure what is the reason,
LocalResult and checkpoint actually worked when I ran the registered flow through the dashboard or cli prefect run -n "your-workflow.py" --watch. It doesn't work when I manually trigger the flow (e.g.: flow.run) in python code.
Try these following two options:
Option 1 : using target argument:
https://docs.prefect.io/core/concepts/persistence.html#output-caching-based-on-a-file-target
#task(target="func_task_target.txt", checkpoint=True, result=LocalResult(dir="~/.prefect"))
def func_task():
return "999"
Option 2 : instantiate LocalResult instance and invoke write manually.
MY_RESULTS = LocalResult(dir="./.prefect").
#task(checkpoint=True, result=LocalResult(dir="./.prefect"))
def func_task():
MY_RESULTS.write("999")
return "999"
PS:
Having same problem as LocalResult doesn't seem to work for mewhen used in decorator . e.g :
#task("func_task_target.txt", checkpoint=True, result=LocalResult(dir="~/.prefect"))
def file_scan():
I am trying to read my mail and see the received time in outlook 2016 using MAPI.
I am able to see the subject of the mail not able to see the receivedTime of the mail. I know that "Receivedtime" is there to get the received time of the mail, but while the program is executed,
a popup is coming, telling that python has stopped working
I know it is not due to any machine problem rather some problem in my code.
Here is my code.
def arrange(mailbox):
global spam
timeperiod() # stores required date in spam[] list
msgs=mailbox.Items
msgs.Sort("[ReceivedTime]", True)
p=msgs.restrict(" [ReceivedTime] >= '"+spam[2]+"'") #and [ReceivedTime] >= '" +spam[1]+"'
print(len(p))
'''
for m in list1:
if m.Unread:
m.Unread=False
'''
return p
#Calling it
ctm1=arrange(ctm)
print(len(ctm1)) #Working fine
for message in ctm1:
print (message.subject) #Also works good
print (message.receivedTime) # Here is the problem, it's not showing
]1
i have tried Senton property as well, but it's not working . So any guesses why the senton or receivedTime properties are not working???
updated code :
def printlist(box1) :
print(len(box1))
for message in box1:
if message.Class==43 :
# print('true')
print (message)
#.senderEmailAddress) #working
#print(message.SentOn.strftime("%d-%m-%y")) #not working
#print(message.body)
#print(message.UnRead)
#print (message.receivedTime) #not working
#print('-----------')
I was also having trouble with the .ReceivedTime breaking the program when I compile the .py script to an .exe using auto-py-to-exe.
Here's where the error occurs underneath this try: statement
try:
received_year = str(email.ReceivedTime)[0:4]
received_month = str(email.ReceivedTime)[5:7]
received_day = str(email.ReceivedTime)[8:10]
This works PERFECTLY well inside of my IDE (PyCharm), but once I've compiled it to .exe, it breaks here.
I've updated pywin32 to the most up-to-date version (228) and also tried using 224 to see if it was a version issue (it wasn't).
BUT!! Going through this, I FOUND THE BUG! When you compile to .exe with auto-py-to-exe, it does not include the package "win32timezone" which the .ReceivedTime portion needs to run correctly. So you need to import this package for it to work.
All you have to do to fix this is include this at the top of your .py script before compiling to .exe:
import win32timezone
Let me know if this works for anyone else facing this issue!
Most likely you run into an item other than MailItem - you can also have ReportItem and MeetingItem objects in your Inbox; none of them expose the ReceivedTime property.
Check that message.Class property == 43 (olMail) before accessing any other MailItem specific properties.
Please try the below:
print([x.ReceivedTime for x in messages])
I have a python app running in a server (using flask) and i am having problems with the current date.
I have a method that calculates current date:
import time
def calcCurrent():
return(time.strftime("%d/%m/%Y"))
It works fine during this day, but, if I try to access to the server the day after and I call this method, it still returns the previous day date. It is, it doesn't recalculate current date.
Any idea on how could I do this?
try using this
import datetime
def calcCurrent():
return(datetime.datetime.now().strftime("%d/%m/%Y"))