I'm receiving this error:
Could not parse 'event_date' as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]
from BigQuery when I'm trying to insert a row.
This is my code:
bigquery_client = bigquery.Client.from_service_account_json(CREDENTIALS_BIGQUERY, 'roas-164016')
dataset = bigquery_client.dataset(BQ_LOGS_DATASET_NAME)
table = dataset.table(BQ_EMAIL_SENDS_TABLE_NAME)
data = {}
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
data['send_id'] = 'test'
data['uid'] = 'test'
data['account_id'] = 'test'
data['subaccount_id'] = 'test'
data['event_id'] = 'test'
data['event_date'] = now
data['html_content'] = 'test'
data['campaign_name'] = 'test'
data['subject'] = 'test'
data['send_type'] = 'test'
json_data = json.dumps(data)
data = json.loads(json_data)
table.reload()
rows = [data]
errors = table.insert_data(rows)
How can I fix the date formatting?
If that's literally what you need.
now = datetime.now().strftime("%Y-%m-%d %H:%M[:%S[.%f]]")
More likely, the square brackets indicate optional parts. So:
now = datetime.now().strftime("%Y-%m-%d %H:%M")
or
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
or
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")
format your date into this YYYY-MM-DD HH:MM:SS
meaning the month day hour minutes and second should be in two digits format.
sample:
2018-02-16 04:39:05 Correct
2018-2-16 4:39:5 wrong
Happy coding!
Related
i have this piece of code:
symbol = 'PSTV'
end_point="https://api.polygon.io/v2/snapshot/locale/us/markets/stocks/tickers/"+symbol+"?apiKey=my_key"
a_json=requests.get(end_point).json()
if a_json['status'] == 'OK':
candle_open = a_json['ticker']['min']['o']
candle_close = a_json['ticker']['min']['c']
candle_high = a_json['ticker']['min']['h']
candle_low = a_json['ticker']['min']['l']
candle_ts = a_json['ticker']['lastQuote']['t']
print(candle_ts/1000000)
candle_ts = datetime.fromtimestamp(candle_ts/1000000).strftime('%Y-%m-%d %H:%M:%S')
print("OK")
im trying to convert timestamp to a readable format like so:
candle_ts = a_json['ticker']['lastQuote']['t'] #get the timestamp
print(candle_ts/1000000)
candle_ts = datetime.fromtimestamp(candle_ts/1000000).strftime('%Y-%m-%d %H:%M:%S')
the print is : 1644529277457.4104
I have no clue why but the error is :
candle_ts = datetime.fromtimestamp(candle_ts/1000000).strftime('%Y-%m-%d %H:%M:%S')
OSError: [Errno 22] Invalid argument
Why do I get such an unusual error??
The value for candle_ts is out of range, as you can see below sample script. The max limit is year 5138 which is around 11digits only. Your value for candle_ts is 13digits.
from datetime import datetime
candle_ts = 1644529277457.4104
try:
candle_ts = datetime.fromtimestamp(candle_ts).strftime('%Y-%m-%d %H:%M:%S')
print(candle_ts)
except Exception as e:
print(e)
Result:
year 54083 is out of range
For the node 'TransactionDate' i have a logic before updating it for policy"POL000002NGJ".
The logic i am trying to implement is If existing 'TransactionDate' < than today, then add 5 days with current value and parse it to xml.
Transaction Date Format in XML : 2020-03-23T10:56:15.00
Please Note that, If i parsing the DateTime value like below, It works good But i dont want to hardcode the value... I want to Parse it as a string object to handle for any datetime in format ""%Y-%m-%dT%H:%M:%S.%f""...
# <TransactionDate>
today = datetime.now()
TransactionDate = doc.find('TransactionDate')
Date = '2020-03-24T10:56:15.00'
previous_update = datetime.strptime(Date, "%Y-%m-%dT%H:%M:%S.%f")
if previous_update < today:
today = previous_update - timedelta(days=-5)
TransactionDate = today.strftime("%Y-%m-%dT%H:%M:%S.%f")
Below code while parsing it as a DateTime Object, I have an issue.. I got struck here and referenced other answers in stackoverflow and python forums, But still i got struct up here and unable to resolve the issue...
if any help to fix will be a great helpful. Thanks. Below code using lxml and getting help to support below code will helpful. Because i already completed for other nodes. My understanding is Date variable is calling as None.. But struck here to fix.. Please help..
# <TransactionDate>
today = datetime.now()
TransactionDate = doc.find('TransactionDate')
Date = str(TransactionDate)
previous_update = datetime.strptime(Date, "%Y-%m-%dT%H:%M:%S.%f")
if previous_update < today:
today = previous_update - timedelta(days=-5)
TransactionDate = today.strftime("%Y-%m-%dT%H:%M:%S.%f")
Full Code is Below
from lxml import etree
from datetime import datetime, timedelta
import random, string
doc = etree.parse(r'C:\Users\python.xml')
# <PolicyId> - Random generated policy number
Policy_Random_Choice = 'POL' + ''.join(random.choices(string.digits, k=6)) + 'NGJ'
# <TransactionDate>
today = datetime.now()
TransactionDate = doc.find('TransactionDate')
Date = str(TransactionDate)
previous_update = datetime.strptime(Date, "%Y-%m-%dT%H:%M:%S.%f")
if previous_update < today:
today = previous_update - timedelta(days=-5)
TransactionDate = today.strftime("%Y-%m-%dT%H:%M:%S.%f")
#Parsing the Variables
replacements = [Policy_Random_Choice , TransactionDate ]
targets = doc.xpath('//ROW[PolicyId="POL000002NGJ"]')
for target in targets:
target.xpath('./PolicyId')[0].text = replacements[0]
target.xpath('.//TransactionDate')[0].text = replacements[1]
print(etree.tostring(doc).decode())
Sample XML
<TABLE>
<ROW>
<PolicyId>POL000002NGJ</PolicyId>
<BusinessCoverageCode>COV00002D3X1</BusinessCoverageCode>
<TransactionDate>2020-03-23T10:56:15.00</TransactionDate>
</ROW>
<ROW>
<PolicyId>POL111111NGJ</PolicyId>
<BusinessCoverageCode>COV00002D3X4</BusinessCoverageCode>
<TransactionDate>2020-03-23T10:56:15.00</TransactionDate>
</ROW>
</TABLE>
Maybe the find method is wrong. Try this one
# <TransactionDate>
today = datetime.now()
TransactionDate = doc.xpath('//ROW/TransactionDate') # Change find to xpath
Date = str(TransactionDate[0].text) # Use the first one
previous_update = datetime.strptime(Date, "%Y-%m-%dT%H:%M:%S.%f")
I have an exsting script which is used to extract data for T and T-1 day. I tried to modify it extract two weeks data, but the script is not able to search the dates other than current date
Check the code section :
def parse_gov():
reject_patterns = generate_reject_patterns()
today_str = date.today().strftime('%Y.%m.%d')
yesterday = date.today() - timedelta(days=14)
yesterday_str = yesterday.strftime('%Y.%m.%d')
#query_date = date.today()
#query_date = yesterday
query_last = '''select last sym, last source, last lastTimeStamp, last objectName...
query_all = '''select objectName, IONrecType, sym, source, lastTimeStamp, objectName, ....
def query(q, query_date):
if query_date = date.today() - timedelta(days=14):
date_clause = "date <= {date}, ".format(date = query_date)
kdbport = '1000' ( historical database)
else:
date_clause = ""
kdbport = '1001' (current database)
Your else part is not triggering because the comparison is not happening over there.
you need to change
query_date = date.today() - timedelta(days=14):
to
query_date == (date.today() - timedelta(days=14)):
You always use == for comparison, = will assign the value to the variable.
I have to set the date and time available as a string in the following format.
"%Y/%m/%d %H:%M:%S"
cur_time = strftime("%H:%M:%S", gmtime())
cur_date = DATA[1]
date_time = cur_date+" "+cur_time
now = QtCore.QDate.fromString(date_time, '%Y/%m/%d %H:%M:%S')
self.dateTimeEdit.setDate(now)
But this is not working.
The format of date and datetime is different from the format of QDate and QDateTime, you should not use% in the Qt format, check the docs for more detail:
Assuming DATA[1] has a format %Y/%m/%d as you try to use, you can use the following code:
cur_time = strftime("%H:%M:%S", gmtime())
cur_date = "2018/11/10"
date_time = cur_date+" "+cur_time
now = QtCore.QDateTime.fromString(date_time, 'yyyy/M/d hh:mm:ss')
self.dateTimeEdit.setDateTime(now)
This is my code:
shift_from = datetime.strptime(shift_change_ob.shift_date, '%Y-%m-%d %H:%M:%S')
shift_to = datetime.strptime(shift_change_ob.shift_date_to, '%Y-%m-%d %H:%M:%S')
shift_time_from = self.get_zone_time(shift_from)
shift_time_to= self.get_zone_time(shift_to)
time_from = datetime.strptime(str(shift_time_from),"%Y-%m-%d %H:%M:%S").strftime("%H.%M")
time_to = datetime.strptime(str(shift_time_to),"%Y-%m-%d %H:%M:%S").strftime("%H.%M")
I want to get only time from shift_time_from and shift_time_to. When I do this I get:
ValueError: unconverted data remains: +00:00
How can I solve this?
If shift_from and shift_to are correct, then use the method time for getting only time part:
shift_time_from = shift_from.time()
shift_time_to = shift_to.time()