I wrote a script that retrieves a date from a textfile, converts that to a datetime and checks if the current time is later than the datetime in the file. I wrote the following code for that:
from datetime import datetime
f = open("token.txt", "r")
expiry_date = f.readline()
f.close()
if datetime.now() >= datetime.strptime(expiry_date, "%Y-%m-%d %H:%M:%S.%f"):
#DO STUFF
However, I get the following error:
ValueError: unconverted data remains:
Anyone knows where I went wrong and how I can fix this?
The line I want to retrieve from the textfile contains a date formatted like this:
2020-05-10 19:29:51.503962
When you call readline(), there is a \n appended to the line. strip the newline first.
Please try:
if datetime.now() >= datetime.strptime(expiry_date.strip(), "%Y-%m-%d %H:%M:%S.%f"):
#DO STUFF
It will work.
Related
I am new to python(Using version 2.7).
I am trying to accept mobile sensor data in Json format via python script. The script is storing data in a text file. The following is a part of my Json data.
{"sensor":"Accelerometer","time":1540532851987,"dataZ":"8.2821044921875"}
{"sensor":"Accelerometer","time":1540532852088,"dataZ":"8.162399291992188"}
{"sensor":"Accelerometer","time":1540532852191,"dataZ":"9.026702880859375"}
I can not understand the meaning of the value "time": . How can I convert it to normal time? I have tried the following
import json
import datetime
with open("acc.json") as data_file:
my_dict = [json.loads(line) for line in data_file]
for acce in my_dict:
date_time_str = str(acce['time'])
date_time_obj = datetime.datetime.strptime(date_time_str, '%Y-%m-%d %H:%M: %S.%f')
print('Date:', date_time_obj.date())
print('Time:', date_time_obj.time())
print('Date-time:', date_time_obj)
##print(datetime.datetime.utcfromtimestamp(acce['time']).strftime('%Y-%m-%d %H:%M:%S'))
Which is giving me error
$python mytest4.py
File "mytest4.py", line 11, in <module>
date_time_obj = datetime.datetime.strptime(date_time_str, '%Y-%m-%d %H:%M:
%S.%f')
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data '1540532851987' does not match format '%Y-%m-%d
%H:%M:%S.%f'
EDIT:
None of the threads mentioned deals with the following together
1. Multiple timestamps received as Json object in UTC format.
2. The format contains 13 digits, unlike the examples discussed with
lesser number of digits.
3. Found a thread based on object_hook but it is too complecated
for me.
Finally I could workout with the partial help of another thread.
I am sharing the final code here in case anyone need it.
import json
import datetime
with open("acc.json") as data_file:
my_dict = [json.loads(line) for line in data_file]
for acce in my_dict:
epoch_time = str(acce['time'])
# get first 10 digits
leading = str(epoch_time)[0: 10]
# get last 3 digits
trailing = str(epoch_time)[-3:]
# print timestamp with milliseconds
print datetime.datetime.fromtimestamp(float(leading)).strftime('%m/%d/%Y -- %H:%M:%S.') + ('%s' % int(trailing))
OUTPUT:
10/26/2018 -- 11:17:31.987
10/26/2018 -- 11:17:32.88
10/26/2018 -- 11:17:32.191
Here is my solution:
import datetime
import json
# {"sensor":"Accelerometer","time":1540532851987,"dataZ":"8.2821044921875"}
# {"sensor":"Accelerometer","time":1540532852088,"dataZ":"8.162399291992188"}
# {"sensor":"Accelerometer","time":1540532852191,"dataZ":"9.026702880859375"}
jsonString = {"sensor":"Accelerometer","time":1540532851987,"dataZ":"8.2821044921875"}
json_data = json.dumps(jsonString)
data = json.loads(json_data)
my_dict = [data]
for item in my_dict:
rawTime = str(item.get('time'))
firstHalf = rawTime[:10]
secondHalf = rawTime[10:]
formatTime = firstHalf + '.' + secondHalf
convertedTime = datetime.datetime.fromtimestamp(float(formatTime)).strftime('%c')
print(convertedTime)
First adding the code:
import csv
from datetime import datetime, date, time
with open('T2.csv') as readcsvfile:
readcsv=csv.reader(readcsvfile)
header=next(readcsv)
data=[]
for row in readcsv:
# if-else construct to read both empty & time string
if row[0]==str():
date=str()
else:date=datetime.strptime(row[0],'%y%m%d')
# stripping the 170101 part with str(ing)p(arse)time and
# changing the style/format into '%Y/%m/%d' format with strftime.
ID=str(row[5])
if row[6]==str():
O_all=str()
else:O_all=datetime.strptime(row[6],'%H:%M').strftime('%H:%M')
Combined_datetime=datetime.combine(date,O_all)
data.append([Combined_datetime,ID])
print(data)
Yields the error:
Combined_datetime=datetime.combine(date,O_all)
TypeError: combine() argument 2 must be datetime.time, not str
But if I check the types, both "date" & "O_all" are 'datetime.datetime' objects. I guess I'm missing something or understood something wrongly. What could be the remedy to get a timetuple named as 'Combined_datetime'?
Update with this code
import csv
from datetime import datetime, date, time
with open('T2.csv') as readcsvfile:
readcsv=csv.reader(readcsvfile)
header=next(readcsv)
data=[]
Combined_datetime =
for row in readcsv:
if len(row[0])<0:
date=str()
else:
date=datetime.strptime(row[0],'%y%m%d')
ID=str(row[5])
if len(row[6])<0:
O_all=str()
else:
O_all=datetime.strptime(row[6],'%H:%M').time()
if date and O_all:
Combined_datetime = datetime.combine(date, O_all).strftime('%y%m%d %H:%M')
data.append([Combined_datetime,ID])
else:
data.append(['',ID])
print(data)
You can explore more pythonic way of writing code. The above code seems very basic and sorry for that.
The problem was elsewhere. While reading my csv, I could read all the row with simple if-else construct (empty time was considered as empty string). But while combining the date & time with datetime.combine(d,t), it could not handle the empty strings.
#amarnath Your suggestion for adding .time() helped though. I removed the rows with empty time this time. Actually it overrides the TypeError: combine() argument 2 must be datetime.time, not str error in this way.
Here is the complete working code:
#Reading T2 & writing new csv
import csv
from datetime import datetime, date, time
with open('T2.csv') as readcsvfile:
readcsv=csv.reader(readcsvfile)
header=next(readcsv)
data=[]
for row in readcsv:
#if-else construct to read both empty & time string
if row[6]!=str():
d=datetime.strptime(row[0],'%y%m%d')
t=datetime.strptime(row[6],'%H:%M').time()
print(type(d))
##stripping the 170101 part with str(ing)p(arse)time and
##changing t from datetime.datetime to datetime.time with `.time()`
print(type(t))
ID=str(row[5])
Combined_datetime = datetime.combine(d, t)
data.append([Combined_datetime,ID])
print(data)
with open('T2_w3.csv','w',newline='') as writecsvfile:
writecsv=csv.writer(writecsvfile)
writecsv.writerow(['Combined_datetime','ID'])
for i in range(len(data)):
ROW=data[i]
CDT=ROW[0]
ID=ROW[1]
Final_list=[CDT,ID]
writecsv.writerow(Final_list)
I have file names in the following format: name_2016_04_16.txt
I'm working with python3 and I would like to extract two things from this file. The prefix, or the name value as a string and the date as a DateTime value for the date represented in the string. For the example above, I would like to extract:
filename: name as a String
date: 04/16/2016 as a DateTime
I will be saving these values to a database, so I would like the DateTime variable to be sql friendly.
Is there a library that can help me do this? Or is there a simple way of going about this?
I tried the following as suggested:
filename = os.path.splitext(filename)[0]
print(filename)
filename.split("_")[1::]
print(filename)
'/'.join(filename.split("_")[1::])
print(filename)
But it outputs:
name_2016_04_16
name_2016_04_16
name_2016_04_16
And does not really extract the name and date.
Thank you!
I would first strip the file extension, then I would split by underscore, removing the 'name' field. Finally, I would join by slash (maybe this value could be logged) and parse the date with the datetime library
import os
from datetime import datetime
file_name = os.path.splitext("name_2016_04_16.txt")[0]
date_string = '/'.join(file_name.split("_")[1::])
parsed_date = datetime.strptime(date_string, "%Y/%m/%d")
To make the date string sql friendly, I found this so post: Inserting a Python datetime.datetime object into MySQL, which suggests that the following should work
sql_friendly_string = parsed_date.strftime('%Y-%m-%d %H:%M:%S')
How about simply doing this?
filename = 'name_2016_04_16.txt'
date = filename[-14:-4] # counting from the end will ensure that you extract the date no matter what the "name" is and how long it is
prefix = filename [:-14]
from datetime import datetime
date = datetime.strptime(date, '%Y_%m_%d') # this turns the string into a datetime object
(However, this works on Python 2.7, if it works for Python 3 you need to find for yourself.)
You can split the filename on "." Then split again on "_". This should give you a list of strings. The first being the name, second through fourth being the year, month and day respectively. Then convert the date to SQL friendly form.
Something like this:
rawname ="name_2016_04_16.txt"
filename = rawname.split(".")[0] #drop the .txt
name = filename.split("_")[0] #provided there's no underscore in the name part of the filename
year = filename.split("_")[1]
month = filename.split("_")[2]
day = filename.split("_")[3]
datestring = (month,day,year) # temp string to store a the tuple in the required order
date = "/".join(datestring) #as shown above
datestring = (year,month,day)
SQL_date = "-".join(datestring ) # SQL date
print name
print date
print SQL_date
Unless you want to use the datetime library to get the datetime date, in which case look up the datetime library
You can then do something like this:
SQL_date = datetime.strptime(date, '%m/%d/%Y')
This is the most explicit way I can think of right now. I'm sure there are shorter ways :)
Apologies for the bad formatting, I'm posting on mobile.
i check many StackOverflow questions. But can't solve this problem...
import pandas as pd
from datetime import datetime
import csv
username = input("enter name: ")
with open('../data/%s_tweets.csv' % (username), 'rU') as f:
reader = csv.reader(f)
your_list = list(reader)
for x in your_list:
date = x[1] # is the date index
dateOb = datetime.strptime(date, '%Y-%m-%d %H:%M:%S')
# i also used "%d-%m-%Y %H:%M:%S" formate
# i also used "%d-%m-%Y %I:%M:%S" formate
# i also used "%d-%m-%Y %I:%M:%S%p" formate
# but the same error shows for every formate
print(dateOb)
i am getting the error
ValueError: time data 'date' does not match format '%d-%m-%Y %I:%M:%S'
in my csv file
ValueError: time data 'date' does not match format '%d-%m-%Y %I:%M:%S'
'date' is not a Date String.
That's why python can not convert this string into DateTime format.
I check in my .csv file, and there i found the 1st line of the date list is not a date string, is a column head.
I remove the first line of my CSV file, and then its works in Python 3.5.1.
But, sill the same problem is occurring in python 2.7
I would like to read in a csv file of dates (shown below) and loop through it using solar.GetAltitude on each date to calculate a list of sun altitudes. (I'm using Python 2.7.2 on Windows 7 Enterprise.)
CSV file: TimeStamp 01/01/2014 00:10 01/01/2014 00:20 01/01/2014 00:30
01/01/2014 00:40
My code gives the following error ValueError: unconverted data remains:. This suggests the wrong date format, but it works fine on a single date, rather than a string of dates.
I've researched this topic carefully on Stack Overflow. I've also tried the map function, np.datetime64 and reading to a list rather than a string but get a different error referring to no attribute 'year'.
I'd really appreciate any help because I'm running out of ideas.
import datetime
from datetime import datetime
import julian
import solar
from solar import *
import os
import csv
# Create lists to hold the records.
dates = []
# Navigate to correct directory
os.chdir('D:\\Di_Python')
filename = 'SPA timestamp small.csv'
# Read through the entire file, skip the first line
with open(filename) as f:
# Create a csv reader object.
reader = csv.reader(f)
# Ignore the header row.
next(reader)
# Store the dates in the appropriate list.
for row in reader:
dates.append(row)
print row
# Change list to string so can use a function on it
lines = []
for date in dates:
lines.append('\t'.join(map(str, date)))
result = '\n'.join(lines)
print result
minutes = []
minutes.append(datetime.datetime.strptime(result,'%d/%m/%Y %H:%M'))
# Inputs
latitude_deg = 52.8
longitude_deg = -1.2
elevation = 0
# i should be 52560 - 10 min interval whole year
for i in minutes:
utc_datetime = i
altitude = solar.GetAltitude(latitude_deg, longitude_deg, utc_datetime)
altitude_list.append(altitude)
print altitude_list
First of all, the code is not indented properly making it harder to guess.
I think the input to datetime.datetime.strptime is not correct. You create result by using a '\n'.join(...) but the format string does not contain the '\n'. Creating a string from the list of dates seems unnecessary to me.
I think what you want is this:
for date in dates:
minutes.append(datetime.datetime.strptime(date, '%d/%m/%Y %H:%M'))
Note that the names you use for the lists are misleading as minutes holds datetime.datetime objects rather than minute values!
Many thanks to Vikramis and Lutz Horn for their help and comments. After experimenting with Vikramis' code, I achieved a working version which I have copied below.
My error occurred at line 40:
minutes.append(datetime.datetime.strptime(result,'%d/%m/%Y %H:%M'))
I found that I needed to create a string from the list to avoid the following error "TypeError: must be string, not list". I have now tidied this up by using (str(date) to replace the for loop and hopefully used more sensible names.
My problem was with the formatting. It needs to be
"['%d/%m/%Y %H:%M']" because I'm accessing items in a list, rather than "'%d/%m/%Y %H:%M'" which works in the shell for a single date.
import datetime
from datetime import datetime
import julian
import solar
from solar import *
import os
import csv
# Create lists to hold the records.
dates = []
datetimeObj = []
altitude_list = []
# Navigate to correct directory
os.chdir('D:\\Di_Python')
filename = 'SPA timestamp small.csv'
# Read through the entire file, skip the first line
with open(filename) as f:
# Create a csv reader object.
reader = csv.reader(f)
# Ignore the header row.
next(reader)
# Store the dates in the appropriate list.
for row in reader:
dates.append(row)
print row
# Change format to datetime
# str(date) used to avoid TypeError: must be string, not list
for date in dates:
datetimeObj.append(datetime.datetime.strptime(str(date),"['%d/%m/%Y %H:%M']"))
for j in datetimeObj:
print j
# Inputs
latitude_deg = 52.8
longitude_deg = -1.2
elevation = 0
# i should be 52560 - 10 min interval whole year
for i in datetimeObj:
utc_datetime = i
altitude = solar.GetAltitude(latitude_deg, longitude_deg, utc_datetime)
print altitude
altitude_list.append(altitude)
# print altitude_list