We Keep Coding

C++ vs. python: daylight saving time not recognized when extracting Unix time?

I'm attempting to calculate the Unix time of a given date and time represented by two integers, e.g.
testdate1 = 20060711 (July 11th, 2006)
testdate2 = 4 (00:00:04, 4 seconds after midnight)
in a timezone other than my local timezone. To calculate the Unix time, I feed testdate1, testdate2 into a function I adapted from Convert date to unix time stamp in c++
int unixtime (int testdate1, int testdate2) {
time_t rawtime;
struct tm * timeinfo;
//time1, ..., time6 are external functions that extract the
//year, month, day, hour, minute, seconds digits from testdate1, testdate2
int year=time1(testdate1);
int month=time2(testdate1);
int day=time3(testdate1);
int hour=time4(testdate2);
int minute=time5(testdate2);
int second=time6(testdate2);
time ( &rawtime );
timeinfo = localtime ( &rawtime );
timeinfo->tm_year = year - 1900;
timeinfo->tm_mon = month - 1;
timeinfo->tm_mday = day;
timeinfo->tm_hour = hour;
timeinfo->tm_min = minute;
timeinfo->tm_sec = second;
int date;
date = mktime(timeinfo);
return date;
}
Which I call from the main code
using namespace std;
int main(int argc, char* argv[])
{
int testdate1 = 20060711;
int testdate2 = 4;
//switch to CET time zone
setenv("TZ","Europe/Berlin", 1);
tzset();
cout << testdate1 << "\t" << testdate2 << "\t" << unixtime(testdate1,testdate2) << "\n";
return 0;
}
With the given example, I get unixtime(testdate1,testdate2) = 1152572404, which according to
https://www.epochconverter.com/timezones?q=1152572404&tz=Europe%2FBerlin
is 1:00:04 am CEST, but I want this to be 0:00:04 CEST.
The code seems to work perfectly well if I choose a testdate1, testdate2 in which daylight saving time (DST) isn't being observed. For example, simply setting the month to February with all else unchanged is accomplished by setting testdate1 = 20060211. This gives
unixtime(testdate1,testdate2) = 1139612404, corresponding to hh:mm:ss = 00:00:04 in CET, as desired.
My impression is that setenv("TZ","Europe/Berlin", 1) is supposed to account for DST when applicable, but perhaps I am mistaken. Can TZ interpret testdate1, testdate2 in such a way that it accounts for DST?
Interestingly, I have a python code that performs the same task by changing the local time via os.environ['TZ'] = 'Europe/Berlin'. Here I have no issues, as it seems to calculate the correct Unix time regardless of DST/non-DST.

localtime sets timeinfo->tm_isdst to that of the current time - not of the date you parse.
Don't call localtime. Set timeinfo->tm_isdst to -1:
The value specified in the tm_isdst field informs mktime() whether or not daylight saving time (DST) is in effect for the time supplied in the tm structure: a positive value means DST is in effect; zero means that DST is not in effect; and a negative value means that mktime() should (use timezone information and system databases to) attempt to determine whether DST is in effect at the specified time.
See the code example in https://en.cppreference.com/w/cpp/chrono/c/mktime

Maxim's answer is correct, and I've upvoted it. But I also thought it might be helpful to show how this can be done in C++20 using the newer <chrono> tools. This isn't implemented everywhere yet, but it is here in Visual Studio and will be coming elsewhere soon.
There's two main points I'd like to illustrate here:
<chrono> is convenient for conversions like this, even if both the input and the output does not involve std::chrono types. One can convert the integral input to chrono, do the conversion, and then convert the chrono result back to integral.
There's a thread safety weakness in using the TZ environment variable, as this is a type of global. If another thread is also doing some type of time computation, it may not get the correct answer if the computer's time zone unexpectedly changes out from under it. The <chrono> solution is thread-safe. It doesn't involve globals or environment variables.
The first job is to unpack the integral data. Here I show how to do this, and convert it into chrono types in one step:
std::chrono::year_month_day
get_ymd(int ymd)
{
using namespace std::chrono;
day d(ymd % 100);
ymd /= 100;
month m(ymd % 100);
ymd /= 100;
year y{ymd};
return y/m/d;
}
get_ymd takes "testdate1", extracts the individual integral fields for day, month, and year, then converts each integral field into the std::chrono types day, month and year, and finally combines these three separate fields into a std::chrono::year_month_day to return it as one value. This return type is simple a {year, month, day} data structure -- like a tuple but with calendrical meaning.
The / syntax is simply a convenient factory function for constructing a year_month_day. And this construction can be done with any of these three orderings: y/m/d, d/m/y and m/d/y. This syntax, when combined with auto, also means that you often don't have to spell out the verbose name year_month_day:
auto
get_ymd(int ymd)
{
// ...
return y/m/d;
}
get_hms unpacks the hour, minute and second fields and returns that as a std::chrono::seconds:
std::chrono::seconds
get_hms(int hms)
{
using namespace std::chrono;
seconds s{hms % 100};
hms /= 100;
minutes m{hms % 100};
hms /= 100;
hours h{hms};
return h + m + s;
}
The code is very similar to that for get_ymd except that the return is the sum of the hours, minutes and seconds. The chrono library does the job for you of converting hours and minutes to seconds while performing the summation.
Next is the function for doing the conversion, and returning the result back as an int.
int
unixtime(int testdate1, int testdate2)
{
using namespace std::chrono;
auto ymd = get_ymd(testdate1);
auto hms = get_hms(testdate2);
auto ut = locate_zone("Europe/Berlin")->to_sys(local_days{ymd} + hms);
return ut.time_since_epoch().count();
}
std::chrono::locate_zone is called to get a pointer to the std::chrono::time_zone with the name "Europe/Berlin". The std::lib manages the lifetime of this object, so you don't have to worry about it. It is a const singleton, created on demand. And it has no impact on what time zone your computer considers its "local time zone".
The std::chrono::time_zone has a member function called to_sys that takes a local_time, and converts it to a sys_time, using the proper UTC offset for this time zone (taking into account daylight saving rules when applicable).
Both local_time and sys_time are std::chrono::time_point types. local_time is "some local time", not necessarily your computer's local time. You can associate a local time with a time zone in order to specify the locality of that time.
sys_time is a time_point based on system_clock. This tracks UTC (Unix time).
The expression local_days{ymd} + hms converts ymd and hms to local_time with a precision of seconds. local_days is just another local_time time_point, but with a precision of days.
The type of ut is time_point<system_clock, seconds>, which has a convenience type alias called sys_seconds, though auto makes that name unnecessary in this code.
To unpack the sys_seconds into an integral type, the .time_since_epoch() member function is called which results in the duration seconds, and then the .count() member function is called to extract the integral value from that duration.
When int is 32 bits, this function is susceptible to the year 2038 overflow problem. To fix that, simply change the return type of unixtime to return a 64 bit integral type (or make the return auto). Nothing else needs to change as std::chrono::seconds is already required to be greater than 32 bits and will not overflow at 68 years. Indeed std::chrono::seconds is usually represented by a signed 64 bit integral type in practice, giving it a range greater than the age of the universe (even if the scientists are off by an order of magnitude).

Capturing RTP Timestamps

I was trying a little experiment in order to get the timestamps of the RTP packets using the VideoCapture class from Opencv's source code in python, also had to modify FFmpeg to accommodate the changes in Opencv.
Since I read about the RTP packet format.Wanted to fiddle around and see if I could manage to find a way to get the NTP timestamps. Was unable to find any reliable help in trying to get RTP timestamps. So tried out this little hack.
Credits to ryantheseer on github for the modified code.
Version of FFmpeg: 3.2.3
Version of Opencv: 3.2.0
In Opencv source code:
modules/videoio/include/opencv2/videoio.hpp:
Added two getters for the RTP timestamp:
.....
/** #brief Gets the upper bytes of the RTP time stamp in NTP format (seconds).
*/
CV_WRAP virtual int64 getRTPTimeStampSeconds() const;
/** #brief Gets the lower bytes of the RTP time stamp in NTP format (fraction of seconds).
*/
CV_WRAP virtual int64 getRTPTimeStampFraction() const;
.....
modules/videoio/src/cap.cpp:
Added an import and added the implementation of the timestamp getter:
....
#include <cstdint>
....
....
static inline uint64_t icvGetRTPTimeStamp(const CvCapture* capture)
{
return capture ? capture->getRTPTimeStamp() : 0;
}
...
Added the C++ timestamp getters in the VideoCapture class:
....
/**#brief Gets the upper bytes of the RTP time stamp in NTP format (seconds).
*/
int64 VideoCapture::getRTPTimeStampSeconds() const
{
int64 seconds = 0;
uint64_t timestamp = 0;
//Get the time stamp from the capture object
if (!icap.empty())
timestamp = icap->getRTPTimeStamp();
else
timestamp = icvGetRTPTimeStamp(cap);
//Take the top 32 bytes of the time stamp
seconds = (int64)((timestamp & 0xFFFFFFFF00000000) / 0x100000000);
return seconds;
}
/**#brief Gets the lower bytes of the RTP time stamp in NTP format (seconds).
*/
int64 VideoCapture::getRTPTimeStampFraction() const
{
int64 fraction = 0;
uint64_t timestamp = 0;
//Get the time stamp from the capture object
if (!icap.empty())
timestamp = icap->getRTPTimeStamp();
else
timestamp = icvGetRTPTimeStamp(cap);
//Take the bottom 32 bytes of the time stamp
fraction = (int64)((timestamp & 0xFFFFFFFF));
return fraction;
}
...
modules/videoio/src/cap_ffmpeg.cpp:
Added an import:
...
#include <cstdint>
...
Added a method reference definition:
...
static CvGetRTPTimeStamp_Plugin icvGetRTPTimeStamp_FFMPEG_p = 0;
...
Added the method to the module initializer method:
...
if( icvFFOpenCV )
...
...
icvGetRTPTimeStamp_FFMPEG_p =
(CvGetRTPTimeStamp_Plugin)GetProcAddress(icvFFOpenCV, "cvGetRTPTimeStamp_FFMPEG");
...
...
icvWriteFrame_FFMPEG_p != 0 &&
icvGetRTPTimeStamp_FFMPEG_p !=0)
...
icvGetRTPTimeStamp_FFMPEG_p = (CvGetRTPTimeStamp_Plugin)cvGetRTPTimeStamp_FFMPEG;
Implemented the getter interface:
...
virtual uint64_t getRTPTimeStamp() const
{
return ffmpegCapture ? icvGetRTPTimeStamp_FFMPEG_p(ffmpegCapture) : 0;
}
...
In FFmpeg's source code:
libavcodec/avcodec.h:
Added the NTP timestamp definition to the AVPacket struct:
typedef struct AVPacket {
...
...
uint64_t rtp_ntp_time_stamp;
}
libavformat/rtpdec.c:
Store the ntp time stamp in the struct in the finalize_packet method:
static void finalize_packet(RTPDemuxContext *s, AVPacket *pkt, uint32_t timestamp)
{
uint64_t offsetTime = 0;
uint64_t rtp_ntp_time_stamp = timestamp;
...
...
/*RM: Sets the RTP time stamp in the AVPacket */
if (!s->last_rtcp_ntp_time || !s->last_rtcp_timestamp)
offsetTime = 0;
else
offsetTime = s->last_rtcp_ntp_time - ((uint64_t)(s->last_rtcp_timestamp) * 65536);
rtp_ntp_time_stamp = ((uint64_t)(timestamp) * 65536) + offsetTime;
pkt->rtp_ntp_time_stamp = rtp_ntp_time_stamp;
libavformat/utils.c:
Copy the ntp time stamp from the packet to the frame in the read_frame_internal method:
static int read_frame_internal(AVFormatContext *s, AVPacket *pkt)
{
...
uint64_t rtp_ntp_time_stamp = 0;
...
while (!got_packet && !s->internal->parse_queue) {
...
//COPY OVER the RTP time stamp TODO: just create a local copy
rtp_ntp_time_stamp = cur_pkt.rtp_ntp_time_stamp;
...
#if FF_API_LAVF_AVCTX
update_stream_avctx(s);
#endif
if (s->debug & FF_FDEBUG_TS)
av_log(s, AV_LOG_DEBUG,
"read_frame_internal stream=%d, pts=%s, dts=%s, "
"size=%d, duration=%"PRId64", flags=%d\n",
pkt->stream_index,
av_ts2str(pkt->pts),
av_ts2str(pkt->dts),
pkt->size, pkt->duration, pkt->flags);
pkt->rtp_ntp_time_stamp = rtp_ntp_time_stamp; #Just added this line in the if statement.
return ret;
My python code to utilise these changes:
import cv2
uri = 'rtsp://admin:password#192.168.1.67:554'
cap = cv2.VideoCapture(uri)
while True:
frame_exists, curr_frame = cap.read()
# if frame_exists:
k = cap.getRTPTimeStampSeconds()
l = cap.getRTPTimeStampFraction()
time_shift = 0x100000000
#because in the getRTPTimeStampSeconds()
#function, seconds was multiplied by 0x10000000
seconds = time_shift * k
m = (time_shift * k) + l
print("Imagetimestamp: %i" % m)
cap.release()
What I am getting as my output:
Imagetimestamp: 0
Imagetimestamp: 212041451700224
Imagetimestamp: 212041687629824
Imagetimestamp: 212041923559424
Imagetimestamp: 212042159489024
Imagetimestamp: 212042395418624
Imagetimestamp: 212042631348224
...
What astounded me the most was that when i powered off the ip camera and powered it back on, timestamp would start from 0 then quickly increments. I read NTP time format is relative to January 1, 1900 00:00. Even when I tried calculating the offset, and accounting between now and 01-01-1900, I still ended up getting a crazy high number for the date.
Don't know if I calculated it wrong. I have a feeling it's very off or what I am getting is not the timestamp.

As I see it, you receive a timestamp of type uint64 which contains to values uint32 in the high and low bits. I see that in a part of the code you use:
seconds = (int64)((timestamp & 0xFFFFFFFF00000000) / 0x100000000);
Which basically removes the lower bits and shifts the high bits to be in the lower bits. Then you cast it to int64. Here I only consider that it should be unsigned first of all, since it should not be negative in any case (seconds since epoch is always positive) and it should be uint32, since it is guarantee it is not bigger (you are taking only 32 bits). Also, this can be achieved (probably faster) with bitshifts like this:
auto seconds = static_cast<uint32>(timestamp >> 32);
Another error I spotted was in this part:
time_shift = 0x100000000
seconds = time_shift * k
m = (time_shift * k) + l
Here you are basically reconstructing the 64 bit timestamp, instead of creating the timestamp usable in other contexts. This means, you are shifting the lower bits in seconds to higher bits and adding the fraction part as the lower bits... This will end in a really big number which may not be useful always. You can still use it for comparison, but then all the conversions done in the C++ part are not needed. I think a more normal timestamp, which you can use with python datetime would be like this:
timestamp = float(str(k) + "." + str(l)) # don't know if there is a better way
date = datetime.fromtimestamp(timestamp)
If you don't care of the fractional part you can just use the seconds directly.
Another thing to consider is, that the timestamp of RTP protocols depends on the camera/server... They may use the clock timestamp or just some other clock like start of the streaming of start of the system. So it may or not be from epoch.

Python: XOR each character in a string

I'm trying to validate a checksum on a string which in this case is calculated by performing an XOR on each of the individual characters.
Given my test string:
check_against = "GPGLL,5300.97914,N,00259.98174,E,125926,A"
I figured it would be as simple as:
result = 0
for char in check_against:
result = result ^ ord(char)
I know the result should be 28, however my code gives 40.
I'm not sure what encoding the text is suppose to be in, although I've tried encoding/decoding in utf-8 and ascii, both with the same result.
I implemented this same algorithm in C by simply doing an XOR over the char array with perfect results, so what am I missing?
Edit
So it was a little while ago that I implemented (what I thought) was the same thing in C. I knew it was in an Objective-C project but I thought I had just done it this way. Totally wrong, first there was a step where I converted the checksum string value at the end to hex like so (I'm filling some things in here so that I'm only pasting what is relevant):
unsigned int checksum = 0;
NSScanner *scanner = [NSScanner scannerWithString:#"26"];
[scanner scanHexInt:&checksum];
Then I did the following to compute the checksum:
NSString sumString = #"GPGLL,5300.97914,N,00259.98174,E,125926,A";
unsigned int sum = 0;
for (int i=0;i<sumString.length;i++) {
sum = sum ^ [sumString characterAtIndex:i];
}
Then I would just compare like so:
return sum == checksum;

So as #metatoaster, #XD573, and some others in the comments have helped figure out, the issue was the difference between the result, which was base 10, and my desired solution (in base 16).
The result for the code, 40 is correct - in base 10, however my correct value I was trying to achieve, 28 is given in base 16. Simply converting the solution from base 16 to base 10, for example like so:
int('28', 16)
I get 40, the computed result.

#python3
str = "GPGLL,5300.97914,N,00259.98174,E,125926,A"
cks = 0
i = 0
while(i<len(str)):
cks^=ord(str[i])
i+=1
print("hex:",hex(cks))
print("dec:",cks)

I created the C version as shown here:
#include <stdio.h>
#include <string.h>
int main()
{
char* str1="GPGLL,5300.97914,N,00259.98174,E,125926,A";
int sum = 0;
int i = 0;
for (i; i < strlen(str1); i++) {
sum ^= str1[i];
}
printf("checksum: %d\n", sum);
return 0;
}
And When I compiled and ran it:
$ gcc -o mytest mytest.c
$ ./mytest
checksum: 40
Which leads me to believe that the assumptions you have from your equivalent C code are incorrect.

How to exchange time between C and Python

I'm writing some C code that needs to embed the current time in its (binary) output file. Later, this file will be read by some other C code (possibly compiled for different architecture) and/or some python code. In both cases calculations may be required on the time.
What I'd like to know is:
How do I get current UTC time in C? Is time() the write call?
What format should I write this to file in? ASN1? ISO?
How do I convert to that format?
How do I read that format in C and Python and convert it into something useful?

You could use rfc 3339 datetime format (a profile of ISO8601). It avoids many pitfalls of unconstrained ISO8601 timestamps.
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
int main(void) {
char buf[21];
time_t ts = time(NULL);
struct tm *tp = gmtime(&ts);
if (tp == NULL || tp->tm_year > 8099 || tp->tm_year < 0) {
perror("gmtime");
exit(EXIT_FAILURE);
}
if (strftime(buf, sizeof buf, "%Y-%m-%dT%H:%M:%SZ", tp) == 0) {
fprintf(stderr, "strftime returned 0\n");
exit(EXIT_FAILURE);
}
exit(puts(buf) != EOF ? EXIT_SUCCESS : EXIT_FAILURE);
}
Output
2014-12-20T11:08:44Z
To read it in Python:
>>> from datetime import datetime, timezone
>>> dt = datetime.strptime('2014-12-20T11:08:44Z', '%Y-%m-%dT%H:%M:%SZ')
>>> dt = dt.replace(tzinfo=timezone.utc)
>>> print(dt)
2014-12-20 11:08:44+00:00

Use the following C code to get a suitable date output:
time_t rawtime;
struct tm *now;
char timestamp[80];
time(&rawtime);
now = gmtime(&rawtime);
strftime(timestamp, sizeof(timestamp), "%Y%m%d%H%M%S", now);
Then use the following python to read it:
start_time = datetime.datetime.strptime(data, "%Y%m%d%H%M%S")
Variations on the format work, as long as it's consistent.

Python strptime ``%z``

EDIT2: This question is assuming a POSIX-ish platform with Python
linked against Glibc.
On my system, round-trip conversion using the %z formatting directive
using Python’s time library fails to parse the offset part of ISO 8601
formatted timestamps. This snippet:
import time
time.daylight = 0
fmt = "%Y-%m-%dT%H:%M:%SZ%z"
a=time.gmtime()
b=time.strftime(fmt, a)
c=time.strptime(b, fmt)
d=time.strftime(fmt, c)
print ("»»»»", a == c, b == d)
print ("»»»»", a.tm_zone, b)
print ("»»»»", c.tm_zone, d)
outputs:
»»»» False False
»»»» GMT 2018-02-16T09:26:34Z+0000
»»»» None 2018-02-16T09:26:34Z
whereas the expected output would be
»»»» True True
»»»» GMT 2018-02-16T09:26:34Z+0000
»»»» GMT 2018-02-16T09:26:34Z+0000
How do I get %z to respect that offset?
Python 3.3.2 and 3.6.4
[Glibc 2.17 and 2.25 ⇒ see below!]
EDIT: Glibc can be acquitted as proven by this C analogue:
#define _XOPEN_SOURCE
#define _DEFAULT_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
/* 2018-02-16T09:59:21Z+0000 */
#define ISO8601_FMT "%Y-%m-%dT%H:%M:%SZ%z"
int main () {
const time_t t0 = time (NULL);
struct tm a;
char b [27];
struct tm c;
char d [27];
(void)setenv ("TZ", "UTC", 1);
tzset ();
daylight = 0;
(void)gmtime_r (&t0, &a); /* a=time.gmtime () */
(void)strftime (b, sizeof(b), ISO8601_FMT, &a); /* b=time.strftime (fmt, a) */
(void)strptime (b, ISO8601_FMT, &c); /* c=time.strptime (b, fmt) */
(void)strftime (d, sizeof(d), ISO8601_FMT, &c); /* d=time.strftime (fmt, c) */
printf ("»»»» b ?= d %s\n", strcmp (b, d) == 0 ? "yep" : "hell, no");
printf ("»»»» %d <%s> %s\n", a.tm_isdst, a.tm_zone, b);
printf ("»»»» %d <%s> %s\n", c.tm_isdst, c.tm_zone, d);
}
Which outputs
»»»» b ?= d yep
»»»» 0 <GMT> 2018-02-16T10:28:18Z+0000
»»»» 0 <(null)> 2018-02-16T10:28:18Z+0000

With the "time.gmtime()" naturally you are getting the UTC time, so the offset will be always +0000, therefore an output string "2018-02-16T09:26:34Z" is correct for the ISO8601. If you want absolutely the "+0000" add it manually because it will be alway the same:
d = time.strftime(fmt, c) + '+0000'

I don't pretend to have the solution to generate the proper hour shift according to the time zone, but I can explain what happens here.
As hinted in Python timezone '%z' directive for datetime.strptime() not available answers:
strptime is implemeted in pure python so it has a constant behaviour
strftime depends on the platform/C library it was linked against.
On my system (Windows, Python 3.4), %z returns the same thing as %Z ("Paris, Madrid"). So when strptime tries to parse it back as digits, it fails. Your code gives me:
ValueError: time data '2018-02-16T10:00:49ZParis, Madrid' does not match format '%Y-%m-%dT%H:%M:%SZ%z'
It's system dependent for the generation, and not for the parsing.
This dissymetry explains the weird behaviour.