C++ vs. python: daylight saving time not recognized when extracting Unix time? - python

I'm attempting to calculate the Unix time of a given date and time represented by two integers, e.g.
testdate1 = 20060711 (July 11th, 2006)
testdate2 = 4 (00:00:04, 4 seconds after midnight)
in a timezone other than my local timezone. To calculate the Unix time, I feed testdate1, testdate2 into a function I adapted from Convert date to unix time stamp in c++
int unixtime (int testdate1, int testdate2) {
time_t rawtime;
struct tm * timeinfo;
//time1, ..., time6 are external functions that extract the
//year, month, day, hour, minute, seconds digits from testdate1, testdate2
int year=time1(testdate1);
int month=time2(testdate1);
int day=time3(testdate1);
int hour=time4(testdate2);
int minute=time5(testdate2);
int second=time6(testdate2);
time ( &rawtime );
timeinfo = localtime ( &rawtime );
timeinfo->tm_year = year - 1900;
timeinfo->tm_mon = month - 1;
timeinfo->tm_mday = day;
timeinfo->tm_hour = hour;
timeinfo->tm_min = minute;
timeinfo->tm_sec = second;
int date;
date = mktime(timeinfo);
return date;
}
Which I call from the main code
using namespace std;
int main(int argc, char* argv[])
{
int testdate1 = 20060711;
int testdate2 = 4;
//switch to CET time zone
setenv("TZ","Europe/Berlin", 1);
tzset();
cout << testdate1 << "\t" << testdate2 << "\t" << unixtime(testdate1,testdate2) << "\n";
return 0;
}
With the given example, I get unixtime(testdate1,testdate2) = 1152572404, which according to
https://www.epochconverter.com/timezones?q=1152572404&tz=Europe%2FBerlin
is 1:00:04 am CEST, but I want this to be 0:00:04 CEST.
The code seems to work perfectly well if I choose a testdate1, testdate2 in which daylight saving time (DST) isn't being observed. For example, simply setting the month to February with all else unchanged is accomplished by setting testdate1 = 20060211. This gives
unixtime(testdate1,testdate2) = 1139612404, corresponding to hh:mm:ss = 00:00:04 in CET, as desired.
My impression is that setenv("TZ","Europe/Berlin", 1) is supposed to account for DST when applicable, but perhaps I am mistaken. Can TZ interpret testdate1, testdate2 in such a way that it accounts for DST?
Interestingly, I have a python code that performs the same task by changing the local time via os.environ['TZ'] = 'Europe/Berlin'. Here I have no issues, as it seems to calculate the correct Unix time regardless of DST/non-DST.

localtime sets timeinfo->tm_isdst to that of the current time - not of the date you parse.
Don't call localtime. Set timeinfo->tm_isdst to -1:
The value specified in the tm_isdst field informs mktime() whether or not daylight saving time (DST) is in effect for the time supplied in the tm structure: a positive value means DST is in effect; zero means that DST is not in effect; and a negative value means that mktime() should (use timezone information and system databases to) attempt to determine whether DST is in effect at the specified time.
See the code example in https://en.cppreference.com/w/cpp/chrono/c/mktime

Maxim's answer is correct, and I've upvoted it. But I also thought it might be helpful to show how this can be done in C++20 using the newer <chrono> tools. This isn't implemented everywhere yet, but it is here in Visual Studio and will be coming elsewhere soon.
There's two main points I'd like to illustrate here:
<chrono> is convenient for conversions like this, even if both the input and the output does not involve std::chrono types. One can convert the integral input to chrono, do the conversion, and then convert the chrono result back to integral.
There's a thread safety weakness in using the TZ environment variable, as this is a type of global. If another thread is also doing some type of time computation, it may not get the correct answer if the computer's time zone unexpectedly changes out from under it. The <chrono> solution is thread-safe. It doesn't involve globals or environment variables.
The first job is to unpack the integral data. Here I show how to do this, and convert it into chrono types in one step:
std::chrono::year_month_day
get_ymd(int ymd)
{
using namespace std::chrono;
day d(ymd % 100);
ymd /= 100;
month m(ymd % 100);
ymd /= 100;
year y{ymd};
return y/m/d;
}
get_ymd takes "testdate1", extracts the individual integral fields for day, month, and year, then converts each integral field into the std::chrono types day, month and year, and finally combines these three separate fields into a std::chrono::year_month_day to return it as one value. This return type is simple a {year, month, day} data structure -- like a tuple but with calendrical meaning.
The / syntax is simply a convenient factory function for constructing a year_month_day. And this construction can be done with any of these three orderings: y/m/d, d/m/y and m/d/y. This syntax, when combined with auto, also means that you often don't have to spell out the verbose name year_month_day:
auto
get_ymd(int ymd)
{
// ...
return y/m/d;
}
get_hms unpacks the hour, minute and second fields and returns that as a std::chrono::seconds:
std::chrono::seconds
get_hms(int hms)
{
using namespace std::chrono;
seconds s{hms % 100};
hms /= 100;
minutes m{hms % 100};
hms /= 100;
hours h{hms};
return h + m + s;
}
The code is very similar to that for get_ymd except that the return is the sum of the hours, minutes and seconds. The chrono library does the job for you of converting hours and minutes to seconds while performing the summation.
Next is the function for doing the conversion, and returning the result back as an int.
int
unixtime(int testdate1, int testdate2)
{
using namespace std::chrono;
auto ymd = get_ymd(testdate1);
auto hms = get_hms(testdate2);
auto ut = locate_zone("Europe/Berlin")->to_sys(local_days{ymd} + hms);
return ut.time_since_epoch().count();
}
std::chrono::locate_zone is called to get a pointer to the std::chrono::time_zone with the name "Europe/Berlin". The std::lib manages the lifetime of this object, so you don't have to worry about it. It is a const singleton, created on demand. And it has no impact on what time zone your computer considers its "local time zone".
The std::chrono::time_zone has a member function called to_sys that takes a local_time, and converts it to a sys_time, using the proper UTC offset for this time zone (taking into account daylight saving rules when applicable).
Both local_time and sys_time are std::chrono::time_point types. local_time is "some local time", not necessarily your computer's local time. You can associate a local time with a time zone in order to specify the locality of that time.
sys_time is a time_point based on system_clock. This tracks UTC (Unix time).
The expression local_days{ymd} + hms converts ymd and hms to local_time with a precision of seconds. local_days is just another local_time time_point, but with a precision of days.
The type of ut is time_point<system_clock, seconds>, which has a convenience type alias called sys_seconds, though auto makes that name unnecessary in this code.
To unpack the sys_seconds into an integral type, the .time_since_epoch() member function is called which results in the duration seconds, and then the .count() member function is called to extract the integral value from that duration.
When int is 32 bits, this function is susceptible to the year 2038 overflow problem. To fix that, simply change the return type of unixtime to return a 64 bit integral type (or make the return auto). Nothing else needs to change as std::chrono::seconds is already required to be greater than 32 bits and will not overflow at 68 years. Indeed std::chrono::seconds is usually represented by a signed 64 bit integral type in practice, giving it a range greater than the age of the universe (even if the scientists are off by an order of magnitude).

Related

Capturing RTP Timestamps

I was trying a little experiment in order to get the timestamps of the RTP packets using the VideoCapture class from Opencv's source code in python, also had to modify FFmpeg to accommodate the changes in Opencv.
Since I read about the RTP packet format.Wanted to fiddle around and see if I could manage to find a way to get the NTP timestamps. Was unable to find any reliable help in trying to get RTP timestamps. So tried out this little hack.
Credits to ryantheseer on github for the modified code.
Version of FFmpeg: 3.2.3
Version of Opencv: 3.2.0
In Opencv source code:
modules/videoio/include/opencv2/videoio.hpp:
Added two getters for the RTP timestamp:
.....
/** #brief Gets the upper bytes of the RTP time stamp in NTP format (seconds).
*/
CV_WRAP virtual int64 getRTPTimeStampSeconds() const;
/** #brief Gets the lower bytes of the RTP time stamp in NTP format (fraction of seconds).
*/
CV_WRAP virtual int64 getRTPTimeStampFraction() const;
.....
modules/videoio/src/cap.cpp:
Added an import and added the implementation of the timestamp getter:
....
#include <cstdint>
....
....
static inline uint64_t icvGetRTPTimeStamp(const CvCapture* capture)
{
return capture ? capture->getRTPTimeStamp() : 0;
}
...
Added the C++ timestamp getters in the VideoCapture class:
....
/**#brief Gets the upper bytes of the RTP time stamp in NTP format (seconds).
*/
int64 VideoCapture::getRTPTimeStampSeconds() const
{
int64 seconds = 0;
uint64_t timestamp = 0;
//Get the time stamp from the capture object
if (!icap.empty())
timestamp = icap->getRTPTimeStamp();
else
timestamp = icvGetRTPTimeStamp(cap);
//Take the top 32 bytes of the time stamp
seconds = (int64)((timestamp & 0xFFFFFFFF00000000) / 0x100000000);
return seconds;
}
/**#brief Gets the lower bytes of the RTP time stamp in NTP format (seconds).
*/
int64 VideoCapture::getRTPTimeStampFraction() const
{
int64 fraction = 0;
uint64_t timestamp = 0;
//Get the time stamp from the capture object
if (!icap.empty())
timestamp = icap->getRTPTimeStamp();
else
timestamp = icvGetRTPTimeStamp(cap);
//Take the bottom 32 bytes of the time stamp
fraction = (int64)((timestamp & 0xFFFFFFFF));
return fraction;
}
...
modules/videoio/src/cap_ffmpeg.cpp:
Added an import:
...
#include <cstdint>
...
Added a method reference definition:
...
static CvGetRTPTimeStamp_Plugin icvGetRTPTimeStamp_FFMPEG_p = 0;
...
Added the method to the module initializer method:
...
if( icvFFOpenCV )
...
...
icvGetRTPTimeStamp_FFMPEG_p =
(CvGetRTPTimeStamp_Plugin)GetProcAddress(icvFFOpenCV, "cvGetRTPTimeStamp_FFMPEG");
...
...
icvWriteFrame_FFMPEG_p != 0 &&
icvGetRTPTimeStamp_FFMPEG_p !=0)
...
icvGetRTPTimeStamp_FFMPEG_p = (CvGetRTPTimeStamp_Plugin)cvGetRTPTimeStamp_FFMPEG;
Implemented the getter interface:
...
virtual uint64_t getRTPTimeStamp() const
{
return ffmpegCapture ? icvGetRTPTimeStamp_FFMPEG_p(ffmpegCapture) : 0;
}
...
In FFmpeg's source code:
libavcodec/avcodec.h:
Added the NTP timestamp definition to the AVPacket struct:
typedef struct AVPacket {
...
...
uint64_t rtp_ntp_time_stamp;
}
libavformat/rtpdec.c:
Store the ntp time stamp in the struct in the finalize_packet method:
static void finalize_packet(RTPDemuxContext *s, AVPacket *pkt, uint32_t timestamp)
{
uint64_t offsetTime = 0;
uint64_t rtp_ntp_time_stamp = timestamp;
...
...
/*RM: Sets the RTP time stamp in the AVPacket */
if (!s->last_rtcp_ntp_time || !s->last_rtcp_timestamp)
offsetTime = 0;
else
offsetTime = s->last_rtcp_ntp_time - ((uint64_t)(s->last_rtcp_timestamp) * 65536);
rtp_ntp_time_stamp = ((uint64_t)(timestamp) * 65536) + offsetTime;
pkt->rtp_ntp_time_stamp = rtp_ntp_time_stamp;
libavformat/utils.c:
Copy the ntp time stamp from the packet to the frame in the read_frame_internal method:
static int read_frame_internal(AVFormatContext *s, AVPacket *pkt)
{
...
uint64_t rtp_ntp_time_stamp = 0;
...
while (!got_packet && !s->internal->parse_queue) {
...
//COPY OVER the RTP time stamp TODO: just create a local copy
rtp_ntp_time_stamp = cur_pkt.rtp_ntp_time_stamp;
...
#if FF_API_LAVF_AVCTX
update_stream_avctx(s);
#endif
if (s->debug & FF_FDEBUG_TS)
av_log(s, AV_LOG_DEBUG,
"read_frame_internal stream=%d, pts=%s, dts=%s, "
"size=%d, duration=%"PRId64", flags=%d\n",
pkt->stream_index,
av_ts2str(pkt->pts),
av_ts2str(pkt->dts),
pkt->size, pkt->duration, pkt->flags);
pkt->rtp_ntp_time_stamp = rtp_ntp_time_stamp; #Just added this line in the if statement.
return ret;
My python code to utilise these changes:
import cv2
uri = 'rtsp://admin:password#192.168.1.67:554'
cap = cv2.VideoCapture(uri)
while True:
frame_exists, curr_frame = cap.read()
# if frame_exists:
k = cap.getRTPTimeStampSeconds()
l = cap.getRTPTimeStampFraction()
time_shift = 0x100000000
#because in the getRTPTimeStampSeconds()
#function, seconds was multiplied by 0x10000000
seconds = time_shift * k
m = (time_shift * k) + l
print("Imagetimestamp: %i" % m)
cap.release()
What I am getting as my output:
Imagetimestamp: 0
Imagetimestamp: 212041451700224
Imagetimestamp: 212041687629824
Imagetimestamp: 212041923559424
Imagetimestamp: 212042159489024
Imagetimestamp: 212042395418624
Imagetimestamp: 212042631348224
...
What astounded me the most was that when i powered off the ip camera and powered it back on, timestamp would start from 0 then quickly increments. I read NTP time format is relative to January 1, 1900 00:00. Even when I tried calculating the offset, and accounting between now and 01-01-1900, I still ended up getting a crazy high number for the date.
Don't know if I calculated it wrong. I have a feeling it's very off or what I am getting is not the timestamp.
As I see it, you receive a timestamp of type uint64 which contains to values uint32 in the high and low bits. I see that in a part of the code you use:
seconds = (int64)((timestamp & 0xFFFFFFFF00000000) / 0x100000000);
Which basically removes the lower bits and shifts the high bits to be in the lower bits. Then you cast it to int64. Here I only consider that it should be unsigned first of all, since it should not be negative in any case (seconds since epoch is always positive) and it should be uint32, since it is guarantee it is not bigger (you are taking only 32 bits). Also, this can be achieved (probably faster) with bitshifts like this:
auto seconds = static_cast<uint32>(timestamp >> 32);
Another error I spotted was in this part:
time_shift = 0x100000000
seconds = time_shift * k
m = (time_shift * k) + l
Here you are basically reconstructing the 64 bit timestamp, instead of creating the timestamp usable in other contexts. This means, you are shifting the lower bits in seconds to higher bits and adding the fraction part as the lower bits... This will end in a really big number which may not be useful always. You can still use it for comparison, but then all the conversions done in the C++ part are not needed. I think a more normal timestamp, which you can use with python datetime would be like this:
timestamp = float(str(k) + "." + str(l)) # don't know if there is a better way
date = datetime.fromtimestamp(timestamp)
If you don't care of the fractional part you can just use the seconds directly.
Another thing to consider is, that the timestamp of RTP protocols depends on the camera/server... They may use the clock timestamp or just some other clock like start of the streaming of start of the system. So it may or not be from epoch.

datetime.datetime.strptime() using variable input length

datetime.datetime.strptime seems to force directive matching regardless of the actual string length used. By using shorter strings, the directives will force the datetime.datetime object to use "something" in the string regardless of actual directives.
This is the correct behavior with enough input to fill the directives
>>> datetime.datetime.strptime('20180822163014', '%Y%m%d%H%M%S')
datetime.datetime(2018, 8, 22, 16, 30, 14)
This directives however will change the previous parsing
>>> datetime.datetime.strptime('20180822163014', '%Y%m%d%H%M%S%f')
datetime.datetime(2018, 8, 22, 16, 30, 1, 400000)
Is there any way to drop rightmost directives if input string is not long enough instead of cannibalizing the left ones?
I've tagged C and ubuntu because documentation says
"The full set of format codes supported varies across platforms,
because Python calls the platform C library’s strftime() function, and
platform variations are common. To see the full set of format codes
supported on your platform, consult the strftime(3) documentation."
EDIT:
man ctime shows the following structure as output. It is interesting that the microseconds ( %f ) precision doesn't seem to be supported.
struct tm {
int tm_sec; /* Seconds (0-60) */
int tm_min; /* Minutes (0-59) */
int tm_hour; /* Hours (0-23) */
int tm_mday; /* Day of the month (1-31) */
int tm_mon; /* Month (0-11) */
int tm_year; /* Year - 1900 */
int tm_wday; /* Day of the week (0-6, Sunday = 0) */
int tm_yday; /* Day in the year (0-365, 1 Jan = 0) */
int tm_isdst; /* Daylight saving time */
};
Well, I guess you have to do it by yourself, which doesn't seems to hard because you know the pattern.
Something like that should to the job
pattern = ""
if len(s) == 0: raise Exception "empty time string"
if len(s) <= 4: pattern += "%Y"
... # as many if as you need here
datetime.datetime.strptime(s, pattern)
Which is very painful to write if you have long date pattern, but I doubt that there is some function doing it already in the datetime module - for the reason that its just a binding with C.
You can try to do something more generic and ask if it could be add to the datetime module.

python mktime(.timetuple()) returns different results in mac and linux

I noticed time.mktime(.timetuple()) returned different time on mac and linux(ubuntu). Why this?
date = ['2016-07-01', '2016-07-05']
xdata = [datetime.datetime.strptime(str(s), "%Y-%m-%d") for s in date]
xdata = [time.mktime(s.timetuple()) * 1000 for s in xdata]
print xdata
# ----mac--
>> [1467356400000.0, 1467702000000.0]
#-----linux---
>> [1467345600000.0, 1467691200000.0]
How to return in UTC?
I marked to close this as a duplicate, but it's really not if you're viewing your original inputs as being in UTC to begin with. If you are (it's not wholly clear), then just replace your time.mktime with calendar.timegm.
>>> d = datetime.datetime(2016, 9, 1)
>>> calendar.timegm(d.timetuple())
1472688000
Or you can do it all yourself:
>>> EPOCH = datetime.datetime(1970, 1, 1)
>>> def dt2utcstamp(d):
... return (d - EPOCH).total_seconds()
and then:
>>> dt2utcstamp(d)
1472688000.0
I generally do the latter, because I find it next to impossible to remember what all the goofy time-and-date functions do ;-) But the timedelta.total_seconds() method doesn't exist before Python 3.2.
Or if you do view the inputs as being in local time, then the other answers apply:
How do I convert local time to UTC in Python?
NOTE
When you ask "How to return in UTC?", you have to realize that your original code already did that: timestamps are always viewed as being seconds from the epoch in UTC. Indeed, that's why you got different results on platforms set to different time zones to begin with: '2016-07-01'(with implied time 00:00:00) is a different real-world instant depending on which time zone it's viewed as being in.
The s.timetuple() part doesn't care about that, but it's a primary purpose of the time.mktime() part to convert the local time to a UTC timestamp.

python struct unpack: Int PIN; char verified; time_t time_second; char status;

I am trying to unpack the following struct from binary data received via UDP from a time attendance clock device(DeviceName=TA8020):
typedef struct _AttLog_{
Int PIN; //U16 PIN, user number
char verified;// verifying method
time_t time_second; //time, time code is user-defined time code.
char status;// attendance state
}TAttLog, *PAttLog;
Trying to unpack the data like this:
uid, state, timestamp, space = unpack( '24s1s4s11s', attendancedata.ljust(40)[:40] )
print "%s, %s, %s, %s" % (uid, state, space, decode_time( int( reverseHex( timestamp.encode('hex') ), 16 ) ) )
yields the following result:
5, , , 2016-02-13 11:55:36
The uid and timestamp are correct but I am unable to get the correct values for char verified and char status as you can see unpacking the struct like above they are returned empty.
When trying to unpack the char status and char verified separately like this:
state = unpack('>3c',attendancedata[17:20])
space = unpack('>2c',attendancedata[21:23])
yields:
statem ('\x00', '\x00', '\x00')
statev 0
every time which are not the correct values.(verified by looking at the web interface logs of the time attendance device)
Unpacking like:
oneSet = unpack('24s1s4s11s',attendancedata.ljust(40)[:40])
yields:
('5\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00','\x01','h\x1b\xe0\x1e','\x01\x00\x00\x00\x00\x00\x00\x00\x00\xb2\x00')
The API documentation gives details as follows:
Only read command can be used to read all attendance logs. Attendance log can be compressed in long or short mode. The compressing method (if char *Buffer is being read now, the hand will be at the first byte) is: the first 2 bytes are used for storing user PIN (U 16 PIN), the front three bits of the third byte are used for storing verifying state. The fourth and fifth bits are to store verifying method. The sixth bit is to store short and long time sign. If it is short time format, time value is the last two bits of the third byte and the third byte plus the recent long time value (therefore, time format is stored as the misregistration value of the former long time value). Then decode it according to time coding mode (refer to user-defined coding mode) to get the correct time.
Any help would be much appreciated.
Ok I figured it out:
state = unpack('c',attendancedata[29:30])
is the value I need.

Why is this call to a dynamic library function so slow?

I am writing a shared library for python to call. Since this is my first time using python's ctypes module, and nearly my first time writing a shared library, I have been writing both C and python code to call the library's functions.
For the heck of it I put some timing code in and found that, while most calls of the C program to the library are very fast, the first is slow, considerably slower than its python counterpart in fact. This goes against everything I expected and was hoping that someone could tell me why.
Here is a stripped down version of the header file from my C library.
typedef struct MdaDataStruct
{
int numPts;
int numDists;
float* data;
float* dists;
} MdaData;
//allocate the structure
void* makeMdaStruct(int numPts, int numDist);
//deallocate the structure
void freeMdaStruct(void* strPtr);
//assign the data array
void setData(void* strPtr, float* divData);
Here is the C program that calls the functions:
int main(int argc, char* argv[])
{
clock_t t1, t2;
t1=clock();
long long int diff;
//test the allocate function
t1 = clock();
MdaData* dataPtr = makeMdaStruct(10, 3);
t2 = clock();
diff = (((t2-t1)*1000000)/CLOCKS_PER_SEC);
printf("make struct, took: %d microseconds\n", diff);
//make some data
float testArr[10] = {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9};
//test the set data function
t1 = clock();
setData(dataPtr, testArr);
t2 = clock();
diff = (((t2-t1)*1000000)/CLOCKS_PER_SEC);
printf("set data, took: %d microseconds\n", diff);
//test the deallocate function
t1 = clock();
freeMdaStruct(dataPtr);
t2 = clock();
diff = (((t2-t1)*1000000)/CLOCKS_PER_SEC);
printf("free struct, took: %d microseconds\n", diff);
//exit
return 0;
}
and here is the python script that calls the functions:
# load the library
t1 = time.time()
cs_lib = cdll.LoadLibrary("./libChiSq.so")
t2 = time.time()
print "load library, took", int((t2-t1)*1000000), "microseconds"
# tell python the function will return a void pointer
cs_lib.makeMdaStruct.restype = c_void_p
# make the strcuture to hold the MdaData with 50 data points and 8 L dists
t1 = time.time()
mdaData = cs_lib.makeMdaStruct(10,3)
t2 = time.time()
print "make struct, took", int((t2-t1)*1000000), "microseconds"
# make an array with the test data
divDat = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], np.float32)
#run the function to load the array into the struct
t1 = time.time()
cs_lib.setData(mdaData, divDat.ctypes.data)
t2 = time.time()
print "set data, took", int((t2-t1)*1000000), "microseconds"
#free the structure
t1 = time.time()
cs_lib.freeMdaStruct(mdaData)
t2 = time.time()
print "free struct, took", int((t2-t1)*1000000), "microseconds"
and finally, here is the output of running the two consecutively:
[]$ ./tester
make struct, took: 60 microseconds
set data, took: 2 microseconds
free struct, took: 2 microseconds
[]$ python so_py_tester.py
load library, took 77 microseconds
make struct, took 3 microseconds
set data, took 23 microseconds
free struct, took 10 microseconds
As you can see, the C call to makeMdaStruct takes 60us and the python call to makeMdaStruct takes 3us, which is highly confusing.
My best guess was that somehow the C code pays the cost of loading the library at the first call? Which confuses me because I thought that the library was loaded when the program was loaded into memory.
Edit: I think there might be a kernel of truth to the guess because I put an extra untimed call to makeMdaStruct and freeMdaStruct before the timed call to makeMdaStruct and got the following output in testing:
[]$ ./tester
make struct, took: 1 microseconds
set data, took: 1 microseconds
free struct, took: 0 microseconds
[]$ python so_py_tester.py
load library, took 70 microseconds
make struct, took 4 microseconds
set data, took 23 microseconds
free struct, took 12 microseconds
My best guess was that somehow the C code pays the cost of loading the library at the first call? Which confuses me because I thought that the library was loaded when the program was loaded into memory.
You are correct in both cases. The library is loaded when the program is loaded. However, the dynamic loader/linker defers symbol resolution until function invocation time.
Calls to shared libraries are done so indirectly, via an entry in the procedure linkage table (PLT). Initially, all of the entries in the PLT point to ld.so. Upon the first call to a function, ld.so looks up the actual address of the symbol, updates the entry in the PLT, and jumps to the function. This is "lazy" symbol resolution.
You can set theLD_BIND_NOW environment variable to change this behavior. From ld.so(8):
LD_BIND_NOW
(libc5; glibc since 2.1.1) If set to a nonempty string, causes the dynamic linker to resolve all symbols at program startup instead of deferring function call
resolution to the point when they are first referenced. This is useful when using a debugger.
This behavior can also be changed at link time. From ld(1):
-z keyword
The recognized keywords are:
...
lazy
When generating an executable or shared library, mark it to
tell the dynamic linker to defer function call resolution to
the point when the function is called (lazy binding), rather
than at load time. Lazy binding is the default.
Further reading:
http://www.macieira.org/blog/2012/01/sorry-state-of-dynamic-libraries-on-linux/

Categories