I've observed this issue while experimenting with test_ksync executable.
I have received the following messages on standard output:
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from TestKSync
[ RUN ] TestKSync.SeqNum_1
[ OK ] TestKSync.SeqNum_1 (1846 ms)
[----------] 1 test from TestKSync (1847 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1849 ms total)
[ PASSED ] 1 test.
Despite that I have received non-zero exit code from the executable (-1073741819).
I checked that last statement from the main function ("return ret;") is called with ret equal to 0.
Short investigation revealed that the problem is related to order of calling destructors. Let's have a look into the following call stack:
00 test_ksync!boost::asio::detail::win_iocp_io_service::cancel_timer<boost::asio::detail::chrono_time_traits<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock> > >+0x4d
01 test_ksync!boost::asio::detail::deadline_timer_service<boost::asio::detail::chrono_time_traits<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock> > >::cancel+0x7d
02 test_ksync!boost::asio::detail::deadline_timer_service<boost::asio::detail::chrono_time_traits<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock> > >::destroy+0x41
03 test_ksync!boost::asio::waitable_timer_service<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock> >::destroy+0x39
04 test_ksync!boost::asio::basic_io_object<boost::asio::waitable_timer_service<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock> >,0>::~basic_io_object<boost::asio::waitable_timer_service<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock> >,0>+0x40
05 test_ksync!boost::asio::basic_waitable_timer<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock>,boost::asio::waitable_timer_service<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock> > >::~basic_waitable_timer<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock>,boost::asio::waitable_timer_service<std::chrono::steady_clock,boost::asio::wait_traits<std::chrono::steady_clock> > >+0x28
06 test_ksync!TimerImpl::~TimerImpl+0x2b
07 test_ksync!TimerImpl::`scalar deleting destructor'+0x2c
08 test_ksync!std::auto_ptr<TimerImpl>::~auto_ptr<TimerImpl>+0x4c
09 test_ksync!Timer::~Timer+0xbc
0a test_ksync!Timer::`scalar deleting destructor'+0x2c
0b test_ksync!intrusive_ptr_release+0x6c
0c test_ksync!boost::intrusive_ptr<Timer>::~intrusive_ptr<Timer>+0x3f
0d test_ksync!boost::intrusive_ptr<Timer>::`scalar deleting destructor'+0x2c
0e test_ksync!std::allocator<std::_Tree_node<boost::intrusive_ptr<Timer>,void * __ptr64> >::destroy<boost::intrusive_ptr<Timer> >+0x2f
0f test_ksync!std::allocator_traits<std::allocator<std::_Tree_node<boost::intrusive_ptr<Timer>,void * __ptr64> > >::destroy<boost::intrusive_ptr<Timer> >+0x32
10 test_ksync!std::_Wrap_alloc<std::allocator<std::_Tree_node<boost::intrusive_ptr<Timer>,void * __ptr64> > >::destroy<boost::intrusive_ptr<Timer> >+0x32
11 test_ksync!std::_Tree<std::_Tset_traits<boost::intrusive_ptr<Timer>,TimerManager::TimerPtrCmp,std::allocator<boost::intrusive_ptr<Timer> >,0> >::_Erase+0xa4
12 test_ksync!std::_Tree<std::_Tset_traits<boost::intrusive_ptr<Timer>,TimerManager::TimerPtrCmp,std::allocator<boost::intrusive_ptr<Timer> >,0> >::clear+0x4c
13 test_ksync!std::_Tree<std::_Tset_traits<boost::intrusive_ptr<Timer>,TimerManager::TimerPtrCmp,std::allocator<boost::intrusive_ptr<Timer> >,0> >::erase+0x173
14 test_ksync!std::_Tree<std::_Tset_traits<boost::intrusive_ptr<Timer>,TimerManager::TimerPtrCmp,std::allocator<boost::intrusive_ptr<Timer> >,0> >::_Tidy+0x88
15 test_ksync!std::_Tree<std::_Tset_traits<boost::intrusive_ptr<Timer>,TimerManager::TimerPtrCmp,std::allocator<boost::intrusive_ptr<Timer> >,0> >::~_Tree<std::_Tset_traits<boost::intrusive_ptr<Timer>,TimerManager::TimerPtrCmp,std::allocator<boost::intrusive_ptr<Timer> >,0> >+0x31
16 test_ksync!std::set<boost::intrusive_ptr<Timer>,TimerManager::TimerPtrCmp,std::allocator<boost::intrusive_ptr<Timer> > >::~set<boost::intrusive_ptr<Timer>,TimerManager::TimerPtrCmp,std::allocator<boost::intrusive_ptr<Timer> > >+0x28
17 test_ksync!`dynamic atexit destructor for 'TimerManager::timer_ref_''+0x21
All instances of Timer managed by TimerManager are deleted when other relevant classes have already been deleted (see frames 0e - 17). Timer (actually TimerImpl) has a reference to an instance of boost timer, which also has to be deleted (frame 05). Boost timer refers to suitable io_service to call its cancel_timer method. This is fairly unfortunate as mentioned io_service has already been deleted earlier.
It means that already freed memory is accessed. It may not be noticed if the process can still access that memory and there's still something meaningful. In such a case an executable might return 0, but this behavior is still incorrect.
Additional details:
Host OS: Windows Server 2016 (64-bit)
It has been tested with development version of Agent running under Windows, but I expect that this is also an issue on Contrail 4.0.
