Re: An explanation of some "Container overflow" ASAN reports: libc++ does sometimes temporarily overflow the used range of a container

> On 2022-Jan-14, at 04:44, Mark Millard <marklmi@yahoo.com> wrote:
> 
> Looks like libc++ does the following sort of thing
> (from lldb list):
> 
> . . .
>   1635	
>   1636	template <class _Tp, class _Allocator>
>   1637	template <class _Up>
>   1638	void
>   1639	#ifndef _LIBCPP_CXX03_LANG
> (lldb) 
>   1640	vector<_Tp, _Allocator>::__push_back_slow_path(_Up&& __x)
>   1641	#else
>   1642	vector<_Tp, _Allocator>::__push_back_slow_path(_Up& __x)
>   1643	#endif
>   1644	{
>   1645	    allocator_type& __a = this->__alloc();
>   1646	    __split_buffer<value_type, allocator_type&> __v(__recommend(size() + 1), size(), __a);
>   1647	    // __v.push_back(_VSTD::forward<_Up>(__x));
>   1648	    __alloc_traits::construct(__a, _VSTD::__to_address(__v.__end_), _VSTD::forward<_Up>(__x));
>   1649	    __v.__end_++;
> (lldb) 
>   1650	    __swap_out_circular_buffer(__v);
>   1651	}
> . . . (the bt points to 1650) . . .
> 
> 1648 constructs into __v at __v.__end_ and 1649 then corrects
> __v.__end_ to cause the constructed object to no longer be
> an example of "Container overflow" but now in the Container.
> (At least that is my interpretation.)
> 
> The compiler's code generation may move the detailed
> place where __v.__end_++ happens relative to some other
> of the activity but the compiler has been told an order
> relative to the construction that would lead to writing
> memory in the capacity of the container __v but outside
> the size of the __v container at the time.
> 
> For reference:
> 
>   970 	template <class _Tp, class _Allocator>
>   971 	void
>   972 	vector<_Tp, _Allocator>::__swap_out_circular_buffer(__split_buffer<value_type, allocator_type&>& __v)
>   973 	{
>   974 	
>   975 	    __annotate_delete();
>   976 	    _VSTD::__construct_backward_with_exception_guarantees(this->__alloc(), this->__begin_, this->__end_, __v.__begin_);
>   977 	    _VSTD::swap(this->__begin_, __v.__begin_);
>   978 	    _VSTD::swap(this->__end_, __v.__end_);
>   979 	    _VSTD::swap(this->__end_cap(), __v.__end_cap());
> (lldb) 
>   980 	    __v.__first_ = __v.__begin_;
>   981 	    __annotate_new(size());
>   982 	    __invalidate_all_iterators();
>   983 	}
> . . . (the bt for this points to 976) . . .
> 
> 
> This suggests to me that using some equivalent of:
> 
> env ASAN_OPTIONS=detect_container_overflow=0
> 
> my be required fairly generally when libc++ can
> be involved.
> 
> 
> Other notes . . .
> 
> I used ld -v as an example for the above via:
> 
> env ASAN_OPTIONS=detect_container_overflow=0 lldb ld
> 
> and used:
> 
> (lldb) env ASAN_OPTIONS=
> (lldb) run -v
> 
> in order to have ld itself not have detect_container_overflow
> disabled.
> 
> lldb suffers the libc++ Container overflow problems via its
> libc++ use and fails to operate without the:
> 
> ASAN_OPTIONS=detect_container_overflow=0
> 

Hmm. The code in:

   772 	template <class _Alloc, class _Ptr>
   773 	_LIBCPP_INLINE_VISIBILITY
   774 	void __construct_backward_with_exception_guarantees(_Alloc& __a, _Ptr __begin1, _Ptr __end1, _Ptr& __end2) {
   775 	    static_assert(__is_cpp17_move_insertable<_Alloc>::value,
   776 	        "The specified type does not meet the requirements of Cpp17MoveInsertable");
   777 	    typedef allocator_traits<_Alloc> _Traits;
   778 	    while (__end1 != __begin1) {
   779 	        _Traits::construct(__a, _VSTD::__to_address(__end2 - 1),
(lldb) 
   780 	#ifdef _LIBCPP_NO_EXCEPTIONS
   781 	            _VSTD::move(*--__end1)
   782 	#else
   783 	            _VSTD::move_if_noexcept(*--__end1)
   784 	#endif
   785 	        );
   786 	        --__end2;
   787 	    }
   788 	}

has the same sort of problem going in the other direction.
__end2 references the __v.__begin_ mentioned earlier for
the context at hand.

line 779 constructs just before where __v.__begin_ refers to
at the time and then 786 updates __v.__begin_ to make the
constructed object no longer be an example of "Container
overflow" but now in the Container.

This is likely the WRITE activity that is actually reported,
although I've not analyzed the machine code at all.

This also suggests to me that using some equivalent of:

env ASAN_OPTIONS=detect_container_overflow=0

my be required fairly generally when libc++ can
be involved.


Important . . .

I'll note that the construct-then-include order is tied
to exception safety. I'm not claiming that the libc++
code is wrong. It just that it and ASAN are currently
mismatched so ASAN by default reports libc++ as having
an addressing issue for which aborting the program is
the handling ASAN does by default.

===
Mark Millard
marklmi at yahoo.com