How to traverse kernel threads?

[Mateusz Guzik]

On Fri, Mar 27, 2015 at 11:20:39PM +0100, Oliver Pinter wrote:
> On Fri, Mar 27, 2015 at 9:55 PM, Yue Chen <ycyc321 at gmail.com> wrote:
> >> Except it seems there routines are supposed to be only used when
> >> execution is 'frozen' (e.g. when escaped to the debugger).
> >
> > It means probably we can run the code in ``smp_rendezvous()'' function,
> > right?
> >
> >> Still nobody knows what you are trying to do.
> >
> > We are trying to enhance FreeBSD's security by randomizing kernel code basic
> > blocks periodically at runtime, to mitigate the attacks like return-oriented
> > programming (ROP). It is basically a stronger form of ASLR.
> > After each randomization procedure, the function return addresses saved in
> > the stack are the ``old'' addresses before randomization, so we need to
> > update them to the new addresses.
> > That's why we need to get all the stack ranges to find those addresses.
> >
> > Also, in kernel, we believe that not only the return addresses in stacks
> > need to be updated, there may exist other ``old'' saved instruction (PC)
> > addresses in memory. Like in exception handling (maybe, do not know),
> > debugging-purpose code and restartable atomic sequences (RAS)
> > implementation. That's why I asked how to traverse all the kernel pages and
> > get their virtual addresses here:
> > https://lists.freebsd.org/pipermail/freebsd-hackers/2015-March/047336.html
> >
> > Now we found that it seems needed to traverse the ``pv_entry'' structure for
> > x86_64 MMU.
> >
> > Another problem is that we do not know if FreeBSD has any form of special
> > encodings or offset form for 64-bit instruction addresses (e.g., saved %RIP)
> > on X86_64, instead of hard-coded addresses. For example, using a 32-bit
> > offset instead of the 64-bit full address; and doing what glibc does for the
> > setjmp/longjmp  jmp_buf (special encodings (PTR_MANGLE) for the saved
> > register values).
> >
> > Any suggestion or correction are highly appreciated.
> >
> > Best,
> > Yue
> 
> (Added HardenedBSD core and PaXTeam to CC.)
> 
> Until you can not fixed all of the infoleaks from kernel (try sysctl
> -a | grep 0x or similar command) the KASLR and other kernel address
> space randomization techniques are easily bypass-able...
> 

I do not believe this can be implemented reliably without some serious
tinkering around the kernel. Surely I'm not a live patching expert (not
any other kind of expert), but hear my out.

It seems proposed approach is to move the kernel around and then updated
all relevant pointers in all pages.

I do not believe this can be done reliably without corrupting data,
unless a major surgery is performed on the kernel.

Consider:
struct meh {
	func_t *m_func;
	size_t	m_len;
	data	*m_buf;
};

Here we have 'm_func' which needs to be updated, but we don't know if
that's the only pointer so we have to scan the entire struct. But m_buf
can have data which looks like kernel pointers (i.e. matches some kernel
funcs) and how will you know not to modify it? What if this is some sort
of a hack and in fact you *should* modify it?

CTF or some other solutions like that don't help if you just traverse
all allocated buffers since you have no idea what the object you found
really is.

So, assuming this has to be done in runtime, the only somewhat workable
solution I see would require leaving function entry points at contant
addresses and only insert jumps to new locations.

This would be a performance hit and would leave a lot of data at
constant addresses, defeating the point to some extent.

Also runtime relocation of everything definitely has a lot of unknown
unknowns, so all in all I would say this is a non-starter.

One could consider a different approach where kernel data is randomly
shuffled around with some granularity and relevant symbol relocated
prior to booting.

This should provide unique enough layout, which paired with big
likelyhood of a kernel panic on first bad exploit attempt may be an
acceptable solution.

But then the kernel may still have enough info leaks for this to not
matter, so I would not be so eager to implement it.

That said, what prompted this entire effort? Is there an operating
system which got this to work or what?

> >
> >
> >
> > On Fri, Mar 27, 2015 at 3:49 PM, Mateusz Guzik <mjguzik at gmail.com> wrote:
> >>
> >> On Fri, Mar 27, 2015 at 02:35:55PM -0400, Yue Chen wrote:
> >> > When using the following code on kernel module loading:
> >> >
> >> > ------------------------------------------------------------------------------------------
> >> > struct thread *td = kdb_thr_first();
> >> > td = kdb_thr_next(td);
> >> >
> >> > ------------------------------------------------------------------------------------------
> >> > The kernel panics.
> >> >
> >>
> >> Panics how?
> >>
> >> Also you can easily see these functions don't lock anything, so it would
> >> be assumed you took appropriate locks.
> >>
> >> Except it seems there routines are supposed to be only used when
> >> execution is 'frozen' (e.g. when escaped to the debugger).
> >>
> >> >
> >> > And when printing all threads in proc0 (all kernel threads?):
> >> >
> >> > ------------------------------------------------------------------------------------------
> >> > struct proc *p = pfind(0);
> >> > FOREACH_THREAD_IN_PROC(p, td) {
> >> >     uprintf("td: %x\n", td);
> >> > }
> >> >
> >>
> >> proc0 is an exported symbol, no need to pfind.
> >>
> >> > td = curthread;
> >> > uprintf("cur td: %x\n", td);
> >> >
> >> > ------------------------------------------------------------------------------------------
> >> > The ``curthread'' (from this kernel module running the above code) is
> >> > not
> >> > in the 0 process group.
> >> >
> >>
> >> There is no 'curthread from kernel module'.
> >>
> >> My guess is you do this work from module initializator, and in that case
> >> curthread is the thread which loads the module, and such a thread is
> >> definitely not linked into proc0.
> >>
> >> Still nobody knows what you are trying to do.
> >>
> >> --
> >> Mateusz Guzik <mjguzik gmail.com>
> >
> >

-- 
Mateusz Guzik <mjguzik gmail.com>