This problem is mostly visible when having Gnome/xorg modular installed, as if so, a buch of ports are getting installed. I think the ports/package managment in FreeBSD is not very scalable in this sense. This has nothing to do with xorg. Maybe this motivates some change in the port system. Working with ~20'000 software is not very easy. good luck!