When is BuildWorld necessary?

[Bob]

Hi:

I recently installed FreeBSD  6.1 over the net from sources. I am keeping 
things up-to-date using CVSup. 

When portaudit tells me I have a security issue; I update/re-install the 
affected port. When a kernel patch comes in, I re-compile the kernel; which 
now stands at FreeBSD 6.1-RELEASE-p6 #3.

From what I can tell, buildworld re-builds the base system, something I have 
yet to do. My thought is to do a buildworld only  when the OS version is 
updated to the next number above 6.1.  I understand this happens at about 4 
month intervals.

My question is, is there a good reason to buildworld before a version change? 
I hate "fixing"  something which is working perfectly, and this system has 
been stellar!
 
Bob