I think it's more fair to say that the people we know tend to move away... or otherwise vanish.

Then there's the problem of people that are nice enough, but associate with people who... well, aren't. Or we've otherwise got reasons to avoid.