I can't comment on how the newest X5 handles as I've never driven one, but I had a 2006 X5 (last year of the first generation) and it handled really good, easily as good as my X1. I did however have the sports package with the tighter suspension on the X5 while my X1 does not have the sports package or the M package.
However, that said, even with the sports suspension, the X5 ride was much smoother over rough pavement than my non-sport X1. I know it looks like a huge vehicle, but it really is a more advanced vehicle. You get much more for the extra 20-25k than just size. I really do miss the X5 as it was much more of a car than my X1. The only time I don't miss it is when I go to the gas pump. In the end, that is the main reason why I got rid of it. 14 mpg is just not acceptable when gasoline is $3-4 per gallon.
But it really is amazing how well that car handled considering how high it was off the ground and how much it weighed. It really did have sports car like handling around turns. You could lean it into a turn like you would a sports car, and it would stay flat. It's just one of those marvels of BMW. And I'm not just blowing smoke here.....my daily driver for the 3 years prior to getting the X5 was an Acura NSX.
The new one is starting to look a little bloated though, so I'm not sure if it still handles as good as the old one did. The old one was closer to the size of the current X3.