Well, lets see...
Go to a go-cart track, line up enough money to run about 40 laps, non stop. Then check and see if you neck, your arms, your torso isnt tired, heavy and/or sore.
Now...
Think about a NASCAR driver, on an oval course diving into turn 1 or turn 3 at or about 180-200mph, the side loads reach somewhere around 3-4 g's. They do this for 3-4 hours with the only breaks being caution laps. Why is it, do you think, that they have seats built with a rest/gaurd on the right side?
On another level...
F1...fighter planes on wheels.
Ever watched an F1 race broadcast and seen the little G meter on the screen? It will go from neutral to left 1.5-2 g's, to negative 2 g's, to positive 1.5-2 g's all in about 2seconds (of course, depending on where they are racing).
Do you think they are athletic? Imo, DUHHHHH, the HAVE to be, forces such as that placed on an unprepared human body can become threatening, not only to you, but to the racers around you, and your pit-crew.
Seriously, the only driver (successfull) I know of that doesnt really fit the "athletic profile is Tony Stewart. But, Tony has been a big ol' boy all his life. His body has adjusted to the requirements of his chosen profession and his driving style have allowed him to be as "athletic" as is required.
Again, if you want to know if they are athletes, go to a go-cart track and drive non-stop for about 40 laps, oval, road-course, it makes no difference. Then, ask yourself that question again.