There is a lot of confusion over the term "life expectancy." One often hears that in the pre-modern world life expectancy was 35, with the assumption that people just keeled over in their mid 30s. Actually, they did not.
This figure of 35 is a modern guess, based on assumptions about the "life span." Span and expectancy are not the same. The span is how long one could reasonably live, barring accident and serious disease, before body parts start wearing out. The average "span" (and remember, this is just an average) is genetic, fairly fixed for humans, and has generally been assumed to be about 70.
Thus, making the assumption that half of all babies died in infancy in the Middle Ages, modern people have assumed that the other half lived to 70, and thus, on average (70 plus zero, divided by two), people could "expect" to live to 35. But this was not in fact the case.
In the modern West, good nutrition and medical care have meant that more people can "expect" to live out the full human life-span, which is probably closer to 80 or 90 than 70. But as we all know, plenty of people die a lot younger, and some live longer. That's why it's an average.
In the Middle Ages, child mortality was certainly higher than it is now but was never half. War, accidents, and (for women) childbirth, as well as the diseases that can easily be cured with modern antibiotics, carried off a lot of people in what we would call the prime of life. Particularly deadly were the two outbreaks of the Black Death, in the sixth century and again in the fourteenth. A few people, very few, did live until 90 or so. But for most, as we can tell from the lives of well-documented individuals, the "expectancy" was probably to live into one's 50s, before the rough life lived by even aristocrats caught up with them.
This is of course not uniquely medieval. Even though my own grandfather lived to 107, his own grandparents did not make it out of their 50s. Their obituary (they died within days of each other) said that they were "quite advanced in years, being near 60."
(Click here for more on growing old in the Middle Ages.)