Averaging One

There are a number of sites that allow users to rate things (books, recipes, hotels, etc.). I've noticed many of them simply sort by average rating in descending order regardless of the number of votes. This is a bad way to do it, because there's a much higher degree of confidence surrounding a 4.9 average coming from 50 votes than there is a 5.0 "average" resulting from a single vote.

So what's a person to do?

I have some ideas, but this has become yet another puzzling bee buzzing around in my head.