I always go through the week, I've never had to buy Fast Lane. For the most part the wait times are minimal and usually its more of a walk on status.
You're absolutely right. I normally only go on Tuesdays or Wednesdays since most people are at work or school.