Season 3 when they were in that Panamanian prison was by far the worst imo. Season 4 was OK, knowing it was the last season made it easier to stick with it.
Just like "Lost", season 1 was so good that you had to keep watching, hoping there would be an episode which is as good as what they did in the first season.