It's an hour and a half from the state line to downtown Atlanta. It's an hour and ten minutes from my house (Buckhead) to Athens. You have to go around the top end of the perimeter, but that's probably canceled out by not coming all the way into the city and back out. If I didn't think traffic would be a factor, I'd allow three hours, which builds in time to make a quick stop.
Do not expect it to take three hours if you're doing this on Saturday morning, though. Getting into Athens can be a pain in the arse. If I were coming on Saturday morning, I'd plan for the trip to take five hours and consider anything less than that bonus beer time.