Pretty much the entire trip's "gist" will be lost on a 1-year old. Colonial Williamsburg is a great family trip -- as long as your kids understand why it exists. Enjoy the buildings/tours there for yourselves. One of my faves was the Governer's Mansion tour. Theres lots of walking in Williamsburg, so be sure to bring everything you need to keep yourselves hydrated!
Busch Gardens with a one-year old -- I don't know how much you'll enjoy there. There's nothing they can do there...so one of you will be enjoying attractions, while the other one waits by the stroller. Really, unless you are theme park enthusiasts who absolutely MUST go, skip it with a one year old.
Enjoy the history and beauty of colonial Virginia, and tailor the trip to yourselves to enjoy while bringing along your baby.