Ages 2 and 4 will have a great time
Your kids will have a lovely time. I am always amazed by what my 2 yr old remembers from holidays and your 4 yr old will remember even more. And, even if they don't remember the trip years from now, they will still have a wonderful time.
Last time I was at Disney World with my 2 yr old, he could go on pretty much anything with the exception of a few obvious rides like Space Mtn and Thunder Mtn. I found that in most cases, the rides he didn't go on, were by my choice (too bumpy or possibly scary) rather than any rule not allowing him. And rides aside, they will love meeting Disney characters and the simple magic of the place.
I definitely recommend either bringing a stroller or borrow one at the park - little legs will inevitably get tired, and they can have a nap if needed along the way.
Have a great time!