We took DD to Disneyland when she was ~20 months. We took DD and DS to Disney World for the first time when they were 4 and 18mo. We took DD, DS, and DD to Disney World a second time when they were 6, 3, and 1.
Having said all that, I can tell you that kids of almost every age have fun at Disney World. My oldest daughter does still remember things from the trip when she was 4, but a lot of that was spurred on by pictures of the trip and videos I had shown her after the trip and before the next trip. She doesn't recall the Disneyland trip at all. Our son, likewise, didn't recall the first Disneyworld trip. They were both able to soak in and remember the second trip though - our oldest, especially, since she was able to ride most of the rides (especially the roller coasters several times). The youngest, obviously, won't remember it.
So kids of any age have fun, but they'll more than likely to forget about the trip if they are 2 or younger. (Unless you are constantly bringing up pictures, stories, etc.)
Child swap is a great option to have and use - along with Fastpass. It allowed us to split up and ride everything just about anytime we wanted. Our oldest got to go on with us multiple times as well.