One big consideration, unless you're flying, HHI is a haul from NJ for a long weekend, especially with a 4-year old. Personally, I wouldn't bother making the trip for less than a week and we usually go for two. I remember this trip as being near torture until my twin girls hit age 7, or so. I'm fortunate to have access to a family place in Sea Pines. So, my only experience with HHI is with the plantations (the term has always made me uncomfortable, as well), which I understand to be different from that of the resorts and condos. I definitely recommend a stay at a plantation (not that it wouldn't also be great to stay elsewhere, again I don't know), but that would likely require a weekly rental.
As for things to do, we're ocean people. So, we always spend the bulk of our time on the beach and doing a bit of kayaking, paddleboarding or surfing (on the very rare occasions when it's going-off). The beaches at HHI are exceptionally great for little kids. The hard-packed sand and almost nonexistent sea-level beach incline mean that you can watch your little one play in the water solo (albeit from a reasonably close distance) without fear that she'll go-out with tide.
Related to the hard-packed sand, one of the cool aspects of HHI is that you can easily ride bicycles on the beach. Better, there's an extensive network of bike paths, which enable you to access much of the worthwhile parts of the Island without need of a car. If you go, you should definitely rent bikes with a tow-behind kid-carrier. My girls loved this experience and still wish that I'd be willing to do the peddling for them. In this regard, if you don't stay at a plantation like Sea Pines (which is fairly large with extensive bike paths and contains the well-recognized (but, fake) lighthouse), I believe that you can essentially sneak-in by simply riding-in on the beach (which is essentially public property), provided the plantation is beach-accessible.
With regard to restaurants, I confer with many of the recommendations suggested in that HHI forum posted above. My favorite is Skull Creek Boathouse (I like the ceviche options). My wife prefers Red Fish. Collectively, my family likes breakfast and lunch at Signe's Bakery (the key lime pound cake is a big deal for some, but not me). Nevertheless, quite honestly, the food on HHI is always solid; but, in my opinion, virtually never as good as the purchase price, no different than most restaurants on the Shore.
For better food (and, cultural and historical options), head to Savannah (about an hour away) and Charleston (a little less than two). Savannah boasts traditional gems like Mrs. Wilkes (fried chicken lunch) and the Olde Pink (not-to-be-missed Americana, can be truly sublime). Charleston is, as many are aware, a foodie capitol, with Husk et al. Honestly, unless you've already taken-in both Savannah and Charleston, you'd be missing two true treasures if you went to HHI and never took-in the two amazing, easily-accessible day-trips.
Just my two cents, either way, GO! I've never known HHI to disappoint. Your family will love it!
