Carter's ineptitude in reaction to foreign events sealed his fate.
Domestically, he wan't that all bad, but the Iranian Revolution and Soviet invasion of Afghanistan were events he bungled. The Three Mile Island incident also struck a blow to his energy policies. Then he went and led the boycott of the Olympics and any popularity had eroded completely. I will say, he found his perfect role post presidency and has almost erased his legacy as a poor president and become a world wide respected advocate for human rights and treatment of disease. Not to mention Habitat for Humanity.
I'll also say that in time, when viewed outside the toxicity of the current political environment, Obama's presidency will be viewed in a much more positive light. Yeah, there will be that whole Bin Laden thing, but that's just blue meat fodder that the red meat eaters should love as well. I think his drone policy, inaction on mass data collection of citizens, and inaction on pressing the issue on the torture findings will be stains. The ACA, while imperfect, will eventually be viewed as a necessary first step toward a more complete healthcare reform (should have happened with Slick Willy and Newt when both sides could hold their noses and actually work together on some stuff). In terms of LGBT rights, he should have pushed it sooner, but he will be the president that was in office when a huge leap was taken. Jobs are coming back after an economic disaster, though incomes still lag. The stock market is back to all time highs. Sure, the president doesn't exactly influence that directly, but it does respond to policies pushed by the government. He will be the president that began to drag the nation into legitimate climate change awareness, which I think will be one of the currently unpopular (for half the electorate) issues that he will be respected for down the line. Also, there is a much more favorable view of America in the majority of the world as we do things like open up to Cuba and attempt negotiation first with countries like Iran instead of shooting first. Three or four presidents down the line, I think he'll be viewed as a general middle of the pack president. Did some good things, did some not so good things, but overall an ok president, with the historical significance of being the first black president. It's just really hard to judge a president through current political climate shaded lenses.
Also... color me surprised by the tone of this discussion. Well done SPS.