The slight majority of Americans say they are worse off than four years ago, while their assessments of the U.S. economy are subpar and inflation remains a key concern.