I've been thinking about this question, and I think I'd have to agree. It's one of those Christmass songs which can go either way, for me. Another good version of it is Karen Carpenter's.
Strange how so many of the secular carols are like that. Even "White Christmas" isn't exactly all cheer and joy, reflecting on memories. I love this topic!