I agree with you that I wouldn't eat GF if I didn't have to do so. But I will say that though I thought I was pretty knowledgeable about food before, I now am much more savvy. When one must read every label, it soon becomes apparent how much junk is in most food!
GF food may have more calories, but I think the body (at least mine) uses it more fully. There are some GF foods with far too much sugar, etc. but most are not filled with long lists of ingredients which come only out of a chem lab.
"It is only with the heart that one sees rightly; what is essential is invisible to the eye."
Antoine de Saint Exupery: The Little Prince
"To the world you may be one person, but to one person you may be the world."