HH, I do take your point that we seem to have created problems for ourselves.
I think one big factor is that whereas letterboxing is a fairly unpublicized, one might almost say underground, activity, caching has been widely publicized by the media. I'm not saying that's a bad thing in itself, but it has created a much wider awareness of caching. The word treasure is often used in that publicity, and I think that to many people it conjures up the idea of buried treasure, which in turn suggests that we have no respect for people's land or for the environment.
I don't think permission is an issue, or at least not a major one. GAGB's revised agreement with the Forestry Commission in the New Forest has been widely discussed recently, but I believe that letterboxers also have an agreement with them, which has been in place for some time.
Do not go gentle into that good night.
Rage, rage against the dying of the light. (Dylan Thomas)