Having a fetish for hats is something that I can understand. Hats have up until recently been falling out of fashion. In recent years they have become a little more common on both men and women again but they are still the exception, rather than the rule they once were.
It is their very unusualness that adds to their effect for me. While hats in everyday life are still relatively uncommon in comparison to the middle of the last century wearing one by definition implies a special event or something outside the run of the mill. The theatricality of hats also has something to do with it I suppose.
Seeing someone in a hat is like seeing a colleague who you’ve never seen outside a work environment in their eveningwear for the first time. It changes your perception of them and you could regard it as a kind of role play scenario where your perceptions roles and relationships change based purely on the garments you wear and the makeup they wear.