Hats should be removed indoors, unless the headgear is part of a costume. Ask yourself: Is it Halloween? Am I at a science fiction convention? If the answers to these questions are no, then the hat comes off indoors.
Unfortunately, this means most of the time you shouldn’t wear a hat, unless you know you’ll be outdoors nearly all the time. Because in public places, there’s nowhere to put the hat when you’re inside, except for leaving it on your head, which is barbaric. And that’s too bad, because hats are excellent.
In the Mad Men days and before, there were hatcheck girls and hat racks for gentlemen and ladies to leave their hats while indoors. But they aren’t around anymore.
Ideally, ball caps should also be removed indoors. But it’s OK to leave them on. It’s no big deal. They’re just caps.
Knit caps should only be worn outdoors, and only during the coldest weather, never for decorative purposes. They are not decorative, they’re functional, for warmth. Worn inside, they make you look like a hipster idiot.