First impression

"At first, when I saw you, I thought you were an haughty smartass."
I've heard that sentence so often, can someone explain?
I know I can look very distant, since when I meet people, especially in a crowed environment, I'm shy and this often passes for arrogance (How the heck is that possible?! I'm not ignoring people when I don't know them, I'm afraid of them and hope they'll ignore me. I start to ignore them when I know them and know that I don't like them, which is quite different, trust me!)
I apparently look like I don't care about people around me (true enough) and because of my attitude (my governess knew her job) people tend to have quite a negative first impression of me, if they don't meet me in an environment I'm comfortable in.
I don't understand.