"At the same time there was a new feeling of freedom. The visitor was no longer controlled by his own mores and he could disregard Japan's. Exceptions were made for the gaijin [foreigner] who could be expected to know nothing. This freedom included the ultimate liberty of finding everything other than himself — walking down the street, he enjoyed the freedom of being manifestly different."
As the late author Donald Richie noted, to be a foreigner in Japan means to experience a paradox. You experience a kind of freedom living in Japan — of being able to experience your surroundings but not your own self. Your status as a foreigner is reflected back at you by the environment you traverse, and it is not only the more-than-occasional stare that reminds you that you are gaijin — here, even the buildings, tall and proud, loom over you, remonstrating you for your foreignness.
Whilst not experienced all of the time by all gaijin, it does lead some to experience themselves schismatically: There is both the person I think I am and the one that is perceived by others. You have to negotiate with the world in order to receive a self-image you can recognize as your own.