Unless blessed with unlimited time and resources, visiting all the buildings around the world that you would like to see is rather unlikely. Even if you do manage to reach some of them, entrance inside may still be prohibited or restricted.
Time also creates a conundrum in that so many places that were once hard to get to (but were then worth the effort) have lost their charm to all that arrives with mass tourism. For those who like old buildings to be difficult to access, a little run-down and ill-kempt, mossy and embraced by creepers, there are not many left. And if you know of any, please keep them secret — as it is, even the most important temples and castles that do remain standing have already had any lingering atmosphere and mystery exorcised with tourist kitsch.
Just about every old building of any architectural merit in Western Europe has already been scrubbed and spotlighted, and even once-seedy parts of London have seen their 19th-century workers' houses gentrified with window boxes and clipped, potted bay-trees on each side of a freshly painted entrance that's adorned with a brightly polished brass door-knocker and matching carriage lamps. At least the buildings still stand.