Much has been made of the alleged difficulty of the Japanese language for non-native and even native speakers. My personal impression is that this view is most commonly cherished by two types of people: those who don't know much Japanese (or any at all), and those who only know Japanese.
Regardless, it is definitely too simplistic to characterize Japanese as a language more difficult than other languages. To start with, its sound system is incredibly simple (14 consonants and five vowels, with very strict rules on their combination), and the basic grammar is so elegantly uncomplicated that it's hard not to wonder why all other languages couldn't be as simple. Maybe the world would be a better place.
Sound and syntax are not everything there is to it, and this brings us to an area where those who champion the difficulty of Japanese certainly have a point: the writing system, especially kanji. There are so many characters and it can be a challenge to know when to use the right one. A particularly tricky issue is the so-called ijidōkun (異字同訓, lit. "different character, same reading") problem. These are words that can be written with more than one character depending on what meaning is intended to be communicated.