More than 30 financial institutions in six countries have been defrauded by sophisticated criminal software that convinces bank customers to install rogue smartphone programs, a major security company reported Tuesday.