Japan's financial watchdog has started examining high-frequency trading, algorithms and dark pools, asking brokers for information about the practices.