When Japan's biggest banks helped finance a $34 billion deal last year for medical supply maker Medline, one of the largest leveraged buyouts since the financial crisis, the famously cautious lenders signaled their ambitions in riskier, and more lucrative, low-grade U.S. debt.

Mitsubishi UFJ Financial Group Inc., Mizuho Financial Group Inc. and Sumitomo Mitsui Financial Group Inc., eagerly hunting yield abroad after years of zero rates at home, have beefed up U.S. operations and are now targeting business there lending to lower-rated borrowers and underwriting junk bonds.

But their timing — when interest rates are rising and the high-yield debt market is slowing — means they will face increasing risks and dwindling opportunities, testing their staying power.