Visible minorities in Japan are in a tough spot in a country where the police have a lot of arbitrary power and few enforceable checks (as we've been witnessing recently with the Carlos Ghosn case). As a result, we are facing two decades of police-promoted narratives of "the foreigner" as a visa overstayer and criminal.
What follows is my advice on what to do if you face a sudden ID check on the street — that is, assuming you don't want to simply surrender your zairyū kādo (residence card) and eventually get on with your day. This is just a brief outline, you can find more details online at debito.org/whattodoif.html.
Ask why you are being stopped: Ask if this is a "shokumu shitsumon" (police questioning of personal details). If yes, the law requires probable cause that a crime has been or is about to be committed, and the display of POLICE ID upon your request. If it is not, ask if you may leave.