Visible minorities in Japan are in a tough spot in a country where the police have a lot of arbitrary power and few enforceable checks (as we've been witnessing recently with the Carlos Ghosn case). As a result, we are facing two decades of police-promoted narratives of "the foreigner" as a visa overstayer and criminal.
What follows is my advice on what to do if you face a sudden ID check on the street — that is, assuming you don't want to simply surrender your zairyū kādo (residence card) and eventually get on with your day. This is just a brief outline, you can find more details online at debito.org/whattodoif.html.
Ask why you are being stopped: Ask if this is a "shokumu shitsumon" (police questioning of personal details). If yes, the law requires probable cause that a crime has been or is about to be committed, and the display of POLICE ID upon your request. If it is not, ask if you may leave.
With your current subscription plan you can comment on stories. However, before writing your first comment, please create a display name in the Profile section of your subscriber account page.