Then you stop the interaction and it’s ultimately harmless. Should we pass up any and all opportunities to have meaningful interaction with people just on the chance that they don’t want to be bothered?

Generally, you pass on the situations where it’s likely that they want to be left alone. For most people, it’s not hard to tell.