I had this set to "only when installing/removing". The other option is "always (when scanning and installing/removing)".
I had made the assumption that it would only show the scan UI when it found a patch or settings it needed to update/install. I guess I was wrong about that.
Thanks for the tip.