UPDATE: it appears that Apple has fixed this problem in the macOS 13.4 update for Ventura. The macOS Installer.app now asks for permission to access special folders, like Downloads, even if you have denied access on a previous install attempt.
2 Likes