If a game also exists on last gen and handheld systems, how does it hold the PS4 version back?

This is something I never quite got. Destiny, Persona 5, and Senran Kagura Estival Versus are a few examples whose later games (Destiny 2 and SK PBS) look better from not being on slower hardware.

Then there's the usual thread I hear where "the game has to run on both consoles" but that doesn't make sense either, because the PS4 has neither a Vita card slot nor the ability to read PS3 discs, and PS4 discs can't be run on either machine either.

I get releasing a game then later porting it to new hardware (Gravity Rush Remastered and both God Eaters), but why didn't most developers target the PS4 during development and downscale everything to the Vita from there (World of Final Fantasy, Sword Art Online Hollow Realization)?

Then there's games like Destiny (pre-The Taken King, that is), and in those cases I just don't get it at all (maybe it has to do with the online infrastructure that game had?).

This is just something I'm legit curious about.