The software your server runs decides what you can add to it later. Picking it is the first real decision, and changing it afterwards is disruptive — so it is worth two minutes now.
Why it happens
There are two separate worlds. Plugins run on the server only and players join with an unmodified client. Mods change the game itself and every player has to install the same set. Nothing bridges the two properly.
What to do
- Paper — plugins, and the default answer
A performance-focused fork of Spigot. It runs Bukkit and Spigot plugins, fixes a long list of vanilla inefficiencies and exposes far more tuning. If you want a survival or minigame server that friends join with a normal client, this is the choice.
- Fabric — mods, light and current
A modern modding platform that updates quickly after each Minecraft release. Good for performance mods and smaller mod sets. Players need the Fabric loader and the same mods you run.
- Forge — mods, and the big packs
The older, heavier ecosystem, and where most large modpacks live. If you want All The Mods or a similar pack, you want Forge (or NeoForge, which many newer packs have moved to).
- Vanilla — the reference, rarely the right pick
Mojang's own server. It accepts neither plugins nor mods, and it is measurably slower than Paper at the same job. Choose it only when you specifically need behaviour identical to single-player.
Your choice matches your goal: plugins and unmodified clients means Paper; a modpack means Forge or Fabric depending on the pack.
Spigot and Purpur are both valid too — Spigot is what Paper is built on, and Purpur is built on Paper with more configuration again. Anything written for Spigot generally runs on all three.