You can install mods only on Forge servers and modpacks.
Spigot, Bukkit, Standard Minecraft don't support mods.
To install mods you will need to get .jar files.
Search and download Minecraft Server mods here: https://mods.curse.com/mc-mods/minecraft
As an example, we will describe the JourneyMap mod installation process: https://www.curseforge.com/minecraft/mc-mods/journeymap
In order to upload your mods go to the Your Gameserver → File manager
page, open folder "mods" and click Upload.
Reboot the server to start using the new mods.