You can install mods only on Forge servers and modpacks.
Spigot, Bukkit, Standard Minecraft don't support mods.

To install mods you will need to get .jar files.
Search and download Minecraft Server mods here:
https://mods.curse.com/mc-mods/minecraft

As an example, we will describe the JourneyMap mod installation process: https://www.curseforge.com/minecraft/mc-mods/journeymap.

In order to upload your mods go to the Your Gameserver → File manager page, open folder "mods" and click Upload.
Reboot the server to start using the new mods.