Your server must support mods.
You can install mods on Forge servers and modpacks.
You can't install mods on Spigot, Bukkit, Standard Minecraft.

To install mods you will need to get .jar files.
Search and download Minecraft Server mods here:
https://minecraft.curseforge.com/mc-mods
https://mods.curse.com/mc-mods/minecraft

As an example, we will describe the Iron Chests mod installation process: https://minecraft.curseforge.com/projects/iron-chests.

Follow the Guide (How to Edit Files Using FileZilla FTP) to connect via FTP by using your Game Server login details.
You can find your login details on the Game Server page (see the "FTP Username" section).



After connection is successfully established, you will see two sections. Files stored on your PC are displayed on the left, and those stored on the server are displayed on the right. By using the left section, navigate to a folder where your downloaded .jar mod files are saved. By using the right section, navigate to the server/mods folder. Next, copy the mod files from the left section (your PC) to the right section (Server).

Reboot the server to start using the new mods.