You can delete custom folders. You cannot delete system folders.
When you delete a folder, its contents are moved to the Trash. You can recover contents from the Trash by dragging and dropping them to another folder.
The Trash folder is purged on a regular basis, so items in the Trash do not stay there forever.
If there are messages in the folder that you want to save, move them to another folder.
Either drag the folder to the Trash or right-click the folder and choose Delete.