Sapphires are the most common Gem. They may be found at random times (see below), gathered from fragments in the Gem Mine and looted from the gem bags dropped by the Gem Goblin.
The primary uses for gems are to upgrade tools, craft keys to unlock Chests or simply to sell.