In the past, the jewelry market was dominated by 3 gemstones, rubies, sapphires and emeralds. These 3 stones have always been the most popular and so the market has always been different for these stones.
When other gemstones started to become popular, jewelers started calling them colored gems. People wanted to be able to distinguish the "big 3" from the "other" colored gemstones because in reality, they could all be called colored gemstones. So the first 3 were called precious gems, and all other gemstones were called semi-precious, or not as precious, as rubies, sapphires and emeralds.
The jewelry industry has stopped using the term semi-precious. Many stones that were considered semi-precious are far more expensive and valuable than the "big 3", and so it would seem they should be more precious.
Basically, the word semi-precious is still used today just because the jewelry industry has not replaced it with another word more fitting.
So don't be fooled. Remember that these lesser known gemstones are no less precious than any other. And anyway, doesn't it come down to personal preference as to what is precious to one person or another?