While finasteride aims to block the production of DHT (DHT is a more potent male hormone than testosterone for hair loss) in hair follicles, unintentionally, it might affect the production of DHT in other parts of the body (e.g. sexual organs). It is generally believed that lower DHT contributes to the sexual side effects. Having said that, the role of “nocebo” effect (i.e. you experience a side effect because you think you might get it) has been investigated and cannot be ruled out.
Reduced conversion of testosterone to DHT by finasteride could lead to an increased conversion of testosterone to estrogen (“female” hormone), resulting in breast tenderness and enlargement in men.
If the side effects are deemed too severe, you could consider alternative forms of hair loss treatment.