Generally when we talk about with-profit contracts, we are saying that the benefit is linked to the profit of the company, not the premiums. So, if the company is doing well, your benefit tends to increase. With-profit contracts tend to work like shares in that the policyholder gets allocated a portion of the company's profit to the policyholder's benefit. But, like with the dividend on a share, once paid they can't necessarily be taken back. Although, note, that this all depends on the term of the contract the policyholder has with the insurer, and that the insurer will likely have accounted for any of these terms in the pricing of the product.
In the case of unit-linked contracts: your benefit is linked to the performance of the fund you are linked to. In this case, your benefit can certainly decrease.