Mixing herbs can be done in two ways; via the inventory or by quick-mixing them into tablets. The latter can be done by pressing RB/R2 plus X (xbox360) or Square (PS3). Mixing in the inventory is done by going to the herb you want to mix and combining it with the second one by selecting it. Always mix Red Herbs with Green Herbs first. Below are all possible combinations:

The maximum amount of pills you can hold in your case are 17. If you have less pills but the herbs mixture would go over your current amount (for example, you have 12 pills and want to use a Green + Red mixture) then the game won’t allow this as you’d go over the maximum amount of pills. Consume a few to use the herbs mixture.