It's not speed that must balance: it's momentum. Momentum depends on speed and mass. While bullets have a very small mass and a very fast speed, rifles have a relatively large mass compared to a bullet, so the recoil has a relatively slow speed. The small and fast bullet balances the larger and slower rifle.
If recoil is a law of physics, it must happen every time a rifle is fired, but we know that hunters aren't beaten black and blue by their own rifles. Holding the rifle tightly to the shoulder makes the law of momentum work for you instead of against you. When you hold a rifle this way, you make your body and the rifle act as one unit. Now, the forward momentum of the bullet must equal the backward momentum of the new rifle-person unit. The mass of the rifle and person together is so huge that the speed of the recoil becomes very small.