I would suggest you look to the ASSRA for advice on better shooting loads for that rifle. Those guys shoot groups with single shot rifles using cast that many of us can only dream of.
Bullet weight isn't you problem. Pinholes aren't either. Not at 50 yards.
...and they do it with bullet weight variances several times the magnitude being discussed.
There is a difference between precision in the places that matter, and the places that it doesn't matter so much.
"I would suggest asking Ben what he does. He shoots lots of very small groups at 50 yards with a wide variety of bullets and rifles."
Yep. And not to diminish Ben in any way, but you'll notice a lot of rounded bands, small voids on band edges, and other inconsistencies in his photos of ready-to-load bullets. He also shoots a lot of one-hole groups at 80 yards using cheap Lee moulds he plain-based himself with an ordinary fractional drill bit on a drill press. No DRO. No Last Word indicator. No 4-jaw chuck. Just Mark 1 Eyeball.
Think about that. Look at his groups shot from old, tired, warhorse rifles. From cheap Savage department store barrels. From Lee moulds. From scrap alloy.
How does he do it?
He understands how to make certain things fit and work together to get good groups.
He doesn't mess around trying to get precision where it isn't needed. I think Ben would get a chuckle if you asked him what his weight bell curve is, or even what his max +/-, or if he even weight sorts or has any need to know what his bullet mould temperatures run when casting.
I'm not saying that these things are unimportant on some levels, because indeed they are, BUT, at 50 yards and cat-sneeze velocities, fractional-percent weight variation and microscopic pinholes in your castings ARE NOT IMPORTANT.
Three things are of critical importance to what you're doing: Match the alloy to the pressure (or vice-versa), make your bullets all release the case in exactly the same manner, and learn how to manage static/dynamic bullet fit.