It's important to understand, while "mucking about", what affects what.
Primers can be too hot, not not enough, inconsistent themselves, or just inconsistent with the powder choice and case volume you're working with. Generally it's a "tenth" thing, if you aren't already under an inch the primers might not make any difference.
Powder. Well, it's all good powder if it's in the right burn and temperature range for your application. Neck tension, bullet weight, and primer brisance affect the "load" a primer sees, think about a spark-ignition internal combustion engine. More compression, need hotter spark. Heavily-deterred powder (high-octane fuel) needs hotter spark. But you don't want to blow the bullet out of the case with just the primer, before the powder has lit off. The only reasons to tweak powder charges is to reach a target velocity, work around barrel vibration nodes you find while shooting and to reach a "flat spot" in the velocity which is consistent with an ideal combustion ratio (ladder testing to find it). Keep in mind bullet seating depth, primer brisance, firing pin power, and case headspace also affect the combustion ratio and barrel harmonics so you have many options when a need to tweak arises. How to know what to tweak? Look at your targets and what your groups are telling you, once you understand the effect of changing things at the loading bench you will know what to do next.
Bullet seating depth. The closer to the lands you go, the peakier the pressure curve. Experiment. Higher-pressure loads need caution if loading close to the lands. Powder burn rate and yes, primer brisance and neck tension also affect the pressure curve so think about those things too when making choices.
Case fit to the chamber. This is the part about getting the bullet in the middle of the hole in the barrel and is achieved by having a chamber and cartridge that line up with that hole when you're ready to pull the trigger. Reduce all un-necessary tolerances by sizing the brass only as much as necessary to function and often by honing the case neck portion of the sizing die so that it doesn't over-work the brass. Sizing down a neck too much and then cramming a bullet into it WILL stretch the brass of the neck to the side of least resistance, which will not be equal all around the neck, ergo your bullet will not be lined up with the hole when you chamber the cartridge. Redding makes a shoulder-bump neck bushing die that works for all but the hottest loads, if you need a FL die and just run it off the shellholder enough to achieve your minimum headspace goal, RCBS makes an excellent FL die that accepts neck bushings so you only work the necks enough to hold the bullet securely....and gives you the ability to adjust neck tension to observe the effect. This die is a must for cast bullets. Again, the Wilson case gauge is invaluable for setting up your sizing die. The "small base" dies are pretty much useless and cause a lot of problems unless you have a tight chamber and a 55-gallon drum of brass fired from a worn-out SAW, in which case you probably aren't expecting sub-moa from the brass anyway.
Bullet fit. Ever notice all the different jacketed bullet profiles? Lots of choices in 22 caliber, not all will work well in everything. A chamber cast is a good way to make selections, just like it is when choosing a bullet mould. You want the bullet to FIND straight and center when it contacts the throat, not GIMBLE AWAY from center when it does. Think about that, it's very important and you need to know the shapes of each and determine how they will work together. How the bullet and throat interact based on their respective shapes will make or break the bullet balance once it exits the muzzle.
A good barrel? Yep, VERY important. If it will already shoot MOA with factory, you likely have a good one.
