I figured brass to hold heat better
Nope.
A cursory look at a specific heat of metals chart reveals that yellow brass has a specific heat of .096 BTU per pound-°F while aluminum (elemental) is .220 and iron is .110. That means that brass takes less energy to warm up, but also will lose that heat just as easily (quickly).
I find that brass moulds have to be run at a much faster pace than aluminum or iron, for an equivalent mass, surface area, bullet volume, etc. The loss of heat to the air is much faster with brass, so pouring "heat" into it has to be a lot faster to compensate, and the heat loss is even greater when the blocks are separated as in when you're fiddling around trying to get the bullets to drop. Brass blocks are constantly trying to rob heat from the source away from the steel pins, so that's working against you too. The only solution is to flood the system with hot lead, and keep it flooded. Lightly-frosted ww alloy with sharp edges and no big voids at the tip of the HP pin takes at least 3.5 pours per minute (four is better) at 725°F pot temperature with most brass moulds to keep them happy.
Get a cheap analog wall clock with a second hand and put it on the wall behind your casting station and time your work. That will give you a frame of reference to this conversation, and a reference to yourself for future work with each mould and alloy combination. You may wish to keep a log for each mould, noting alloy, humidity, ambient temperature, alloy temperature, pour pace in pours per minute over at least a ten-minute period, and so forth.