Sure alloy matters with regards to hardness.
Freshly, cast (air cooled) bullets don't change much, after three weeks. They've reached their hardness potential, for all practical purposes.
I have lubed and sized bullets that are 25-30 years old, sitting on shelves. Size only increased by .001, on average. Had no way to test hardness, way back then. However, testing them now with an LBT tester, they are still pretty much in the expected range of the current similar alloy.