Straight from the horse’s mouth in a Reddit AMA Kevin Feige says Cap was always worthy and was being polite in Age of Ultron.
I would say the same if I were him. But it's a simplified explanation. It is not a detailed one. A detailed one would be something like this: The story of Captain America is a classic coming of age story of a person who tried to prove himself worthy (in general, not to the Mjolnir). Which is why he became a soldier and refused to live a peaceful life (The First Avenger movie). But he made the self-sacrifice. Which is one of the three Mjolnir's feats to complete. In the Age of Ultron he is still trying to prove it. Since he is still trying, he is not there yet. Which is why he could not lift the hammer. In the Endgame:
2) he stopped trying to prove it:
a) "Hail hydra" = "I don't wanna fight and prove it"
b) "I am not Loki" = "I don't wanna fight and prove it"
c) "Yeah I know" = "What an idiot I was, now I finally grew up and I don't wanna fight and prove it"
(Classic 3 times^)
and
3) accepted the family life with Peggy instead of a life of the soldier.
Then he became worthy of the power of Thor.
Why is that? Well, this is the plot of the Thor-1 movie:
1) to make a self-sacrifice
2) to make a choice of a peaceful life over war
3) to become adult = you don't need to prove yourself anymore
But yeah, you can put it simply "he refused to lift it", if you don't wanna go that much into details.
You can even copy-paste this and ask him if this is what the writers meant. Cause this is what they wrote or very very close to it.