Colors look better for me when my astigmatism isn’t bothering me.
Do you play poker? A big part of the game of poker is figuring out what cards the other player has by “putting a player on a hand”. My guess is that when you’re looking at something unclear your brain can’t decide quite what it is. It might be a T, might be an F, might be a P. If it can’t decide at all what you’re looking at it may just be a blur. If it thinks you’re looking at F or P you might see one or the other but not very clearly. If you can see it clearly enough that your brain decides it’s certainly P and not F, then it probably presents you with a clear image of P that maybe looks bigger and brighter. The letter on the eye chart hasn’t changed any, but your perception of it has.
At some level your brain doesn’t know exactly what’s printed there on the eye chart and like a poker player it’s trying to use all the information it can gather to make the best possible guess. Eventually there’s enough information that to decide “he has a Pair of Aces” or “the letter on the eye chart is P and not F”. If you get close enough to the eye chart it’s like turning over the cards and revealing the hand.
Anyway, I certainly don’t know how it all works, that’s my best guess as to what’s going on. Once you’ve figured out what a letter is with enough certainty is you probably still see it clearly even if you move a little further back to a place where you couldn’t see it clearly in the first place.