How do you know which subways are uptown or downtown in NYC?

You see the red stations and green ones, does that matter too? Specifically I need to go from downtown to times square most of the time. I know what trains and everything, just I always forget this bit.