Change it.
Most of the debris resulting from the wear of hundreds of engine and transmission component surfaces gets generated in the first few hundred miles. Also, small metal chips lost parts from machining and assembly operations and excess gasket materials show up on the filter screen during this time, so clean that off too.
You don't want that stuff floating around causing additional wear and damage to precision parts and clogging small orifices. That's why they tell you 600 miles, not "whenever you get around to it."
20,000 miles, no problem ... don't you just love the internet?