Why do we think that pulsars, at least those that spin the fastest (i.e. the millisecond pulsars), are neutron stars?