There are two ways to connect your phone, Bluetooth and Car Play\Android Auto.
With BT, you can specify in the BT settings for each connection whether or not to sync contacts, and on iOS, all, recents or just Favorites. You can even specify specific contact lists. Not sure about Android.
With Car Play, no control over what syncs. Again, not sure about Android.