You may find it helpful to come armed with some information. Even if a doctor has no experience, there's LOADS of information available to them, and if they're made aware of that and your willingness to do some of the footwork yourself, you might be able to avoid getting bounced around with referrals.

My doctors also had no trans experience, so I had a copy of Medical Therapy and Health Maintenance for Transgender Men: A Guide For Health Care Providers printed and spiral bound, and brought that in. That eased their concerns of the unknown, and I think it also inspired some trust--they know that I'm reading the medical lit, not just getting my info from YouTube or something.

There are a number of additional resources for providers here: http://www.trans-health.com/resources/

ps. The Medical Therapy and Health Maintenance guide is an excellent manual for patients as well. I had one spiral bound for myself too.