CRM & ADFS woes (the connector story)

So I have been doing some ADFS 3.0 work as of late. Part of this work included getting CRM 2016 to use ADFS for authentication instead of the normal AD (IWA) approach. Getting the user side of it set up is not that hard, and there are many good sources for correct information. Now it turns out CRM has some hidden integration complexity. Though not directly from CRM itself, but the Connector for Microsoft Dynamics (which has not changed since CRM 2011). This was making a stink and would not connect to ADFS. At the time I did not know that the error was caused by the connector not being able to talk to ADFS server because of a firewall issue, so I wrongfully assumed that the error had something to do with ADFS configuration. This analysis was done on internal federation configuration and not using IDF (published external using ADFS Proxy), i.e using the Intranet configuration for the CRM RP with Windows Authentication only active. read more