Troubleshooting RPC over HTTPS (Part 1)

RPC over HTTPS is one of the most useful features of Exchange 2003/Outlook 2003. It is also one of the most frustrating as there are numerous issues that come up causing the connection to fail.

Introduction

In the past remote users where forced to use a VPN to connect Outlook to the corporate Exchange servers or be forced to use the limited features available in Outlook Web Access. With the release of Exchange 2003 and Outlook 2003 a new connectivity option was introduced: RPC over HTTPS. RPC over HTTPS tunnels remote procedure calls through an HTTPS connection allowing you to connect to the Exchange server when outside the corporate LAN without needing to establish a VPN connection. To understand how to troubleshoot issues, you need to be aware of what is going on when an RPC connection is made. Figure 1 shows a typical RPC over HTTPS scenario, and you can refer to it as we go over the steps required to make a connection.


Figure 1: RPC over HTTPS Scenario

  1. The client computer resolves the DNS name of the RPC Proxy server and connects via SSL using Internet Explorer to process the certificate and the client computer authenticates to the RPC Proxy server.
  2. The RPC Proxy server connects to the Exchange 2003 back-end server and establishes a TCP session with the destination computer
  3. The RPC Proxy server passes the remote users credentials to be authorized by the Active Directory domain and logs the remote user on to Exchange.

In order to troubleshoot RPC over HTTPS issues there are a few steps to take that solve the most common issues:

  • Verify the Outlook 2003 configuration
  • Verify the firewall configuration
  • Verify the SSL certificate authority trust (part 2)
  • Verify the RPC Proxy configuration (part 2)

Verifying the Outlook 2003 Configuration

In my experience one of the most common issues has to do with the client configuration. Often the issue is as simple as Outlook not using basic authentication or mis-configured server names and connection options. One common issue is that the remote user does not have the correct version of Outlook installed or they are running the wrong base OS. In order to establish an RPC over HTTPS connection the client computer must meet these minimum requirements:

  • Outlook 2003 (Service Packs are optional but recommended)
  • Windows XP SP1 (SP2 is optional but recommended)

Often, administrators have to let remote users configure the profile themselves and something as simple as a typo or a missed checkbox is often the cause. To verify the settings:

  1. Browse to the Control Panel and launch the mail applet.
  2. Click on Email Accounts to start the configuration wizard.
  3. Click the View or change existing e-mail accounts radio button and press Next.
  4. On the e-mail accounts page, select the RPC over HTTPS profile and press Change.
  5. On the Exchange server settings page, click on More settings.
  6. Switch to the Connections tab.
  7. Ensure the box next to Connect to my Exchange mailbox using HTTP is checked and then press the Exchange Proxy Settings button.

On this screen (see Figure 2) we want to ensure that the following settings are correct:

  • URL to connect to my proxy server for Exchange is the correct fully qualified domain name (FQDN) used to access the RPC proxy from outside your organization
  • Connect using SSL only checkbox is checked
  • Mutually authenticate the session when connecting with SSL box is checked
  • Principal name for proxy server is the FQDN used to access the RPC proxy server from outside your organization
  • The Principal Name is preceded by msstd:
  • Basic Authentication is selected as the proxy authentication method
  • Check On fast networks, connect using HTTP first, then connect using TCP/IP (optional, but recommended)
  • Check On slow networks, connect using HTTP first, then connect using TCP/IP (optional, but recommended)


Figure 2: Outlook Connection Settings

If you have chosen to use a wildcard SSL certificate, you should enter the URL as follows in the Principal name for proxy server dialog box:

msstd:*.yourdomain.tld

Two other common issues that occur on the client side appear when a user enters a UPN (username@yourdomain.tld) is that they constantly get prompted for credentials and Outlook 2003 performance issues/hang ups. Both of these issues are easily resolved by upgrading the client to Windows XP SP2.

Outlook 2003 allows you to test the RPC connection by launching Outlook from the command line with the following command:

Outlook.exe /rpcdiag

You will see a window appear with connection information (see Figure 3), which will display HTTPS as the connection type if you have connected via RPC over HTTPS.


Figure 3: Outlook.exe /rpcdiag

Finally, Outlook 2003 SP1 is known to disable the Exchange over the Internet area on the Connection tab of the Microsoft Exchange Server setup in the mail profile. This will prevent you from configuring or editing the RPC profile. If this is disabled on your client, open up the registry with regedit and drill down to:

HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\RPC

Create a REG_DWORD called EnableRPCtunnelingUI and set the value to 1.

Verifying the firewall configuration

If you are lucky enough to be using an ISA 2004 server as your firewall, configuring RPC over HTTPS becomes a whole lot easier.  ISA 2004 has built in support for RPC over HTTPS connections, which allows you to easily create an access rule to allow this traffic.  However there is some confusion when you publish your Exchange server in ISA 2004.  A lot of times, administrators believe enabling RPC over HTTPS access is as simple as checking the box on the services selection page (see Figure 3).


Figure 4: ISA RPC Access Rule

In fact this is not the method used to publish RPC over HTTPS.  Microsoft has an in-depth article on publishing RPC over HTTPS with ISA 2004 which is linked at the end of this article.  It is possible to configure RPC over HTTPS on ISA 2000 and you can find out how to accomplish this task in the links provided at the end of this article.

Conclusion

So far we have looked at some common client side issues and validated our firewall configuration. Part two will look at issues that occur with SSL certificates as well as mis-configured RPC proxy server and Exchange server settings.