This behavior was observed Windows Server 2003 SP2 32-bit and Windows Server 2003 R2 Standard x64 Edition SP2.
A 32-bit Visual C++ program would start and then call RpcServerUseProtseqEp() which succeeded, then RpcServerRegisterIf() which also succeeded, then RpcServerListen(). The latter was called only once during the program runtime. Most of the times RpcServerListen() succeeded but once in a while RpcServerListen() returned RPC_S_ALREADY_LISTENING error code.
Furthermore it was noted that the program would consistently work okay when run under an ordinary domain user account and would consistently fail as described above when run under "Local system" built-in account. That was in a corporate network where only domain users were granted access to the Internet.
Microsoft Professional Support helped with this.
Turned out that before calling RpcServerListen() the program would also load a bunch of third-party DLLs for image processing. One of those DLLs would depend on wmphoto.dll - an image codec shipped with Windows. When wmphoto.dll was loaded Windows would try to validate a certificate of wmphoto.dll. To do that it would attempt a network access to one of Microsoft sites via RPC.
That RPC access would fail in the middle under "Local system" account (because the corporate firewall would not allow Internet access to a program not under a domain user account). That would leave RPC runtime in inconsistent state so that further calls to RpcServerListen() would fail with RPC_S_ALREADY_LISTENING.
If validation was done under a domain user account it would pass and RPC runtime would be left in consistent state and further calls to RpcServerListen() would succeed.
So basically
1. Windows tried to call hookers
2. calling hookers was not allowed without being this tall and married
3. Windows failed to hang up properly and that left RPC runtime in inconsistent state.
There're two ways of dealing with this.
Way 1 - treat RPC_S_ALREADY_LISTENING as "success". That's kind of risky - it broke for a weird reason so counting on this exact behavior can backfire.
Way 2 - use auto-listen interfaces by passing RPC_IF_AUTOLISTEN flag into RpcServerRegisterIfEx() and get rid of StartListening() and friends. That's the recommended way.
A 32-bit Visual C++ program would start and then call RpcServerUseProtseqEp() which succeeded, then RpcServerRegisterIf() which also succeeded, then RpcServerListen(). The latter was called only once during the program runtime. Most of the times RpcServerListen() succeeded but once in a while RpcServerListen() returned RPC_S_ALREADY_LISTENING error code.
Furthermore it was noted that the program would consistently work okay when run under an ordinary domain user account and would consistently fail as described above when run under "Local system" built-in account. That was in a corporate network where only domain users were granted access to the Internet.
Microsoft Professional Support helped with this.
Turned out that before calling RpcServerListen() the program would also load a bunch of third-party DLLs for image processing. One of those DLLs would depend on wmphoto.dll - an image codec shipped with Windows. When wmphoto.dll was loaded Windows would try to validate a certificate of wmphoto.dll. To do that it would attempt a network access to one of Microsoft sites via RPC.
That RPC access would fail in the middle under "Local system" account (because the corporate firewall would not allow Internet access to a program not under a domain user account). That would leave RPC runtime in inconsistent state so that further calls to RpcServerListen() would fail with RPC_S_ALREADY_LISTENING.
If validation was done under a domain user account it would pass and RPC runtime would be left in consistent state and further calls to RpcServerListen() would succeed.
So basically
1. Windows tried to call hookers
2. calling hookers was not allowed without being this tall and married
3. Windows failed to hang up properly and that left RPC runtime in inconsistent state.
There're two ways of dealing with this.
Way 1 - treat RPC_S_ALREADY_LISTENING as "success". That's kind of risky - it broke for a weird reason so counting on this exact behavior can backfire.
Way 2 - use auto-listen interfaces by passing RPC_IF_AUTOLISTEN flag into RpcServerRegisterIfEx() and get rid of StartListening() and friends. That's the recommended way.
