macOS Server 5.3 contains a few traps for the unwary—traps which don’t appear to be mentioned in the release notes

1. It only installs on 10.12.4 (or later, one assumes). This is mentioned in the release notes, but not in the App store notes. NOTE: 10.12.3 and earlier are not supported.
2. It will control a remote server running version 5.2 on macOS 10.12
3. It will not control a remote server, running any version, on OS X 10.11!

This last point is a horrible gotcha! If you are running a server on a previous OS X version—such as because it is on older hardware which can’t be updated to macOS 10.12, and you update to sever 5.3 on another machine, you can no longer control/manage the server instance running on the El Capitan machine.

Server 5.2 and Open Directory

Server 5.2 has its own version issues, or rather Open Directory does.
Server 5.2 will happily run on 10.11 and 10.12. However, if you had a Master/Replica on OS X 10.11/Server 5.2, and you upgrade to a mixture of OS X 10.11/Server 5.2 and macOS 10.12/Server 5.2, the Master/Replica breaks, simply because Open Directory insists (for no good reason that I can see) that they be running the same OS X version!

Server 5.2 can control Server 5.3

If you have a system running Server 5.2, it will happily control remote instances of:

• Server 5.2 on OS X 10.11
• Server 5.2 on macOS 10.12
• Server 5.3 on macOS 10.12