External Time Reference (ETR) Requirements for z990
Published 19 June 2003, updated 14 July 2003
Authors: Bill White
The IBM External Time Reference (ETR) architecture facilitates the synchronization of server time-of-day (TOD) clocks to ensure consistent time stamp data across multiple servers and operating systems. The ETR architecture provides a means of synchronizing TOD clocks in different servers with a centralized time reference, which in turn may be set accurately on the basis of an international time standard (External Time Source). The architecture defines a time-signal protocol and a distribution network, called the ETR network, that permits accurate setting, maintenance, and consistency of TOD clocks.
A new function is introduced with the z990 server, implemented in the server's Support Element code, which now requires the ETR Network ID of the attached Sysplex Timer Network to be manually set in the Support Element at installation time. This new function checks that the ETR Network ID being received in the timing signals via each of the server's two ETR ports matches the ETR Network ID manually set in the server's Support Element (SE).
The IBM 9037 Sysplex Timer is a mandatory hardware requirement for a Parallel Sysplex consisting of more than one zSeries or G5/G6 server.
The Sysplex Timer provides the synchronization for the time-of-day (TOD) clocks of multiple servers, and thereby allows events started by different servers to be properly sequenced in time. When multiple servers update the same database, all updates are required to be time stamped in proper sequence.
There are two models of the IBM 9037 Sysplex Timer Unit: Model 1, and Model 2. The zSeries and G5/G6 servers can attach to either an IBM 9037 Model 1 or Model 2 Sysplex Timer Unit.
Note: IBM 9037 Model 1 is withdrawn from marketing. Service support for the IBM 9037 Model 1 will be discontinued year-end 2003.
z990 ETR (Sysplex Timer) attachment
Two optional External Time Reference (ETR) cards are features (feature code 6154) on the
z990 server. These cards, located in the processor cage, provide attachment to the 9037 Sysplex Timer. Each ETR card should connect to a different 9037 Sysplex Timer in an Expanded Availability configuration.
Sysplex Timer ETR Network ID
As part of the installation of a Sysplex Timer in either Basic or Expanded Availability configuration, each IBM 9037 Sysplex Timer Unit is assigned a ETR Network ID (0 to 31decimal) and ETR Unit ID (0 to 31decimal). The ETR network ID and ETR Unit ID values are arbitrary (within the valid range) and can be chosen by the customer to uniquely identify an ETR network and a unquie ETR unit (Sysplex Timer) within the ETR Network.
For example, in a Sysplex Timer Expanded Availability configuration there are two IBM 9037 Sysplex Timer Units. The first IBM 9037 may have a ETR Network ID value of 0, and a ETR Unit ID value of 0. The second IBM 9037 will have the same ETR Network ID value of 0, but a different ETR Unit ID value of 1. The two Sysplex Timer Units in an Expanded Availability configuration must have the same ETR Network ID value defined. However, their ETR Unit ID values must be unique within the Sysplex Timer ETR Network.
A Sysplex Timer Unit’s ETR Network ID, ETR Unit ID, and Port number are transmitted along with timing signals to an attached server’s ETR port. This information is available to sysplex systems running on the server and can be displayed using the D ETR command.
A new function is introduced with the z990 server, implemented in the server’s Support Element code, which now requires the ETR Network ID of the attached Sysplex Timer Network to be manually set in the Support Element at installation time. In addition, on the same panel, the ETR ports have to be enabled for stepping the TOD. This new function checks that the ETR Network ID being received in the timing signals via each of the server’s two ETR ports matches the ETR Network ID manually set in the server’s Support Element (SE).
This function provides greater checking, helping eliminate cabling errors where a server’s ETR port may be incorrectly connected to a Sysplex Timer Unit in an incorrect Sysplex Timer ETR Network, and allows verification of cabling connectivity from the Sysplex Timer to the z990 server prior to IPL of z/OS or OS/390.
If the ETR Network ID received on an ETR ports does not match the value set in the server’s Support Element, that ETR port state is made semi-operational by the server. In the semi-operational state, timing signals are still received by the ETR port, but are not used for stepping the server TOD clock. This has some important operational considerations at system IPL time as well as for running sysplex systems:
- If one server ETR port receives a ETR Network ID which does not match the server defined value, that ETR port state is made semi-operational by the server:
- Systems running with ETRMODE=YES on the server will continue to operate normally using the server TOD clock, which is stepping to the timing signals being received on the second server ETR port. A server hardware error message is generated.
- At each IPL of a system with ETRMODE=YES on the server, the operating system will alert the server of the state of the ETR port as semi-operational, generating a server hardware error message.
- If both server ETR ports receive a ETR Network ID which does not match the server defined value, both ETR port states are made semi-operational by the server:
- Systems with ETRMODE=YES already running on the server will immediately enter a non-restartable disabled wait state. A server hardware error message is generated.
- At each IPL of a system with ETRMODE=YES on the server, the operating system will not be able to enter the sysplex configuration. The customer may choose to continue the IPL in local mode using the server’s local TOD clock. The operating system will alert the server of the state of the ETR ports as semi-operational, generating a server hardware error message.
Important: Changing the ETR Network ID to an incorrect value on a z990 server’s SE will cause all operating systems with ETRMODE=YES on the server to immediately enter a non-restartable disabled wait state.
This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment.
Follow IBM Redbooks
Follow IBM Redbooks