.. _redundant-system-clock-synchronization-89ee23f54fbb: ====================================== Redundant System Clock Synchronization ====================================== .. rubric:: |context| The ``phc2sys`` application can be configured to accept multiple source clock inputs. The quality of these sources are compared to user-defined priority values and the best available source is selected to set the system time. The quality of the configured sources is continuously monitored by ``phc2sys`` application and will select a new best source if the current source degrades or if another source becomes higher quality. The following section describes how to configure ``phc2sys`` with multiple clock sources, and manage and monitor the ``phc2sys`` application. |HA| Phc2sys configuration -------------------------- .. rubric:: |proc| #. Create a ``phc2sys`` instance with **multiple source selection** enabled. .. note:: See the :ref:`ha-phc2sys-cong-param` section for additional information on parameters required for configuration. .. code-block:: none # Set bash variables to simplify commands INSTANCE=phc2sys1 TYPE=phc2sys IFNAME1=phc1-iface1 IFNAME2=phc1-iface2 HOST=controller-0 .. code-block:: none # Create instance system ptp-instance-add $INSTANCE $TYPE # Create interface(s) system ptp-interface-add $IFNAME1 $INSTANCE system ptp-interface-add $IFNAME2 $INSTANCE # Assign host interfaces system host-if-ptp-assign $HOST data0 $IFNAME1 system host-if-ptp-assign $HOST data1 $IFNAME2 #. Assign global instance parameters. The parameters described in :ref:`ha-phc2sys-cong-param` are required for |HA| ``Phc2sys`` functionality. The ``domainNumber`` value can be included if all interfaces use the same ``ptp4l`` domain. If ``ptp4l`` instances use different domains, see the interface configuration options for each interface domain numbers. .. code-block:: none # Assign required global instance parameters system ptp-instance-parameter-add $INSTANCE \ ha_enabled=1 \ ha_phc2sys_com_socket=/var/run/$INSTANCE \ domainNumber=24 #. Assign optional global instance parameters. These parameters determine what clock quality values are required by ``phc2sys`` to consider a source valid for selection. See :ref:`ha-phc2sys-cong-param` for more details on the behaviour of these parameters. .. code-block:: none system ptp-instance-parameter-add $INSTANCE \ ha_max_gm_clockClass=6 #. Assign CLI options. This parameter is required for |HA| ``Phc2sys`` operation. .. code-block:: none system ptp-instance-parameter-add $INSTANCE \ cmdline_opts='-w' #. Assign required interface parameters. The ``ha_domainNumber`` parameter can be omitted if a shared domain number is used and is configured under the global instance parameters. .. code-block:: none system ptp-interface-parameter-add $IFNAME1 \ ha_priority=100 \ ha_uds_address=/var/run/ptp4l-ptp1 \ ha_domainNumber=24 system ptp-interface-parameter-add $IFNAME2 \ ha_priority=90 \ ha_uds_address=/var/run/ptp4l-ptp2 \ ha_domainNumber=25 #. Assign the instance to a host and apply the |PTP| configuration. .. code-block:: none system host-ptp-instance-assign $HOST $INSTANCE system ptp-instance-apply .. _ha-phc2sys-cong-param: |HA| Phc2sys Configuration Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: The parameters listed here can be used in combination with the default ``phc2sys`` parameters. Use the ``phc2sys`` help page for additional parameters. Global Instance Parameters ************************** ``ha_enabled`` The ``ha_enabled`` is a global setting to enable or disable the |HA| multiple ``ptp4l`` inputs support. The default is 0 (disabled). Set 1 to enable |HA| multiple input support. ``ha_stability_timer`` The ``ha_stability_timer`` option is a global setting used to configure the stability timer. The value is expressed in seconds, and the value 0 disables the timer. The default value is 0 (disabled). When the timer is enabled, ``phc2sys`` waits a few seconds after a source becomes a candidate for selection before allowing it to be selected as the synchronization source. This can be useful to prevent frequent source switching if a given source is frequently changing quality. A value of 300 would cause ``phc2sys`` to ensure that a source has had stable quality for 300 seconds before it is eligible to be used as a synchronization source. The stability timer only applies to sources that were in a degraded state and have recovered to meet the selection threshold. If the active source degrades and ``phc2sys`` determines that a new source must be selected, this happens without delay. The ``ha_stability_timer`` is important when the clock has different priorities (ha_priority) to prevent multiple and consecutive clock switches. When the service starts with both clocks in a valid state, the higher priority source is selected, and when it degrades the lower priority is selected quickly. Then when the higher priority recovers it will only become active after the stability timer expires. ``ha_phc2sys_com_socket`` Provides a path for the communication socket used for monitoring and maintenance activities. This value is required for system alarming to check the ``phc2sys`` state. It is recommended to create the socket in the ``/var/run`` directory, for example, ``ha_phc2sys_com_socket=/var/run/`` See :ref: ``manage-phc2sys`` for managing interaction with ``phc2sys`` via this socket. Source Selection Parameters *************************** The parameters in this section are applied at the instance level. For example, ``system ptp-instance-parameter-add ha_max_gm_clockClass=7``. These parameters set the threshold for a source to be considered eligible for use in setting the system time. A source that does not meet one or more of the requirements will not be selected. ``Phc2sys`` then selects the best available source that meets the threshold. The interface parameter ``ha_priority`` is used as a tiebreaker for sources of equal quality, with the higher priority being selected. The default value for each parameter is in effect even if the parameter is not explicitly defined in the configuration file. .. note:: In most cases, users may use the "|GM| clock values" set of parameters to configure the selection threshold. The local clock values are available for additional fine tuning if required, but are generally not a good indicator of the quality of a clock source compared to the |GM| values. .. Cole we need values for GM +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Parameter | Default Value | Range | Description | +=================================+===============+===============+=====================================================================================================================================================================================================================+ | **GM clock values** | | | | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ha_max_gm_clockClass | 6 | 6-255 | The ha_max_gm_clockClass option is a global setting used to configure the maximal GM clock class requirement. It ranges from 6 to 255 and its default is 6. A source with a higher clockClass will not be selected. | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ha_max_gm_clockAccuracy | 0xfe | 0x00-0xff | A source must have a clockAccuracy value equal or lower than this parameter to be selected. | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ha_max_gm_offsetScaledLogVar | 0xffff | 0x0000-0xffff | A source must have an offsetScaledLogVariance value equal or lower than this parameter to be selected. | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ha_gm_timeTraceable | 0 | 0/1 | If this parameter is set to 1, then a source must have the timeTraceable flag set in order to be selected. | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ha_gm_frequencyTraceable | 0 | 0/1 | If this parameter is set to 1, then a source must have the frequencyTraceable flag set in order to be selected. | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | **Local clock values** | | | | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ha_max_local_clockClass | 255 | 6-255 | Same as ha_max_gm_clockClass, but only considers the local clock class, not the class received from the parent time source. | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ha_max_local_clockAccuracy | 0xfe | 0x00-0xff | Only considers the local clockAccuracy. | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ha_max_local_offsetScaledLogVar | 0xffff | 0x0000-0xffff | Only considers the local offsetScaledLogVariance | +---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Interface Parameters ******************** ``ha_uds_address`` The ``ha_uds_address`` is a **required** parameter for each source configured in ``phc2sys``. This provides a path to the associated ``ptp4l`` instance on that source which allows ``phc2sys`` to determine the quality of the source. This value should match the ``uds_address`` parameter of the associated ``ptp4l`` instance. For example, ``ha_uds_address=/var/run/ptp4l-ptp1``. ``ha_priority`` The ``ha_priority`` option is an interface setting used to configure the clock priority. The lowest priority is 0, highest is 254, and the default value is 0. If all clock quality parameters are equal between multiple sources, then the highest priority source is selected. If multiple sources have the same clock quality and the same priority, then ``phc2sys`` will select the first interface in the configuration file. An interface configured with ``ha_priority`` 254 will force ``phc2sys`` to select that source, regardless of its status. If more then one interface is configured with ``ha_priority`` 254 the first interface in the configuration file with priority 254 is selected as clock source. ``ha_domainNumber`` The ``ha_domainNumber`` option is an interface setting to configure the domain number for a |UDS| socket. It ranges from 0 to 127. The ``ha_domainNumber`` has no default value, if the ``ha_domainNumber`` is not configured for a given interface, the global ``domainNumber`` setting is used. This allows for configurations where multiple ``ptp4l`` instances providing clock source quality information may be in different domains. If all ``ptp4l`` instances on the system are in the same domain, it is recommended to use the domainNumber parameter in the global section. Example Configuration File ~~~~~~~~~~~~~~~~~~~~~~~~~~ The following is an example configuration file for |HA| ``Phc2sys`` which shows how services can be configured with multiple interfaces tracking separate ``ptp4l`` instances for quality information. .. code-block:: none [global] ## ## Default Data Set ## domainNumber 24 ha_enabled 1 ha_gm_frequencyTraceable 1 ha_gm_timeTraceable 1 ha_max_gm_clockAccuracy 0x20 ha_max_gm_clockClass 6 ha_max_gm_offsetScaledLogVar 0x4e5d ha_max_local_clockAccuracy 0x20 ha_max_local_clockClass 255 ha_max_local_offsetScaledLogVar 0xffff ha_phc2sys_com_socket /var/run/phc1 ha_stability_timer 60 logging_level 7 message_tag phc1 [enp81s0f1] ## ## Associated interface: enp81s0f1 ## ha_priority 100 ha_uds_address /var/run/ptp4l-ptp11 [enp138s0f0] ## ## Associated interface: enp138s0f0 ## ha_priority 101 ha_uds_address /var/run/ptp4l-ptp12 .. _manage-phc2sys: Manage Phc2sys -------------- The ``ha_phc2sys_com_socket`` displays information about the state of the source selection algorithm, as well as performs test and troubleshooting operations. To perform these operations, locate the socket file for the ``phc2sys`` instance. This should be configured in the global instance parameters for the ``phc2sys`` instance. For example, ``/var/run/``. These commands must be run on the specific host being managed, as access to the socket file is required. For example: .. code-block:: none [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "" | socat -t 1 - UNIX-CONNECT: Informational Commands ~~~~~~~~~~~~~~~~~~~~~~ ``status`` The ``status`` command prints a list of configured sources along with the quality characteristics that have been read from the associated ``ptp4l`` instances. The ``status`` command also indicates which source has been selected. .. code-block:: none [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "status" | socat -t 1 - UNIX-CONNECT:/var/run/phc0 act interface priority clockClass clockAcc offset time freq gm.clockClass gm.clockAcc gm.offset * enp138s0f0 100 248 0xfe 0xffff yes yes 6 0x20 0x4e5d enp81s0f1 100 248 0xfe 0xffff yes yes 6 0x20 0x4e5d ``forced lock`` The ``forced lock`` command prints "False" if ``phc2sys`` has not been forced to select a specific interface via a management operation. It prints "True" if a force lock has been set via a management operation. .. code-block:: none [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "forced lock" | socat -t 1 - UNIX-CONNECT: ``valid sources`` The ``valid sources`` command prints the interfaces that meet the quality threshold for selection. If none of the configured interfaces meet the quality threshold then the string "None" is printed. .. code-block:: none [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "valid sources" | socat -t 1 - UNIX-CONNECT: ``clock source`` The ``clock source`` command returns the current source that is selected by ``phc2sys``. .. code-block:: none [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "clock source" | socat -t 1 - UNIX-CONNECT: Management Operations ~~~~~~~~~~~~~~~~~~~~~ ``enable lock`` The ``enable lock `` command forces ``phc2sys`` to use the given interface as the time source. This operation can be helpful for debugging operations that require the clock source to stay fixed. .. note:: ``phc2sys`` can also be forced to select an interface by configuring it with ``ha_priority=254`` which is preferred if the lock needs to persist between reboots and restarts of the ``phc2sys`` service. ``Phc2sys`` can only be locked to a single interface at a time. .. code-block:: none [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "enable lock enp81s0f0" | socat -t 1 - UNIX-CONNECT: ``disable lock`` The ``disable lock`` command returns ``phc2sys`` to normal operations where it selects the best quality clock source based on the configured thresholds. .. code-block:: none [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "disable lock" | socat -t 1 - UNIX-CONNECT: ``disable source `` The ``disable source `` command removes a given interface from the list of sources considered by ``phc2sys`` for selection. This can be used for troubleshooting and testing by allowing sources to be removed and re-enabled without restarting the ``phc2sys`` service. ``Phc2sys`` will not allow a source to be disabled if it is the only source currently configured and enabled. .. code-block:: none [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "disable source enp81s0f0" | socat -t 1 - UNIX-CONNECT: ``enable source `` The ``enable source `` command returns a previously disabled interface to the selection pool, allowing ``phc2sys`` to consider it for selection based on its quality. .. code-block:: none [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "enable source enp81s0f0" | socat -t 1 - UNIX-CONNECT: |HA| Phc2sys Alarms ------------------- New alarms have been added to report the state of |HA| ``Phc2sys``. To view system alarms on |prod|, use the :command:`fm alarm-list` command. **Phc2sys HA has no source clock** Example EID: ``host=controller-0.phc2sys=phc0.phc2sys=no-source-clock`` The "no source clock" alarm is raised when ``phc2sys`` does not have any sources that meet the specified threshold in the user configuration. This is due to the quality of the available sources not meeting specified thresholds. The alarm clears when an available source meets the specified quality requirements. The ``ha_phc2sys_com_socket`` "status" and "valid sources" commands can be used to assist with troubleshooting this alarm. **phc2sys HA source clock is not locked to a PRC** Example EID: ``host=controller-0.phc2sys=ptp01.interface=enp81s0f1.phc2sys=source-clock-no-prc-lock`` The "source clock is not locked to a PRC" alarm is raised when one of the |HA| ``phc2sys`` interfaces is indicating a clockClass that is of lesser quality than the specified maximum clockClass in the user config. The alarm indicates which interface is degraded. The presence of this alarm should coincide with ``phc2sys`` selecting a different source if a better one is available. The alarm will clear when the clockClass for the source meets the selection threshold. The ``ha_phc2sys_com_socket`` "status" and "valid sources" commands can be used to assist with troubleshooting this alarm. **phc2sys HA has selected a lower priority clock source** Example EID: ``host=controller-0.phc2sys=phc1. phc2sys=source-clock-low-priority`` The "selected lower priority clock source" alarm is raised when ``phc2sys`` is configured with multiple sources and the selected one that is not the highest priority. This alarm indicates that there is degraded quality for the highest priority source and that ``phc2sys`` has taken action to switch. The alarm will be cleared when the highest priority source becomes available for selection. **phc2sys HA automatic source selection has been disabled** Example EID: ``host=controller-0.phc2sys=phc0.phc2sys=forced-clock-selection`` The "automatic source selection disabled" alarm is raised when an operator has used the ``ha_phc2sys_com_socket`` :command:`enable lock `command to force ``phc2sys`` to use a specific source. This alarm is informational and can be cleared by running the :command:`disable lock` command via the ``ha_phc2sys_com_socket``. **phc2sys HA source selection algorithm selected new active source** Example EID: ``host=controller-0.phc2sys=ptp02.interface=enp138s0f0.phc2sys=source-failover`` The "selected new active source" event is raised when ``phc2sys`` switches between sources for any reason. This is not raised as a persistent alarm, but rather as an event, viewable via :command:`fm event-list`. This event enables users to see when source selection events have occurred. There is no action required to clear this event. Phc2sys Source Selection Behaviour ---------------------------------- When ``phc2sys`` is configured with multiple sources and "ha_enabled", a selection algorithm runs repeatedly to determine which source to sync the system time to. The source selection algorithm operates by determining a list of valid sources (i.e. sources that meet the configured selection requirements), and then choosing the source from that list with the highest priority. When ``phc2sys`` detects that a source's quality has changed, the list is re-calculated and a new source is selected if a better one has become available. The source selection operates in the following way: .. code-block:: none For each configured clock source: if local.clockClass > ha_max_local_clockClass -> disqualify source if local.clockAccuracy > ha_max_local_clockAccuracy -> disqualify source if local.offsetScaledLogVariance > ha_max_gm_offsetScaledLogVar -> disqualify source if source.timeTraceable = 0 AND ha_gm_timeTraceable = 1 -> disqualify source if source.frequencyTraceable = 0 and ha_gm_frequencyTraceable = 1 -> disqualify source if gm.clockClass > ha_max_gm_clockClass -> disqualify source if gm.clockAccuracy > ha_max_gm_clockAccuracy -> disqualify source if gm.offsetScaledLogVariance > ha_max_gm_offsetScaledLogVar -> disqualify source From list of qualified sources, select the source with highest "ha_priority". If qualified sources have the same "ha_priority", select the first one as it appears in the ``phc2sys`` config file. If there is no valid source, select the source with lowest "local.clockClass". If "local.clockClass" is tied, select the first source in ``phc2sys`` config file from among the tied sources.