Product Overview
This is an overview description of the product the covers basic functionality, key features, configuration options, and possible limitations. These limitations may be temporary and only for the first prototypes and the associated DVT.
IPm Redundancy allows a user to configure two RTUs to act as hot standby units for one another. The user configures a primary controller, chooses to configure a secondary controller, and enables built-in redundancy.
The Tool Kit then creates similar configuration for both units so that when they are configured and run, one controller assumes an active role and the other acts as a backup, monitoring the active controller’s status and assuming the active role if the active controller fails. When that failover occurs, the secondary controller is the active controller and the primary is the backup (though it may be off line and there is no backup until the primary is restored to operation).
The active and backup roles are controlled by heartbeat bits in each controller’s I/O database. The backup monitors the active controller’s heartbeat bit and if it stops toggling for a specified period, it assumes the active role. It is assumed that the heartbeat bit is set by an ISaGRAF program though it could be done by a compiled program which used IODB calls to update the value.
The active/backup status may be monitored in a virtual I/O module (the Redundancy Module) created automatically by the Tool Kit. It is also reflected in the state of the module’s Status LED. The active controller has it on most of the time (on for two seconds, flashed off for 100ms, then repeat) and the backup controller has it off most of the time (off for two seconds, flashed on for 100ms, then repeat).
The Tool Kit also automatically creates I/O transfers to keep the active and backup controller virtual values in sync. The active controller periodically updates the virtual I/O in the backup controller so that if it needs to take over, it has current values.
When IPm Redundancy is enabled, a Redundant Controller Options tab is displayed.
The backup controller may use the same station number as the primary or it may have a unique station number. When both controllers use the same station number, the backup does not respond to that number.
Each controller has its own IP address. ST-IPm controllers have two Ethernet interfaces and have an additional IP address on the second interface. In IPm Redundancy in both VT and ST controllers, a Common IP address may be specified and when configured is used by the active controller. In this way, an outside system such as a SCADA system may use one IP address and always access the active controller, regardless of whether that is the primary or secondary.
Equipment Requirements
- 2 ST-IPm-x350 RTUs
- 2 VT-IPm-x410 RTUs with at least 16MB of NAND flash storage.
- 2 SIXNET ring switches or managed switches with ring protocol.
- 1 or more ET-2 I/O modules with bases
- Enough computers, network interfaces, and switches to connect to both IPms and the I/O ring. This may be as simple as one computer with two addresses bound to the same interface and an external 4-port switch.
- Ethernet cables to connect the equipment listed above
- Power supplies for the equipment listed above
- A way to change physical inputs to the ET-2 (perhaps as simple as looping DOs back to DIs on a mix module).
Tool Kit Testing
Configure an ST-IPm-6530 as follows:
- On the General tab
- Check Automatically configure a secondary controller
- Check Enable built-in redundancy
- On the I/O Modules tab
- Select the Redundancy Module
- Click Configure Module
-
- Select the Discrete Options tab
- Enter 100 for First DO Register
- Select the Discrete Tags tab. Verify that Self_Heartbeat matches the register just configured.
- Click OK to close the Module Configuration dialog.
-
- Click Add New Module
-
- Select Remote I/O Link as the Module type
- Select the Part number based on the ET-2 module you have to test with.
- Click OK.
- Finish configuring the ET-2 module as appropriate for its type. Make sure that the locations used to concentrate the values from the remote I/O link do not conflict with the locations used for the Redundancy Module (the Tool Kit should detect such an overlap and display a warning).
-
- Click Add Virtual I/O
-
- Create at least a couple values of each I/O type and click OK.
-
- On the Advanced tab
- Under Services, check Enable Telnet
- On the Files to Load tab
- On the ISaGRAF tab, pick the Project “4.0 - Redundancy DVT Plan.any.x6m”. This program toggles Y100 as a heartbeat for built-in redundancy.
- On the Redundant Controller Options tab
- Configure a common IP address
- Select Use a unique station number and specify a number.
- On the Virtual I/O Synch tab
-
- Set up output registers to be synced with the backup controller.
-
- Click OK
Repeat the above for a VT-IPm-1410.
Configure the network interfaces on the ET-2 I/O module to be a ring switch.
Save the project next to this test plan document for future reference.
ST-IPm Testing
The ST-IPm RTU has two equivalent Ethernet interfaces, eth0 (Ethernet1) and eth1 (Ethernet2). IPm Redundancy uses eth0 to connect to a ring of ET-2 I/O modules and eth1 to connect to the SCADA system. The test setup looks like:
Network load
Establish a load on the ring network and the IPm network stack by running a constant ping in the Test Access connection to one of the IPms. Leave this running throughout the rest of the tests. On a linux system the following command can be used to generate traffic:
ping –f –i 0.020 <ip address>
The command needs to have root privilege to run with these options.
Basic convergence
- Cycle power to all equipment and let it come to steady state.
- Verify that within 60 seconds, the primary controller is active and the secondary controller is backup
Communications loss between SCADA and RTU
- Cycle power to all equipment and let it come to steady state.
- Disconnect the Ethernet cable between SCADA1 and its IPm.
- Verify that the active/backup status of the IPms does not change.
- Verify that changes to the active controller’s virtual I/O are reflected in the backup controller.
- Reconnect SCADA1 and its IPm
- Verify that the active/backup status of the IPms does not change.
- Disconnect SCADA2 and its IPm
- Verify that the active/backup status of the IPms does not change.
- Verify that changes to the active controller’s virtual I/O are reflected in the backup controller.
I/O Ring Break
- Cycle power to all equipment and let it come to steady state.
- Break the I/O by disconnecting a cable from one ring switch
- Verify that the active/backup status of the IPms does not change.
- Verify that changes to the active controller’s virtual I/O are reflected in the backup controller.
- Verify that changes to the physical I/O in the ET-2 module are reflected in both controllers
- Restore the ring
- Verify that the active/backup status of the IPms does not change.
- Disconnect a cable from the other ring switch
- Verify that the active/backup status of the IPms does not change.
- Verify that changes to the primary controller’s virtual I/O are reflected in the backup controller.
- Verify that changes to the physical I/O in the ET-2 module are reflected in the both controllers
- Restore the ring
Ring switch failure
- Cycle power to all equipment and let it come to steady state. The primary unit should be active.
- Set a value in the primary controller’s virtual I/O and wait a few seconds to give it time to propagate to the backup controller.
- Turn off the ring switch connected to the primary controller
- Verify that the Other_Heartbeat stops toggling in the secondary controller
- Verify that the Self_Active becomes true and Self_Backup becomes false in the secondary controller
- Verify that the Self_Active becomes false and the Self_Backup becomes true in the primary controller
- Verify that data is available on the SCADA system via the common IP address
- Verify that the virtual I/O value set above is in the secondary (now active) controller’s virtual I/O.
- Turn the ring switch back on
- Verify that the Other_Heartbeat starts toggling in the secondary controller
- Verify that the active and backup status of the IPms does not change.
- Verify that data is available via the common IP address
- Turn off the ring switch connected to the secondary controller
- Verify that primary becomes active again.
RTU failure after both start up
- Cycle power to all equipment and let it come to steady state.
- Disconnect power from the primary controller
- Verify that the Other_Heartbeat stops toggling in the secondary controller
- Verify that the Self_Active becomes true and Self_Backup becomes false in the secondary controller
- Verify that data is available via the common IP address
- Set a value in the secondary controller’s virtual I/O.
- Restore power to the primary controller and let it come to a steady state
- Verify that the Other_Heartbeat starts toggling in the secondary controller
- Verify that the Self_Active remains false and Self_Backup remains true in the primary controller
- Verify that the Self_Active remains true and the Self_Backup remains false in the secondary controller
- Verify that the virtual I/O values set above in the secondary controller are visible in the primary controller
- Disconnect power from the secondary controller
- Verify that the Other_Heartbeat stops toggling in the primary controller
- Verify that the primary controller becomes active
- Verify that data is available via the common IP address
- Restore power to the secondary controller
- Verify that the Other_Heartbeat starts toggling in the primary controller
- Verify that the primary controller remains active
Primary RTU failure at power up
- Turn off all equipment. Turn on all but the primary controller. Let the equipment come to steady state.
- Verify that the secondary controller comes up active
- Turn on the primary controller
- Verify that the Other_Heartbeat starts toggling in the secondary controller
- Verify that active and backup status in the IPms does not change
Secondary RTU failure at power up
- Turn off all equipment. Turn on all but the secondary controller. Let the equipment come to steady state.
- Verify that the primary controller comes up active
- Turn on the secondary controller
- Verify that the Other_Heartbeat starts toggling in the primary controller
- Verify that the primary controller remains active
ISaGRAF failure
- Cycle power to all equipment and let it come to steady state.
- Telnet to the primary controller and remove the ISaGRAF program (project).
- Verify that the Self_Heartbeat stops toggling in the primary controller.
- Verify that the Other_Heartbeat stops toggling in the secondary controller
- Verify that the Self_Active becomes true and Self_Backup becomes false in the secondary controller
VT-IPm Testing
The VT-IPm RTU has only one Ethernet interface, eth0, used to connect to the ring of ET-2 I/O modules. The SCADA system and RTU also communicate through the ring switch. The test setup looks like:
Network load
Establish a load on the ring network and the IPm network stack by running a constant ping in the Test Access connection to one of the IPms. Leave this running throughout the rest of the tests.
Basic convergence
- Cycle power to all equipment and let it come to steady state.
- Verify that within 60 seconds, the primary controller is active and the secondary controller is backup
I/O ring break
- Cycle power to all equipment and let it come to steady state.
- Break the I/O by disconnecting a cable from one ring switch
- Verify that the active/backup status of the IPms does not change.
- Verify that changes to the active controller’s virtual I/O are reflected in the backup controller.
- Verify that changes to the physical I/O in the ET-2 module are reflected in both controllers
- Restore the ring
- Verify that the active/backup status of the IPms does not change.
- Disconnect a cable from the other ring switch
- Verify that the active/backup status of the IPms does not change.
- Verify that changes to the primary controller’s virtual I/O are reflected in the backup controller.
- Verify that changes to the physical I/O in the ET-2 module are reflected in the both controllers
- Restore the ring
Ring switch failure
- Cycle power to all equipment and let it come to steady state.
- Set a value in the primary controller’s virtual I/O and wait a few seconds to give it time to propagate to the backup controller.
- Turn off the ring switch connected to the active controller
- Verify that the Other_Heartbeat stops toggling in the secondary controller
- Verify that the Self_Active becomes true and Self_Backup becomes false in the secondary controller
- Verify that the Self_Active becomes false and the Self_Backup becomes true in the primary controller
- Verify that data is available on the SCADA system via the common IP address
- Verify that the virtual I/O value set above is in the secondary (now active) controller’s virtual I/O.
- Turn the ring switch back on
- Verify that the Other_Heartbeat starts toggling in the secondary controller
- Verify that the active and backup status of the IPms does not change.
- Verify that data is available via the common IP address
- Turn off the ring switch connected to the backup controller
- Verify that the active/backup status doesn’t change
RTU failure after both start up
- Cycle power to all equipment and let it come to steady state.
- Disconnect power from the primary controller
- Verify that the Other_Heartbeat stops toggling in the secondary controller
- Verify that the Self_Active becomes true and Self_Backup becomes false in the secondary controller
- Verify that data is available via the common IP address
- Set a value in the secondary controller’s virtual I/O.
- Restore power to the primary controller and let it come to a steady state
- Verify that the Other_Heartbeat starts toggling in the secondary controller
- Verify that the Self_Active remains false and Self_Backup remains true in the primary controller
- Verify that the Self_Active remains true and the Self_Backup remains false in the secondary controller
- Verify that the virtual I/O values set above in the secondary controller are visible in the primary controller
- Disconnect power from the secondary controller
- Verify that the Other_Heartbeat stops toggling in the primary controller
- Verify that the primary controller becomes active
- Verify that data is available via the common IP address
- Restore power to the secondary controller
- Verify that the Other_Heartbeat starts toggling in the primary controller
- Verify that the primary controller remains active
Primary RTU failure at power up
- Turn off all equipment. Turn on all but the primary controller. Let the equipment come to steady state.
- Verify that the secondary controller comes up active
- Turn on the primary controller
- Verify that the Other_Heartbeat starts toggling in the secondary controller
- Verify active and backup status in the IPms does not change
Secondary RTU failure at power up (only one starts up)
- Turn off all equipment. Turn on all but the secondary controller. Let the equipment come to steady state.
- Verify that the primary controller comes up active
- Turn on the secondary controller
- Verify that the Other_Heartbeat starts toggling in the primary controller
- Verify that the primary controller remains active
ISaGRAF failure
- Cycle power to all equipment and let it come to steady state.
- Telnet to the primary controller and remove the ISaGRAF program (project).
- Verify that the Self_Heartbeat stops toggling in the primary controller.
- Verify that the Other_Heartbeat stops toggling in the secondary controller
- Verify that the Self_Active becomes true and Self_Backup becomes false in the secondary controller
Common Station Number
- Configure one of the primary controllers. On the Redundant Controller Options tab
- Select Use the primary controller station number
- Load the configuration into the controllers.
- Cycle power to all equipment and let it come to steady state.
- Verify that within 60 seconds, the primary controller is active and the secondary controller is backup
- Disconnect power from the primary controller
- Verify that the Other_Heartbeat stops toggling in the secondary controller
- Verify that the Self_Active becomes true and Self_Backup becomes false in the secondary controller
- Verify that data is available via the common IP address
- Set a value in the secondary controller’s virtual I/O.
- Restore power to the primary controller and let it come to a steady state
- Verify that the Other_Heartbeat starts toggling in the secondary controller
- Verify that the Self_Active remains false and Self_Backup remains true in the primary controller
- Verify that the Self_Active remains true and the Self_Backup remains false in the secondary controller
- Verify that the virtual I/O values set above in the secondary controller are visible in the primary controller
- Disconnect power from the secondary controller
- Verify that the Other_Heartbeat stops toggling in the primary controller
- Verify that the primary controller becomes active
- Verify that data is available via the common IP address
- Restore power to the secondary controller
- Verify that the Other_Heartbeat starts toggling in the primary controller
- Verify that the primary controller remains active
Disclaimer
It is the customer's responsibility to review the advice provided herein and its applicability to the system. Red Lion makes no representation about specific knowledge of the customer's system or the specific performance of the system. Red Lion is not responsible for any damage to equipment or connected systems. The use of this document is at your own risk. Red Lion standard product warranty applies.
Red Lion Technical Support
If you have any questions or trouble contact Red Lion Technical Support by clicking here or calling 1-877-432-9908.
For more information: http://www.redlion.net/support/policies-statements/warranty-statement