Testing of the HP G-Link Chip Set for an Event Builder Application Osamu Sasaki1, Jeffry Andresen, Hector Gonzalez, Masaharu Nomachi1, and Ed Barsotti Fermi National Accelerator Laboratory Document # ESE-DAQ-940115 Batavia, IL. 6051 Abstract The Hewlett Packard HDMP-1000 G-Link transmitter and receiver chip set was tested for an event builder application. The re-lock time of the serial link when a data path is changed is less than 30 µs provided fill frames are transmitted. With a 1 kHz data path switching rate, this results in less than a 3% data rate inefficiency due to re- synchronization, which makes the HP G-Link chip set effective for use in a large event builder for a hadron collider experiment. I. INTRODUCTION A large high energy physics detector requires a large scale data acquisition system which could require several thousand data links running at a speed of several tens of MBytes per second per link. A transparent switch network with global traffic control being used as an event builder for such a large experiment has already been proposed [1]. The event builder receives event data fragments from many sources via serial links. The transparent switch performs the event reconstruction by the switching of the serial data links through the use of global traffic control. One of the most important hardware parameters of an event builder switch is the time that is taken to re-configure the switch connections and to recover a synchronized clock to a new data stream. This re-synchronization time needs to be a small percentage of time in comparison to the switching interval to achieve sufficient data throughput. The Hewlett Packard HDMP-1000 G-Link chip set was selected to be tested as it one of the candidates for a serial data link protocol chip set [2]. The purpose of this test was to measure the re-lock time of the HP G-Link chip set and to understand what other design parameters have to be satisfied to build a switch based on using G-Links. The HDMP-1000 consists of a HDMP-1002 (transmitter) and a HDMP-1004 (receiver). The transmitter serializes either 16 bits or 20 bits of parallel data, adds 4 coding bits, and transmits the data at a serial speed as high as 1.4 Gbps. The G-Link receiver converts the serial data to its original parallel form. The G- Link chip set has three kinds of frames (data sets) which are a data frame, a control frame, and a fill frame. The transmitter sends fill frames if Data Available (DAV*) and Control Word Available (CAV*) are false or if Enable Data (ED) is false. Data frames and control frames are treated in the same way by the G-Link chip set. Fill frames are sent by the transmitter at start up, whenever data frames or control frames are not being sent by the user, and whenever there is an error condition. A fill frame has the same duration as a data frame or a control frame with one master transition to allow the receiver to acquire frequency lock (both frame synchronization and bit synchronization). Once the receiver has frequency lock, data transmission can begin. The 4-bit coding field that is sent with the data has one master rising/falling edge that the receiver's Phase Lock Loop (PLL) circuitry uses to maintain bit synchronization as it recovers the clock from the serial data stream. This phase lock has a narrow frequency detection range. If phase lock is lost, fill frames are transmitted until re-lock can occur through the use of frequency lock. It should be noted that the receiver can never be re-locked when data or control frames are being received. The G-Link system maintains DC balance on the serial data line by inverting the transmitted data or control frame whenever necessary. Figure 1 is HP's example of a simplex G-Link configuration. Figure 1: HP's Simplex Configuration Example II. TESTS A. Testing Hardware This testing utilized a HDMP-1000 G-Link evaluation board simulating a simplex configuration [3]. In the following tests, all serial signals were transmitted over 50½ SMA coaxial cable. The G-Link TX device on the evaluation board was clocked from the STRBIN input by an external pulse generator at 42 MHz. This frequency corresponds to a 1 Gbps bit stream when the 20 bit data transfer mode is selected. The KEK PECL 4X4 switch [1] was used to receive a 1 Gbps bit stream from the HP HDMP-1002 TX device. PECL fanout/level adapter boards were used to adapt the G- Link logic levels to PECL levels since the serial outputs from the TX are not ECL but are buffer line logic (BLL) [2]. A clock divider board and a NIM signal discriminator provided a 1 kHz NIM level pulse that was used to change the KEK switch configuration. The bit stream exited the switch and was sent to the HP HDMP-1004 RX device. A 20 bit random data generator board provided a random data pattern when data was being transmitted. The eye patterns of the serial bit stream and the receiver strobe out signal were observed on a sequential sampling oscilloscope. The re-lock time was measured by observing the Error and Link Ready signals from the RX on a digital oscilloscope. B. Random Data Pattern Test A 20-bit random data pattern, along with the 4 coding bits that the G-Link generates, was transmitted to the switch at 1 Gbps. As can be seen in Fig. 2, the oscilloscope eye pattern clearly shows the 4 coding bits along with the data bits. Figure 2: Coding and Data Bit Eye Pattern The coding bit field has a master transition of a single rising/falling edge which serves as a fixed timing reference for the G-Link receiver's clock recover circuit. For this test, in order to send a continuous random data pattern regardless of the lock condition, DAV* and ED were always enabled through the use of jumpers. In a non-switching mode, the G- Link chip set transmitted and received the 20-bit data pattern without losing lock. When the switch was in its 1 kHz switching mode, the G-Link receiver would lose lock and never re-lock. For the RX G-Link to synchronize with the TX G-Link when only one clock source is used in the simplex configuration as shown in Fig. 1, the G-Link system requires the TX to send fill frames to the RX G-Link. C. Fill Frame Phase Shift Test In this test, the G-Link evaluation board continuously transmitted and received fill frames. The phase of the serial signal was changed along with the timing of the changing of the switch in relation to the fill frame. The top scope trace in Fig. 3 shows the G-Link fill frame. Figure 3: G-Link Fill Frame There are two types of fill frames [4]. At startup, fill frame FF0 is transmitted which has a single falling edge in the data field going from a high D9 to a low D10. Once frequency lock occurs, fill frames FF1L and FF1H are transmitted. With the FF1 fill frames, the position of the falling edge in the data field is shifted forward or backward by one bit. This is accomplished by toggling data bits D9 and D10. FF1L transmits zeros for D9 and D10, and FF1H transmits ones for D9 and D10. The transmitter sends either FF1L or FF1H to reduce the cumulative serial DC offset. FF0 maintains DC balance as it is a square wave with a 50% duty cycle. The rising edge of the fill frame is used by the receiver to achieve frequency lock. The lower scope trace in Fig. 3 is the receiver's strobe output (RSTRBOUT). This is the clock that has been recovered from the receiver's serial input. The TX serial link was connected to the input of a PECL fanout board. Figure 4 is a block diagram of the test setup. Figure 4: Cable Delay Test Configuration The PECL KEK 4X4 switch was used as a "2 to 1 SELECTOR" which changed the signal path every millisecond. The serial data cable lengths from the fanout to the switch were varied to create signal delays. The delay was incremented in steps of 5 ns for the serial signal from the fanout. Delays in 4 ns increments were added to the 1 kHz signal for the selector switching to change the point in the fill frame at which the changing of the switch occurs. There are two parameters affecting the re-lock time in the configuration of Fig. 4. The first is the switch changing the signal path which shifts the phase of the serial signal. The second parameter is where within the fill frame the changing of the switch occurs. The re-lock times were measured as a function of these two parameters. Figure 5 is an example of a re-lock time reading. Figure 5: G-Link Re-Lock Time To begin the test, all serial data cables were the same 1 meter length. The G-Link evaluation board operated without losing lock which indicated that the switching did not create errors when the phase difference between the two serial signals is zero. The switching from a 0 ns delay to a 24 ns serial data delay results in a zero phase shift as the frame has a 24 ns period. The system did not lose lock when this 24 ns delay was tested. The system did lose lock as the 5 ns delay steps where introduced. The re-lock time was as short as 8.1 µs and as long as 27.4 µs. Adding delay to the 1 kHz NIM signal did not affect the maximum re-lock time. Figure 6 is a graph of the re-lock time in relationship to the delay time. Figure 6: Re-Lock Time vs. Phase Change Figure 6 shows that if the clock is in the correct phase, the G-Link operates without losing lock. The negative time readings were when the switch went to a shorter cable length. The maximum re-lock time did not occur when the phase difference was the greatest. The -5 ns and 20 ns points are sensitive points in that either a small re-lock time or the maximum time occurs at these two points. The -10 ns and 15 ns delays also produced two different re-lock times, but the difference between the times was smaller. The re-lock times shown in Fig. 6 follow a pattern that repeated in succeeding frames. D. Fill Frame Frequency Change Test Figure 7 is a diagram of the frequency change test. Figure 7: Frequency Change Test This test did not use the G-Link TX device. The first clock generator was connected to one input of the switch with a fixed frequency of 42 MHz. A second clock generator was connected to another input of the switch with a frequency that was changed in incremental steps on both sides of the 42 MHz frequency of fo. Since the G-Link RX device recovers its clock from the serial stream, the 32 MHz to 58 MHz frequencies were chosen to be in the same frequency range as the fill frame rate that the RX can receive. In other words, the two clock generators transmit similar serial signals as that of the fill frame signals of the TX. The G-Link RX device did not lose lock when the two switch inputs were at the same frequency of 42 MHz without a phase difference. Figure 8 shows that as the difference in frequency increased, the re-lock time increased. Figure 8: Frequency Change Re-Lock Time The sequence of switching from fo to f or from f to fo resulted in the same re-lock times. By tying the f input of the selector low, the time for the RX's PLL circuit to re-lock from its disabled state was measured. The result was 1.8 ms. E. Simplex Dual Clock Data Pattern Test In the simplex configuration with one clock generator as shown in Fig. 1, when RX unexpectedly loses lock, RX can not re-lock without receiving fill frame signals from the TX. This configuration does not provide the TX a way of knowing that RX lock has been lost and that fill frames need to be transmitted. Figure 9 is a diagram of a simplex data pattern test with dual clocks that was done. The RX in this dual clock simplex configuration has the ability to recover the lock by itself. Figure 9: Simplex Dual Clock Data Pattern Test In this test, a 1 Gbps data pattern (a data frame) was sent from the TX G-Link to the RX G-Link. The TX and the RX have dual ports for the serial signal. The Loopen signal controls whether the Dout or Lout output and the Din or Lin input are currently enabled. When Stat1 is low, Dout and Din are active. Clk2 provides the frequency pattern which simulates a fill frame pattern for the receiver to lock upon. When lock occurs, Stat1 goes high which activates Lin instead of Din. The TX Lock output was connected directly to the TX Loopen input. A fixed precision crystal clock was used for the transmitter clock. A HP 8110A pulse generator was used as the variable receiver clock as it has better than 0.1% stability, period steps of 10 ps, and duty cycle steps of 0.1%. The fixed clock was disabled and then enabled with a second pulse generator. When the TX clock is disabled, TX Lock and TX Loopen go low which disables the Lout and Lout* outputs. This results in the RX losing lock and switching to the Din input. The RX clock then supplies a fill frame like clock for the receiver to re-lock upon. Fixed clock frequencies of 40.0000 MHz and 25.002 MHz were tested with Fig. 10 showing the 40.0000 MHz re-lock times. The 25.002 MHz test produced similar results. Figure 10: Simplex Test Re-Lock Times Figure 10 indicates that the two clocks need to be within 0.1% of having the same clock frequencies. This result confirms HP's specification [6] and agrees with the test that was done at LBL [5]. The points at the top of the graph are actually points going beyond the graph indicating that re- lock did not occur at those test points. When the same clock generator was used for the transmitter and receiver in the Fig. 9 setup, there was no difference in frequencies with re- lock never occurring. When the two different clocks were within a 0.1% difference in clock frequency, the G-Link regained lock provided the duty cycle of the variable RX clock was 50%. However, there were duty cycles that would cause the RX G-Link to never re-lock. Figure 11 is a diagram of the test setup that was used to test the duty cycle of the receiver's clock. Figure 11: Simplex Receiver's Clock Duty Cycle Test The HP 8110A pulse generator provided a stable clock at the selected test frequencies. The duty cycle of the clock was changed in 0.1% steps from a 20% duty cycle to a 80% duty cycle. Figure 12 shows the frequencies that were tested and the duty cycle ranges in which the RX G-Link would not re- lock. Figure 12: Duty Cycle Error Ranges of G-Link RX Clock The test results in Fig. 12 show that there is a narrow duty cycle range on each side of 50% for each frequency in which the G-Link receiver will not re-lock. When this same test was repeated except that the duty cycle of the G-Link transmitter clock was varied, the G-Link never lost lock. The G-Link transmitter compensates for its clock not having a 50% duty cycle. The inability to re-lock occurred in 16-bit mode and 20-bit mode and is when the G-Link is used in the two clock simplex mode. This is important as the duty cycle of a clock used in an actual G-Link data system may not be exactly 50%. Further testing needs to be done on newer versions of the G-Link chip, set including the HDMP-1012 transmitter and HDMP-1014 receiver, to verify if the re-lock time is still sensitive to the clock duty cycle. III. Conclusion In an event builder switch setup, it is required that a serial data link system have the ability to regain lock within a reasonable time on the change of the switch configuration. Our tests demonstrate that the G-Link can re-lock in less than 30 µs provided fill frames are sent. In the application of the event builder, the single clock simplex mode can be used if the TXs are forced to send fill frame signals on each change of the switch configuration. With an expected minimum switching interval of 1 ms, devoting a period of 30 µs to send fill frames to assure re-lock would result in only a 3% data throughput inefficiency. In the dual-clock simplex mode, re- lock performance is critically dependent on relative clock frequencies and the receiver's clock duty cycle, which makes this mode unsuitable for the event-builder application. IV. Acknowledgments We would like to thank J. Butler, Prof. S. Iwata, T.K. Ohska, and Y. Watase for their support. V. References [1] O. Sasaki, M. Nomachi, T.K. Ohska and H. Fujii, "A VME Barrel Shifter System for Event Reconstruction for up to 3 Gbps Signal Trains", IEEE trans. Nucl. Sci., NS-40 (1993) 603. [2] "Gigabit Rate Transmit Receive Chip Set Technical Data", Hewlett Packard. [3] "Reference Guide for the Gigabit Rate Transmit/Receive Chip Set (HDMP-1000) Evaluation Board", Hewlett Packard, October 1992. [4] "G-Link: A Chipset for Gigabit-Rate Data Communication", Hewlett-Packard Journal, October 1992. [5] B. Turko, M. Wong, "Measurements With External Clock At Receiver To Re-Establish Lock in Simplex Mode", LBL, March 1993. [6] "Low Cost Gigabit Rate Transmit/Receive Chip Set", Hewlett Packard, November 1993. 1 National Laboratory for High Energy Physics (KEK) Tsukuba, Ibaraki-Ken, 305 Japan