Communications infrastructure for a teraflop computer

Building on the successful ACPMAPS technology, we propose to build a prototype switching network that would lay the groundwork for a next generation ACPMAPS supercomputer. The project can be done now with a minimum budget (~$50,000) and minimal effort (2 FTE for 1 year) using FPGAs and commercial switching chips. The following table lists the prototype specifications in comparison to the final target specifications and the existing ACPMAPS system:
Target System PrototypeCurrent ACPMAPS
Cycle Time:2 to 10 ns 20 ns200 ns
Word Width:64 bits32 bits 32 bits
Bandwidth Per Port:800 to 4000 Mbytes/Sec 200 Mbytes/Sec20 Mbytes/Sec
Number of ports:6420 or 40 16
Total Bandwidth:51.2 to 256 Gbytes/Sec 4 or 8 Gbytes/Sec.32 Gbytes/Sec
Connection Latency:40 ns 80 ns1200 ns
Software user-to-user connection latency: 300 ns500 ns30,000 ns

The prototype switch will consist of the following PC-boards:

Switch Core: Parts $200 Fabrication $1,500 Design effort:1 week Layout: 1 week
Contains an 8-bit slice of the 40-port switch. Five of these will be required for to implement a 40-bit data+control path.
Consists of 1 or 2 I-Cube® switch chips and connectors.

Switch Arbiter: Parts $200 Fabrication $1,500 Design effort 2 Months. Layout: 1-2 weeks.
Services requests from 64 slots and configures switch core.
Consists of a small FPGA and connectors.

CPU card: Parts $2,000 Fabrication $1,500 Design effort4-6 Months Layout: 1 Month.
This is a mock CPU card that interfaces to the switch. Several of these will be required for a system test.
Consists of a port-arbiter chip (FPGA), DMA Queue processor Chip, RAM, and a cheap microprocessor with memory.

Additional efforts:

Mechanical: $10,000 4-6 Months
Substantial effort needs to be spent on mechanical design. This includes design of the cylindrical crate and card guides, cooling, and connector design. The prototype can be built using off-the shelf connectors, but special connectors may be needed to achieve the bandwidth and reliability required by the target system.

System Design: 1 Month
Some time should be spent in meetings with ACPMAPs alumni discussing architecture, hardware and software protocols.

Software: 4 Months
A minimal amount of software needs to be written to do basic functionality and performance tests. This includes porting of a small operating system and writing a hardware interface. It also includes high level software that will simulate the communication requirements of a user-level program.

Conversion from prototype to target:
Several major design efforts are required to convert the prototype system to the target system. Work should be done now to specify these efforts. Major efforts include:

Conversion of FPGAs to higher speed gate arrays.
Development of a switch chip and electrical protocol that can handle sub 10ns cycle times.
Develpment of a crate-to-crate link that can handle up to 4 gigabytes/Second.
Choosing a high speed microprocessor and putting several on a board with large amounts of RAM.
Developing the operating software and user libraries.

It will take approximately 5 full time engineers two years to convert the prototype into the target system.


The following documents were written about two years ago when we last considered this effort. They are in MS-Word format.

System Overview
Details of switch core
Communication processor


For more information, you can contact:
Don Husby, Fermilab.

Back to ESE home page