Some NSFNET reflections
by Hans-Werner Braun
Co-Principal Investigator (1987-1991), NSFNET Backbone

November 13, 2007

The NSFNET did not start with the Backbone award to Merit (University of Michigan) in November 1987, it was significantly predated by the 56kbps NSFNET backbone built around mid-1986 to interconnect six NSF-funded supercomputing sites, i.e., the five then new supercomputer centers at

  • General Atomics -- San Diego Supercomputer Center, SDSC
  • University of Illinois at Urbana-Champaign -- National Center for Supercomputing Applications, NCSA
  • Carnegie Mellon University -- Pittsburgh Supercomputer Center, PSC
  • Cornell University -- Cornell Theory Center, CTC
  • Princeton University -- John von Neumann National Supercomputer Center, JvNC

plus existing supercomputing facilities at the National Center for Atmospheric Research (NCAR).

But that does not mean that I was not involved prior to my being Co-Principal Investigator on the NSFNET Backbone award to Merit. Two new NSF-funded network infrastructure activities extended to the Computing Center at the University of Michigan, which is where I worked at that time. Those were NCAR's University Satellite Network (USAN) project and SDSC's SDSCnet connection, both via geostationary satellite.

This 1986 photo shows the roof of the Computing Center at the University of Michigan with the USAN satellite antenna on the left, and the SDSCnet antenna on the right.

USAN and SDSCnet antennas at UMich

My being responsible for the Michigan-local site allowed me to attend many related meetings, one of the more important ones being a meeting at NCAR between staff of the supercomputing centers and Dennis Jennings of the NSF, which discussed plans for a national backbone network based on the Internet Protocol. At that time considering IP was somewhat gutsy. The federal government had just issued a mandate not to use IP, but to embrace GOSIP (OSI protocol suite) instead. This made the days of the Internet, with their applications generally confined to a United States Department of Defense network-research context via the ARPAnet, seem to reach their close. Even the network protocol use by the supercomputing centers was inconsistent. SDSCnet embraced the Department of Energy MFEnet protocols, and USAN used Ethernet bridging. Both were problematic, as MFEnet was non-interoperable with other networking protocols, while bridged networks were hard to manage (we had "ARP wars in the sky"). And, for that matter, Merit for it's own network across multiple universities used home grown protocols. Without NSF's decision to embrace the Internet Protocol as the common denominator for an interoperable platform, it is unlikely the the Internet would have developed into the global cyberinfrastructure that it is today, and at least not that early on. My speculation is that instead phone companies would likely dominate the landscape, using X.25 or other OSI protocols for data communications. Plus perhaps there would be islands of non-interoperable network protocols. But, one never knows what would have happened.

56kbps NSFNET backbone

Topology of the original 56kbps NSFNET backbone

Glenn Ricart and I were sitting in at the NCAR meeting with Dennis Jennings, since we were really waiting for a different meeting to happen, and kind of did not have anything better to do. I really misbehaved during the meeting, as I just could not accept the technology the supercomputing centers wanted to use for the 56kbps NSFNET backbone to be. I was thinking they should use LSI11 nodes with Dave Mills' Fuzzball software, as they were the most evolved Internet Protocol engines I knew then. But I was not gutsy enough to specifically mention Fuzzballs. But Glen Ricard did, and to our surprise, the group adopted that notion.

This photo shows the specific machine where almost all the software for the original 56kbps NSFNET backbone was compiled on, before it was transfered to and loaded onto the backbone nodes.

Software compilation machine for 56kbps NSFNET backbone

8 inch floppies from the 56kbps NSFNET backbone

Original 8" floppy disks for the 56kbps NSFNET backbone nodes

Fuzzball-based 56kbps NSFNET backbone node

56kbps NSFNET Fuzzball

The responsibility for the 56kbps NSFNET backbone was awarded by the NSF to UIUC (Ed Krol), with operations being with Cornell University (Alison Brown and Scott Brim). When the links and nodes were deployed, the people responsible had problems making them work, so I made it work, and from then on ran (though inofficially but broadly known) the 56kbps NSFNET backbone via the USAN satellite link from the University of Michigan until 1988, when it was replaced with the backbone network that was part of the new award to Merit. I recall the 56kbps NSFNET time as being extremely stressful to me, because I had no support staff and no one else understanding the infrastructure at its systemic level that I could have talked to. That does not mean others were not helpful. Dave Mills was just great in supporting the software on the backbone nodes (and he should get a lot more credit than he does for much of his pioneering Internet work), and Scott Brim and his group working on statistics and software for a routing processor (which turned into gated), and Ed Krol representing NSFNET centrally and providing reports. But all of that just seemed like components to me. The unpleasant scare was in the system!

The award to Merit via the University of Michigan happened in November 1987, with Eric Aupperle being the Project Director, and me being the Co-Principal Investigator. The award was met by the Internet community with teasing and some outright hostility. Some believed the task at hand was impossible (some wanted to go "trout fishing" rather than us expecting us to succeed, one person told me the project would make me end up with a screaming fit or go quietly insane), others were upset about the IBM and MCI project partners as largely an unknown in the Internet realm. Some of it was perhaps justified.

Earlyon when we talked with MCI about wanting unchanneled T1 (1.5Mbps) links for the infrastructure (rather than multiplexing it down to 56kbps voice channels), they thought we are crazy. When we then told them that a few years later we would want unchanneled DS3 (45Mbps) they thought we are completely insane. But, to their credit, they worked with us through those issues, and IBM made the backbone nodes work, to deliver the new NSFNET backbone on the date promised: July 1, 1988.

Evidence for the delivery of the new backbone are a summary of node reachability on June 11, 1988 to the mailing list (Network Working Group, NWG) that included NSFNET backbone-related staff, NSF, and staff of the to-be-connected regional networks:

 From hwb Sat Jun 11 21:38:26 1988
 Subject: No Trout Fishing, Bob!
 netstat -rn
 Routing tables
 Destination          Gateway              Flags    Refcnt Use        Interface
 129.140.1          UG       0      361        lan0
 129.140.2          UG       0      0          lan0
 129.140.3          UG       0      701        lan0
 129.140.4          UG       0      0          lan0
 129.140.5          UG       0      0          lan0
 129.140.6          UG       0      0          lan0
 129.140.7          UG       0      0          lan0
 129.140.8          UG       0      0          lan0
 129.140.9          UG       0      0          lan0
 129.140.10         UG       0      0          lan0
 129.140.11         UG       0      236        lan0
 129.140.12         UG       0      0          lan0
 129.140.13         UG       0      956        lan0
 129.140.14         UG       0      0          lan0
 129.140.15         UG       0      0          lan0
 129.140.16         UG       0      0          lan0
 129.140.17          U        12     191714     lan0
 129.140.45         UG       0      0          lan0
 129.140.46         UG       0      0          lan0

and the announcement about letting operational traffic onto the network on June 30, 1988:

 Date: Thu, 30 Jun 88 20:02:36 EDT
 From: Hans-Werner Braun 
 Subject: Network to be operational.
 Cc: hwb
 The NSFNET Backbone has reached a state where we would like to more
 officially let operational traffic on. There are some sites who are
 doing that inofficially for a while already. We would like to still
 view this phase as "operational testing" for the near term future
 and will require some maintenance windows to update software and such.
 We will try to have those times generally at off hours and coordinated
 by our Network Operation Center. The routing phasein from our site is 
 coordinated by Susan Hares, who is the Routing Coordinator we proposed
 in the response to the NSFNET solicitation. I would like to encourage
 sites to closely coordinate the phaseins with her and it would be
 extremely helpful to inform her of changes, additions and such. While
 she can be reached at we would prefer if you would send
 updates and questions to (for now) so that
 several people see it and routing problems get resolved sooner (like at 
 off hours or when Susan is not around). Also problems and suggestions
 could for now be sent to the nsfnet-admin email address. In any case,
 Susan will call individual sites to coordinate the phasein of the new
 Generally, problem reports at this point of time should be called in
 to 313-764-9430 which is a Merit receptionist desk. Someone there can
 then route the call to the NOC or wherever else is appropriate. We
 also have set up a mailing list
        -- Hans-Werner

Later in July 1988 I sent email out about discontinuing the old 56kbps Fuzzball-based NSFNET backbone:

 Date: Mon, 18 Jul 88 21:05:44 EDT
 From: Hans-Werner Braun 
 Subject: Cooling Fuzzballs.
 Cc: hwb
 On Wednesday, 20 July 1988, 13:00EDT we would like to physically turn the
 Fuzzballs off to see whether we have remaining issues with regional 
 routing. It is important that they are really off, in case people still
 listen to Hello packets and/or are using static routes. The phone lines will
 then stay for another few days, during which we can turn the Fuzzballs back
 on, if there is a requirement which cannot be resolved immediately. The
 phone lines will, I think, go around the 25th, after which we can turn 
 the Fuzzballs back on, if desired at some sites (as time servers, for
 example). Dave Mills has expressed interest to leave the nodes where they
 are, if only as an Internet research tool and/or servers for accurate time.
 They will obviously NOT be supported by Merit/IBM/MCI at that time, but I
 think Dave Mills will gladly do care and feeding, software wise.
        -- Hans-Werner

Another difficulty around that time was that control of the Internet evolution was pretty much with the United States Department of Defense, via groups such as their Internet Engineering Task Force. To address those concerns, Scott Brim and I met with people in the Pentagon to convince the DoD to at least open up the IETF to a larger community, specifically the NSFNET and its associated regional networks. To our surprise, one meeting was all it took, and they agreed, which lead to a rapid expansion of the IETF with a lot of involvement from many constituents over time.

NSFNET introduced a complexity into the Internet, which the existing network protocols could not handle. Up to the NSFNET, the Internet consisted basically of the ARPAnet, with client networks stubbed off the ARPAnet backbone. I.e., the hierarchy between so-called Autonomous Systems (AS) was linear, with no loops/meshes, with the Exterior Gateway Protocol (EGP) used for for inter-AS routing carrying the AS Number of the routing neighbor. This made it impossible to detect loops in an environment where two or more separate national backbones with multiple interconnections exist, specifically the ARPAnet and the NSFNET. I defined that I needed an additional "previous" AS Number for the inter-AS routing to allow supporting a meshed Internet with many administrations for its components. Meetings with various constituents did not get us anywhere, and I needed it quickly, rather then creating a multi-year research project. In the end, Yakov Rekhter (IBM/NSFNET) and Kirk Lougheed (Cisco) designed a superset of what I needed on three napkins alongside an IETF meeting that included not just the "previous" AS Number but all previous AS numbers that an IP network number route had encountered since its origin. This protocol was called the Border Gateway Protocol (BGP) and versions of it are in use to this day to hold the Internet together. BGP used the Transmission Control Protocol (TCP) to make itself reliable. Use of TCP as well as general "not invented here" caused great problems with the rest of the Internet community, which we somewhat ignored as we had a pressing need, and soon with NSFNET, Cisco and gated implementations at hand, the Internet community did not have much of a choice. Eventually and after long arguments, BGP got adopted by the IETF.

While the 1988 NSFNET backbone ran on a T1 substrate, the links were multiplexed via a switch into three 448kbps circuits. This created a connectivity-richer logical network on top of the physical T1 network. The reason for this were initial performance limits with the IBM backbone nodes, specifically their Almaden line cards, until they were replaced about a year later with full-T1-capable ARTIC cards.

logical initial T1 NSFNET backbone draft

Hand-drawn draft from 1987 about the mapping of the 448kbps circuits into the T1 NSFNET backbone network

A pair of 448kbps Almaden cards from the T1 NSFNET backbone

T1 NSFNET Almaden cards

physical initial T1 NSFNET backbone

Physical topology of the initial T1 NSFNET backbone, before being multiplexed into the 448kbps logical topology

Resulting logical topology of the initial T1 NSFNET backbone

logical initial T1 NSFNET backbone

T1 NSFNET backbone node

T1 NSFNET backbone node at the Network Operations Center with the IDNX switch on the right

Closeup of the T1 NSFNET backbone node at the Network Operations Center

T1 NSFNET backbone node closeup


View of the NSFNET Network Operations Center in the Computing Center building at the University of Michigan. The machine room is seen in the back on the right.

After the transition to the fully-T1-capable ARTIC line cards, the separation between physical and logical topologies was abandoned for obvious reasons, as no multiplexing was necessary any more. Hence the switches were being removed at that time. This email reflects some of the transition strategy, with the numbers in the graphs being node identifiers of the backbone routers:

 Date: Fri, 30 Jun 89 15:44:04 EST
 From: Hans-Werner Braun 
 Message-Id: <>
 Subject: Reconfiguration updates.
 As of the 8 June 1988 we completed Phase A of the reconfiguration, resulting 
 in the following fully redundant T1 topology. The numbers are NSS node numbers.
        +---14         +-------------17----8---+
        |    |         |              |    |   |
        |    |         |              |    |   |
        |    |         |              |    |   |
        |   13---15----7---16---12----5---10   |
        |    |         |         |    |        |
        |    |         |         |    |        |
        |    |         |         |    |        |
        +----6---------+         |   11--------9
             |                   |
 Today the first link of the reconfiguration Phase B was installed:
                  |                   |
        +---14    |    +-------------17----8---+
        |    |    |    |              |    |   |
        |    |    |    |              |    |   |
        |    |    |    |              |    |   |
        |   13---15----7---16---12----5---10   |
        |    |         |         |    |        |
        |    |         |         |    |        |
        |    |         |         |    |        |
        +----6---------+         |   11--------9
             |                   |

 The remaining four steps of the reconfiguration are tentatively
 to begin on the 5 July 1989:

 Step 1         remove   Boulder to San Diego
                install  Boulder to Houston
                  |                   |
        +---14    |    +-------------17----8---+
        |    |    |    |              |    |   |
        |    |    |    |              |    |   |
        |    |    |    |              |    |   |
        |   13---15----7---16---12----5---10   |
        |    |         |         |    |        |
        |    |         |         |    |        |
        |    |         |         |    |        |
        +----6-------------------+   11--------9
                       |              |
 Step 2         remove   Pittsburgh to Houston
                install  San Diego to Houston
                remove   San Diego to Champaign
                install  Seattle to Champaign
                  |                   |
                  |    +-------------17----8---+
                  |    |              |    |   |
                  |    |              |    |   |
                  |    |              |    |   |
        +---13---15----7---16---12----5---10   |
        |    |         |         |             |
       14----|---------|---------+             |
        |    |         |                       |
        +----6         |                       |
             |         |                       |
 Step 3         remove   Ann Arbor to Denver
                install  Ann Arbor to Ithaca
                even exchange   College Park to Princeton
                even exchange   Ann Arbor to Princeton
                (for routing DRS capable Cross Connects)
                  |                   |\   |   |
                  |                   | \  |   |
                  |                   |  \ |   |
        +---13---15----7---16---12----5---10   |
        |    |         |         |             |
       14----|---------|---------+             |
        |    |         |                       |
        |    |         |                       |
        |    |         |                       |
 Step 4         remove   Princeton to Ithaca
                install  Ithaca to College Park
                remove   Ann Arbor to Pittsburgh
                install  Pittsburgh to Princeton
                  |                   |    |   
                  |                   8----|---+   
                  |                   |    |   |
        |    |         |         |             |
       14----|---------|---------+             |
        |    |         |                       |
        |    |         |                       |
        |    |         |                       |

ARTIC card for the non-multiplexed T1 NSFNET backbone


Non-multiplexed T1 NSFNET backbone

Non-multiplexed T1 NSFNET backbone after the 1989 transition

During 1990 the plans for the new 45Mbps (DS3) backbone upgrade matured with additional funding from the National Science Foundation.

 Date: Sun, 9 Dec 90 20:08:42 EST
 From: Hans-Werner Braun 
 Message-Id: <>
 Subject: T3 NSFNET update.
 I learned today that during the IETF meeting a few people expressed concerns
 about the lack of information flow regarding the NSFNET T3 upgrade. There has
 been no real intention not to keep people informed, it is that we are quite
 busy getting the work done and also there have been quite a few open issues
 where things could change very easily. However, given that the concern had been
 expressed, let me try to summarize where we stand today.
 We are in the midst of implementing an eight node T3 network right now. This
 is done as an overlay on top of the T1 network, rather than an integration
 between the two. Lateron we hope to be able to upgrade the remaining T1 node
 sites to T3 and phase out the T1 network.
 The T3 architecture that we implement consists of major nodes (Core Nodal
 Switching System, C-NSS) co-located in MCI junction places. Junctions are
 major Points Of Presence (POP). The intention there is to have a very robust
 central infrastructure. Robustness includes things like availability of
 bandwidth, uninterrupted power and constant availability of operators. This
 central infrastructure forms a two dimensional cloud where the internal
 connectivity may change over time. People will be able to use things like
 traceroute to figure the internal connectivity, but there is no guarantee
 that the connectivity will stay the same over longer periods of time. NSF
 in fact liked this kind of a model and requested from Merit to present the
 network in that fashion to the outside.
 The cloud will initially contain eight C-NSS connected to each other via
 clear channel T3 links. Eight Exterior NSS (E-NSS) at individual sites will
 connect into this cloud. Some C-NSS connect to more than one E-NSS, some C-NSS
 do not connect to any E-NSS and just allow for more robust infrastructure.
 The T3 NSS are based on IBM's RS-6000 technology. Specific cards, which will
 evolve over time, have been developed to connect to the T3 links as well as
 T1 links, Ethernets and FDDI rings. The T3 nodes are rack mounted.
 The C-NSS will be located in the Boston, Washington DC, Cleveland, Chicago (2),
 Houston, Los Angeles and San Francisco areas.  The eight E-NSS will be in
 Boston, Ithaca, Pittsburgh, Ann Arbor, Chicago, Urbana-Champaign, San Diego
 and Palo Alto.
 One of our major obstacles was the early availability of specific cards in the
 nodes. We knew that we could not implement all nodes at the same time and
 therefore had to stage things. We also had to implement a T3 testbed to
 do system integration and testing. The T3 testbed today consists of two C-NSS
 (in Michigan and New York) as well as E-NSS in Ann Arbor, Richardson (TX, MCI)
 and Milford (CT, IBM (T1)). A further node should be operational soon at IBM in
 Yorktown (NY). For the operational network it turned out to be easier (fewer
 sites) to start from the west coast. At this point of time San Diego (SDSC),
 Urbana-Champaign (NCSA) and Ann Arbor (Merit/NOC) have E-NSS working on the
 T3 network and connected to the co-location places in Chicago and Los Angeles.
 The Houston C-NSS is likely to start working this week. There is a chance
 that we will get the C-NSS and the E-NSS in the bay area working before the
 end of the year. That would then allow for half of the T3 sites to be running
 before the year is over. We expect to be able to install the remaining nodes
 in January. Besides Ann Arbor, the first E-NSS on the T3 NSFNET was installed
 at SDSC perhaps a week or two ago and the one at NCSA is working since Friday.
 At this point of time there is still some significant work going on to make
 further improvements to particularely the software in the nodes and, with
 the exception of the node in Ann Arbor, none of the E-NSS on the T3 backbone
 are connected to the external network yet.


 Date: Thu, 20 Dec 90 13:28:07 EST
 From: Hans-Werner Braun 
 Message-Id: <>
 Subject: Update on the T3 status.

 It seems timely to send out an update as to where we are on the T3 network.

 At this point of time E-NSS equipment is in San Diego, Palo Alto,
 Urbana-Champaign and Ann Arbor for the operational network and connected.
 Five C-NSS are in place in the San Francisco, Los Angeles, Houston and
 Chicago areas (two in Chicago). All are up and running on T3 links, except
 that the Palo Alto connection to the C-NSS is still running some T3 tests.

 We have done significant testing for dynamic routing and network management.

 On Tuesday we had the first coast to coast demonstration from an exterior
 workstation at the San Diego Supercomputer Center to a machine in White Plains
 (New York) where the serial links were running at T3 speeds. There was an
 Ethernet in the middle in Ann Arbor. To get coast-coast at this point of
 time we had to utilize our test network which exists between Merit, IBM and
 MCI locations. The workstation at SDSC had an EGP session with the E-NSS and
 we exchanged a "symbolic first file and email message."

 We hope to very soon be in a preproduction mode, followed by initial
 operational traffic on the new T3 network.


Near the end of the year 1990, while I visited SDSC in San Diego, we demonstrated a multi-node 45Mbps backbone capability:

 Date: Sun, 30 Dec 90 23:02:01 EST
 From: Hans-Werner Braun 
 Message-Id: <>
 Subject: I thought this message from Gerard Newman may be of interest...

 Date:    Sun, 30 Dec 90 02:19:27 GMT
 From: gkn@M5.Sdsc.Edu (Gerard K. Newman)
 Subject: T3!
 To:, moreland@M5.Sdsc.Edu, kael@M5.Sdsc.Edu, hwb@M5.Sdsc.Edu
 Organization: San Diego Supercomputer Center

 As of a few minutes ago H-W and I here at SDSC, and Bilal and Elise at
 MERIT managed to get "production" traffic routed over the T3 backbone.

 At present, SDSC is routing traffic for the following nets over the T3

        35              MERIT
        128.174         NCSA
        130.126         NCSA
        140.222         T3 backbone

 (nevermind that the packets take a hop thru a lowly Sun-3/110 -- that
 will be fixed in a better way in the near term).

 To wit:

 y1.gkn [13]% ping
 PING 56 data bytes
 64 bytes from icmp_seq=0. time=81. ms
 64 bytes from icmp_seq=1. time=79. ms
 64 bytes from icmp_seq=2. time=86. ms
 64 bytes from icmp_seq=3. time=89. ms
 64 bytes from icmp_seq=4. time=84. ms
 64 bytes from icmp_seq=5. time=86. ms PING Statistics----
 6 packets transmitted, 6 packets received, 0% packet loss
 round-trip (ms)  min/avg/max = 79/84/89
 y1.gkn [14]% telnet
 Connected to
 Escape character is '^]'.

 Cray UNICOS (u2) (ttyp003)

     For help, send e-mail to or call (217)244-1144
                On nights and weekends, call (217)244-0710

 login: Connection closed by foreign host.
 y1.gkn [15]%

 (traffic between the SDSC Y-MP and the NCSA CRAY-2)


and Elise Gerich, who was on my staff at Merit, sent out a note to more collaborators:

 Date: Mon, 31 Dec 90 15:52:39 EST
 From: Elise Gerich 
 Message-Id: <>
 Subject: Introducing Production Traffic on the T3 NSFNET

 December 31, 1990

 The NSFNET partnership has been in the process of building an
 eight node T3 backbone to complement the T1 NSFNET backbone.
 The T3 end nodes which are currently installed are San Diego,
 Urbana-Champaign, Ann Arbor, and Palo Alto.  We already have
 begun to route some traffic between San Diego, Urbana-Champaign,
 and Ann Arbor, and are in the process of phasing in more
 production traffic.

 We are pleased to announce that the installed end nodes,
 constituting half of the initial T3 NSFNET backbone, are
 ready for operational traffic, with the remaining nodes to
 become operational over the coming weeks.

 Happy New Year everyone!

         The NSFNET Partnership Holiday Crew

I left the University of Michigan and Merit in January 1991 for largely personal reasons, and moved to San Diego to work on network research activities at the San Diego Supercomputer Center.

Some earlier NSFNET background summaries are at and