- Part two -
Oh, here you are. It turned out quickly, didn't it? With just a click or tap on the screen, if you have a 21st century connection, you are instantly on this page.
But how does it work? Have you ever thought about how a picture of a cat gets to your computer in London from a server in Oregon? We're not just talking about the wonders of TCP / IP or the ubiquitous Wi-Fi hotspots, although these are all important too. No, we're talking about large infrastructure: huge submarine cables, vast data centers with all their redundancy of power systems, and giant, labyrinthine networks that directly connect billions of people to the Internet.
Perhaps more importantly, as we increasingly rely on ubiquitous connectivity to the Internet, the number of connected devices is growing, and our thirst for traffic knows no boundaries. How do we make the Internet work? How do Verizon and Virgin (the largest Internet service providers in the US - approx. New) manage to consistently transfer one hundred million bytes of data to your home every second, around the clock, every day?
Well, after reading the next seven thousand words, you will know about it.
Secret places of exit of cables on land
British Telecom (BT) can lure customers with the promise of fiber to every home (FTTH) for faster speeds, and Virgin Media has good service quality - up to 200Mbps for individuals thanks to its hybrid fiber-coaxial (GVC) network … But, as the name suggests, the World Wide Web is truly a worldwide network. Ensuring the Internet is beyond the power of one single provider on our island, or indeed anywhere in the world.
Promotional video:
First of all, we will for once look at one of the most unusual and interesting cables that carry data, and how it reaches the British coast. We are not talking about any ordinary wires between ground data centers a hundred kilometers apart, but about a contact station in a mysterious place on the west coast of England, where, after a 6500 kilometers journey from American New Jersey, the Atlantic submarine cable Tata ends.
A US connection is essential for any major international communications company, and Tata's Global Network (TGN) is the only single-owner fiber network around the planet. This is 700 thousand kilometers of submarine and terrestrial cables with more than 400 communication nodes around the world.
Tata, however, is willing to share. It does not exist just so that the children of the director can play Call of Duty without delay, but a select group can watch Game of Thrones online without delay. Tata's Tier 1 network accounts for 24% of the world's Internet traffic every second, so the chance to get to know TGN-A (Atlantic), TGN-WER (Western Europe) and their cable friends is not to be missed.
The station itself - quite a classic data center in appearance, gray and nondescript - may generally seem like a place where, for example, cabbage is grown. But inside, everything is different: to move around the building you need RFID cards, to enter the premises of the data center - give your fingerprint to be read, but first - a cup of tea and a conversation in the conference room. This is not your usual data center, and some things need to be explained. In particular, submarine cable systems require a lot of energy, which is provided by numerous standby units.
Protected submarine cables
Carl Osborne, Tata's VP of Worldwide Networking, joined us on the tour to share his thoughts. Before Tata, Osborne worked on the ship laying the cable and oversaw the process. He showed us samples of submarine cables, demonstrating how their design changes with depth. The closer you are to the surface, the more protective sheathing will be needed to withstand potential shipping damage. Trenches are dug in shallow water where cables are laid. However, at greater depths, as in the Western European Basin with a depth of almost five and a half kilometers, protection is not required - commercial shipping does not threaten the cables at the bottom.
At this depth, the cable diameter is only 17 mm, it is like a felt-tip pen in a thick insulating polyethylene sheath. The copper conductor is surrounded by a plurality of steel wires that protect the fiber optic core, which is embedded in a steel tube less than three millimeters in diameter in soft thixotropic jelly. The shielded cables are the same internally, but in addition are clad with one or more layers of galvanized steel wire wrapped around the entire cable.
Without a copper conductor, there would be no submarine cable. Fiber optic technology is fast and can carry almost limitless amounts of data, but fiber cannot operate over long distances without a little help. To enhance light transmission along the entire length of a fiber-optic cable, repeater devices are needed - in fact, signal amplifiers. On land, this is easily done with local electricity, but at the ocean floor, the amplifiers draw direct current from the copper conductor of the cable. And where does this current come from? From stations at both ends of the cable.
While consumers don't know this, TGN-A is actually two cables running across the ocean in different ways. If one is damaged, the other will provide continuity of communication. The alternative TGN-A lands 110 kilometers (and three ground amplifiers) from the main one and gets its energy from there. One of these transatlantic cables has 148 amplifiers, while the other, longer one, has 149.
Station leaders try to avoid publicity, so I'll call our station guide John. John explains how the system works:
“To power the cable, there is a positive voltage at our end, but in New Jersey it is negative. We try to maintain the current: voltage can easily bump into resistance on the cable. A voltage of about 9 thousand volts is divided between the two ends. This is called bipolar feeding. So about 4,500 volts from each end. Under normal conditions, we could keep the entire cable running without any help from the United States."
Needless to say, the amplifiers are built to last 25 years without interruption, since no one will send divers down to change contact. But looking at the sample of the cable itself, inside which there are only eight optical fibers, it is impossible not to think that with all these efforts there must be something more.
“Everything is limited by the size of the amplifiers. Eight fiber pairs require amplifiers twice the size,”explains John. And the more amplifiers, the more energy is needed.
At the station, the eight wires that make up the TGN-A form four pairs, each containing a receive fiber and a transmit fiber. Each wire is painted in a different color, so that in the event of a breakdown and the need for repairs at sea, technicians can understand how to reassemble everything in its original state. Likewise, onshore workers can figure out what to insert when connected to a subsea line terminal (SLTE).
Repair of cables at sea
After touring the station, I spoke with Peter Jamieson, Fiber Support at Virgin Media, to learn more about making submarine cables work.
“As soon as the cable is found and brought to the ship for repair, a new piece of undamaged cable is installed. The remote-controlled device then returns to the bottom, finds the other end of the cable, and makes a connection. Then the cable is buried into the bottom for a maximum of one and a half meters using a high-pressure water jet, he says.
“Usually, the repair takes about ten days from the date of departure of the repair vessel, of which four to five days are work directly at the site of the breakdown. Fortunately, this is rare: Virgin Media has only encountered two in the past seven years.”
QAM, DWDM, QPSK …
With cables and amplifiers in place - likely for decades - nothing else in the ocean can be adjusted. Bandwidth, latency and everything related to quality of service is regulated at the stations.
“Forward error correction is used to understand the signal being sent, and modulation techniques have changed as the amount of traffic carried by the signal increased,” says Osborne. “QPSK (Quadrature Phase Shift Keying) and BPSK (Binary Phase Shift Keying), sometimes called PRK (Double Phase Shift Keying), or 2PSK, are long range modulation techniques. 16QAM (Quadrature Amplitude Modulation) would be used in shorter submarine cable systems, and 8QAM technology is being developed, intermediate between 16QAM and BPSK.
DWDM (Dense Wavelength Division Multiplexing) technology is used to combine different data channels and transmit these signals at different frequencies - through light in a specific color spectrum - over fiber optic cable. In fact, it forms many virtual fiber optic links. This increases the fiber throughput dramatically.
Today, each of the four pairs has a bandwidth of 10 Tbit / s and can reach 40 Tbit / s in a TGN-A cable. At the time, 8 Tbps was the maximum potential available on this Tata cable. As new users begin to use the system, they use spare capacity, but this will not make us impoverished: the system still has 80% of the potential, and in the coming years, with the help of another new coding or increased multiplexing, it will almost certainly be possible to increase throughput.
One of the main problems affecting the application of photonic communication lines is dispersion in optical fibers. This is what the designers consider when designing the cable, since some sections of the fiber have positive dispersion and some have negative dispersion. And if you need to make repairs, you need to be sure to have a cable with the right type of dispersion on hand. On land, electronic dispersion compensation is a task that is constantly being optimized to handle the weakest signals.
“We used to use coils of fiber to force dispersion compensation,” says John, “but now it's all done electronically. It is much more accurate to increase the throughput."
So now, instead of initially offering users 1-, 10-, or 40-gigabit fiber, thanks to technologies that have improved in recent years, you can prepare “drops” of 100 gigabits.
Cable masking
Despite the fact that the bright yellow gutter makes them hard to miss, at first glance, both Atlantic and East European submarine cables in the building can easily be mistaken for some elements of the power distribution system. They are wall-mounted and do not need to be fiddled with, although in the event that new fiber cable routing is required, they will be directly connected via underwater fiber from the shield. The red and black stickers sticking out of the floor in the place of the bookmark read "TGN Atlantic Fiber"; on the right is a TGN-WER cable equipped with a different device in which the fiber pairs are separated from each other in a junction box.
To the left of both boxes are power cables enclosed in metal pipes. The two most robust ones are for the TGN-A, the two thinner ones are for the TGN-WER. The latter also has two submarine cable routes, one ending in the Spanish city of Bilbao and the other in the Portuguese capital, Lisbon. Since the distance from these two countries to the UK is shorter, much less power is required in this case and therefore thinner cables are used.
Speaking about the cable management, Osborne says:
“The cables that run from the beach have three main parts: the fiber that carries the traffic, the power line, and the ground. The fiber on which the traffic goes is the one that stretches over that box over there. The line of force branches off on another segment within the territory of this object"
An overhead yellow fiber trough crawls towards distribution panels, which will perform a variety of tasks, including demultiplexing incoming signals so that different frequency bands can be separated. They represent a potential “loss” site where individual links can be cut off without entering the terrestrial network.
John says, "There are 100 Gbps channels coming in, and you have 10 Gbps clients: 10 to 10. We also offer customers a clean 100 Gbps."
“It all depends on the wishes of the client,” adds Osborne. “If they need a single 100 Gbps channel that comes from one of the dashboards, it can be directly provided to the consumer. If the client needs something slower, then yes, they will have to supply traffic to other equipment, where it can be split into parts at a lower speed. We have clients who buy a 100 Gbps leased line, but there are not that many of them. Any small provider who wants to buy transmission capability from us would rather choose a 10 Gbps line.”
Submarine cables provide many gigabits of bandwidth that can be used for leased lines between two company offices so that, for example, voice calls can be made. All bandwidth can be expanded to the service level of the Internet backbone. And each of these platforms is equipped with various separately controlled equipment.
“Most of the bandwidth provided by cable is either used to power our own Internet or sold as transmission lines to other wholesale Internet companies like BT, Verizon and other international operators that do not have their own cables on the seabed and therefore buy access to the transmission of information from us."
Tall distribution boards support a jumble of optical cables that share a 10 Gigabit connection with customers. If you want to increase throughput, it's almost as easy as ordering additional modules and cramming them into shelves - that's what the industry says when they want to describe how large rack arrays work.
John points to the customer's existing 560Gbps system (built on 40G technology), which has recently been updated with an additional 1.6Tbps. The additional capacity has been achieved with two additional 800 Gbps modules, which operate on 100G technology with traffic of more than 2.1 Tbps. When he talks about the task at hand, it seems that the longest phase of the process is waiting for new modules to appear.
All infrastructure facilities of the Tata network have copies, therefore there are two premises SLT1 and SLT2. One Atlantic system, internally named S1, is to the left of SLT1, and the Eastern Europe to Portugal cable is called C1, and is located to the right. On the other side of the building are SLT2 and Atlantic S2, which, together with C2, are connected to Spain.
In a separate compartment nearby is a ground-based room, which, among other things, is responsible for controlling the flow of traffic to London's Tata data center. One of the transatlantic fiber pairs actually drops data at the wrong place. It is an extra pair that continues on its way to Tata's London office from New Jersey to minimize signal latency. Speaking of which: John checked the latency data for the signal going over the two Atlantic cables; the shortest path achieves a Packet Data Delay (PGD) rate of 66.5 ms, while the longest reaches 66.9 ms. So your information is transported at a speed of about 703,759,397.7 km / h. So fast enough?
He describes the main problems that arise in this regard: “Every time we change from optical to low current cable, and then again to optical, the delay time increases. Now, with high quality optics and more powerful amplifiers, the need to reproduce the signal is minimized. Other factors include a limitation on the level of power that can be sent over submarine cables. Crossing the Atlantic, the signal remains optical all the way."
Testing submarine cables
On one side is the surface on which the testing equipment rests, and since, as the saying goes, the eyes are the best witness, one of the technicians plunges the fiber into the EXFO FTB-500. It is equipped with the FTB-5240S Spectrum Analysis Module. The EXFO itself runs on Windows XP Pro Embedded and has a touch screen. It reloads to show the installed modules. After that, you can select one of them and start the available diagnostic procedure.
“You simply divert 10% of the light output from this cabling system,” explains the technician. "You create an access point for the spectral analysis device, so you can then return that 10% back to analyze the signal."
We are looking at the highways stretching to London, and as this section is in the midst of a decommissioning process, we can see that it has an unused section showing up on the display. The device cannot determine in more detail what amount of information or a particular frequency it is talking about; to find out, you have to look at the frequency in the database.
“If you look at the underwater system,” he adds, “it's also full of sidebands and all kinds of other things, so you can see how the device works. However, you know that there is a mixing of the meter readings. And you can see if it is moving to a different frequency band, which lowers the efficiency.
Never leaving the ranks of the heavyweights of information transmission systems, the Juniper MX960 universal router acts as the backbone of IP telephony. Actually, as John confirms, the company has two of them: “We will soon have all sorts of things from overseas, and then we can launch STM-1 [Synchronous Transport Module Level 1], GigE, or 10GigE clients - it will kind of multiplexing will allow providing various consumers with IP-networks”.
The equipment used on terrestrial DWDM platforms takes up much less space than a submarine cable system. It looks like the ADVA FSP 3000 hardware is pretty much the same as the Ciena 6500 kit, however, since it's land-based, the electronics quality shouldn't be high. In fact, the ADVA shelves used are simply cheaper versions as it works at shorter distances. In submarine cable systems, there is a relationship that the further you send information, the more noise appears, so there is a growing reliance on Ciena photonic systems that are installed at the cable site to compensate for this noise.
One of the telecommunication racks contains three separate DWDM systems. Two of them are connected to the London center by separate cables (each of which goes through three amplifiers), while the other leads to the information center located in Buckinghamshire.
The cable site also provides a site for the West African Cable System (WACS). It was built by a consortium of about a dozen telecommunications companies and runs all the way to Cape Town. Submarine junction blocks help split the cable and bring it to the surface at various locations along the coast of the African South Atlantic.
Energy of nightmares
You can't visit a cabling site or data center and notice how much energy is needed there: not only for equipment in telecommunication racks, but also for coolers - systems that prevent servers and switches from overheating. And since the submarine cable installation site has unusual energy requirements due to its submarine repeaters, its backup systems are not ordinary either.
If we go into one of the Yuasa batteries, instead of racks with spare batteries, Yuasa - whose form factor is not particularly different from those seen in a car - we will see that the room is more like a medical experiment. It is filled with huge lead-acid batteries in transparent tanks that look like alien brains in jars. Maintenance-free, this set of 2V batteries with a 50-year lifespan adds up to 1600 Ah for 4 hours of guaranteed battery life.
Chargers, which are, in fact, current rectifiers, provide an open-circuit voltage to maintain the charge of the batteries (sealed lead-acid batteries must sometimes be recharged at idle, otherwise they lose their useful properties over time due to the so-called sulfation process - approx. Newthat). They also conduct the DC voltage for the shelving to the building. Inside the room, there are two power supplies housed in large blue cabinets. One powers the Atlantic S1 cable, the other the Portugal C1. The digital display reads 4100 V at approximately 600 mA for an Atlantic power supply, the second shows slightly more than 1500 V at 650 mA for a C1 power supply.
John describes the configuration:
“The power supply consists of two separate converters. They each have three power levels and can supply 3000 VDC. This single cabinet can power a whole cable, that is, we have n + 1 reserves, since we have two of them. Although, more likely even n + 3, because even if both converters fall in New Jersey, and one more here, we will still be able to power the cable."
Revealing some very sophisticated switching mechanisms, John explains the control system: “This is basically how we turn it on and off. If there is a problem with the cable, we have to work with the ship to fix it. There are a number of procedures that we must go through to ensure safety before the ship's crew starts work. Obviously, the voltage is so high that it is lethal, so we have to send messages about energy security. We send a notification that the cable is grounded and they respond. Everything is interconnected, so you can make sure everything is safe."
The facility also has two 2 MVA (megavolt-ampere - approx. New than) diesel generators. Of course, since everything is duplicated, the second is a spare. There are also three huge cooling units, although apparently they only need one. Once a month, the spare generator is checked off load, and twice a year, the entire building is started up on load. Since the building is also a data processing and storage center, this is required for accreditation to a Service Level Agreement (SLA) and an International Organization for Standardization (ISO).
In a typical month at the facility, the electricity bill easily reaches 5 digits.
Next stop: data center
In a Buckinghamshire data center, there are similar requirements for the volume of reserves, albeit of a different scale: two giant colocations (colocation is a service that a provider places client equipment on its territory and ensures its operation and maintenance, which saves on channel organization connections from the provider to the client - approx. New than) and managed hosting halls (S110 and S120), each of which occupies a square kilometer. Dark fiber connects the S110 to London, and the S120 connects to the cable exit on the west coast. There are two installations - stand-alone systems 6453 and 4755: Multi-Protocol Label Switching (MPLS) and Internet Protocol (IP)
As the name suggests, MPLS uses labels and assigns them to data packets. There is no need to study their content. Instead, decisions to send a packet are made based on the contents of the tags. If you want to learn more about how MPLS works, MPLSTutorial.com is a good place to start.
Likewise, Charles Cozierock's TCP / IP Guide is a great online resource for anyone looking to learn more about TCP / IP, its different layers, its equivalent, the Open Systems Interconnection (OSI) model, and more.
In a sense, the MPLS network is the crown jewel of Tata Communications. Because packets can be tagged with priority, this form of switching technology allows a company to use this flexible transport system to provide assurance in customer service. Labeling also allows data to be directed along a specific path, rather than a dynamically assigned one, which allows you to define requirements for quality of service or even avoid high tariffs for traffic from certain territories.
Again, as the name suggests, multi-protocol allows for multiple communication methods. So, if a corporate client wants a VPN (virtual private network), personal internet, cloud applications or some kind of encryption, these services are easy enough to provide.
For the duration of this visit, we'll call our Buckinghamshire guide Paul, and his colleague at the Network Operations Center, George.
“With MPLS we can provide any BIA (Security Address) or Internet - any service the customer wants. MPLS feeds our dedicated server network, which is the largest service area in the UK. We have 400 locations with a large number of devices connected to one large network, which is a single autonomous system. It provides IP, Internet and P2P services to our clients. Since it has a mesh topology (400 interconnected devices), each new connection will take a new path to the MPLS cloud. We also provide network services: on-net and off-net. Providers like Virgin Media and NetApp provide their services directly to customers,”says Paul.
In the spacious Data Room 110, Tata's dedicated servers and cloud services are positioned on one side, and collocation on the other. Data Room No. 120 is also equipped. Some clients keep their racks in cages and only allow their own personnel to access them. Being here, they get a place, energy and a certain environment. By default, all racks have two sources: A UPS and B UPS. Each of them travels on a separate network, passing through the building on different routes.
“Our fiber, which comes from SLTE and London, ends here,” says Paul. Pointing to the rack of the Ciena 6500 kit, he adds: “You may have seen similar equipment at the cable exit site. This takes the main dark fiber that enters the building and then distributes it to the DWDM equipment. Dark fiber signals are distributed across different spectra, and then it goes to ADVA, after which it is distributed to clients. We do not allow clients to connect to our network directly, so all network devices end here. From here we spread our connection.
- Part two -