Carl Osborne, Tata's vice president of global network development, explains the details.
The closer you are to the surface, the more containment you need to withstand potential shipping damage. Trenches are dug in shallow water where cables are laid. However, at greater depths, as in the Western European Basin with a depth of almost five and a half kilometers, protection is not required - commercial shipping does not threaten the cables at the bottom.
At this depth, the cable diameter is only 17 mm, it is like a felt-tip pen in a thick insulating polyethylene sheath. The copper conductor is surrounded by a plurality of steel wires that protect the fiber optic core, which is embedded in a steel tube less than three millimeters in diameter in soft thixotropic jelly. The shielded cables are the same internally, but in addition are clad with one or more layers of galvanized steel wire wrapped around the entire cable.
Without a copper conductor, there would be no submarine cable. Fiber optic technology is fast and can carry almost limitless amounts of data, but fiber cannot operate over long distances without a little help. To enhance light transmission along the entire length of a fiber optic cable, repeater devices are needed - in fact, signal amplifiers. On land, this is easily done with local electricity, but at the ocean floor, the amplifiers draw direct current from the copper cable conductor. Where does this current come from? From stations at both ends of the cable.
While consumers don't know this, TGN-A is actually two cables running across the ocean in different ways. If one is damaged, the other will provide continuity of communication. The alternative TGN-A lands 110 kilometers (and three ground amplifiers) from the main one and gets its energy from there. One of these transatlantic cables has 148 amplifiers, while the other, longer one, has 149.
Station leaders try to avoid publicity, so I'll call our station guide John. John explains how the system works:
Promotional video:
“To power the cable, there is a positive voltage at our end, but in New Jersey it is negative. We try to maintain the current: voltage can easily bump into resistance on the cable. A voltage of about 9 thousand volts is divided between the two ends. This is called bipolar feeding. So about 4,500 volts from each end. Under normal conditions, we could keep the entire cable running without any help from the United States."
Needless to say, the amplifiers are built to last 25 years without interruption, since no one will send divers down to change contact. But looking at the sample of the cable itself, inside which there are only eight optical fibers, it is impossible not to think that with all these efforts there must be something more.
“Everything is limited by the size of the amplifiers. Eight fiber pairs require amplifiers twice the size,”explains John. And the more amplifiers, the more energy is needed.
At the station, the eight wires that make up the TGN-A form four pairs, each containing a receive fiber and a transmit fiber. Each wire is painted in a different color, so that in the event of a breakdown and the need for repairs at sea, technicians can understand how to reassemble everything in its original state. Likewise, onshore workers can figure out what to insert when connected to a subsea line terminal (SLTE).
Repair of cables at sea
Peter Jamieson, Fiber Support Specialist at Virgin Media, reports on cable repairs.
“As soon as the cable is found and brought to the ship for repair, a new piece of undamaged cable is installed. Then the remote control device returns to the bottom, finds the other end of the cable and makes a connection. Then the cable is buried into the bottom for a maximum of one and a half meters using a high pressure water jet,”he says
“Usually, the repair takes about ten days from the date of departure of the repair vessel, of which four to five days are work directly at the site of the breakdown. Fortunately, this is rare: Virgin Media has only encountered two in the past seven years.”
QAM, DWDM, QPSK …
With cables and amplifiers in place - likely for decades - nothing else in the ocean can be adjusted. Bandwidth, latency and everything related to quality of service is regulated at the stations.
“Forward error correction is used to understand the signal being sent, and modulation techniques have changed as the amount of traffic carried by the signal increased,” says Osborne. “QPSK (Quadrature Phase Shift Keying) and BPSK (Binary Phase Shift Keying), sometimes referred to as PRK (Double Phase Shift Keying), or 2PSK, are long range modulation techniques. 16QAM (Quadrature Amplitude Modulation) would be used in shorter submarine cable systems, and 8QAM technology is being developed, intermediate between 16QAM and BPSK.
DWDM (Dense Wavelength Division Multiplexing) technology is used to combine different data channels and transmit these signals at different frequencies - through light in a specific color spectrum - over fiber optic cable. In fact, it forms many virtual fiber optic links. This increases the fiber throughput dramatically.
Today, each of the four pairs has a bandwidth of 10 Tbit / s and can reach 40 Tbit / s in a TGN-A cable. At the time, 8 Tbps was the maximum potential available on this Tata cable. As new users begin to use the system, they use spare capacity, but we will not be impoverished from this: the system still has 80% of the potential, and in the coming years, with the help of another new coding or increased multiplexing, it will almost certainly be possible to increase throughput.
One of the main problems affecting the application of photonic communication lines is dispersion in optical fibers. This is the name of what the designers consider when creating the cable, since some sections of the fiber have positive dispersion, and some - negative. And if you need to make repairs, you need to be sure to have a cable with the right dispersion on hand. On land, electronic dispersion compensation is a task that is constantly being optimized to handle the weakest signals.
“We used to use coils of fiber to force dispersion compensation,” says John, “but now it's all done electronically. It is much more accurate to increase the throughput. So now, instead of initially offering users 1-, 10-, or 40-gigabit fiber, thanks to technologies that have improved in recent years, 100 gigabit drops can be prepared.
Speaking of cable management, Osborne says:
“The cables that run from the beach have three main parts: the fiber that carries the traffic, the power line, and the ground. The fiber on which the traffic goes is the one that stretches over that box over there. The line of force branches off on another segment within the territory of this object"
An overhead yellow fiber optic chute crawls towards distribution panels that will perform a variety of tasks, including demultiplexing incoming signals so that different frequency bands can be separated. They represent a potential “loss” site where individual links can be cut off without entering the terrestrial network.
John says, "There are 100 Gbps channels coming in, and you have 10 Gbps clients: 10 to 10. We also offer customers a clean 100 Gbps."
“It all depends on the wishes of the client,” adds Osborne. “If they need a single 100 Gbps channel that comes from one of the dashboards, it can be directly provided to the consumer. If the client needs something slower, then yes, they will have to supply traffic to other equipment, where it can be split into parts at a lower speed. We have clients who buy a 100 Gbps leased line, but there are not that many of them. Any small provider that wants to buy transmission capability from us would rather choose a 10 Gbps line.”
Submarine cables provide many gigabits of bandwidth that can be used for leased lines between two company offices so that, for example, voice calls can be made. All bandwidth can be expanded to the service level of the Internet backbone. And each of these platforms is equipped with various separately controlled equipment.
“Most of the bandwidth provided by the cable is either used to power our own Internet or sold as transmission lines to other wholesale Internet companies like BT, Verizon and other international operators that do not have their own cables on the seabed and therefore buy access to the transmission of information from us."
Tall distribution boards support a jumble of optical cables that share a 10 Gigabit connection with customers. If you want to increase throughput, it's almost as easy as ordering additional modules and cramming them into shelves - that's what the industry says when they want to describe how large rack arrays work.
John points to the customer's existing 560Gbps system (built on 40G technology), which was recently updated with an additional 1.6Tbps. The additional capacity has been achieved with two additional 800 Gbps modules, which operate on 100G technology with traffic of more than 2.1 Tbps. When he talks about the task at hand, it seems that the longest phase of the process is waiting for new modules to appear.
All infrastructure facilities of the Tata network have copies, therefore there are two premises SLT1 and SLT2. One Atlantic system, internally named S1, is to the left of SLT1, and the Eastern Europe to Portugal cable is called C1, and is located to the right. On the other side of the building are SLT2 and Atlantic S2, which, together with C2, are connected to Spain.
In a separate compartment nearby is a ground-based room, which, among other things, is responsible for controlling the flow of traffic to London's Tata data center. One of the transatlantic fiber pairs is actually dumping data at the wrong place. It is an extra pair that continues on its way to Tata's London office from New Jersey to minimize signal latency. Speaking of which: John checked the latency data for the signal going over the two Atlantic cables; the shortest path achieves a Packet Data Delay (PGD) rate of 66.5 ms, while the longest reaches 66.9 ms. So your information is transported at a speed of about 703,759,397.7 km / h. So fast enough?
He describes the main problems that arise in this regard: “Every time we change from optical to low current cable, and then again to optical, the delay time increases. Now, with high quality optics and more powerful amplifiers, the need to reproduce the signal is minimized. Other factors include a limitation on the level of power that can be sent over submarine cables. Crossing the Atlantic, the signal remains optical all the way."
Energy of nightmares
You can't visit a cabling site or data center and notice how much energy is needed there: not only for equipment in telecommunication racks, but also for coolers - systems that prevent servers and switches from overheating. And since the submarine cable installation site has unusual energy requirements due to its submarine repeaters, its backup systems are not ordinary either.
If we go into one of the batteries, instead of the shelves with spare batteries from the Yuasa - the form factor of which is not particularly different from those seen in the car - we will see that the room is more like a medical experiment. It is filled with huge lead-acid batteries in transparent tanks that look like alien brains in jars. Maintenance-free, this set of 2V batteries with a 50-year lifespan adds up to 1600 Ah for 4 hours of guaranteed battery life.
Chargers, which are, in fact, current rectifiers, provide an open-circuit voltage to maintain the charge of the batteries (sealed lead-acid batteries must sometimes be recharged at idle, otherwise they lose their useful properties over time due to the so-called sulfation process - approx. Newthat). They also conduct the DC voltage for the shelving to the building. Inside the room, there are two power supplies housed in large blue cabinets. One powers the Atlantic S1 cable, the other the Portugal C1. The digital display reads 4100 V at approximately 600 mA for an Atlantic power supply, the second shows slightly more than 1500 V at 650 mA for a C1 power supply.
John describes the configuration:
“The power supply consists of two separate converters. They each have three power levels and can supply 3000 VDC. This single cabinet can power a whole cable, that is, we have n + 1 reserves, since we have two of them. Although, more likely even n + 3, because even if both converters fall in New Jersey, and one more here, we will still be able to power the cable."
Revealing some very sophisticated switching mechanisms, John explains the control system: “This is how, in essence, we turn it on and off. If there is a problem with the cable, we have to work with the ship to fix it. There are a number of procedures that we must go through to ensure safety before the ship's crew starts work. Obviously, the voltage is so high that it is lethal, so we have to send messages about energy security. We send notification that the cable is grounded and they respond. Everything is interconnected, so you can make sure everything is safe."
The facility also has two 2 MVA (megavolt-ampere - approx. New than) diesel generators. Of course, since everything is duplicated, the second is a spare. There are also three huge cooling units, although apparently they only need one. Once a month, the spare generator is checked off load, and twice a year, the entire building is started up on load. Since the building is also a data processing and storage center, this is required for accreditation to a Service Level Agreement (SLA) and an International Organization for Standardization (ISO).
In a typical month at the facility, the electricity bill easily reaches 5 digits.
How an infrastructure provider works
As an international cable system, service providers around the world face the same challenges: damage to terrestrial cables, which most often occurs on construction sites in less closely monitored areas. These are, of course, the anchors at the bottom of the sea that have lost their trajectory. Plus, don't forget about DDoS attacks, in which systems are attacked and all available bandwidth is filled with traffic. Of course, the team is well equipped to deal with these threats.
“The equipment is set up to track the typical traffic patterns that are expected during a particular period of the day. They can consistently check traffic between 4pm last Thursday and now. If the inspection reveals anything unusual, the equipment can proactively eliminate the intrusion and reroute traffic with another firewall, which can weed out any intrusion. This is called productive DDoS mitigation. Its other type is reciprocal. In this case, the consumer can tell us: “Oh, I have a threat in the system on this day. You'd better be on the alert. " Even so, we can filter out as a proactive measure. There is also legal activity that we will be notified of, for example, Glastonbury (UK Music Festival - approx. New),so when tickets go on sale, the increased level of activity is not blocked."
System latency also has to be proactively monitored by clients like Citrix who run virtualization services and cloud applications that are sensitive to significant network latency. The need for speed is appreciated by such a client as Formula 1. Tata Communications operates a racing network infrastructure for all teams and various broadcasters.
And by the way, if you're curious about how backup systems work, they have 360 batteries per UPS and 8 uninterruptible power supplies. This adds up to over 2,800 batteries, and since they each weigh 32 kg, their total weight is about 96 tons. The service life of the batteries is 10 years, and each of them is individually monitored for temperature, humidity, resistance and other indicators, checked around the clock. When fully loaded, they will be able to keep the data center running for about 8 minutes, which will give a lot of time for the generators to turn on.
The center has 6 generators - three for each hall of the data center. Each generator can handle the full load of the center - 1.6 MVA. Each of them produces 1280 kilowatts of energy. In general, it receives 6 MVA - this amount of energy, perhaps, would be enough to provide power to half of the city. There is also a seventh generator in the center, which covers the energy demand needed to maintain the building. The room contains about 8000 liters of fuel - enough to survive a day in full conditions. With full combustion of fuel per hour, 220 liters of diesel are consumed, which if this were a car moving at 96 km / h could take the modest 235 liters per 100 km to a new level - the numbers that make the Humvee look like like a Prius.
The NewWho team worked on the translation: Vlada Olshanskaya, Nikita Pinchuk, Alexander Pozdeev, Georgy Leshkasheli, Olya Kuznetsova and Kirill Kozlovsky. Editors: Anna Nebolsina, Roman Vshivtsev and Artyom Slobodchikov