Preserving internet in hotspots: Painful stories from Kramatorsk, Chernigiv, Kherson, and so on
What life hacks have emerged from local and national operators during the war?
A few hours before the full-scale invasion, russian terrorists initiated a massive cyberattack aimed at completely disabling communication in Ukraine. However, the subscribers hardly noticed the attack, as the operators were already prepared for the unwelcome 'guests'. Subsequently, the physical destruction of infrastructure began. What crisis management measures and pre-planned actions saved the networks? How did ordinary employees save valuable equipment from looters in Kherson and, on a half-ruined boat, lay cables after the destruction of a bridge in the Mykolaiv Oblast? Why do subscribers in the non-occupied territories of Donetsk and Lugansk oblasts still lack access to services like eBay, Zoom, and other Western platforms? What lessons have the operators learned, and what else are they preparing for? The owners and CEOs of the operators shared these and many other stories at the industry conference Ukraine: Resilience of Communication and Internet During War. Mind has recorded five interesting case studies.
Mykola Kucheruk,
Owner of Elite-Line (Kramatorsk):
– In 2014, Kramatorsk was under russian occupation for over two months. Our network suffered greatly during that time, losing over 50% of subscribers. However, that experience helped us work during the full-scale aggression. We were already prepared and had about 50 kilometres of cable and equipment in stock. These reserves were very helpful because financially, we wouldn't have been able to afford network restoration purchases in the first months of the war.
Currently, we have over 100 kilometres of backbone lines, not counting the subscriber lines. This means we have replaced more than 100 kilometres of optical cables in Kramatorsk, Druzhkivka, and surrounding villages. Our network began to be destroyed from the first day of the full-scale invasion. Most of the russian rockets were falling on roads, private houses, and civilian infrastructure.
For the first few months, none of our installation technicians left the city. However, emotional exhaustion gradually took its toll. It was very difficult to endure. We were among the first to arrive at the site of missile strikes, seeing injured civilians and destroyed homes… So, around June 2022, employees started resigning. It was precisely when the intensity of rocket shelling increased. If in March-May, we were targeted about twice a week, in summer, it was almost daily. Those who remained initially worked without days off, from morning till night. Gradually, we managed to hire new technicians to restore the network.
Currently, there are few people in Kramatorsk, but all the structures are functioning: hospitals, local administration… They need the internet, so we tried to restore the network as quickly as possible. Now we do it automatically: if a rocket hits, it takes two hours to restore the optical section. If the damage affects the sewage system, it takes a little more time.
However, we still encounter the problem of resource blocking in the Ukraine-controlled territory of Donetsk oblast. Last year, the governments of the USA and the EU imposed sanctions against the so-called LPR ("Lugansk People's Republic") and DPR ("Donetsk People's Republic"). However, there is no distinction between the temporarily occupied and unoccupied territories of Donetsk and Lugansk oblasts. Many resources are inaccessible to us: our subscribers cannot access Ebay, cannot make purchases on AliExpress, cannot use Zoom, and so on. Therefore, students and teachers are forced to connect to VPNs to use Zoom for learning.
Last year, we reached out to the Ministry of Digital Transformation and members of parliament. Unfortunately, this issue is still unresolved. And last month, more sanctions were added: the number of blocked resources increased. It is very difficult for us to explain to our subscribers why the websites of the local operator do not open, while everything works for the national player. Some consumers accuse us of traffic allegedly passing through the occupied territory, calling us "separatists", and so on. We have once again appealed to the Ministry of Digital Transformation. I hope the problem will be resolved. It is very unpleasant that the Donetsk Oblast is considered 'different' compared to other regions.
By the way, it took me a year before the full-scale invasion to install Google cache servers in Kramatorsk. Google support argued that Kramatorsk is an occupied territory because it is located in the Donetsk oblast. Finally, on February 22, 2022, we received the servers and installed them the next day. I don't know if we would have been able to do this after the full-scale invasion. In March, Google blocked the Donetsk oblast. We received notifications that all services would stop working. Fortunately, the Ministry of Digital Transformation intervened. We regained the ability to work with Google. But there was a risk that operators on the controlled territory could be left without Google services.
Another lesson we learned from 2014 is the need for an adequate number of uplinks (ports for connecting the local network to trunk channels. – Mind). In 2014, we had only two uplinks. Then, after the arrival of russian shells, we were left without communication for a day, meaning our network operated without external channels. After that, we connected all the uplinks available in Kramatorsk. However, when the full-scale aggression began, the uplinks surrounding Kramatorsk started to 'drop off'. At one point, only one remained out of seven. Fortunately, thanks to the cooperation of all backbone providers, we managed to establish a connection to Pokrovsk. Currently, we have four independent channels in different directions. This is very reassuring because it is very difficult to 'shut us down' and leave the city without internet now.
Overall, the full-scale invasion taught us that all services must have backups. We need a minimum of two servers, preferably underground, and connections to multiple backbone lines. Minimising human factors is also important. Everything must work self-sufficiently: cables – in the ground, the generator starts automatically because there may not be enough staff physically available to handle all the recovery tasks.
Dmytro Samsonenko,
Director of Osnova Operator (Chernigiv):
– We were not prepared for a full-scale invasion. We didn't believe it was possible in the 21st century. However, we had plans to expand our network, so we had purchased cables and had equipment in stock. All of this helped us later in restoring the networks.
On the evening of 24th February, the russian onslaught was already looming near Chernigiv and began systematically shelling the city. It lasted for a month, a week, and one day. There was no electricity, communication, water, or heating in the city. Panic ensued, and people started evacuating. Before the war, Chernigiv had over 300,000 residents, but after it began, only around 60,000 remained.
We were able to continue functioning thanks to the backbone provider UAR-net.They managed to somehow provide us with their services. We lived without electricity, relying on generators. I am very grateful to the military personnel who brought us fuel in barrels. They relied on us, and we provided them with our services.
How badly was the network affected? In Chernigiv, around 80% of the network is laid underground in cable channels. It remained intact. However, in the outskirts of the city and suburban villages, everything was destroyed. A lot of equipment was burnt due to constant shelling of the thermal power plant and voltage surges. Our fleet was also hit, with 14 vehicles destroyed. The office windows were shattered, and the premises were severely damaged. Luckily, there were no employees present at that time.
Currently, the network has been restored by replacing over 100 km of fibre optics. Essentially, it was not just restoration but a new construction: sections of the network were completely destroyed. We invested millions of dollars in this endeavour. We still need to repair the office and restore the fleet. Through our professional association, our Polish colleagues provided us with two vehicles, which was immensely helpful since we had nothing to drive on at all.
Oleksandr Savchuk,
Director of Atrakom Backbone Operator, Chairman of the Board of Ukrainian Internet Association (UIA):
– All major operators have been preparing to work with risks since 2021. These risks were discussed within companies and with relevant government bodies. However, we need to continue preparing for these risks and improving measures. So, what exactly are we talking about?
The first risk is the physical safety of employees: mines, shelling, and so on. We were prepared in advance, had appropriate personal protective equipment, and conducted tactical medicine courses. Unfortunately, we have had some injured specialists in certain areas.
The second risk is the absence of technological communication. If it is not essential for a provider working in a specific micro-district to have connectivity with other regions (they can restore their local network), it becomes a significant problem for backbone networks.
Imagine if there is no communication between oblasts in Ukraine, with damaged backbone lines, then restoring them becomes a major issue. Since there is no communication with emergency brigades, you don't know what has actually happened. Currently, we have provided our units with satellite communication, phones, and Starlink. It is an emergency measure in case of a telecommunication blackout. Fortunately, we did not encounter that, although there were local outages in certain regions for a few days.
The third risk is the disruption of supply chains for materials, equipment, and so on. By February 15, we had formed reserves, distributed equipment, vehicles, and so on, so we were ready to continue working even if everything came to a halt.
The fourth risk is fuel supply problems. Servicing backbone lines requires vehicles to cover long distances, so we need to refuel our vehicles every day. Therefore, we stocked up on fuel before February 24. Initially, during the war, it was not allowed to store more than five litres. Fortunately, this restriction has been lifted, so we can legally tackle this issue.
A similar situation arises with energy blackout. We stocked up on generators and fuel.
Another risk is the restriction of movement for employees during curfew hours. Before the war, this issue was not resolved, and we did not receive the necessary passes. As a result, we initially faced significant delays in restoring the lines.
In conclusion, the measures taken by operators were sufficient to support the networks at an adequate level of quality. However, it is too early to relax. Currently, in my opinion, it is crucial to enhance the reliability of telecom networks through redundancy and duplication. Yes, these are highly expensive measures. But it needs to be done not only in eastern regions but throughout the territory of Ukraine. International assistance can be considered precisely in this direction.
Mykhailo Shelemba,
CEO of Datagroup-Volia:
– We had various scenarios in case of a full-scale invasion. We tried to diversify our equipment and distribute routes. But when it all started, we had to act according to the situation. (Previously, Mind explained how exactly.)
I believe it was thanks to the heroism and resourcefulness of our people, who took initiative and made quick decisions on the ground, that we managed to save a lot and prevent any major consequences.
Let me give you an example. In 2020, we began upgrading our network, and in late 2021, we received a lot of Cisco equipment. In January 2022, we installed the new equipment in Kherson, which cost around a hundred thousand dollars. So, when the city was occupied, our regular technician with a modest salary took that Cisco equipment and hid it at his home. And when our forces retook control, he returned it and connected it to the network.
To be honest, no one would have condemned this person if they had sold that Cisco equipment worth $100,000 for $5,000 in order to survive, feed their family, or evacuate them from the occupation. But this person didn't do that. And it allowed us to restore communication in Kherson, among the first. There are many such examples of preserving equipment, generators, cars, and more. People treated the company's property as their own.
Regarding survival under occupation… We made a decision to continue paying full salaries in all occupied territories. And we still do that. After the de-occupation of certain regions, people said that it was psychologically important for them and enabled them not to cooperate with local collaborators, fake administrations, and all that mess that came to our land. By the way, when those 'locals' seized our office in Kherson, they made a report there, or rather spread dishonest lies using our logos: "The biggest operator in Kherson, Volia, has already restored everything and is operating. Come, sign up." We had to issue a denial, stating that we had no connection whatsoever to this 'work'.
Yuriy Zadoya,
Head of Technology Management and Control Provision Department at lifecell:
– We learned several lessons from direct military aggression by the russian federation.
The first lesson is not to keep essential equipment on the frontlines. We started evacuating it even before the full-scale invasion began. We transported around 10 tons of equipment from our data centres in Kharkiv and Dnipro to other regions.
Another initiative that we haven't been able to implement yet is the possibility of moving critical equipment (base transceiver station controllers, registers) abroad. Current legislation prohibits us from doing so. However, I hope that for the sake of security during aggression, we will be allowed to relocate the management system to protected cities.
The second lesson is to protect engineers and all market players, from the smallest to the largest. We had an interesting case. In a small village in the Chernigiv Oblast that came under occupation, russian military cut the cable to our base transceiver station (BTS). In that village, there lived a former engineer from Ukrtelecom, who contacted us and soldered patch cords to restore the BTS.
So, it's great to have a communicator in every populated area. For this to happen, there needs to be at least one small provider in every village. Therefore, it is important to preserve our ecosystem with thousands of small players.
Another example: in Mykolaiv Oblast, a bridge was blown up, which carried a backbone cable with our fibres. Our contractor had to restore the power supply. However, the locals burned all the boats so that the orcs couldn't reach them. Miraculously, we found a semi-deflated boat. And the contractor used the 'water laying' method to lay the cable, which allowed us to restore communication in the city.
The third lesson is that the state can be a 'healthy state'. That means making quality decisions quickly and delaying excessive regulation. There are many examples here. One of them is receiving the first international aid – equipment from Telefónica – thanks to the regulator. It immediately went for network restoration. For example, in Kharkiv, with regular incoming strikes, we kept restoring the BTSs: as soon as a new set arrived, we took it, went, installed it, and if it got hit again, we restored it. And so on endlessly…
The fourth lesson is that we should not only be an operator but also become a generating company and fuel trader. As they say, in every barrel of oil, there's a drop of human blood. Similarly, in every gigabyte of information, there's a drop of an operator's sweat. Once I calculated that producing one gigabyte requires around 100 ml of fuel.
If you have read this article to the end, we hope that means it was useful for you.
We work to ensure that our journalistic and analytical work is of high quality, and we strive to perform it as competently as possible. This also requires financial independence. Support us for only UAH 196 per month.
Become a Mind subscriber for just USD 5 per month and support the development of independent business journalism!
You can unsubscribe at any time in your LIQPAY account or by sending us an email: [email protected]
