Telecom front: How mobile operators resist blackouts, restore communication, rebuild wreckage and help the Armed Forces

"During the war, the word ‘competition’ practically disappeared from the telecom industry. Technical teams help each other to restore base transceiver stations (BTS), jointly connect to generators, share fibre optics, diesel fuel and transport," mobile operators describe the military everyday life.

How exactly do they "deal with" the consequences of destruction by russian terrorists? What do they do to ensure that during blackouts, the communication does not disappear? How much do they invest in their own "fleets" of generators, fuel, and purchase of batteries of larger capacity? What problems cannot operators overcome on their own and expect support from the state? Experts from Kyivstar, Vodafone Ukraine and lifecell told Mind about this and much more.

Why does the connection "go down" during blackouts?

"Now mobile networks suffer more from power outages than from attacks and destruction. Operators are connected to the power grid on the same basis  as ordinary households. There are no exceptions for mobile communications, which is a critical service. Therefore, after missile attacks on the power grid, mobile networks work for some time, then the load increases due to a large amount of subscriber traffic. Base transceiver stations (BTS) are gradually shutting down. At the same time, the network capacity decreases, and the signal level decreases. Therefore, subscribers begin to have difficulties with communication," Kyivstar explained.

Lifecell clarifies that when the power supply is lost, the rechargeable batteries are automatically turned on. They can support the operation of the base station from 2 to 5 hours after a power outage (provided they are fully charged, which is not always the case in case of frequent outages). "Thus, the stability of the network may be disturbed if the power supply is absent for a long time over a large area. Then it is difficult for the operator to ensure the operation of all equipment on autonomous power supply", lifecell says.

Kyivstar cites statistics: if in September–October there were about 6,000 incidents in the operator's network per week (i.e. emergencies on the cell towers or digital infrastructure), then after the shelling in November, 80,000 per week have already been recorded. The main reason is the disruption of the power supply.

What exactly do operators do to counteract blackouts?

In an "ideal universe" all equipment should be connected to generators, but the "Big Three" have a huge number of them. For example, Kyivstar in Ukraine has more than 60,000 PCAs (points of collective Internet access in homes), more than 30,000 cellular BTSs, and 13,000 technical sites (locations where switching equipment and BTSs are located). Vodafone Ukraine and lifecell also have tens of thousands of them.

"It is physically impossible to provide all BTSs with generators during large-scale power outages. To power thousands of BTSs, you need to have not only thousands of generators, but also the same number of people who could deliver fuel for them every day and be near them to refuel them regularly. To maintain the operation of one BS, 100 litres of diesel fuel per day is needed", lifecell explains.

Vodafone Ukraine adds: many BTSs are located on roofs, high-rise buildings, and complex architectural objects, where it is quite problematic or almost impossible to deliver or install a generator. Therefore, first of all, at the beginning of the war, the players "insured" the viability of the most important objects.

Kyivstar provided a geographical backup of key services in Western Ukraine. All critical infrastructure facilities: switches, traffic routers, and BTSs node exchanges are connected to stationary diesel generators. For autonomous power supply of other equipment, the operator uses mobile diesel generators. Now, Kyivstar has almost 700 generators. Their operation requires more than 10 tonnes of petrol and diesel fuel every day. "Even in the worst-case scenario, if the energy infrastructure is completely damaged, the communication will work, although it will not be a ‘carpet’ coverage, but standalone islands where there is a connection of the BTS to the generators," Kyivstar reassures.

The operator is going to purchase additional 230 generators soon, and is also increasing the number of specialists who service them.

The "blue" (the color of Kyivstar's logo is blue) operator is also modernizing the fleet of batteries of base transceiver stations. "We are installing new, more capacious batteries that can work without power supply for up to 7 hours," Kyivstar says, "Such work is already underway in Kyiv, Poltava and Sumy oblasts. We have already replaced 3,000 batteries, and are waiting for new supplies from foreign partners."

Vodafone Ukraine currently has all key network elements reserved. "No matter how long the outage lasts, we can ensure the operation of the network core, maintain the operation of the transport network connecting the network core and networks in different parts of Ukraine, the country's hubs," the company explains.

The operator attracts partners and other operators to share stationary generators. "This is painstaking work, but it will increase the coverage map in case of outages and increase the availability of services. We agreed to reserve the daily fuel demand at gas stations for the needs of operators. We have also developed special scenarios for the network operation for each city during prolonged outages: to maintain coverage in all districts of the city with the help of a much smaller number of BTSs. Of course, in this case, we are not talking about the usual quality of services and continuous coverage. However, there is an opportunity to stay in touch. These calculations, unfortunately, have already been tested in practice, so we have to adjust our plans and adapt to the need for additional generators", Vodafone Ukraine says.

This year, the company has increased investments in improving the reliability of power supply systems. In addition to the purchase of generators, it is updating the "fleet" of batteries to more modern ones in order to increase the autonomy time of the BTS without resorting to generation. The total unplanned amount of additional investments for these technical measures only for the last month is about UAH 120 million.

"Like everyone else, we are facing with a shortage of generators and batteries. After all, the established logistics chains, in particular the production and supply of materials and components, were disrupted. It affects the delivery time. However, because we are a part of an international holding, we have managed to quickly provide the minimum needs for equipment and build up a potential," Vodafone Ukraine said.

Lifecell also increases the number of generators and batteries at technological sites. The operator ensured the constant functioning of the most important sites – the backbone sites that can maintain the highest percentage of coverage. "Now the company's need for generators significantly exceeds their availability. We are constantly trying to increase their number, buy new batteries and fuel to maintain communication during power outages as long as possible. For example, in the coming months, we will buy another 150 generators. However, it is not enough if we are talking about the entire network. This is a very large-scale problem because the whole country suffers from the lack of electricity", lifecell explains. This operator is also looking for alternative sources. It has started testing power from solar panels.

And also asks for help from subscribers.

How much extra money do you have to invest in power supply?

Lifecell says that large-scale attacks on energy infrastructure have significantly increased operators' costs.
Vodafone Ukraine cites statistics: in January – July 2022, the operator's generators throughout Ukraine consumed an average of more than 10,000 litres of fuel per month. Since August, the company has been increasing fuel costs every month: in August and September – by 1.9 times, in October – by 7.8 times.

In addition, from January to October inclusive, the cost of diesel fuel has more than doubled. If in the first half of the year the average monthly fuel costs were about UAH 400,000, in October they reached UAH 3.8 million. "To fully assess the costs, one should also take into account the significantly increased frequency of service and repairs of generators due to heavy 24/7 operation, transportation services, etc.," Vodafone Ukraine adds.

How do operators continue to restore their own destroyed infrastructure?

Players say that the russian terrorists’ level of vandalism is simply off the scale now. For example, if at the beginning of the war in the temporarily occupied Kyiv, Chernigiv, and Sumy oblasts the scale of destruction has reached 30%, now in Kharkiv and Kherson regions it exceeds 80%. So now operators are increasingly faced with the need to build from scratch.

"The occupants take out the equipment. What they cannot take out, they blow up. They cut out fiber-optic cables in telecom wells and leave mines everywhere on technical sites. They do everything to deprive the population in the occupied territories of communication with Ukraine," Kyivstar says. "We are beginning to restore the infrastructure as soon as we receive permits from local military administrations, and after sappers ‘clean’ the territory. If the object is not badly destroyed, the restoration takes several days. For heavily destroyed ones, the work can last several months. Therefore, when we enter a de-occupied city, we launch several mobile cell towers connected to generators, and simultaneously begin to rebuild what the occupiers destroyed."

According to lifecell, every day 90 repair teams all over Ukraine make two or three trips, sometimes under fire and with risk to the lives of specialists. Repairs of base transceiver stations "in the fields" for 9 months of war have turned into serial episodes of domestic and professional heroism of engineers. Wherever possible, the company uses the connection of BTS to Starlink. It allows restoring the network quickly in regions where the transport network is severely physically damaged, and its repair takes time. Especially if there are problems with the power supply. There is a need for generators and fuel, which is quite difficult to find in the newly liberated areas, as well as people who will bring this fuel and regularly replenish the generator.

In general, according to the telecom regulator, operators manage to maintain communication. "The latest attacks of the russian federation were aimed at destroying the energy and telecommunications infrastructure of Ukraine, but even under such conditions, we observe that operators are taking effective measures to ensure the sustainable functioning of their own networks and continue to provide services to Ukrainians under martial law. The state constantly monitors the quality of communication services and the factors that affect it. The results of the analysis of the tests of the quality of mobile communication services in cities and on highways generally indicate that the quality indicators are met. However, in some cases they are provided at the minimum available level for objective reasons," the regulator explains.

How much damage and losses have been recorded since the beginning of the full-scale invasion?

Operators say it is still difficult to give an accurate estimate. "Often the same object has to be restored several times. An example is the village of Senkivka, Chernigv Oblast, where the "Three Sisters" monument is located, the monument to the "friendship" of Ukrainians, belarusians and russians. We would restore the base transceiver station in this village six or seven times, as this settlement was constantly shelled by russian troops. There are also many such cases in Kharkiv and Kharkiv oblast when today we restore the base station, and tomorrow it is destroyed again, and we have to do everything from the beginning", Kyivstar gives examples.

In total, since February, the “blue” operator's engineers have carried out more than 25,000 repairs on the network. "Over 9 months, the company's expenses for network maintenance and development have increased by almost 30% – up to UAH 3.5 billion compared to the same period before the war. The main costs are the restoration of the destroyed by the aggressor russian federation and investments in network protection," Kyivstar said.

Vodafone Ukraine still estimates losses from the destruction of the network at more than UAH 2 billion: "Unfortunately, they are still growing. We have already restored 242 facilities that have been destroyed or damaged as a result of hostilities. 500 settlements have been returned to communication. In Kherson Oblast, we have managed to restore communication in 45 settlements, engineers are held back only by demining works," the company explains.

How many BTS are now working in "hot" oblasts?

Kyivstar has already restored 90% of its network in Kharkiv Oblast, 15% in Kherson Oblast, 50% in Zaporizhzhia Oblast, and 30% in Donetsk Oblast.

Vodafone Ukraine has restored 84% of its network in the Kharkiv Oblast, 5% in the Kherson Oblast, and 44% in the Donetsk Oblast.

Lifecell restored 64% of BTSs in the Kharkiv Oblast, 33% in the Kherson Oblast, 94% in the Zaporizhzhia Oblast, and 54% in the Donetsk Oblast. In Lugansk Oblast, all lifecell cell towers have no connection to the network.

According to the regulator, in the de-occupied territories, operators restore communication as soon as possible wherever they can. "At present, the availability of base transceiver stations in Ukraine exceeds 83%, says the regulator, the best situation with mobile communication is in Kyiv, Chernivtsi, Cherkasy, Khmelnytsky, Sumy, Rivne, Poltava, Kirovograd, Dnipropetrovsk, Zhytomyr, and Zakarpattia oblasts. Service coverage in these regions exceeds 90% of the total number of tests conducted."

How much has the number of subscribers decreased?

Due to the war, all players recorded a drop in subscriber base. In Kyivstar for 9 months it decreased by 7%, to 24.4 million. Vodafone Ukraine as of the end of September had 15.8 million customers (19 million a year ago). The active 3-month lifecell subscriber base in the third quarter of 2022 decreased by 7.9% compared to the same period last year – to 8.2 million.

Kyivstar predicts that by the end of the year, the decline may reach a double-digit mark, which is 12% year-on-year, and the players may lose more than 7 million subscribers in total.

How does a reduction in the number of subscribers affect the operators' revenue?

"The decline in the subscriber base directly affected the decline in industry revenues. If in the first quarter the market grew by 14.1% year-on-year, then according to the results of the third quarter the growth fell significantly and amounted to +1.4%. We believe that this trend will worsen by the end of the year. The fall in the IV quarter may be minus 2% year-on-year," Kyivstar forecasts.

In addition, according to the "blue" operator, the costs and revenues are also affected by many external migrants in the roaming and interconnect segments. In roaming, all operators now have special offers for subscribers in order to preserve the base as much as possible. However, the base will still gradually decrease as subscribers will organically switch to local networks of the countries they are in.

How many BTSs have operators built as a result of internal migration?

Since spring, players have started investing in redistributing the network load and improving communication in the regions chosen by the IDPs (Internally Displaced Persons). "Kyivstar has built 470 4G base transceiver stations in 2022. And 6,400 BTSs have had data transmission speed increased (previously they worked on 3G technology, now 4G).

"Vodafone Ukraine built 147 BTSs, launched about 3,000 4G BTSs, and upgraded 1,422 3G BTSs to increase capacity in densely populated areas of the western regions, where most internal migrants settled.

Lifecell upgraded almost 280 BTSs and built 230 stations.

Did operators have to reduce staff?

The number of employees of the operators remained almost unchanged, the staff was not reduced. As of November 1, Kyivstar had 3,900 employees and 80 open vacancies. Vodafone Ukraine currently employs about 2,800 specialists, and lifecell has more than 1,020 specialists.

Do they continue helping the Armed Forces and refugees?

Kyivstar donated about UAH 130 million for humanitarian aid to the Armed Forces, hospitals, elderly people, and victims of war. In addition, it provided subscribers with free services for more than UAH 574 million. The operator is currently allocating UAH 300 million (to the United24 account) to restore digital infrastructure and develop a system to counter cyber threats.

As of the end of September, Vodafone Ukraine transferred UAH 275 million of humanitarian and direct financial aid, provided free services worth UAH 1.15 billion to subscribers abroad in February-September, and launched the League of Warmth challenge to provide alternative heating to maternity hospitals and clinics. Currently, two more companies have joined the operator, each contributing EUR 20,000 for heating.

Lifecell also provided millions of free GB and minutes to subscribers, separately – within the framework of special service packages for the military, doctors, and critical infrastructure workers. Lifecell also donated UAH 37 million worth of telecom equipment and machinery to the Armed Forces of Ukraine and pledged to transfer UAH 40 million for the digital development and restoration of mobile infrastructure.

What "war" problems of the industry cannot be solved without the state?

Operators cannot overcome all problems on their own, and therefore they hope for the help of the authorities.

Cable thefts. Currently, this is the second-largest problem (the first one is power outages). Only Kyivstar has already recorded 326 vandal damages since the beginning of the year. The most crime-prone oblasts are Dnipropetrovsk, Zaporizhzhia, and Odesa oblasts. Most thefts occur at night, even though there is a curfew.

Slowing down the launch of new cell towers. According to the players, it currently takes an average of 5–6 months to get access to state-owned infrastructure facilities for the construction of base transceiver stations. This time is spent on obtaining various permits, in particular, holding an auction on ProZorro, where operators get the right to lease a place on the roof of a state-owned facility for a base transceiver station. Kyivstar believes that this problem can be solved by applying telecom access agreements (it takes 3–4 weeks to execute) instead of lease agreements, and also offers to reduce the land allocation procedure for the construction of new communication towers from six to three months. Support from the state authorities is needed for the adoption of relevant initiatives.

Taxes on free services. Since the beginning of the war, companies from various industries have been providing free services to their users, the military, doctors, and volunteers, but they do not receive any income from them, although they have to pay taxes. "Together with other operators, we initiate changes to the Tax Code.

Licensing of free additional frequency bands. Kyivstar says that operators need new frequencies both to develop 4G and to increase capacity, especially in those regions where many residents from the eastern and southern oblasts of Ukraine have moved since the beginning of the war. The state has already partially helped. In the summer, the telecom regulator decided to allocate 2100 MHz bands to three operators for the period of martial law to increase the capacity of existing 3G networks or improve 4G coverage. Also, Vodafone Ukraine was allowed to use its 35 MHz band in the 2500-2690 MHz range, using LTE 2600-TDD technology, and Kyivstar was allocated 40 MHz in the 2300 MHz range.

The "blue" operator wants to buy the licence for this band now. "Today this resource does not bring funds to the budget. According to our estimates, it is UAH 50 million annually, which the state treasury could receive only through rent. In addition, the development of these frequency bands will allow mobile operators to maintain a high quality of services in the regions where many subscribers and Ukrainian enterprises have migrated due to the war," Kyivstar said.

Lifecell has the opposite position. The operator believes that all players have received a sufficient number of radio frequencies in these bands for the period until the end of martial law. "Despite this, we are now witnessing actions that indicate an attempt to distribute valuable spectrum, in which 4G and 5G can be introduced, already on commercial terms: for decades to come, without an auction or tender, at a reduced price. As a result, the state budget may lose from UAH 760 million to UAH 2.8 billion or more compared to the transparent purchase of spectrum through a tender after the war. Smaller and potential new market players will lose access to valuable 5G frequencies, competition in the telecom market will be destroyed, and subscribers themselves will suffer the most. Because without competition, both technologies are introducing more slowly and prices are higher," lifecell believes.

Rent for frequencies that cannot be used yet. On the temporarily occupied territories, russian terrorists have disconnected the equipment of Ukrainian operators. "We appeal to the state with a proposal to exempt companies from the rent for the use of radio frequency resources of Ukraine on the temporarily occupied territories and territories where military operations have been conducted, lifecell suggests. Taking into account this initiative will allow operators to restore the network in the newly liberated territories more efficiently and provide preferential communication to people who have lived under occupation."

 

Матеріал у партнерстві з Київстар