Today’s utilities face many threats they didn’t face at the turn of the century. Cyberattacks on utilities jumped 70% in 2023, according to research reported by Reuters. Such dramatic increases in risk go back to 2018, says the International Energy Agency. It calculates that cyberattacks doubled between 2020, when the average number of weekly attacks per organization was 500, and 2022, when that number passed 1,000 hits per week.
To make matters worse, Power Grid International reported that more than a quarter of U.S. utilities have “weak cybersecurity management programs.” Having watched the Aclara staff help utilities recover from cybersecurity breaches, I know firsthand how devastating this can be.
Loss of an AMI system can impact the outage management system, mapping, load management, network operations such as voltage regulation and more. Even billing, which is the primary use for AMI, could be wiped out. As a result, the AMI system can’t send new reads over to the customer information system or a meter data management system, which directly impacts billing and utility revenue.
If you’re using your AMI communications for sensors, you lose visibility into your system. Other scenarios have a more direct impact on your members: If they have been disconnected for non-payment, you can’t reconnect them once payment has been made. These events can unfold without any physical hazard at all.
According to research conducted by IBM, ransomware attacks are one of the leading forms of attack on organizations, second only to malware. When ransomware gets onto a utility server or network, it discovers network devices and encrypts all the files on network machines. The files are still there, and you can see them. You just can’t access or use them.
Many utilities refuse to fund criminal organizations by paying the ransom. They restore their systems instead. This has happened to some of the utilities Aclara works with, and we’ve had instances where we could simply restore systems from a backup the utility had on site. We’ve also had a few cases where we've had to rebuild the systems from the ground up with new hardware and servers, reinstall everything, and treat the event like we were deploying a brand-new system. Which course of action would you rather face after disaster strikes?
If you want to recover a downed system quickly, here are a few best practices you can follow:
Aclara works with many utilities that have large, sophisticated IT departments, and those organizations generally have good, well-tested plans in place. We also work with smaller organizations, and some of them have no backup plan at all. Every utility should ensure they’re taking regular backups of the current system and its configuration and storing that data in a safe location, off the server and offsite.
You can store your backup data on another server, removable media, or even in the cloud. Along with having timely backups, your plan should cover equipment procurement in case of failure. Sometimes, you can quickly purchase equipment off-the-shelf, but in other cases, the equipment requires orders and lead times that can delay arrival. You can also have Aclara manage your system as a hosted solution. When you have a hosting contract, we back up your system, keep it patched and updated, and maintain redundancies within the environment. All of these challenges fall to us.
No matter how you choose to back up your system, you’ll also need to perform walk-through testing of your plan regularly so you know how it operates and verify that it will work for you. One utility we worked with made regular backups, but no one ever tested the backup methodology. Imagine putting all your backups on tape for three years, and when a disaster hits, you can’t use them. That’s what happened to this organization.
Along with testing your plan, review it regularly – at least once a year. System updates and redesigns take place, and personnel changes occur. Regular reviews ensure the plan remains relevant. If you’re not sure how to test the system or don’t have enough staff to manage the job, Aclara offers services and expertise to help you craft a plan.
For proper disaster recovery, Aclara offers hosting solutions involving geographically redundant systems with processes designed to deliver a two-hour return to operations (RTO.) We perform an annual disaster recovery test that takes the utility’s production system and switches operations to their disaster recovery environment within the RTO. The customer runs off the recovery environment for a week, and then Aclara switches operations back to the production environment. This annual test proves that everything works as it should.
Another best practice is to make sure you’ve taken a backup of your system configuration. That is, ensure you know the configurations you’ve made for the system, the servers, the databases and the interconnections. You want a snapshot of your virtual environments to rebuild all or part of the system. An entire system backup or snapshot can technically eliminate the need for a backup specifically of the database, but we still recommend taking the database backups for redundancy and specific server restoration efforts.
When Aclara sets your system up, backups are part of the configuration. Ask us how they work, where the data is stored and what it will take to restore your system if necessary. Aclara can also work with you to create a disaster recovery program for your utility. With the Aclara team’s knowledge of the systems, we can help you identify what needs to be backed up and validate that your strategy is sound.
The important thing is to be ready, and you need to be ready for bad actors hacking into your system or simple system issues. Among the customers I’ve helped recover from disasters in my 19 years with Aclara, 10% to 20% have suffered some form of ransomware attack, and 50% have experienced some form of hardware failure. Organizations that are ready for these problems can recover very quickly. Isn’t that what you want for your utility, too?