ADP Board Emergency Meeting - April 17, 2023
Meeting Details: The Automatic Data Processing Board (ADP) met in emergency session at 1 pm on April 17, 2023 in the 3rd Floor Conference Room at the Geauga County Office Building at 12611 Ravenwood Drive, Chardon, OH. Virtual access was available via MS Teams and the agenda was provided in advance. This observer attended in person.
Public Comment: It is unclear whether ADP has a written policy. In practice, public comment is solicited at the end of the meeting.
Board Attendance:
-
Auditor/Chief Admin - Charles E. Walder
-
Board of Elections - Michelle Lane, Director
-
Clerk of Courts - Sheila Bevington
-
County Commissioner - Timothy Lennon and Jim Dvorak (Mr. Dvorak left the meeting approximately 30 minutes after it began)
-
Engineer - Joe Cattell
-
Prosecutor - Jim Flaiz
-
Recorder - Celesta Mullins
-
Sheriff – Scott Hildenbrand
-
Treasurer - Caroline Mansfield, Alternate for Christopher Hitchcock
ADP Staff Attendance:
Other Government Representatives:
-
Ron Leyde – Chief Deputy Auditor
-
Pam McMahan - Fiscal Office Manager, Auditor’s Office
-
Kate Jacob - Chief Compliance Officer, Auditor’s Office (Virtual)
-
Paul Pestello - Auditor’s Office
-
Nora McGinnis - Deputy Director, Board of Elections
-
Scott Daisher - Administrator, Board of Elections
-
Tom Rowan - Chief Deputy Sheriff
-
Gerry Morgan - County Administrator
-
Nicholas Gorris - Sanitary Engineer, Dept. of Water Resources
-
Katie Taylor - Engineering
Others Present:
-
Elise von Gunten - Geauga Times Courier/Chagrin Valley Times (Virtual)
-
Brian Doering - Geauga County Maple Leaf
-
John Karlovec - Geauga County Maple Leaf
-
Amy Patterson - Geauga County Maple Leaf (Virtual)
-
Diane Jones (Virtual)
-
Shelly Lewis - LWV Geauga (Virtual)
The meeting was called to order by Mr. Antenucci at 1 pm.
Agenda: To discuss additional issues regarding the cyberattack on the Water Resources exchange server. This was first discussed at an ADP Board Emergency Meeting on April 13, 2023.
Background: A summary of the ADP incident report regarding the cyberattack is as follows: On the morning of April 12, 2023, Crowdstrike Falcon services (the endpoint cybersecurity product installed by ADP on all servers and workstations touching the County's network) noticed possible nefarious activity involving a significant and persistent threat attack on a Water Resources Microsoft Exchange server. This server is an end of life/end of support server operating Microsoft email for Water Resources which was not properly patched by Water Resources. This vulnerability likely permitted the exploitation of an outside actor to externally penetrate the server through Exchange and attempt to run a series of tasks or commands through PowerShell scripting. ADP personnel immediately notified Water Resources of the attack, blocked all inbound Water Resource domain traffic, removed Water Resources from all shared ISP switches, and began a deep scan of all County systems to ensure that the County's environment, under ADP control, was secure and unaffected. There was no indication that the attack bridged beyond Water Resources, and they remain off-line awaiting remediation. CrowdStrike and ADP were successful in containing the attack with no disruption to other County services or systems under ADP control. Reviewer Comment: for more details see LWV Geauga Observer Report for the April 11, 2023 ADP Emergency Meeting.
According to Mr. Walder, this would not have happened if Water Resources had migrated their server to M365 which is a cloud-based system in use by most other agencies and offices in the county. This migration was originally planned for February 2023 but was put on hold by Gerry Morgan until mediation for a different dispute between the Board of County Commissioners (BOCC) and ADP was resolved. The parties underwent mediation through the Ohio Supreme Court without reaching resolution. BOCC filed a Notice of Voluntary Dismissal on April 4, 2023.
At the April 13, 2023 emergency meeting, the Board authorized ADP to immediately commence the migration of Water Resources email to Office 365 and perform any other services necessary to get Water Resources operational. Any cost incurred will be covered up front by Water Resources. ADP will attempt to recover historical Water Resources email data if possible. ExpertIT, a 3rd party vendor, will conduct the conversion of Water Resources email to Office 365. Observer Comment: On September 13, 2022 the BOCC filed a Complaint for Declaratory and Injunctive Relief (this was subsequently dismissed by the BOCC on April 4, 2023 as described above) against Charles E. Walder, Allen Keener and Frank Antenucci as individuals in their official capacities with ADP. This precedent by the BOCC of filing a complaint against individuals in ADP has made some reluctant to work directly with Water Resources.
Discussion: Mr. Walder started the meeting by explaining that the ADP team began to immediately migrate Water Resources to Office 365 as directed by the Board on April 13, 2023. As they conducted this process, certain items arose that he felt were necessary to bring to the Board’s attention:
-
Status of Water Resources Exchange Server - Zach McLeod provided an update on the cyberattack and current status of the Water Resources Exchange server. A review by Crowdstrike of the activity on the network log indicated that the attack originated from Russia and that no other servers were penetrated. The affected server is still in containment status with no email activity being processed. Mr. Flaiz shared that he was informed that Water Resources employees believe that the cyberattack was concocted by the ADP team and was not a real event. Mr. McLeod explained that Crowdstrike alerted ADP of the attack, and that ADP was not able to duplicate such an attack utilizing the same malware and with its origination in Russia.
-
Domain Name Service (DNS) at Water Resources - Allen Keener explained that the DNS for Water Resources was controlled by a 3rd party. The credentials were not provided to ADP which required a workaround to transfer the DNS to a single account within network solutions before being able to add current records. The Water Resources email accounts will be set up consistently with all other county employee emails. Aliases of old accounts will be attached to the new accounts to ensure that all email activity is captured.
-
M365 Migration Status - Corey Thomspon explained that during the migration process they discovered that the intended Office 365 tenant - “gcdwr.org” - already existed. Mike Kurzinger, Network Administrator at Water Resources, communicated that neither he nor the 3rd party contractor recalled setting up this tenant and could not provide any credentials. This required additional action through an ‘internal admin takeover’ and delayed the migration by about 4 to 5 hours. This step was completed and ExpertIT was working on setting up the new Water Resources email accounts.
-
M365 Database Setup - Mr. Thompson and Mr. Antenucci described the process to obtain the list of necessary emails for the Water Resources employees. They initially planned to access the existing backup email database to obtain the list. After twice requesting assistance from Mike Kurzinger to access this database, Mr. Kurzinger informed Mr. Antenucci that the repeated requests constituted harassment. The ADP team instead compiled an approximate list of email accounts from timesheets available through payroll, and the Office 365 licenses were assigned accordingly. Other non-user email accounts are likely to be needed, such as those used to automatically notify staff of system problems at the facilities. They expect email to be functional by midnight April 17th. The timing of the transition of email history is to be worked out.
-
Other IT Servers - Mike Adams indicated that as a result of the work conducted, an additional 6 servers were identified that were involved in the migration process. All of the servers were up to date with patches. Each of these servers will be assessed during migration.
-
Firewalls - Andy Haines provided an assessment of the current firewall protections on the Water Resources exchange server. The current firewall subscription expired on February 17, 2023 and only basic firewall operations are in place. As a result, for example, certain antivirus and spyware protections, content filtering, and intrusion prevention were not enabled. Other IO devices on the network were identified showing vulnerabilities. One was a DVR at McFarland Creek Wastewater Treatment Plant (McFarland). ADP will access the new firewall technology purchased earlier by Water Resources and onsite at former Water Resources offices at 470 Center Street and utilize this on the new M365 server.
-
Geauga County ADP Board Multi-Factor Authentication (MFA) and Password Policy Compliance - At this time the current ADP policy and a chart illustrating the vulnerability of passwords based on length and complexity were passed out. The requirement for 25-character password length for all Geauga County users went into effect on November 1, 2022 and the MFA requirement went into effect on March 1, 2023. Admin, service accounts, and servers have much longer passwords. Mr. McLeod indicated that the Water Resources users, admin accounts, service accounts and server passwords were well below 25-characters and no MFA was in place. There was concern about bringing servers with vulnerable passwords back online. To strengthen the passwords in concurrence with policy and ADP practice will require coordination with McFarland plant and OT vendors to avoid any unexpected operational issues.
-
Wonderware Server - Mr. Antenucci reviewed the current location and structure of the server that maintains Wonderware (industrial operations software) archived data. This server currently resides at 470 Center Street which complicates the flow of data from McFarland and represents the risk of a potential entry point into the county’s network or the Water Resources operations system. Mr. Walder made a motion to require Water Resources to deploy an operational technology vendor to expeditiously relocate this server to McFarland at a cost not to exceed $25,000. The motion was approved.
-
End User Patching and Vulnerabilities - Mr. McLeod explained that there currently were no scheduled patch applications in place for the Water Resources servers and workstations, exposing them to security and operational vulnerabilities. ADP currently utilizes a program scheduler that regularly reviews for available patches and schedules their application. It was recommended that the Water Resources servers be transitioned to this scheduling process. Per Mike Kurzinger, 4 workstations (2 at McFarland, 1 at Water Resources shop/garage, and 1 at central office) should be excluded to ensure operational continuity and instead be subject to regular manual updates. Discussion concluded that the personal workstation in the central office should be included in scheduled patching.
Mr. Walder emphasized the importance of having operational technology vendors available and access to Water Resources employees and sites to ensure that all necessary work was completed in a thorough and effective manner. He mentioned that he wanted to avoid finger pointing from Water Resources if issues subsequently arise by ensuring that all necessary functions are involved along the way.
A motion was made by Mr. Walder to require that all user, admin, service accounts, and server passwords at Water Resources comply with the Geauga County Multi-Factor Authentication and Password Policy. An upfront assessment of password changes will be made to avoid operational continuity issues. Sheriff Hildenbrand asked why policy compliance should be approved by the Board since it was already a requirement. Mr. Walder explained that the Board’s upfront support was necessary in case of future allegations that password changes caused operational interruptions. The motion was approved.
A motion was made by Mr. Flaiz that all Water Resources servers and workstations be subject to scheduled patching with the exceptions of operational technology workstations noted in item 9 above. Mr. Walder explained that the Board’s upfront support for this move was necessary to ensure Water Resources cooperation. The motion was approved.
Mr. Morgan agreed to ensure access to Water Resources sites, employees, and workstations to make necessary firewall updates, server moves, and software updates. Nick Gorris at Water Resources agreed to obtain a complete list of necessary email accounts by the end of day. The necessary Office 365 software and the Gatekeeper MFA software will be installed on the estimated 50 workstations at Water Resources on April 18, 2023. This will enable the new email to go live at Water Resources as soon as possible.
Mr. Walder communicated to the Board that the ADP team has been working around the clock to complete the migration of Water Resources to Office 365 and to address other related issues since the cyberattack first occurred on the morning of April 13, 2023. They are looking forward to completing this project.
Public Comment: None.
There was a motion to adjourn, and the meeting was adjourned at 2:19 pm.
Minutes when posted are available here: Meeting Minutes | Geauga County Automatic Data Processing
Virtual Meeting Information: Contact Pamela McMahan at pmcmahan@gcauditor.com
Observer: Carol Benton
Reviewer: Anne Ondrey
Reviewer: Shelly Lewis
Submitted: 4/18/2023
The League of Women Voters of Geauga is a 501(c)(3) nonpartisan political organization that encourages informed and active participation in government, works to increase understanding of major public policy issues, and influences public policy through education and advocacy. They do not support or oppose individual candidates or parties. Learn more about the LWVG at www.lwvgeauga.org.