ADP Board Emergency Meeting - April 13, 2023
Meeting Details: The Automatic Data Processing Board (ADP) met in emergency session at 12 pm on April 13, 2023 in the Appraisal Office, 231 Main Street, Suite 1A, Chardon, OH. Virtual access was available via MS Teams and the agenda was provided in advance. This observer attended virtually.
Public Comment: it is unclear whether ADP has a written policy. In practice, public comment is solicited at the end of the meeting.
Board Attendance:
-
Auditor/Chief Admin - Charles E. Walder
-
Board of Elections - Michelle Lane, Director
-
Clerk of Courts - Sheila Bevington
-
County Commissioner -Jim Dvorak and Tim Lennon
-
Engineer - Joe Cattell
-
Prosecutor - Jim Flaiz
-
Recorder - Celesta Mullins
-
Sheriff – Scott Hildenbrand
-
Treasurer - Christopher Hitchcock - ABSENT
ADP Staff Attendance:
Other Government Representatives:
-
Pam McMahan - Fiscal Office Manager, Auditor’s Office
-
Kate Jacobs - Chief Compliance Officer, Auditor’s Office (Virtual)
-
Ron Leyde – Chief Deputy Auditor (Virtual)
-
Scott Daisher - Administrator, Board of Elections
-
Tom Rowan - Chief Deputy Sheriff
-
Gerry Morgan - County Administrator
-
Steven Oluic - Director, Dept. of Water Resources
-
Nicholas Gorris - Sanitary Engineer, Dept. of Water Resources
-
Mike Kurzinger - Network Administrator, Dept. of Water Resources
Others present:
-
Elise von Gunten - Geauga Times Courier/Chagrin Valley Times (virtual)
-
Brian Doering - Geauga County Maple Leaf
-
Diane Jones (virtual)
-
Joe Comino - CSJ Technologies
The meeting was called to order by Mr. Walder at 12 pm
Agenda: Crowdstrike Critical Threat Escalation on Water Resources’ Exchange Server.
Background: The ADP Board and the Board of County Commissioners (BOCC) have engaged in multiple conflicts over which body controls automatic or electronic data processing or record-keeping equipment, software, or services. The subject of this meeting centered on a cybersecurity attack on a server maintained and operated by the Department of Water Resources, which reports to BOCC.
-
ADP Position: In a January 28, 2021 letter to BOCC, ADP asserted its authority under ORC 307.84, which states: After the initial meeting of the automatic data processing board, no county office shall purchase, lease, operate, or contract for the use of any automatic or electronic data processing or record-keeping equipment, software, or services without prior approval of the board.
-
BOCC Position: During this meeting, Mr. Morgan of BOCC appeared to assert authority under ORC 307.843, which states: The board of county commissioners may purchase, lease, or otherwise acquire any automatic or electronic data processing or record-keeping equipment, software, or services that the board determines is necessary, or that the county automatic data processing board recommends, from funds budgeted and appropriated by the board of county commissioners for such purposes.
Notably, only ADP is statutorily authorized to operate the equipment and services referred to in these code sections.
Action Taken: Following extensive discussion that lasted about an hour (see below) Mr. Flaiz made the following motion, which passed by roll call vote unanimously:
to authorize ADP to immediately commence performing the migration of Water Resources email to Office365 and perform any other services necessary to get Water Resources operational. Any cost incurred will be covered up front by Water Resources. ADP will attempt to recover historical Water Resources email data if possible.
Discussion: Mr. Walder began by giving a summary of what led up to this emergency meeting.
Observer Note: Rather than report the description given by Mr. Walder, we are including the text of the incident report that was provided after the meeting by Pam McMahon. Most of this information was discussed at some point during the meeting and it will be easier for the readers of this report to understand what happened if all of this information is included here.
On Wednesday, April 12, 2023, in the early morning hours (approximately 4am) CrowdStrike Falcon services began noticing possible nefarious script and command-line activity on a critical water resource server. CrowdStrike Falcon is an endpoint cybersecurity product installed on all servers and workstations touching the County's network by ADP.
Shortly before 8am, ADP staff began receiving a series of serious high priority alerts through ADP's cyber-security center from CrowdStrike indicating what appeared to be a significant and persistent threat attack on this Water Resource server.
CrowdStrike observed, what appeared to be nefarious activity, attempting to access and control the server. Given the persistent nature of the attack, CrowdStrike elevated the incident to "Critical", automatically blocked its execution, isolated the server's communication, and put in motion a series of procedures and instructions for ADP to further isolate and protect the County's network infrastructure.
ADP personnel immediately notified Water Resources of the attack, blocked all inbound Water Resource domain traffic, removed Water Resources from all shared ISP switches, and began a deep scan of all County systems to ensure that the County's environment, under ADP control, was secure and unaffected.
It appears that the Water Resource server in question is an end of life/end of support server operating Microsoft Exchange for Water Resources which was not properly service patched by Water Resources. This vulnerability likely permitted the exploitation of an outside actor to externally penetrate the server through Exchange and attempt to run a series of tasks or commands through PowerShell scripting. The server was ultimately powered off by Water Resource staff preventing any further analysis by ADP or CrowdStrike.
As of now, there is no indication that the attack bridged beyond Water Resources and they remain off-line awaiting remediation. CrowdStrike and ADP were successful in containing the attack with no disruption to other County services or systems under ADP control.
According to Mr. Walder, this would not have happened if Water Resources had migrated their server to MS 365 which is a cloud-based system in use by most other agencies and offices in the county.
Water Resources maintains its own servers and independently contracts for IT support. Their contractor, Joe Comino of CSJ Technologies, was present to answer the ADP Board’s questions. Mr. Flaiz asked whether any of Water Resources’s vendors had experienced ransomware attacks or issues over the past year. It was revealed that:
-
Mr. Comino’s company had suffered a ransomware attack on a CSJ Technologies email server in December, 2022 that was resolved after paying a ransom.
-
Mr. Comino informed Water Resources of the incident but did not specify who was told. It is unclear whether BOCC was informed. No one informed ADP.
-
After initially insisting there were no connections between his CSJ servers and the Water Resources servers, Mr. Comino conceded there was a connection that allows for remote access. He said it had not been used in over a year.
-
Water Resources’s compromised server was not patched with necessary software updates.
After some additional back and forth, Mr. Flaiz gave the following timeline of events regarding the attempts by ADP to get Water Resources to do the migration:
-
September 2021 - ADP started migrating all county servers to MS 365
-
September 2022 - Quote was ready for a purchase order to be done by Water Resources. The work was to begin in February 2023
-
December 2022 - CSJ Technologies suffered a ransomware attack that appears to be similar to the attack on Water Resources’s Exchange server. Water Resources’s servers were not patched with the necessary software updates.
-
February 2023 - Mike Kurzinger at Water Resources informed ADP via email that Administrator Gerry Morgan instructed them not to go through with the migration until mediation regarding the lawsuit between the BOCC and the ADP Board was completed.
-
Current status - Water Resources’ Exchange server was attacked and staff cannot access their email. Gerry Morgan is now agreeing to allow Water Resources to proceed with the migration.
Mr. Walder stated that he didn’t want ADP staff to do the conversion. Instead it was to be farmed out to a vendor, Expert IT. Since this process was put on hold in February 2023, the vendor’s schedule may have changed, and a new quote and purchase order may be necessary.
Mr. Flaiz and Mr. Morgan got into a dispute about a lawsuit between BOCC and ADP. Mr. Flaiz stated that the lawsuit was illegally filed by Mr. Morgan without having a public vote by the BOCC. Mr. Morgan asserted he was not required to get approval in a public session. Mr. Flaiz stated further that because ADP staff were individually named in that lawsuit, staff are reluctant to work directly with Water Resources for fear of having additional lawsuits filed against them.
Reviewer Comment: Regarding Geauga County Common Pleas Court Case 22M000541, on September 13, 2022 the BOCC filed a Complaint for Declaratory and Injunctive Relief against Charles E. Walder, Allen Keener and Frank Antenucci as individuals in their official capacities with ADP. The dispute centered on whether access control systems at the new Geauga County Office Building fell under the purview of ADP or BOCC’s Maintenance Department. The parties underwent mediation through the Ohio Supreme Court without reaching resolution. BOCC filed a Notice of Voluntary Dismissal on April 4, 2023.
At this point in the meeting Mr. Flaiz made the motion described above, and the motion passed.
Mr. Lennon suggested that in light of apparent communication problems between Water Resources and ADP, each office could appoint one person to handle the communications. Mr. Walder was reluctant to do this, given that the person ADP appointed might be subject to future lawsuits.
Mr. Walder stated that the affected server was “one wire away” from possibly exposing Board of Elections data to a breach. It was mentioned this could be a big concern with the Secretary of State’s office. Reviewer Comment: After moving into the new County Office Building in mid-2022, Water Resources continued to independently operate and maintain its own servers at the former 470 Center Street location. The compromised server shares a switch with the Board of Elections.
There was a discussion about whether ADP staff would work on the 365 migration first, or work on recovering data from the exchange server first. It was decided the 365 migration would be done first.
At 1:13 pm, the Microsoft Teams virtual connection ended. People had started leaving the meeting and Pam McMahon asked about a motion to adjourn, but as of 1:13 there was no motion.
Minutes when posted are available here: Meeting Minutes | Geauga County Automatic Data Processing
Virtual Meeting Information: contact Pamela McMahan at pmcmahan@gcauditor.com
Observer: Nina Lalich
Reviewer: Shelly Lewis
Submitted: 4/13/2023
The League of Women Voters of Geauga is a 501(c)(3) nonpartisan political organization that encourages informed and active participation in government, works to increase understanding of major public policy issues, and influences public policy through education and advocacy. They do not support or oppose individual candidates or parties. Learn more about the LWVG at www.lwvgeauga.org.