In March 2017, Montclair’s Town Council unanimously voted to buy a subscription to Microsoft Office 365 that would give township employees easy access to the company’s ubiquitous mix of office services. Included at no extra charge was a piece of software that experts say can prevent most cyber attacks.
The software, known as multi-factor authentication, or MFA, creates a process in which anyone signing on to a computer network is required to confirm their identity on their cell phone or through some other means. It prevents the most common form of hacking in which an intruder manages to steal someone else’s username and password.
The federal government has used MFA as standard procedure for years. In 2015, the Obama administration launched a nationwide campaign to persuade businesses and local governments to use it.
Those warnings went unheeded in Montclair, a Montclair Local investigation shows. In the six years after the township bought Office 365, officials repeatedly passed up opportunities to activate its MFA feature or put in place a comparable product. By May of 2023, the town’s insurers were threatening to cancel Montclair’s cybersecurity policy if officials did not immediately implement MFA for all employees.
Officials were still arguing over how to pay for a competing product when hackers struck, seizing key files and shutting down basic town services for weeks. In April 2024, Montclair acknowledged that the damage was even worse than initially thought. It sent letters to an unknown number of residents warning that their personal data, including social security numbers, had been compromised in the incident.
In Montclair, MFA Was MIA
Records show that Montclair spent $96,000 on a three-year Office 365 subscription in 2017 and has been paying for it ever since. The resolution approving the initial purchase said it would “improve security” and “reliability” and save an estimated $90,000. All Office 365 subscriptions sold to governments include MFA features that meet federal standards, according to Microsoft’s website.
In 2022, the holes in Montclair’s cyber defenses were identified by the company that insures the town against cyber attacks and other liabilities. The Garden State Municipal Joint Insurance Fund, known as JIF, sent a report recommending that Montclair implement MFA along with other security improvements. Township officials denied a public records request for this document as did JIF. Both stated that communications between a public agency and its insurance carrier are “confidential” and could not be obtained as government records through OPRA.
Tony Fan, Montclair’s chief information officer, did not implement Microsoft’s MFA features. Fan told the Local he preferred Cisco, a competing product, and first tried to purchase it in in 2022. He said he had requested $30,472 for the “Cisco license and service fee” and another $2,412 to cover an estimated 100 “hardware tokens.”
Fan preferred Cisco’s MFA software because he had been “testing the product for a long time and they allowed a free account.’’
“I had a lot of settings set up,’’ Fan said in an interview.
A former township employee who worked in the IT department said he urged Fan to use the Microsoft MFA, but Fan wanted Cisco.
“This delayed things quite a bit since he wanted to spend all that money instead of using the free one that was included with Microsoft,’’ the former employee said.
Fan said he kept extending his trial account with Cisco while trying to secure the money to pay for it. “Once funding was available sometime in May 2023, the funding was not enough. So I requested additional funding,” Fan told the Local.
‘MFA Can Make You 99% Less Likely To Get Hacked‘
As officials wrangled over money, disaster struck. Hackers broke into Montclair’s systems, sometime between May 22 and June 1, 2023. They shut down access to key files and demanded a hefty ransom for their release.
Fan said the town had a backup system in place. For reasons he declined to detail, that precaution failed. “We did have the backups but the hacker got to it so we couldn’t restore it and had to pay the settlement.”
Experts said it is standard practice to store backup files in a secure, offline location or in what is known as the “cloud,’’ virtual data storehouses protected by multiple layers of cyber defenses. When asked directly where the backups had been sent, Fan declined to answer, saying he did not want to reveal the inner workings of Montclair’s computer networks.
The hack unleashed chaos. Township workers could not sign on to their computers, halting the delivery of basic services. They struggled to reconstruct from memory lists of the files that they needed.
A month later, in July of 2023, the insurance company negotiated a settlement with the hackers with Montclair paying $450,000 for return of its files. Nearly all of the cost was covered by the town’s insurance, according to Michael Lapolla, the interim town manager.
Lapolla said the town’s “out of pocket” expenses amounted to approximately $25,000.
Montclair officials announced in July of 2023 that the files had been restored and the town’s cyber defenses had been strengthened. “To guard against future incidents, the township has installed the most sophisticated dual authentication system available to its own system, and it is currently up and running,’’ said Joseph Harnett, who was then the interim town manager.
The statement was meant to be reassuring. Harnett did not disclose that the town had previously had no system for “dual authentication,’’ which is a type of MFA.
Brandon Pugh, policy director and resident senior fellow for the Cybersecurity and Emerging Threats team at the R Street Institute, told the Local that the value of multi-factor protections has long been understood.
“The director of the Cybersecurity & Infrastructure Security Agency has said that implementing MFA can make you 99% less likely to get hacked,’’ Pugh said. “Similarly, the New Jersey Cybersecurity & Communications Integration Cell highly recommends that MFA be utilized.’’
It is not known how the hackers broke into Montclair computer networks. When asked if there was evidence that MFA would have blocked the attack, Fan did not provide an answer.
A ‘Very, Very Basic’ Security Measure
The fallout from the cyber-attack on Montclair’s computers continues to reverberate.
In April of this year, the township notified residents that their personal information, including social security numbers, might have been stolen. This, too, contradicted the town’s initial assurance that there was “no indication or any breach of personal data.’’
Fan refused to comment on how many individuals received this notice. A government data breach notification shows 17,835 individuals were affected.
From the very beginning, Montclair officials insisted that Montclair was the victim of a sophisticated group of hackers and that nothing could have been done to prevent the attack.
Brett Callow, a cybersecurity expert and threat analyst at Emisoft said, “This attack probably wasn’t sophisticated at all. Dozens of governments fall victim to them every year and they are usually the result of shoddy security.”
“I don’t think any government can be prepared for this,’’ Lapolla said in September 2023. “In reviewing how it was handled, it was done efficiently and amazingly. This is not something anyone is prepared for.”
Callow disagreed.
“Organizations should absolutely be prepared for incidents and should have incident response plans in place,” Callow said.
“Using some form of MFA is probably the single biggest thing an organization can do to reduce the likelihood of it falling victim to an attack. It’s a very, very basic security measure, and it should be used everywhere it can be used. If they didn’t have MFA on one of their systems, some people would probably consider that as quite negligent,” Callow said.
The Local looked into surrounding townships and found that Glen Ridge installed MFA three years before Montclair.
Montclair Was Not Prepared
The 2017 purchase of Office 365 was the first opportunity to implement MFA on Montclair’s systems, but it didn’t happen.
In 2022, Fan wanted to buy MFA from Cisco. In an interview with the Local last month, Fan said he had been requesting funds to implement Cisco since January 21, 2022, but did not get a response from Chief Financial Officer Padmaja Rao.
Fan detailed a series of emails to support his contention that the purchase of MFA had been blocked by others in the township. (The Local filed an OPRA request to access a complete set of emails on this subject. Fan responded that staff shortages prevented him from responding in the legally required 30 business days.)
In a March 2022 email Fan mentioned, he wrote to Rao and said the insurance company, JIF, was continuing to recommend that Montclair install MFA, calling it an “urgent item.” Timothy Stafford, who was then Township Manager, was copied on the message. On May 9, 2023, Fan sent an email to Rao and Deputy Township Manager Brian Scantlebury mentioning he had a meeting with the insurance company’s IT consultant the previous day. The consultant again stressed the importance of MFA.
“I indicated we cannot wait any longer and time is running out,” Fan told the Local. In his email to Rao and Scantlebury, Fan stated that MFA had to be put in place before July 1, 2023, or the township would lose its cybersecurity coverage.
Fan told the Local that after this email, Rao replied with available funds but it wasn’t enough.
However, speaking to the Local, Rao said the IT department has always had a surplus of leftover funds each year. These funds could be used for IT projects at any time.
More Questions Than Answers
Fan told the Local that MFA implementation was completed in July 2023, after the attack.
The Local sent follow-up questions to Fan, Mayor Sean Spiller, Lapolla and Interim Township Attorney Paul Burr.
- Montclair paid $96,000 in March 2017 for Microsoft Office 365, which included MFA. Why was MFA not implemented in all those years?
- Fan stated that the IT department lacked sufficient funds for MFA implementation in 2022. However, CFO Rao has shown that there were leftover funds available each year for IT projects. Why were these funds not utilized for implementing MFA?
In an email response to the Local on June 17, Fan said, “The MFA implementation was not required by the GSJIF insurance until 2023.”
Fan did not explain why the IT department did not proceed with the existing MFA software from Microsoft instead.
Fan refused to answer the second question. He said Rao violated a township policy set by Lapolla when she told the Local about the IT department’s surplus funds.
The Local spoke to Rao in May. Fan refers to a “policy” in a June 11 email obtained by the Local. Lapolla sent the email to town department heads.
“Please be advised that Township employees are to direct ALL press inquiries to the Manager’s Office. Township employees may not speak directly to the press on behalf of the Township unless authorized by me.” Michael Lapolla, in an email to department heads.
Fan stated that he can’t talk to the press but he is listed as the media contact for the town. In addition to being the town’s chief information officer, Fan is the interim communications specialist, taking over for Ananmay Uttara.
Montclair hired Uttara at a salary of $85,000 in March 2024. She left the position less than two months later on May 10. On Wednesday, the Township advertised a job posting for a new communications specialist on social media.
I hope you found this article valuable. Before you go, I want to ask you to consider supporting Montclair Local’s journalism.
Montclair Local plays a crucial role in our community by holding our government leaders accountable. With a brand new town council set to be sworn in this July, your support is more critical than ever. Your contribution ensures we can continue providing in-depth coverage and keep our community well-informed.
Montclair Local is able to offer paywall-free journalism with the support of readers like you. Please consider making a one-time donation or, better yet, become a sustaining donor and make a recurring monthly annual donation today. – Liz George, Publisher
All else aside, so Fan is the chief media person but he won’t speak to the press? Is something wrong is his head?
But back when he did speak to the press, he tried to blame the CFO for the fact that he failed to put in place a basic IT safety feature. Why is this department head lying AND trying to harm another employee?
Township’s fried servers aside, 17,000 people had their personal information compromised. Why a professional forensic investigation of the Town’s IT wasn’t ordered and why wasn’t this clearly incompetent employee dispatched?
A bunch of useless employees, Fan and Lapolla, gotta get the boot ASAP. It’s obvious that residents gotta step up, do the employees’ work, and slash those crazy high taxes.
Jesus, 2-factor authentication has been an industry standard for years! What is this guy, a caveman? Also, what kind of “communications specialist” is this Fan guy if he never picks up his phone? It’s a joke. I managed to get him on the phone once, and couldn’t understand a word he was saying because his voice was so garbled as to be unintelligible.
His maneuver with the CFO is no doubt nasty, but I’m equally concerned by his seeming penchant for obfuscation and misrepresentation. The whole thing seems so shady. Discipulus is right that there should be a thorough investigation because perhaps Fan should be on the hook for some of those massive damages his negligence caused.
Bottom line, it is now clear why we got hit. It wasn’t sophisticated hackers, it was a painfully ‘unsophisticated’ and conniving IT guy. Cleaning of the house is in place.
Parenthetically, I’m impressed by the quality of this article. Very thorough. I’m even more impressed because Fan and Lapolla are extremely secretive and good for not responding to any inquiries. Hats off to Ms. Fernandes for somehow unearthing it all for us in what, I have no doubt, is an uncooperative environment. Well done!
Two caveats first. The article does not make it clear that Mr Lapolla started in August, 2023 and the Council hired hime for something like 24 hrs/week. Mr Lapolla’s communication policy is pretty standard in both the private & public sector. The BoE voted to formalize its comms policy that only the President can speak for the Board. And make special note that the Board members are each our elected representatives and they mostly agreed to muzzle themselves! Imagine if the Council tried to pass such a resolution.
Anyway, before we suspended Mr Stafford and filled his spot with Acting and Interim Managers, people often referred to the Township Manager’ role as like a CEO. OK, I think we have all drop making that analogy.
But, our Township Manager is expected and does act as the example. They should & do set the values and standards for interfacing with and communicating with the townspeople. He has clearly failed in this regard. He has failed to give us accurate & timely information, proper notifications and documents, and really has made little attempts to improve I these areas. In fact, I think he is getting worse by the week.
Maybe only being here intially part-time and limited in what he could take on/absorb hid his poor communication skills and penchant to run roughshod over process and tradition. He has made no secret he wants to revamp the organization and has reached out to a select group he has confidence are on his team and qualified. And that may be his other deficiency. He has not proven to me he is a good judge of talent in his key assignments and hires. He might know the organization we should be moving towards, but he is not the one to realize it. He is an interim manager. A bridge hire. The new Council has to decide what it wants in its Manager, look at MR Lapolla and see if he has enough of these qualities to continue as the Interim Manager. But, it should be dawning on the new Council that they need to go a new direction or resign themselves to requiting the job requirements to fit Mr Lapolla’s strengths…and how to mitigate his weaknesses.
The whole IT thing is obviously was and still is a disaster and how a leader handles a crisis say a lot.
Sherry Fernandes is one of the best things to happen to Montclair journalism in years. Congratulations! Just one question: why is Tony Fan still employed by us?
Typical of our township: abuse of tax money and obvious nonsense. Why did we keep the CIO after last year’s disaster, or why are we giving him a bigger budget? This guy needs to go, and Lapolla should go with him. Lapolla’s lies and lack of judgment are very costly. “In reviewing how it was handled, it was done efficiently and amazingly.” What a joke this guy is!
Of course, blaming the CFO is a “go-to strategy” to hide corrupt practices. Anyone who blames the CFO is part of the problem, and I agree with other comments, Sherry is doing an amazing job!
Is there any criminal component to it? If so, forensic investigation would be in place. I mean one conducted by Federal Government, not by Spiller/Lapolla – picked vendors paid for by our own taxes-to which they have unlimited access.
I don’t always agree with EyeCaramba, but I agree with him/her about Sherry Hernandes. She kills!
Excellent reporting. Not a sophisticated cyber attack. Not using part of a product already purchased. What is wrong with these people? Is it just being lazy? Is it because they know from how the people at the top work, they can get away with incredibly sloppy work? They just punch the clock? Not using a product that was included in the purchase, because you want the product you want? Sitting in your hands, when Jim is telling you to get moving? Lapolla was not here for the attack, but chose to lie about all of this?
Ah, the smell of government incompetence in the morning. 😌
Some observations in no particular order:
1) MFA, User policies & procedures, and training are 1st Line of Defense strategies. Like locking your car around here. This IT also a failure of safeguards redundancy.
2) Comprehensive data security is heavily weighted towards protections & redundancies that assume actors will be successful circumventing the locked car. I assume our insurance carrier’s systems audit recommendations were similarly weighted & aligned. Our backup plan failed, too. The actor accessed 17K households, not just necessarily individual records of the public.
3) 99% prevention? I don’t know what this means and how it relates to what is an acceptable failure rate. I keep using Boeing Company as my favorite example. Does 1 failure out of every 100 incidents (whatever the unit of measure) an acceptable loss of countless labor hours and sensitive information? Should it be 1 in 1,000? What is the true cost? Can we afford it? Can we even achieve it? Knowing this, what information should we be collecting, retaining, how stored and who internally can access? Where are the Township’s privacy & use disclosures to the public?
5) Excel spreadsheets? This is a whole security discussion in itself. In the hands of typical users, many now deputized as record custodians for OPRA efficiency, supervised and authorized by department managers who may be only slightly more conversant on security risk?
I think we are lucky in a way. It has been exposed with great help from Montclair Local. The ransom and repairs is a large, but digestible cost. This six-figure bill doesn’t make our Top 5. But, it is of a sufficient impact and with potential future impact to warrant a whole reexamination of the Township’s master technology plan. And just for fun, maybe this plan should look at a shared services/shared cost component. Anyway, just some random thoughts.
Can someone remind me, why is Fan employed again?
R-22-279
The resolution you mentioned above can be found on page 19 at this link to the pdf file: https://ecode360.com/MO0769/document/752989222.pdf
What does it have to do with the price of eggs?
The guy is toast. No reasonable CEO would keep him.
Sorry, I took you question literally and provided a 15 min Council discussion on 12/20/22 to showcase all the canaries falling dead to the coal mine floor. It will explain his longevity. I wasn’t addressing his future prospects.
Frank Rubacky wrote:
“The ransom and repairs is a large, but digestible cost. This six-figure bill doesn’t make our Top 5. But, it is of a sufficient impact and with potential future impact to warrant a whole reexamination of the Township’s master technology plan.”
IMO there is clearly gross negligence involved here, when most others using data moved to some kind of two-step verification requirement. Toward that end, I just read a post in the Montclair Politics Facebook site that the job description for the Head of IT here originally called for a college degree requirement. And that the individual in the office now did not have one, when hired.
If that is correct, it should be pretty clear to top town management following this what action should be taken. And if not, the decision-making and job evaluation skills of that Manager, then seriously called into question. If unwilling to act and if defending the actions to date — given the new Council coming on board.
An Administration department reality check is needed across the 205 building board July 1…now that this crew of Council “head in the sand” crazies’ is departing…
Martin,
Steve Jobs, Bill Gates, and Zuckerberg do not have college degrees. Hiring someone in IT without one is no big deal. I have a family member that runs the IT department of a municipality. No degree but impeccable record of keeping his town hack free. He couldn’t believe Montclair was so incompetent but said other towns have been hacked as well. He said the attacks are constant and you need robust systems and protocols to fend them off. Also, he felt that our system was hacked months before our IT Department was aware of the breach. But hey, this town knows how to have great festivals!
“Fan stated that he can’t talk to the press but he is listed as the media contact for the town.”
What people really need to understand is that for many of the top people who run this town (and many of their assistants and underlings) there’s nothing but comtempt for members of the press as well as the residents as a whole. There is a pervasive and highly toxic “us versus them” mentality at 205 Claremont — especially from Lapolla and the team he has brought in. These people are not used to residents digging around and calling them out on their corruption and inability to play by the rules. And they’re clearly not used to having to actually answer difficult questions from us taxpayers. Unfortunately for them, this town is filled with a lot of people with strong journalism backgrounds. Combine that with runaway property taxes and you’re going to have a lot of people uncovering a lot of dirt. Sadly for us, this is just the tip of the iceberg.
Bill Looney,
I have no idea if Fan has a college degree. I won’t argue with you that one can be technologically competent without one. However, I think the issue though may be that having a degree WAS a job requirement. If he doesn’t have one, it just makes the whole thing worse, in particular if he got “creative” on his resume.
Of course, the biggest issue is that he is incompetent and he exposed 17,000 people to identity theft and costed taxpayers $500,000. Of what we know.
Lastly, why is Fan taking on Communications responsibility (and secretarial ministrations during council meetings – of all things!) if he can’t handle his own job? Lapolla is financially incompetent and unable to properly delegate. Keep an eye on those taxes – we’re devolving into a fiscal mess a la Jerry Fried. Meanwhile, Fan is ‘guarding’ your social security number.
@Frank Rubacky ” And just for fun, maybe this plan should look at a shared services/shared cost component.” Wow talk about going back into the archives. If you recall we did have a shared services agreement, albeit an informal one, with the BOE for data, communication, equipment and ultimately for the multiple GigaBit fiber network that currently exists throughout the township and BOE. If memory serves me correctly those shared services were dismantled by a new superintendent whose name escapes me. I also recall it saved the Township hundreds of thousands of dollars in yearly subscriptions, shared servers and network expertise. I do not recall any breach that exposed Township citizens or employees to data theft or ransom for data recovery. Times have changed but perhaps a revisit to the past is in order before we embark on a similar course. I’m not sure when Tony Fan was employed but I don’t think the communication between entities was all that cooperative around that time. Maybe he can shed some light on what happened if there is an additional stipend for communication positions.
Anyway, this is all water under the bridge. But with a new council in place there needs to be some serious discussions about the past, the current, and the future of Montclair technology across the board. Good luck to the new council.