Any connection between KART 5 (3660) and BSOD Critical Service Failed

[pdwk]

@steve_paul_quinn No offence taken. Honestly I forgot that was even my quote. 

For me, I was always successful doing:

msconfig

Boot tab → Safe boot ->minimal
General tab → Selective startup → Uncheck EVERYTHING

This prevents anything but the very core system, only Microsoft signed and certified dlls being loaded. Then I could either disable the Kaspersky service or rename the folder. Then restart back to a normal mode and VSS should be back up and running and no Kaspersky.

[alyo]

???!!!

 

[steve_paul_quinn]

Thanks PDWK.  My concern is locking myself out of helping my customer remotely.

Guiding them to restore a normal msconfig would be hard blind.

I’m going to experiment with KART on a VM.  I know TeamViewer has a Safe Mode boot that obviously includes the Safe Mode with Networking option.  It sometimes works which is cool.

I hope to find the minimal adjustments required to prevent KART from starting.

 

 

[pdwk]

???!!!

 

 

It probably just happened. It WAS there, the system booted, the vss drivers were loaded, and now they’ve been deleted. If you restart now the problem will start occurring. If you also have KART installed I’d be curious if it also deletes the *cat files upon restart. 

[steve_paul_quinn]

Alyo.  Is that the state of one one your machines?

No UpperFilters yet System Protection appears?  If so, my advice is wrong

 

There might be a less lazy way to query the state of VSS but I use the Free Macrium Reflect

The GUI has a convenient View VSS Events and Fix VSS Problems

 

It would be interesting to see if your VSS is happy or not.

 

 

[steve_paul_quinn]

My Bad.  I did not see your comment Aylo.  I only saw the screen shot. It looks like you caught it in an interesting state.  What an interesting problem this is.  Gotta put a positive spin on it as we are all learning quite a bit here.  Albeit with less sleep  :-)

[steve_paul_quinn]

A quick update

I wanted a fool proof KART uninstall from a working or recently fixed machine, just in case the KART uninstaller failed.  This is such an ugly issue, I don't want to take any chances.

From what I found

The anti_ransom_gui.exe task can easily be stopped
The AntiRansom4 Service CANNOT be set to manual or disabled without further investigation
Even sc config AntiRansom4 start=disabled fails
Manually attempting to change Start in Computer\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\AntiRansom4 appears protected

I resorted to using Kaspersky's own tool kavremvr.exe
https://support.kaspersky.com/common/uninstall/1464
It's fun to get in Canada, I need to use TOR Browser, YMMV
It worked great, doing more tests

Ugly notes here
https://docs.google.com/document/d/1gDzDaWPk4s8L2eqP6qERVojFXUtevCaca2G7SmO0wK8/edit

[pdwk]

A quick update

I wanted a fool proof KART uninstall from a working or recently fixed machine, just in case the KART uninstaller failed.  This is such an ugly issue, I don't want to take any chances.

From what I found

The anti_ransom_gui.exe task can easily be stopped
The AntiRansom4 Service CANNOT be set to manual or disabled without further investigation
Even sc config AntiRansom4 start=disabled fails
Manually attempting to change Start in Computer\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\AntiRansom4 appears protected

I resorted to using Kaspersky's own tool kavremvr.exe
https://support.kaspersky.com/common/uninstall/1464
It's fun to get in Canada, I need to use TOR Browser, YMMV
It worked great, doing more tests

Ugly notes here
https://docs.google.com/document/d/1gDzDaWPk4s8L2eqP6qERVojFXUtevCaca2G7SmO0wK8/edit

 

Just wanted to commend you, a fellow Canadian, on your notes and the things you tried. I read them all and it’s like re-living my past few weeks. (Including the attempts to disable the KART service via unchecking in msconfig and forcing the service’s registry entries. I even tried changing the Security Permissions on the registry keys).

As you said, despite this system-stopping problem, we all learned some things and together were able to brainstorm some solutions. 

[steve_paul_quinn]

Just wanted to commend you, a fellow Canadian, on your notes and the things you tried. I read them all and it’s like re-living my past few weeks. (Including the attempts to disable the KART service via unchecking in msconfig and forcing the service’s registry entries. I even tried changing the Security Permissions on the registry keys).

As you said, despite this system-stopping problem, we all learned some things and together were able to brainstorm some solutions. 

 

Thanks PDWK.  Awesome to hear you too are Canadian.  I’ve learned a bunch.  I’m sure the Kaspersky lurkers have as well.  We wound up doing a bunch of their support, R&D and QA in this thread.

[steve_paul_quinn]

WTF.  I just watched an upgrade to KART 3660 occur live and UpperFilter is now gone.  Now I await the deletion of CatRoot and DriverStore.

I should be LiveStreaming this lol

https://docs.google.com/document/d/1gDzDaWPk4s8L2eqP6qERVojFXUtevCaca2G7SmO0wK8/edit

I’m glad I started uninstalling KART on my customer machines.  It almost seems intentional.

First kill VSS so no backups can be made, then later render the system unbootable by deleting CatRoot and DriverStore. 

Is this intentional or has the update mechanism been compromised by an external actor?
 

 

[pdwk]

@steve_paul_quinn 

What’s your KART install process ? I have the same experience of the 3660(h) update removing the UpperFilters however I have NOT observed it deleting the *cat files lately (yet). Those still remain and the system functions normally (except for system restore obviously). The 3660 auto-update must be flawed. My process was always to install a version of KART4 (that doesn’t ask for registration) and let it update itself.

I also have a test system where I downloaded the most recent version of KART5 direct installer and it stays at version: 2058(d) and will not auto-update past that.

[steve_paul_quinn]

My current KART torture chamber is my personal HP ZBook. 

I had restored a working Macrium image onto it that had KART 3409 already installed.

I watched it upgrade in front of my eyes after about 4 hours of waiting to 3660

UpperFilters is now toast.

 

I had a customer who was in this exact situation.  Backups had failed for 4 to 5 days.

I got called in for a BSOD and was first introduced to this mess.

The last working backup had no UpperFilter.

I made a backup of the machine in this BSOD state.

I just checked now

CatRoot/DriverStore are significantly smaller than his current working system.

 

I’ll leave my Zbook on and watch if for days.  I have a funny feeling 3660 is still messed up and CatRoot and DriverStore will soon be gone.

 

[steve_paul_quinn]

Made a little batch file

1 Query the registry to see if UpperFiler is sad

2 Pop open System32 so I can lazily inspect the size of CarRoot and DriverStore

3 Pop open System Properties so I can if System Protection is sad

4 Open Macrium if installed to query the last good backups and VSS state

 

It’s not perfect but it saves me time.  YMMV

Save below as whatever.bat and have fun

reg query HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f} >UpperFiltersResult.txt & notepad UpperFiltersResult.txt
pause

explorer C:\Windows\System32
pause

sysdm.cpl
pause

cd C:\Program Files\Macrium\Reflect
reflect
pause

[CF93]

Hello,

I’m happy to find this topic after quite 2 weeks of intense searching and troubleshooting because I have the same problem with KART on Windows 2016 server.

I managed to restore the server and to have it booting but VSS is still not working, I’ve seen the reg fix in this topic but I’ve not tried it yet because when I look into windows 2016 registry I see that it lacks UpperFilters and LowerFilters key too (compared to a Win10), see Attachments.

 

Is LowerFilters missing from someone else Windows 2016 ?

Is LowerFilters needed on Windows 2016 ?

If yes, what is the its good value ?

Thanks for your help

[steve_paul_quinn]

Hi CF93

Those are all good questions.  I don’t have a Windows 2016 to compare sorry.  Hopefully you have one.  It should not take too long to build a similar vintage 2016 VM to validate as a reference.

Good luck

[CF93]

I’m not ready yet to throw Kaspersky under the bus.  This issue may be in combination with a faulty Microsoft KB Patch which we all know have been terrible for years.  IE https://www.askwoody.com/ms-defcon-system/

I am a paying Kaspersky customer and I will reach out to their support channels regarding this issue.  My priority is to communicate to my customers to remove KART (for now) and establish a working recovery process.

I’ll post whatever new and helpful information I can here.

 

Again I agree. I *know* it was KART directly causing the issue (ProcMon log), but the reason why remains a mystery. Since its internal workings are unknown, maybe it thought it was protecting my servers and workstations from a ransomware attack. Maybe a Windows Update updated/changed just the right amount of files that triggered KART to “protect” me. 

After our environment became “normal” again I spent some time trying to replicate the issue. I had Reflect backups of two machines from a week prior to the problem so I restored those and tried to let the problem re-occur. It never did. Windows updated as expected, KART updated and everything worked normally.

I also installed Win10 in a VM and installed KART4. I let everything update (Win10 20H2 and KART5) but still the problem did NOT reoccur. An unofficial inventory of our environment (and including friends and family) shows that not every KART5 system suffered and I cannot find any common links between those that did. 

My negative side thinks that a bad KART5 update was released, pulled and re-released without telling anyone. Thus only a few unlucky systems were affected. The files are deleted at boot so it’s possible a bugged KART5 was released and only systems that were rebooted between that release and the bugfix release were impacted.

It’s unfortunate but I knew this was free software. I knew there was no support. I accept that. I get the bonus of free but there actually was a cost. I’m not yelling, I’m not demanding Kaspersky be held accountable. I came here to get help and to possibly help others with what I learned. I appreciate all the input that was given. I do hope @steve_paul_quinn does manage to get an official response from K and can share it with us.

Hello,

First thank you for your posts,

I’ve read that KART deletes Catroot and Driverstore folders and Upperfilters regkey,

can you tell us if KART does other modifications to Windows ?

Maybe you could share your procmon log.

Thank you again, your posts, and all others in this topic, helped me not going crazy...

[CF93]

Hi CF93

Those are all good questions.  I don’t have a Windows 2016 to compare sorry.  Hopefully you have one.  It should not take too long to build a similar vintage 2016 VM to validate as a reference.

Good luck

 

Thanks for your good advice, I will try this and tell the result if no one can answer.

[steve_paul_quinn]

Hi CF93

Those are all good questions.  I don’t have a Windows 2016 to compare sorry.  Hopefully you have one.  It should not take too long to build a similar vintage 2016 VM to validate as a reference.

Good luck

 

Thanks for your good advice, I will try this and tell the result if no one can answer.

 

No problemo.  This thread has been really quiet lately so it might be faster to build a VM.  Please post your results here so others can benefit :-)

[pdwk]

@CF93 Pardon my late reply. If you haven’t had a chance to create a 2016 VM yet, I can confirm that the UpperFilters entry DOES exists in Server 2016. By default it is the same as all other versions. 
UpperFilters   REG_MULTI_SZ   volsnap

[steve_paul_quinn]

Thanks PDWK.  The question was regarding LowerFilters :-)

[pdwk]

Thanks PDWK.  The question was regarding LowerFilters :-)

LOL. So it was. Oops.

I checked again and there is NO LowerFilters on any of our 2016 installations. Those servers have also never been touched by KART so I am confident that there is no LowerFilters entry by default.

[steve_paul_quinn]

That’s good news.  Nothing to delete I suppose :-)

[CF93]

Thanks PDWK.  The question was regarding LowerFilters :-)

LOL. So it was. Oops.

I checked again and there is NO LowerFilters on any of our 2016 installations. Those servers have also never been touched by KART so I am confident that there is no LowerFilters entry by default.

Hello,

Thank you for your answer, you have been quicker than be, I was doing some testing on a VM yesterday and today :-)

So I've seen too that on a fresh setup of Windows 2016 there is no LowerFilters regkey, there is only the UpperFilters as you say.
I join a pdf with the number of files into Catroot/Driverstore folders/subfolders and the VSS regkey values of this fresh setup.

I add that "UpperFilters   REG_MULTI_SZ   volsnap" is the default value but backup softwares can add there value to this key too, so if there are not working with this value, their value must be added too.

 

On my VM, I have also tried to setup KART again... and it harms again...

On my prod server, I've seen that the b*st*rd update was installed on 11/03/2021, it has made my server unbootable after reboot on 02/04/2021 with "critical service failed" bsod.
I have restored my server many times until 08/04/2021 before suspecting KART and deleting it from the OS (my worst 2 weeks of the year), and now I have banned KART to hell forever with all Kaspersky stuffs !
Because it seems that, more than a month after, maybe nothing has been done by Kaspersky to solve this...

On my VM, I've used KART_4.0.0.861.0.4497162.0_en_US.exe (version 3.0.1.1807) for the setup, the same setup I used a year ago to install it on my prod server.
I've made the setup twice :

-On the first setup yesterday, I have seen a KART update notice some minutes after the end of the initial setup and the UpperFilters regkey has disappeared before reboot but the files into catroot and Driverstore hasn't been deleted even after many reboots. KART version is 3.0.1.861 (or 3.0.1.3660 on properties) after the update.

-On the second setup today, after the update notice after the end of the initial setup, the
UpperFilters regkey and the files into catroot and Driverstore hasn't been deleted even after reboots but the KART version has not changed (still 3.0.1.1807 after "update")...
I've joined some screenshots of all this.
I will try to monitor the VM on the next days.

For the joke, one year ago the 01/04/2020 my prod server was crypted by a ransomware and this is why I installed KART on it... this year the 02/04/2021, KART has corrupted my server, so ransomware and antiransomware are allied... I hate these April Fools...

 

[steve_paul_quinn]

Great work CF93.  Thanks for sharing your details.  Hopefully it will help others who land here and find this thread.  😎

[CF93]

Great work CF93.  Thanks for sharing your details.  Hopefully it will help others who land here and find this thread.  😎

 

@pdwk and you have done a great work on this topic and it has helped me a lot too, thank you.

 

For my tests, regarding the second setup of KART on my VM, despite the KART update notice, Windows log says that there was no update and KART is not updating yet.