Any connection between KART 5 (3660) and BSOD Critical Service Failed

[lnet]

Did somebody of you find out why certain machines are affected and others not ?

I did not.

There are Servers with SBS 2011 with exactly same configurations than others affected (empty catroot folder) and others are not.

They were having the same KART installed.

[lnet]

I just checked the folder C:\Windows\System32\DriverStore on the SBS 2011 machine which has an empty catroot folder.

The folder C:\Windows\System32\DriverStore is completely missing.

I will now restore the catroot folder and the diverstore folder from a backup and wait for your suggestions if i can than savely restart the server.

Should i also check the registry for missing keys ?

[lnet]

@pdwk by the way: mmc is opening on the sbs 2011 with empty catroot but telling “There are no items to show in this view”

[lnet]

Has anybody of you asked yourself how it is possible that a software which should prevent ransomevirus from encrypting files and folders is actually deleting important system files and in our case also folders (C:\Windows\System32\DriverStore completely missing) ?

@pdwk did you test if you let KART run and delete files for longer if it completely deletes the folder DriverStore and if it deletes even more folders after emptying the catroot and Driverstore ?

[lnet]

Hi Inet,

On a few computers where I reinstalled KART the version remains at 5.0.0.3660. That is after I installed KART4 and let it auto-update. I have not tried the KART5 direct install package. I stopped installing it though.

As for recovery, yes. The first few were complete reinstalls but after some testing I discovered that the BSOD was caused by all the missing *.cat files. My fix involves
1) Copying all the *.cat files from c:\Windows\servicing\Packages\ into c:\Windows\System32\CatRoot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\  either via a command prompt.

2) That allows the computer to boot. THEN once the computer boots back up normally I need to find all the various files to refill the C:\Windows\System32\DriverStore folder. Either from a backup of that workstation or a donor computer with a similar configuration. The computer works ok without these files but you won’t be able to add new devices.

3) Final step is quickly reinstalling the VC++ runtimes from Microsoft. 

All that gets the computer back to a working state. I have also successfully done a different process of:
1) Copy *.cat files as above
2) Install or force re-install 20H2 via the MediaCreationTool20H2 and the option “Keep files AND apps”. This takes longer but makes me feel better about the system as a whole.

 

I have been trying to recreate the exact scenario that causes this by setting up various virtual machines but so far they are working great.

Hi @pdwk pdwk. May i ask how u made it to find out initially that the BSOD was caused by missing *.cat files ?

And thanks again for all your precious help !

[lnet]

@pdwk

Thank You for your solution, following your steps solved my BSoD problem (Kaspersky antiransomware was the culprit) on my Windows Server 2016, thank You a lot.👏

I have only a question: did You find the shadow copy service not working? ie, if I right click on my C disk… Shadow Copies tab… blank with “NO VOLUME IS ELIGIBLE FOR SHADOW COPIES” notice?

Thank You,

Alessandro

 

I think this is the same as the System Restore bug mentioned earlier by myself and a few others. Try the Registry fix AND a restart:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f}
UpperFilters        REG_MULTI_SZ     volsnap
 

https://www.reddit.com/r/GoroHome/comments/c9i6rl/kaspersky_removal_tool_kavremover_chkdsk_volume/

or 

https://www.sysnative.com/forums/threads/system-restore-not-working-error-0x81000203-kaspersky-removal-tool.28983/

Should i check in advance before i restart my still up servers if this Registry entry is correct/present ?

i checked my servers in regarding this key (UpperFilters). It is not present on the one with the empty catroot folder and it is not present on the other servers (i everywhere uninstalled KART but did not yet restart the servers).

Do i have to create this key ? is there a regfile file which does the work ?

[pdwk]

Should i check in advance before i restart my still up servers if this Registry entry is correct/present ?

You can check after. It is NOT a requirement for boot.

 

 

Did somebody of you find out why certain machines are affected and others not ?

I did not.

There are Servers with SBS 2011 with exactly same configurations than others affected (empty catroot folder) and others are not.

They were having the same KART installed.

I was never able to reproduce this on a fresh install nor was I able to find the reason why KART5 started doing this. I played with a laptop for a long time, every reboot would delete the files. I would re-copy them and try various KART5 settings. After a reboot it would always delete them.

I checked an old SBS2011 server I had here. It works fine. It only has 524 *.cat files and NO DriverStore folder (only an empty DRVSTORE folder which is normal) 

 

 

[pdwk]

@pdwkby the way: mmc is opening on the sbs 2011 with empty catroot but telling “There are no items to show in this view”

 

That’s normal for just the plain MMC. (You’d usually add snap-ins). On Windows 10 and Sever 2012 I never got it to run. Just gave an error that it wasn’t allowed to run.

 

 

Has anybody of you asked yourself how it is possible that a software which should prevent ransomevirus from encrypting files and folders is actually deleting important system files and in our case also folders (C:\Windows\System32\DriverStore completely missing) ?

@pdwkdid you test if you let KART run and delete files for longer if it completely deletes the folder DriverStore and if it deletes even more folders after emptying the catroot and Driverstore ?

 

I asked myself this. I even have the full KART5 log where it show that it is looking at each file, but it does not say why it was deleted. Letting KART5 stay running is fine. The delete happens only at boot. If I copy all the files back to catroot they will stay there forever until the next reboot. Only deleted at reboot. For the files that remain in catroot, it is because they are also being used by another process and KART5 cannot delete them.

 

Hi @pdwk pdwk. May i ask how u made it to find out initially that the BSOD was caused by missing *.cat files ?

And thanks again for all your precious help !

 

Searching for only the specific “Critical Service Failed” led me to this article https://rquintino.wordpress.com/2017/05/11/recovering-from-windows-10-boot-blue-screen-critical-service-failed-disable-drivers-signature-enforcement-unsigned-drivers/

where it talks about an empty catroot folder.

[pdwk]

i checked my servers in regarding this key (UpperFilters). It is not present on the one with the empty catroot folder and it is not present on the other servers (i everywhere uninstalled KART but did not yet restart the servers).

Do i have to create this key ? is there a regfile file which does the work ?

 

This key is required for most backup programs and system restore to work so YES you should create it.  I will paste the text below so you can create a regfile for it. open Notepad, copy/paste and save as *.reg

 

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f}]
"UpperFilters"=hex(7):76,00,6f,00,6c,00,73,00,6e,00,61,00,70,00,00,00,00,00

[alios]

i checked my servers in regarding this key (UpperFilters). It is not present on the one with the empty catroot folder and it is not present on the other servers (i everywhere uninstalled KART but did not yet restart the servers).

Do i have to create this key ? is there a regfile file which does the work ?

 

This key is required for most backup programs and system restore to work so YES you should create it.  I will paste the text below so you can create a regfile for it. open Notepad, copy/paste and save as *.reg

 

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f}]
"UpperFilters"=hex(7):76,00,6f,00,6c,00,73,00,6e,00,61,00,70,00,00,00,00,00

Hi, about that registry fix, if it could help: 

I was not able to add the key in that way (it did not work, ie no key added), so I added it manually: new “Multi-String Value” named “UpperFilters”, with value “volsnap”, restart the server.

[lnet]

Should i check in advance before i restart my still up servers if this Registry entry is correct/present ?

You can check after. It is NOT a requirement for boot.

 

 

Did somebody of you find out why certain machines are affected and others not ?

I did not.

There are Servers with SBS 2011 with exactly same configurations than others affected (empty catroot folder) and others are not.

They were having the same KART installed.

I was never able to reproduce this on a fresh install nor was I able to find the reason why KART5 started doing this. I played with a laptop for a long time, every reboot would delete the files. I would re-copy them and try various KART5 settings. After a reboot it would always delete them.

I checked an old SBS2011 server I had here. It works fine. It only has 524 *.cat files and NO DriverStore folder (only an empty DRVSTORE folder which is normal) 

 

 

Hi @pdwk 

Thanks for your reply.

I have few SBS 2011 servers. All of them have more than 1000 cat files and all of them do in fact have the DriverStore folders. Also the one where the cat folder is empty has had a DriverStore folder at least back in the registry Backup of January (unfortunately i cannot check the tape backup since it does not work anymore and i would need to restart with the filled catroot folder to get it back online.

Now i am not sure if i should restart with the filled catroot folder and without a DriverStore folder or if i should use the DriverStore folder from the backup of January.

[lnet]

@pdwkby the way: mmc is opening on the sbs 2011 with empty catroot but telling “There are no items to show in this view”

 

That’s normal for just the plain MMC. (You’d usually add snap-ins). On Windows 10 and Sever 2012 I never got it to run. Just gave an error that it wasn’t allowed to run.

 

 

Has anybody of you asked yourself how it is possible that a software which should prevent ransomevirus from encrypting files and folders is actually deleting important system files and in our case also folders (C:\Windows\System32\DriverStore completely missing) ?

@pdwkdid you test if you let KART run and delete files for longer if it completely deletes the folder DriverStore and if it deletes even more folders after emptying the catroot and Driverstore ?

 

I asked myself this. I even have the full KART5 log where it show that it is looking at each file, but it does not say why it was deleted. Letting KART5 stay running is fine. The delete happens only at boot. If I copy all the files back to catroot they will stay there forever until the next reboot. Only deleted at reboot. For the files that remain in catroot, it is because they are also being used by another process and KART5 cannot delete them.

 

Hi @pdwk pdwk. May i ask how u made it to find out initially that the BSOD was caused by missing *.cat files ?

And thanks again for all your precious help !

 

Searching for only the specific “Critical Service Failed” led me to this article https://rquintino.wordpress.com/2017/05/11/recovering-from-windows-10-boot-blue-screen-critical-service-failed-disable-drivers-signature-enforcement-unsigned-drivers/

where it talks about an empty catroot folder.

Very interesting.

This does indeed look like a malicious or tampered Software trying to harm a system. The fact it does not happen immideatly but only at the next reboot makes sure it is found only after the system gets restarted which on server could be after only weeks/months.

 

Cool u found the “Crititcal Service Failed” article. How did u make it to find out after that this folder is getting deleted by KART ? Did u do this on a VM ?

[pdwk]

Very interesting.

This does indeed look like a malicious or tampered Software trying to harm a system. The fact it does not happen immideatly but only at the next reboot makes sure it is found only after the system gets restarted which on server could be after only weeks/months.

 

Cool u found the “Crititcal Service Failed” article. How did u make it to find out after that this folder is getting deleted by KART ? Did u do this on a VM ?

 

I had a low-priority laptop exhibiting the same problem brought to me and I was able to do repeated tests on it (reboot, copy cats, reboot, copy cats, etc)

  

I am now 100% positive that it is KART causing this. But I don't know why. I have used the laptop and copied the *cat files over and over again.
- I used a program named ProcMon to log all processes at boot. It shows that KART tried to delete every file in the 'catroot' folder and then tries to delete the folder (fails as the folder isn't empty).  Screenshots: https://ibb.co/2jmkZbJ   and  https://ibb.co/B6qgJjQ
- Disabling KART service from Safe Mode stops the file deletion. Starting it again causes the files to delete.
- My version is KART 5.0.0.3660    (aka 3.0.1.3660) Screenshot https://ibb.co/5sry0Rf

 

[pdwk]

Hi @pdwk 

Thanks for your reply.

I have few SBS 2011 servers. All of them have more than 1000 cat files and all of them do in fact have the DriverStore folders. Also the one where the cat folder is empty has had a DriverStore folder at least back in the registry Backup of January (unfortunately i cannot check the tape backup since it does not work anymore and i would need to restart with the filled catroot folder to get it back online.

Now i am not sure if i should restart with the filled catroot folder and without a DriverStore folder or if i should use the DriverStore folder from the backup of January.

 

I am not an expert and have been testing my way through this very slowly and with many mistakes, but if I had that scenario I would

1. FIll catroot 
2. Copy the DriverStore from a backup onto a live machine.
3. Reboot

That scenario has worked every time for me this past week. Again, I am not an expert and can only offer my opinion. But I have confidence in that process.

There will still be missing files. Depending on how you do your backups, you could use a “folder compare” program to compare the current Windows folder with the backup to see if there are any *other* removed files.

As mentioned before, the Microsoft C++ runtimes (usually in the system32 folder) were also deleted on my servers and workstations. Some non-essential programs need these to run. After the reboot, I also install the runtimes which can be downloaded freely. (It was mainly programs such as Adobe Reader and some Xerox utilities).

[steve_paul_quinn]

Hi Folks

I'm experiencing this issue as well.  First it was from a friend/customer.  Then I had it myself.
I'm most grateful for the helpful hints in this thread.

I use Macrium Reflect so recovery was not that painful for us.
I've intentionally recreated the issue on another laptop to learn recovery without the benefit of backups.

I believe a third step is necessary.  I can replicate the need for this 3rd step repeatedly.

If the files in C:\Program Files (x86)\Kaspersky Lab are not dealt with, the repaired CatRoot and DriverStore may be impacted on the next reboot.

I think the KART application directory needs to be renamed so it’s files are not found upon the next boot.

I rename them rather than delete them, just in case.

C:\Program Files (x86)\Kaspersky Lab
C:\Program Files (x86)\Kaspersky Lab Old

After a successful boot, I then delete KART

I hope this helps 😎

Take care

Steve

[Intrepid]

Kaspersky Anti-Ransomware Tool caused numerous business computers and servers to blue screen and become completely unbootable due to a faulty update. Kaspersky provided no solution and ignored the problem. Discraceful.

[steve_paul_quinn]

I’m not ready yet to throw Kaspersky under the bus.  This issue may be in combination with a faulty Microsoft KB Patch which we all know have been terrible for years.  IE https://www.askwoody.com/ms-defcon-system/

I am a paying Kaspersky customer and I will reach out to their support channels regarding this issue.  My priority is to communicate to my customers to remove KART (for now) and establish a working recovery process.

I’ll post whatever new and helpful information I can here.

 

[pdwk]

Hi Folks

I'm experiencing this issue as well.  First it was from a friend/customer.  Then I had it myself.
I'm most grateful for the helpful hints in this thread.

I use Macrium Reflect so recovery was not that painful for us.
I've intentionally recreated the issue on another laptop to learn recovery without the benefit of backups.

I believe a third step is necessary.  I can replicate the need for this 3rd step repeatedly.

If the files in C:\Program Files (x86)\Kaspersky Lab are not dealt with, the repaired CatRoot and DriverStore may be impacted on the next reboot.

I think the KART application directory needs to be renamed so it’s files are not found upon the next boot.

I rename them rather than delete them, just in case.

C:\Program Files (x86)\Kaspersky Lab
C:\Program Files (x86)\Kaspersky Lab Old

After a successful boot, I then delete KART

I hope this helps 😎

Take care

Steve

 

Steve is correct. Funny how this was never directly pointed out. As long as KART remained installed, on every reboot it would delete the same files over and over again. (A few of my earlier posts discussed how (for testing) I kept re-copying the *cat files after every reboot and that a ProcMon log showed that it was KART deleting the files). Renaming the Kaspersky folder would definitely solve that problem.

[pdwk]

I’m not ready yet to throw Kaspersky under the bus.  This issue may be in combination with a faulty Microsoft KB Patch which we all know have been terrible for years.  IE https://www.askwoody.com/ms-defcon-system/

I am a paying Kaspersky customer and I will reach out to their support channels regarding this issue.  My priority is to communicate to my customers to remove KART (for now) and establish a working recovery process.

I’ll post whatever new and helpful information I can here.

 

Again I agree. I *know* it was KART directly causing the issue (ProcMon log), but the reason why remains a mystery. Since its internal workings are unknown, maybe it thought it was protecting my servers and workstations from a ransomware attack. Maybe a Windows Update updated/changed just the right amount of files that triggered KART to “protect” me. 

After our environment became “normal” again I spent some time trying to replicate the issue. I had Reflect backups of two machines from a week prior to the problem so I restored those and tried to let the problem re-occur. It never did. Windows updated as expected, KART updated and everything worked normally.

I also installed Win10 in a VM and installed KART4. I let everything update (Win10 20H2 and KART5) but still the problem did NOT reoccur. An unofficial inventory of our environment (and including friends and family) shows that not every KART5 system suffered and I cannot find any common links between those that did. 

My negative side thinks that a bad KART5 update was released, pulled and re-released without telling anyone. Thus only a few unlucky systems were affected. The files are deleted at boot so it’s possible a bugged KART5 was released and only systems that were rebooted between that release and the bugfix release were impacted.

It’s unfortunate but I knew this was free software. I knew there was no support. I accept that. I get the bonus of free but there actually was a cost. I’m not yelling, I’m not demanding Kaspersky be held accountable. I came here to get help and to possibly help others with what I learned. I appreciate all the input that was given. I do hope @steve_paul_quinn does manage to get an official response from K and can share it with us.

[steve_paul_quinn]

Thanks PDWK.  Honestly my first exposure to this issue was VSS failure.  I was making careful notes when I discovered this (your) excellent post in relation to CatRoot and DriverStore.

I’ve shared my notes below for anyone to read.  They start nice and clean and slowly get messy as I learn more.  I’ll clean it up once things settle.

I have a suspicion this issue came up when the KART application version automatically upgraded to 3.0.1.3660.  I say this because I have compared application versions of working and dead machines.  I’ve got screen shots in my messy notes to exemplify.  The Product Version of anti_ransom_gui.exe is helpful to query on dead machines.

https://docs.google.com/document/d/1gDzDaWPk4s8L2eqP6qERVojFXUtevCaca2G7SmO0wK8/edit

Regarding reaching out to Kaspersky Support, I am struggling to find a Support URL for the Free KART, but I’ll keep looking.  I’d bet the non-free KART has this same issue so I might try that.  It’s kinda sad Kaspersky has said nothing here.

Hey Kaspersky, who cares if we are dealing with a free product?   It’s possibly implicated in wacking computers.  These free KART customers of mine and now afraid of Kaspersky.  If the purpose of your free software is to transform them into paying customers, you had better act quick if you want to save mine.

[steve_paul_quinn]

Sorry folks.  I cannot find any support process for the free KART.  I only hope Kaspersky see this thread and reaches out.  I’ve got to reach out to multiple customers to uninstall it for now.

[Intrepid]

• steve_paul_quinn “ I’m not demanding Kaspersky be held accountable.”

Kaspersky should be held accountable for the damage they caused and the expense incurred to correct it.  It’s negligence and their lack of response is disgraceful.

[steve_paul_quinn]

Hi folks

Intrepid, not to throw PDWK under the bus but you misquoted me.  It easy to do with the layout of forum posts. In terms of accountability, you may want to first review the KART EULA.  It may in fact absolve Kaspersky from any damage.  Just being honest.

 

Back to the solution ...

 

I’ve spent much of yesterday running around cleaning up from the mess this caused so I’ve had little time to troubleshoot.

I have now seen 3 computers with a missing UpperFilter Registry entry in the following.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f}

These computers were still working but without UpperFilter and the volsnap entry, VSS is silently failing.  Automated Macrium backups are thus failing and soon I think CatRoot and DriverStore will be wacked on a future reboot.  I have 1 workstation with proof this happened.

I have a customer now who has yet to reboot in this exact state.  I cannot rename C:\Program Files (x86)\Kaspersky Lab as they are in use.  I think I’ll visit using msconfig to manually disable the software so at least it’s not running on the next reboot.

I think I’ve got 10 very unhappy customers now who are afraid of Kaspersky.  I’ve got a bunch more ticking time bombs out there as we speak. What a nightmare.

 

Folks even if you have a fixed or working machine, please double check VSS is happy.  An easy way I found is to try to open the System Properties thingy.  It normally look something like this.

A Happy System Protection


 

If UpperFilter/volsnap missing from HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f} opening System Protection reveals this;

 

A Sad System Protection

 

FYI I believe fixing HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f} requires a reboot.  This adds to the fun of this issue.  I’ll try to find a way around that if needed.

[pdwk]

@steve_paul_quinn 

Note sure why you thought I’d be under the bus. I agree. There are certain risks that come when using a free product. I accepted those risks and the results. I can remember 2 instances right now where KART in-fact saved those user’s files from encryption. So a reinstall or recovery can be an inconvenience it’s WAY better then losing all their files. Everyone’s situation is different. I just compare this to the times in the past where I’ve looked friends and coworkers in the eye and told them their files are gone unless they want to pay a ransom.

- Anyway, relating to the UpperFilters issue. I also arrived at that conclusion. The KART5 update probably had an uninstall component to it that removed the registry entry. It seems to be a known trend with Kaspersky products that it removes that registry key:

https://www.reddit.com/r/GoroHome/comments/c9i6rl/kaspersky_removal_tool_kavremover_chkdsk_volume/

- I found no way to reactive VSS without a reboot. It’s a filesystem driver so it needs to be active before Windows starts accessing the disk

-I found now way to disable KART5 from within Windows. It is a protected system service and it fights all task-enders and force renames and Service disables and registry editors. 

Instead of renaming the KART5 folder, why not just uninstall it? Have the *cat files been deleted yet ? If not, then a removal of KART5 before the next reboot should save them. Again, my experience says those files are only deleted on reboot. You could then do a Reflect backup and then reinstall KART

[steve_paul_quinn]

The bus quote was because Intrepid misquoted your comment as mine.  I did not mean to offend if I did.

 

Interestingly my daily drive laptop has many other UpperFiler entries.  It’s a popular place for other applications it seems

mrcbt appears to be Macrium

eudcpepm appears to be EaseUS

 

 

To answer your question about renaming/uninstalling KART, it depends on where I am

If I’m in a CATRoot DriverStore recovery process,  I will be in the Macrium Reflect PE environment.  It’s easy to rename/delete it there :-)

Thanks for the VSS restart info, it will save me time trying

 

The perfect storm is my current customers machine.

Windows is working, I’m logged in with TeamViewer !

UpperFilter was gone and is now fixed

VSS was failing for x days and there are no current Macrium backups

KART is installed, I have uninstalled it.

CatRoot is ok

DriverStore is ok

I’d really like to do a Macrium backup but I cannot without VSS

 

To be super duper safe, I hope some startup management, safe mode, msconfig Kung Fu can at least prevent the next reboot from using KART if for whatever reason it’s still present

Some of my customer are a 100 KM return trip !

Hope that makes sense