pdwk

LOL. So it was. Oops. I checked again and there is NO LowerFilters on any of our 2016 installations. Those servers have also never been touched by KART so I am confident that there is no LowerFilters entry by default.

@CF93 Pardon my late reply. If you haven’t had a chance to create a 2016 VM yet, I can confirm that the UpperFilters entry DOES exists in Server 2016. By default it is the same as all other versions. UpperFilters REG_MULTI_SZ volsnap

@steve_paul_quinn What’s your KART install process ? I have the same experience of the 3660(h) update removing the UpperFilters however I have NOT observed it deleting the *cat files lately (yet). Those still remain and the system functions normally (except for system restore obviously). The 3660 auto-update must be flawed. My process was always to install a version of KART4 (that doesn’t ask for registration) and let it update itself. I also have a test system where I downloaded the most recent version of KART5 direct installer and it stays at version: 2058(d) and will not auto-update past that.

Just wanted to commend you, a fellow Canadian, on your notes and the things you tried. I read them all and it’s like re-living my past few weeks. (Including the attempts to disable the KART service via unchecking in msconfig and forcing the service’s registry entries. I even tried changing the Security Permissions on the registry keys). As you said, despite this system-stopping problem, we all learned some things and together were able to brainstorm some solutions.

It probably just happened. It WAS there, the system booted, the vss drivers were loaded, and now they’ve been deleted. If you restart now the problem will start occurring. If you also have KART installed I’d be curious if it also deletes the *cat files upon restart.

@steve_paul_quinn No offence taken. Honestly I forgot that was even my quote. For me, I was always successful doing: msconfig Boot tab → Safe boot ->minimal General tab → Selective startup → Uncheck EVERYTHING This prevents anything but the very core system, only Microsoft signed and certified dlls being loaded. Then I could either disable the Kaspersky service or rename the folder. Then restart back to a normal mode and VSS should be back up and running and no Kaspersky.

@steve_paul_quinn Note sure why you thought I’d be under the bus. I agree. There are certain risks that come when using a free product. I accepted those risks and the results. I can remember 2 instances right now where KART in-fact saved those user’s files from encryption. So a reinstall or recovery can be an inconvenience it’s WAY better then losing all their files. Everyone’s situation is different. I just compare this to the times in the past where I’ve looked friends and coworkers in the eye and told them their files are gone unless they want to pay a ransom. - Anyway, relating to the UpperFilters issue. I also arrived at that conclusion. The KART5 update probably had an uninstall component to it that removed the registry entry. It seems to be a known trend with Kaspersky products that it removes that registry key: https://www.reddit.com/r/GoroHome/comments/c9i6rl/kaspersky_removal_tool_kavremover_chkdsk_volume/ - I found no way to reactive VSS without a reboot. It’s a filesystem driver so it needs to be active before Windows starts accessing the disk -I found now way to disable KART5 from within Windows. It is a protected system service and it fights all task-enders and force renames and Service disables and registry editors. Instead of renaming the KART5 folder, why not just uninstall it? Have the *cat files been deleted yet ? If not, then a removal of KART5 before the next reboot should save them. Again, my experience says those files are only deleted on reboot. You could then do a Reflect backup and then reinstall KART

Again I agree. I *know* it was KART directly causing the issue (ProcMon log), but the reason why remains a mystery. Since its internal workings are unknown, maybe it thought it was protecting my servers and workstations from a ransomware attack. Maybe a Windows Update updated/changed just the right amount of files that triggered KART to “protect” me. After our environment became “normal” again I spent some time trying to replicate the issue. I had Reflect backups of two machines from a week prior to the problem so I restored those and tried to let the problem re-occur. It never did. Windows updated as expected, KART updated and everything worked normally. I also installed Win10 in a VM and installed KART4. I let everything update (Win10 20H2 and KART5) but still the problem did NOT reoccur. An unofficial inventory of our environment (and including friends and family) shows that not every KART5 system suffered and I cannot find any common links between those that did. My negative side thinks that a bad KART5 update was released, pulled and re-released without telling anyone. Thus only a few unlucky systems were affected. The files are deleted at boot so it’s possible a bugged KART5 was released and only systems that were rebooted between that release and the bugfix release were impacted. It’s unfortunate but I knew this was free software. I knew there was no support. I accept that. I get the bonus of free but there actually was a cost. I’m not yelling, I’m not demanding Kaspersky be held accountable. I came here to get help and to possibly help others with what I learned. I appreciate all the input that was given. I do hope @steve_paul_quinn does manage to get an official response from K and can share it with us.

Steve is correct. Funny how this was never directly pointed out. As long as KART remained installed, on every reboot it would delete the same files over and over again. (A few of my earlier posts discussed how (for testing) I kept re-copying the *cat files after every reboot and that a ProcMon log showed that it was KART deleting the files). Renaming the Kaspersky folder would definitely solve that problem.

I am not an expert and have been testing my way through this very slowly and with many mistakes, but if I had that scenario I would FIll catroot Copy the DriverStore from a backup onto a live machine. RebootThat scenario has worked every time for me this past week. Again, I am not an expert and can only offer my opinion. But I have confidence in that process. There will still be missing files. Depending on how you do your backups, you could use a “folder compare” program to compare the current Windows folder with the backup to see if there are any *other* removed files. As mentioned before, the Microsoft C++ runtimes (usually in the system32 folder) were also deleted on my servers and workstations. Some non-essential programs need these to run. After the reboot, I also install the runtimes which can be downloaded freely. (It was mainly programs such as Adobe Reader and some Xerox utilities).

I had a low-priority laptop exhibiting the same problem brought to me and I was able to do repeated tests on it (reboot, copy cats, reboot, copy cats, etc)

This key is required for most backup programs and system restore to work so YES you should create it. I will paste the text below so you can create a regfile for it. open Notepad, copy/paste and save as *.reg Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f}] "UpperFilters"=hex(7):76,00,6f,00,6c,00,73,00,6e,00,61,00,70,00,00,00,00,00

That’s normal for just the plain MMC. (You’d usually add snap-ins). On Windows 10 and Sever 2012 I never got it to run. Just gave an error that it wasn’t allowed to run. I asked myself this. I even have the full KART5 log where it show that it is looking at each file, but it does not say why it was deleted. Letting KART5 stay running is fine. The delete happens only at boot. If I copy all the files back to catroot they will stay there forever until the next reboot. Only deleted at reboot. For the files that remain in catroot, it is because they are also being used by another process and KART5 cannot delete them. Searching for only the specific “Critical Service Failed” led me to this article https://rquintino.wordpress.com/2017/05/11/recovering-from-windows-10-boot-blue-screen-critical-service-failed-disable-drivers-signature-enforcement-unsigned-drivers/ where it talks about an empty catroot folder.

You can check after. It is NOT a requirement for boot. I was never able to reproduce this on a fresh install nor was I able to find the reason why KART5 started doing this. I played with a laptop for a long time, every reboot would delete the files. I would re-copy them and try various KART5 settings. After a reboot it would always delete them. I checked an old SBS2011 server I had here. It works fine. It only has 524 *.cat files and NO DriverStore folder (only an empty DRVSTORE folder which is normal)

Hi again @Inet I’m no expert but I’ll try to offer my opinion on your situation and what I would do. The empty catroot subfolder is definitely a problem. The external hard drive that isn’t recognized is because of the empty DriverStore folder. All the built-in Windows drivers are in that folder. You’ll find that different USB mice, keyboards and USB sticks all will not work. If you plug the external hard drive into one of the other servers (with 1000 files, 3000 or more) does it get recognized ? As Windows does updates and gets new system files it adds more and more *cat files therefore the number can change over time and also why I’m worried about copying *cat files from older backups. My experience is that all important *cat files are duplicated in the Packages folder and that’s why I recommend copying all of them from there into the {F750E6C3-38EE-11D1-85E5-00C04FC295EE} folder. If you copy *cat from Packages on the server with empty catroot folder, do the services start ? HOWEVER files from the DriverStore folder should be perfectly good to copy from an older backup since the DriverStore files do NOT change that often. My testing indicates that KART5 deletes all the catroot files at boot and stops. It does not keep deleting. That and the fact that most of my servers have between 1000 - 3000 *cat files seem to indicate that those servers will be fine for reboot. Have you compared the number of files in the catroot folder from current servers to older backups? Are they approximately the same count ? I am absolutely sure that it is only a catroot folder with less than 100 *cat files that will prevent a server or workstation from booting. Driverstore didn’t stop my computers from booting. They booted but I could *not* add devices and the printers were corrupted. Copying DriverStore from a backup helped solve that on a server.

I think this is the same as the System Restore bug mentioned earlier by myself and a few others. Try the Registry fix AND a restart: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f} UpperFilters REG_MULTI_SZ volsnap https://www.reddit.com/r/GoroHome/comments/c9i6rl/kaspersky_removal_tool_kavremover_chkdsk_volume/ or https://www.sysnative.com/forums/threads/system-restore-not-working-error-0x81000203-kaspersky-removal-tool.28983/

I also wondered this. At first I thought it might be just the auto-update version “3660”. Our process has always been: 1) Install KART4 and let it auto-update. You mention that you install KART5 directly from the downloadable installer. I tested this and the version currently available is 3039. But it NEVER auto updates for me and stays are 3039. I thought maybe the auto-update 3660 was compromised but if you are experiencing the same problem from the direct download then I don’t know. I can say for certain that on the machines I was able to investigate it was always KART5 3660 that had the problem.

In my experience some network services stop to work and connected devices. Like Remote desktop, Web cameras. Same. We also noticed that we cannot run mms. It seems like the catroot folder contains the digital signatures for many Windows drivers. Without the digital signature, Windows will not allow programs and services to run. The DriveStore also gets deleted so we noticed that on machines where we recovered the catroot folder we could not add new devices and most printers were corrupt. This is where either a copy-from-backup DriverStore helped OR reinstalling the Windows 10 20H2 update using the Media Creation Tool rebuilt the DriverStore folder.

Pardon my late reply. For us the catRoot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE} folder was not protected so there was no issue copying. Are you logged in as an Admin? Are you running a cmd prompt as Admin? Maybe try taking ownership of the folder and giving your user account full access to the folder.

The catroot folder contains the “catalog” of digital signatures and certificates used by Windows. In my understanding you are correct. Without these files there is no way for Windows to “verify” that certain programs are allowed to run. We also noticed that a machine suffering with this problem cannot run “mmc”. Again, you’ll need to refer back to a few of my initial posts where I talk about copying the *cat files from the Packages folders right back into the catroot sub folder “{F750E6C3-38EE-11D1-85E5-00C04FC295EE}”. copy /y c:\Windows\servicing\Packages\*.cat c:\Windows\System32\CatRoot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE} The catroot folder shouldn’t change much so copying from the January backup should also be good. Since you have a backup from January, you should also copy all the contents of the DriverStore folder from the backup onto the live machine. In one of my first few posts I discovered that the C:\Windows\System32\DriverStore\en-US and C:\Windows\System32\DriverStore\FileRepository were missing MANY files as well. (Those two folders are protected so you’ll have to take ownership of them and give your user account Full Access to them). Emptying the catroot2 folder has not helped us. catroot2 is automatically rebuilt by Windows, not catroot.

I’m sorry to hear that. I was hoping my solution would be able to help others. No, for us it was only the *cat files. On a fresh Win10 install the {71a27cdd-812a-11d0-bec7-08002be2092f} folder only contains *cat files

Interesting. I don’t no for sure. Possibly KART will not update since it is 1909. BUT if the computer updates to 20H2 then KART will probably auto-update to version 5 then.

I had also seen that forum post. I feel it is exactly the same. But why Kaspersky would cause this? Sadly that user also had to do a full reinstall. I also talked about this problem and forum discussion when uninstalling KART. Hopefully they read it and take it seriously. It is very strange that even with all the *cat files your computer won’t boot. I was very happy when that simple solution worked for us. I’m sad that it didn’t help you. To check the driver signing on a working PC. Use cmd prompt , Run As Admin, then type: bcdedit IF it is disabled on the working PC you will see a row for nointegritychecks ON. Screenshot: https://ibb.co/XJw7rZ9 On important laptops we pay for ESET ( about 12 laptops). We were using KART to help with less-important laptops. We are in the process of KART removal and trying to encourage backup to either Google Drive or OneDrive. Not sure what the next steps will be.

Hi again Inet. Thank you for all your input. -We’ve been lucky that copying the *cat files always works. We identify affected PCs simply by that. IF the c:\Windows\System32\CatRoot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\ has less than 100 files in it, the PC will not boot. -I have heard in other (unrelated) BSOD that turning OFF “Disable drivers signature enforcement” allows the PC to boot but that did not work in our case. Do some of your PCs have that permanently turned off for other reasons? -I did not write to Kaspersky. I guessed because it was a free product there would be no support. I wish you luck. Please let me know if they respond. -Yes. ALL our Windows KARTs update to 5.0.0.3660 but some PCs with that version work fine and never experience the problem. -We do not have it on our servers. We do pay for another product for our small amount of servers. We have many workstations so a free option was the best choice. -I have not found any other product as small, simple and free as KART. It was great for years until this problem. We have also added the folder protection in Windows Defender Antivirus.

I am now 100% positive that it is KART causing this. But I don't know why. I have used the laptop and copied the *cat files over and over again. - I used a program named ProcMon to log all processes at boot. It shows that KART tried to delete every file in the 'catroot' folder and then tries to delete the folder (fails as the folder isn't empty). Screenshots: https://ibb.co/2jmkZbJ and https://ibb.co/B6qgJjQ - Disabling KART service from Safe Mode stops the file deletion. Starting it again causes the files to delete. - Changing ANY setting in KART (user mode, protection, malware, etc) does NOT help. Files are still deleted. - Accepting or reading the new license agreement does NOT help. Files are still deleted. - My version is KART 5.0.0.3660 (aka 3.0.1.3660) Screenshot https://ibb.co/5sry0Rf