steve_paul_quinn

Hi Vasily/Folks I've had to adjust the procedure slightly but was able to recreate and hope fully capture something helpful. I was able to get one reboot without a BSOD and the next reboot with a BSOD. Hopefully yay. The timestamps to look for are from May 05 2021 at 7:33 AM EST and are hopefully in AntiRansom.3.0.1.3660_05.05_07.33_4084.SRV.log Along with the logs, I've included a copy of the files in C:\Program Files (x86)\Kaspersky Lab in case they are helpful. I've used 7z with ultra compression to get the files down from 783 MB to 208 MB The files are password protected and shared below, PM me for the password https://drive.google.com/file/d/1WErYxo9mJR19OT8tD9KAHcOjM1_c01bY/view?usp=sharing Here is the procedure I used 1 Boot with Macrium Reflect PE Rescue Environment 2 Restore the BSOD Macrium Image 3 Restore the CatRoot and DriverStore files 4 Rename C:\Program Files (x86)\Kaspersky Lab to C:\Program Files (x86)\Kaspersky Lab Old 5 Restart the machine 6 Rename C:\Program Files (x86)\Kaspersky Lab Old to C:\Program Files (x86)\Kaspersky Lab 7 Enable product logging on maximum level 8 Reboot and witness BSOD 9 Boot with Macrium Reflect PE Rescue Environment 10 Restore the CatRoot and DriverStore files 11 Reboot and witness no issues 12 Reboot and witness BSOD 13 Boot with Macrium Reflect PE Rescue Environment 14 Copy AllUsersProfile Kaspersky Lab\Logs and files from From C:\Program Files (x86)\Kaspersky Lab 15 Compress with 7z 16 Upload with Google Drive 17 Drink Vodka and Pray Take care Steve

Hi, Steve! Sorry for delay, in Russia we have small holidays :-) We still can’t reproduce this issue :-( Thank you for submitting the playback algorithm. I suggest the following way: Restore the Macrium Image Enable product logging on maximum level. Restart the product Restore the CatRoot and DriverStore files Restart the computer and get the BSOD After memory dump is created - restart computer again in safe mode Save product logs (please see “Log application events” chapter in online help) and memory dump to another location. Please write here the message if you succeed. I will consult with our legal department about the method of transferring traces to us. Thanks! Hi Folks Thanks for the update. I hope you had a nice holiday. I only have a Macrium image of the machine AFTER a BSOD. I will need to adjust the process slightly. I will try this tomorrow, it was a long day. 1 Restore the BSOD Macrium Image 2 Restore the CatRoot and DriverStore files 3 Rename the Kaspersky Lab files so they do not delete CatRoot and DriveStore on the next reboot 4 Restart the machine 5 Somehow enable product logging on maximum level. I hope I can. 6 Rename the Kaspersky Lab files back to their original names 7 Reboot, hope for a BSOD and get you the logs Any adjustments or suggestions?

Hi Vasily/Kaspersky Team It’s been several days with no official response from Kaspersky. Can you please give us an update? Steve

Hi Vasily/Kaspersky Team Perhaps my original post got lost in this thread. I will repost and await a reply I have an idea to accelerate your research. I have a Macrium backup of my Zbook after the BSOD occurred. In restoring this image to investigate I have to do the following process 1 Restore the Macrium Image 2 Restore the CatRoot and DriverStore files 3 Manually recursively delete C:\Program Files (x86)\Kaspersky Lab\ If I skip step 3, CatRoot and DriverStore are deleted on the next reboot. Would a copy of this Macrium image not be helpful for your team to investigate? I can somehow upload it to you for a Physical restore or VM. It wont reveal the triggers to cause the issue but it may be helpful for a post mortem analysis. Just an idea :-) Take care Steve

Hello, can you please tell us how we can make sure if a computer is affected and will start as soon we restart it ? We have many Servers (SBS 2011, Windows 2008R2, Windows 2016 Server, Windows 2019 Server) which have never been restarted since the problem arised. Please do respond soon to this urgent issue ! If somebody else has a definitive answer to this question i am happy to hear about. Regards, Mike Hi Mike This forum thread is getting kinda messy and hard to follow. Hopefully I can help you. I’ll summarize if you have not reviewed this entire thread. I’ve got a laptop with a Macrium backup of the BSOD issue on Windows 10 x64 1909. Perhaps what I know will help you with your Windows Servers. The first clue of a KART issue is the removal of the UpperFilters registry entry. This is easy to check for. Several days later, it appears a BSOD is caused by the removal of CatRoot and DriverStore during the next reboot. I know of no way predict this. It did happen for many of us with KART Application version 3660 which has been since upgraded. Time will tell if the BSOD risk remains. For all my customers, I am removing KART from their “working” machines to prevent the BSOD risk. If the KART files are not present during the next reboot, CatRoot and DriverStore “should” remain intact. If I was you, I would create a small repo of CatRoot and DriverStore files for all your system variants from working machines. Just in case they are needed. I would also prepare and test a working PE recovery environment proactively so recovery is not in a panic when needed. I like Macrium for this and there are certainly others to choose from. Hope this helps Steve I hope this helps

Did you installed the product in non-default path? Oops my bad sorry. I manually recursively deleted C:\Program Files (x86)\Kaspersky Lab\

Hi, Steve Thanks for very useful info! KART update was installed and popup asking to restart is displayed - when you restarted the PC after that (immediately or with delay)? Can you write the sequence of your actions? Hi Vasily I’m sorry but I cannot recall the exact sequence of events prior to the BSOD as the issue occurred on March 28 2021. I do recall turning on my Zbook while I was working with a customer experiencing the same issue. I’m pretty sure I was prompted to restart for a KART update and I did restart immediately. I hope this helps. Steve

Hi Vasily I have an idea to accelerate your research. I have a Macrium backup of my Zbook after the BSOD occurred. In restoring this image to investigate I have to do the following process 1 Restore the Macrium Image 2 Restore the CatRoot and DriverStore files 3 Manually recursively delete C:\Kaspersky Lab If I skip step 3, CatRoot and DriverStore are deleted on the next reboot. Would a copy of this Macrium image not be helpful for your team to investigate? I can somehow upload it to you for a Physical restore or VM. It wont reveal the triggers to cause the issue but it may be helpful for a post mortem analysis. Just an idea :-) Take care Steve

Hi Vasily It is confirmed. Windows Update Blocker v1.5 was used and Windows updates is disabled. In my situation the BSOD issue was independent of KB500802/KB500808. I’ve included the KB that were installed and the Windows Version as 1909

Hi, Mike! I don’t know what computers will be affected by this issue. In our labs we don’t have the same problem :-( We try to reproduce it. At this moment we think that fix of the UpperFilters registry value in last product update will fix the BSOD too. Can anybody to tell about the sequence of updates install that led to BSOD: Windows KB500802/KB500808 update was installed, then update for KART and then computer was restarted or KART update was installed, then Windows KB500802/KB500808 update and then computer was restarted or KART update was installed and then computer was restarted but windows KB500802/KB500808 update installation was still in progress or It does not matter? I appreciate you for help. Hi Vasily I created for reference, a Macrium backup of my physical Zbook after I experienced the BSOD. I will recover it and look at the status of KB500802/KB500808. I have honestly been shell shocked by poor MS patch stability for quite some time now. There is a possibility that I had disabled Windows Updates using Windows Update Blocker. https://www.sordum.org/9470/windows-update-blocker-v1-6/ I will confirm for you. Hope this helps Steve

Hi, Steve! It will be very helpful! Please try to reproduce this BSOD with product logs on maximum level. Can you list here the value of UpperFilters registry parameter on restored machine before product update? Thanks. Hi Vasily Sure. Here you go. UpperFilters has a single Data entry of volsnap

Hi, Steve! Thanks for you reply. I agree that response time was long and hope that this situation will not repeats again. In our lab we can not reproduce this BSOD at the moment. I need some time to think about how we can reproduce this on your configuration, I will be discuss that with colleagues tomorrow. I really want to catch this BSOD. 😡 Thanks. Hi Vasily I have a parallel test idea to keep momentum on troubleshooting this issue since we may be in different time zones. I keep a small repo of files at home for offline use. I noticed I have KART_5.0.0.92320-Home.exe This installer is Product version 3.0.1.3660 which we have seen be a problem. I have a second ZBook G2 I use for testing stuff with VMware Workstation I'll spin up a similar vintage Windows 10 x64 Pro VM to test out 3660 I'll try to block KART or disable networking so as to keep it at 3660 FYI the Zbook G1 physical testing and now virtual testing will be done with Windows 10 x64 Pro 1909 If you have adjustments or suggestions I'm all ears Take care Steve

Hi, Steve! Thanks for you reply. I agree that response time was long and hope that this situation will not repeats again. In our lab we can not reproduce this BSOD at the moment. I need some time to think about how we can reproduce this on your configuration, I will be discuss that with colleagues tomorrow. I really want to catch this BSOD. 😡 Thanks. Hi Vasily You are very welcome. I too want to catch it. It’s a very nasty bug. Many of us noticed that the BSOD occurred with 3660 a few days after UpperFilters was removed. I am very tempted to restore my test machine to 3660, disable the network interface to prevent an upgrade and see if I can recreate the problem. Do you think this would be helpful? Steve

Hi Vasily I really appreciate an official Kaspersky representative reaching out. I’m sure you would agree the response time has been very slow. I hope that whatever the cause of the slow response has been addressed. I have many customers who are now afraid of Kaspersky software. You may also notice there are others who voiced these same opinions in this forum thread. I ‘d like to work together to gain back their trust. As you may see from my contributions in this thread, I have a HP ZBook G1 that had experienced the BSOD issue. I reverted to a previous backup and have monitored the machine for many days. I am hoping to assist with catching the BSOD bug. As expected, the machine automatically upgraded to 5.0.0.3886(i). I have enabled Event Logging with Maximum detail. I’ve also created a reboot task for every 30 minutes. It has rebooted without issue for 3 days. If there are any changes or suggestions to my test environment, I would appreciate your input Take care Steve Quinn

Hi Folks. I noticed 3.0.1.3886 was pushed out and upgraded in the last 24 hours. I’ve rebooted, all is ok and continue to test. Notes here https://docs.google.com/document/d/1gDzDaWPk4s8L2eqP6qERVojFXUtevCaca2G7SmO0wK8/edit

@pdwkand you have done a great work on this topic and it has helped me a lot too, thank you. For my tests, regarding the second setup of KART on my VM, despite the KART update notice, Windows log says that there was no update and KART is not updating yet. Thanks. It feels good to help others save hair. :-) Regarding your tests, it sounds like Kaspersky may have wisely paused the problematic update.

Great work CF93. Thanks for sharing your details. Hopefully it will help others who land here and find this thread. 😎

That’s good news. Nothing to delete I suppose :-)

Thanks PDWK. The question was regarding LowerFilters :-)

Thanks for your good advice, I will try this and tell the result if no one can answer. No problemo. This thread has been really quiet lately so it might be faster to build a VM. Please post your results here so others can benefit :-)

Hi CF93 Those are all good questions. I don’t have a Windows 2016 to compare sorry. Hopefully you have one. It should not take too long to build a similar vintage 2016 VM to validate as a reference. Good luck

Made a little batch file 1 Query the registry to see if UpperFiler is sad 2 Pop open System32 so I can lazily inspect the size of CarRoot and DriverStore 3 Pop open System Properties so I can if System Protection is sad 4 Open Macrium if installed to query the last good backups and VSS state It’s not perfect but it saves me time. YMMV Save below as whatever.bat and have fun reg query HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f} >UpperFiltersResult.txt & notepad UpperFiltersResult.txt pause explorer C:\Windows\System32 pause sysdm.cpl pause cd C:\Program Files\Macrium\Reflect reflect pause

My current KART torture chamber is my personal HP ZBook. I had restored a working Macrium image onto it that had KART 3409 already installed. I watched it upgrade in front of my eyes after about 4 hours of waiting to 3660 UpperFilters is now toast. I had a customer who was in this exact situation. Backups had failed for 4 to 5 days. I got called in for a BSOD and was first introduced to this mess. The last working backup had no UpperFilter. I made a backup of the machine in this BSOD state. I just checked now CatRoot/DriverStore are significantly smaller than his current working system. I’ll leave my Zbook on and watch if for days. I have a funny feeling 3660 is still messed up and CatRoot and DriverStore will soon be gone.

WTF. I just watched an upgrade to KART 3660 occur live and UpperFilter is now gone. Now I await the deletion of CatRoot and DriverStore. I should be LiveStreaming this lol https://docs.google.com/document/d/1gDzDaWPk4s8L2eqP6qERVojFXUtevCaca2G7SmO0wK8/edit I’m glad I started uninstalling KART on my customer machines. It almost seems intentional. First kill VSS so no backups can be made, then later render the system unbootable by deleting CatRoot and DriverStore. Is this intentional or has the update mechanism been compromised by an external actor?

Thanks PDWK. Awesome to hear you too are Canadian. I’ve learned a bunch. I’m sure the Kaspersky lurkers have as well. We wound up doing a bunch of their support, R&D and QA in this thread.