Jump to content
Kaspersky Support Forum

steve_paul_quinn

Members
 View Profile  See their activity

  • Posts

    35

  • Joined

  • Last visited

    Never

 Content Type 

Posts posted by steve_paul_quinn

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hi Vasily/Folks

    I've had to adjust the procedure slightly but was able to recreate and hope fully capture something helpful.  I was able to get one reboot without a BSOD and the next reboot with a BSOD.  Hopefully yay.

    The timestamps to look for are from May 05 2021 at 7:33 AM EST and are hopefully in AntiRansom.3.0.1.3660_05.05_07.33_4084.SRV.log
    Along with the logs, I've included a copy of the files in C:\Program Files (x86)\Kaspersky Lab in case they are helpful.

    I've used 7z with ultra compression to get the files down from 783 MB to 208 MB
    The files are password protected and shared below, PM me for the password
    https://drive.google.com/file/d/1WErYxo9mJR19OT8tD9KAHcOjM1_c01bY/view?usp=sharing

    Here is the procedure I used

    1 Boot with Macrium Reflect PE Rescue Environment

    2 Restore the BSOD Macrium Image

    3 Restore the CatRoot and DriverStore files

    4 Rename C:\Program Files (x86)\Kaspersky Lab to C:\Program Files (x86)\Kaspersky Lab Old

    5 Restart the machine

    6 Rename C:\Program Files (x86)\Kaspersky Lab Old to C:\Program Files (x86)\Kaspersky Lab

    7 Enable product logging on maximum level

    8 Reboot and witness BSOD

    9 Boot with Macrium Reflect PE Rescue Environment

    10 Restore the CatRoot and DriverStore files

    11 Reboot and witness no issues

    12 Reboot and witness BSOD

    13 Boot with Macrium Reflect PE Rescue Environment

    14 Copy AllUsersProfile Kaspersky Lab\Logs and files from From C:\Program Files (x86)\Kaspersky Lab

    15 Compress with 7z

    16 Upload with Google Drive

    17 Drink Vodka and Pray

    Take care

    Steve

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hi Vasily/Kaspersky Team

    It’s been several days with no official response from Kaspersky.

    Can you please give us an update?

    Steve

     

    Hi, Steve!

    Sorry for delay, in Russia we have small holidays :-) We still can’t reproduce this issue :-( Thank you for submitting the playback algorithm. I suggest the following way:

    1. Restore the Macrium Image
    2. Enable product logging on maximum level.
    3. Restart the product
    4. Restore the CatRoot and DriverStore files
    5. Restart the computer and get the BSOD
    6. After memory dump is created - restart computer again in safe mode
    7. Save product logs (please see “Log application events” chapter in online help) and memory dump to another location.

    Please write here the message if you succeed. I will consult with our legal department about the method of transferring traces to us.

    Thanks!

     

     

    Hi Folks

    Thanks for the update.  I hope you had a nice holiday.

    I only have a Macrium image of the machine AFTER a BSOD. 

    I will need to adjust the process slightly.  I will try this tomorrow, it was a long day.

     

    1 Restore the BSOD Macrium Image

    2 Restore the CatRoot and DriverStore files

    3 Rename the Kaspersky Lab files so they do not delete CatRoot and DriveStore on the next reboot

    4 Restart the machine

    5 Somehow enable product logging on maximum level.  I hope I can.

    6 Rename the Kaspersky Lab files back to their original names

    7 Reboot, hope for a BSOD and get you the logs

     

    Any adjustments or suggestions?

     

     

     

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hi Vasily/Kaspersky Team

    Perhaps my original post got lost in this thread.  I will repost and await a reply

     

    I have an idea to accelerate your research.

    I have a Macrium backup of my Zbook after the BSOD occurred.  In restoring this image to investigate I have to do the following process

    1 Restore the Macrium Image

    2 Restore the CatRoot and DriverStore files

    3 Manually recursively delete C:\Program Files (x86)\Kaspersky Lab\

    If I skip step 3, CatRoot and DriverStore are deleted on the next reboot.

    Would a copy of this Macrium image not be helpful for your team to investigate?  I can somehow upload it to you for a Physical restore or VM.   It wont reveal the triggers to cause the issue but it may be helpful for a post mortem analysis. Just an idea :-)

    Take care

    Steve

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hi, All!
    Sorry for late response. Thank you for your patience and for reporting about these issues. The problem with UpperFilters registry value was fixed in last product update. We continue investigation of BSOD problem but we think that this may be due to the first problem with the registry.

    P.S. The actual version of the product can be found here: 
    Menu -> Get Support


    Thanks.

     

    Hello, can you please tell us how we can make sure if a computer is affected and will start as soon we restart it ?

    We have many Servers (SBS 2011, Windows 2008R2, Windows 2016 Server, Windows 2019 Server) which have never been restarted since the problem arised.

    Please do respond soon to this urgent issue !

     

    If somebody else has a definitive answer to this question i am happy to hear about.

     

    Regards,

     

    Mike

     

     

    Hi Mike

    This forum thread is getting kinda messy and hard to follow.  Hopefully I can help you.  I’ll summarize if you have not reviewed this entire thread.  I’ve got a laptop with a Macrium backup of the BSOD issue on Windows 10 x64 1909.  Perhaps what I know will help you with your Windows Servers.

    The first clue of a KART issue is the removal of the UpperFilters registry entry.  This is easy to check for.   Several days later, it appears a BSOD is caused by the removal of CatRoot and DriverStore during the next reboot.  I know of no way predict this.  It did happen for many of us with KART Application version 3660 which has been since upgraded.  Time will tell if the BSOD risk remains.

    For all my customers, I am removing KART from their “working” machines to prevent the BSOD risk.  If the KART files are not present during the next reboot, CatRoot and DriverStore “should” remain intact.

    If I was you, I would create a small repo of CatRoot and DriverStore files for all your system variants from working machines.  Just in case they are needed.  I would also prepare and test a working PE recovery environment proactively so recovery is not in a panic when needed.  I like Macrium for this and there are certainly others to choose from.

    Hope this helps

    Steve

     

     

     

    I hope this helps

     

     

     

     

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hi Vasily

    It is confirmed.  Windows Update Blocker v1.5 was used and Windows updates is disabled.  In my situation the BSOD issue was independent of KB500802/KB500808.  I’ve included the KB that were installed and the Windows Version as 1909

     

     

     

     

     

     

    Hi, Steve

    Thanks for very useful info! 

    KART update was installed and popup asking to restart is displayed - when you restarted the PC after that (immediately or with delay)? Can you write the sequence of your actions?  

     

    Hi Vasily

    I’m sorry but I cannot recall the exact sequence of events prior to the BSOD as the issue occurred on March 28 2021.  I do recall turning on my Zbook while I was working with a customer experiencing the same issue.  I’m pretty sure I was prompted to restart for a KART update and I did restart immediately.  I hope this helps.

    Steve

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hi Vasily

    I have an idea to accelerate your research.

    I have a Macrium backup of my Zbook after the BSOD occurred.  In restoring this image to investigate I have to do the following process

    1 Restore the Macrium Image

    2 Restore the CatRoot and DriverStore files

    3 Manually recursively delete C:\Kaspersky Lab 

    If I skip step 3, CatRoot and DriverStore are deleted on the next reboot.

    Would a copy of this Macrium image not be helpful for your team to investigate?  I can somehow upload it to you for a Physical restore or VM.   It wont reveal the triggers to cause the issue but it may be helpful for a post mortem analysis. Just an idea :-)

    Take care

    Steve

     

     

     

     

     

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hello, can you please tell us how we can make sure if a computer is affected and will start as soon we restart it ?

    We have many Servers (SBS 2011, Windows 2008R2, Windows 2016 Server, Windows 2019 Server) which have never been restarted since the problem arised.

    Please do respond soon to this urgent issue !

     

    If somebody else has a definitive answer to this question i am happy to hear about.

     

    Regards,

     

    Mike

     

    Hi, Mike!

    I don’t know what computers will be affected by this issue. In our labs we don’t have the same problem :-( We try to reproduce it. At this moment we think that fix of the UpperFilters registry value in last product update will fix the BSOD too.

    Can anybody to tell about the sequence of updates install that led to BSOD:
    Windows KB500802/KB500808 update was installed, then update for KART and then computer was restarted
    or
    KART update was installed, then Windows KB500802/KB500808 update and then computer was restarted
    or
    KART update was installed and then computer was restarted but windows KB500802/KB500808 update installation was still in progress
    or
    It does not matter?

    I appreciate you for help.

     

     

    Hi Vasily

    I created for reference, a Macrium backup of my physical Zbook after I experienced the BSOD.  I will recover it and look at the status of KB500802/KB500808.  I have honestly been shell shocked by poor MS patch stability for quite some time now.  There is a possibility that I had disabled Windows Updates using Windows Update Blocker. https://www.sordum.org/9470/windows-update-blocker-v1-6/  I will confirm for you.

    Hope this helps

    Steve

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hi Vasily

    You are very welcome.  I too want to catch it.  It’s a very nasty bug.

    Many of us noticed that the BSOD occurred with 3660 a few days after UpperFilters was removed.  I am very tempted to restore my test machine to 3660, disable the network interface to prevent an upgrade and see if I can recreate the problem.  Do you think this would be helpful?

    Steve

     

    Hi, Steve!

    It will be very helpful! Please try to reproduce this BSOD with product logs on maximum level. 

    Can you list here the value of UpperFilters registry parameter on restored machine before product update?

    Thanks.

     

    Hi Vasily

    Sure.  Here you go.  UpperFilters has a single Data entry of volsnap

     

     

     

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

     

    Hi Vasily

     

    I really appreciate an official Kaspersky representative reaching out.  I’m sure you would agree the response time has been very slow.  I hope that whatever the cause of the slow response has been addressed.  I have many customers who are now afraid of Kaspersky software.  You may also notice there are others who voiced these same opinions in this forum thread.  I ‘d like to work together to gain back their trust.

     

    As you may see from my contributions in this thread, I have a HP ZBook G1 that had experienced the BSOD issue.  I reverted to a previous backup and have monitored the machine for many days.  I am hoping to assist with catching the BSOD bug.  As expected, the machine automatically upgraded to 5.0.0.3886(i).  I have enabled Event Logging with Maximum detail.  I’ve also created a reboot task for every 30 minutes.  It has rebooted without issue for 3 days.

     

    If there are any changes or suggestions to my test environment, I would appreciate your input

     

    Take care

     

    Steve Quinn

     

     

    Hi, Steve!

    Thanks for you reply. I agree that response time was long and hope that this situation will not repeats again. 

    In our lab we can not reproduce this BSOD at the moment. I need some time to think about how we can reproduce this on your configuration, I will be discuss that with colleagues tomorrow. 
    I really want to catch this BSOD. 😡

    Thanks.

     

     

    Hi Vasily

    I have a parallel test idea to keep momentum on troubleshooting this issue since we may be in different time zones.  I keep a small repo of files at home for offline use.

    I noticed I have KART_5.0.0.92320-Home.exe  This installer is Product version 3.0.1.3660 which we have seen be a problem.

    I have a second ZBook G2 I use for testing stuff with VMware Workstation

    I'll spin up a similar vintage Windows 10 x64 Pro VM to test out 3660
    I'll try to block KART or disable networking so as to keep it at 3660

    FYI the Zbook G1 physical testing and now virtual testing will be done with Windows 10 x64 Pro 1909

    If you have adjustments or suggestions I'm all ears

    Take care

    Steve

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

     

    Hi Vasily

     

    I really appreciate an official Kaspersky representative reaching out.  I’m sure you would agree the response time has been very slow.  I hope that whatever the cause of the slow response has been addressed.  I have many customers who are now afraid of Kaspersky software.  You may also notice there are others who voiced these same opinions in this forum thread.  I ‘d like to work together to gain back their trust.

     

    As you may see from my contributions in this thread, I have a HP ZBook G1 that had experienced the BSOD issue.  I reverted to a previous backup and have monitored the machine for many days.  I am hoping to assist with catching the BSOD bug.  As expected, the machine automatically upgraded to 5.0.0.3886(i).  I have enabled Event Logging with Maximum detail.  I’ve also created a reboot task for every 30 minutes.  It has rebooted without issue for 3 days.

     

    If there are any changes or suggestions to my test environment, I would appreciate your input

     

    Take care

     

    Steve Quinn

     

     

    Hi, Steve!

    Thanks for you reply. I agree that response time was long and hope that this situation will not repeats again. 

    In our lab we can not reproduce this BSOD at the moment. I need some time to think about how we can reproduce this on your configuration, I will be discuss that with colleagues tomorrow. 
    I really want to catch this BSOD. 😡

    Thanks.

     

    Hi Vasily

    You are very welcome.  I too want to catch it.  It’s a very nasty bug.

    Many of us noticed that the BSOD occurred with 3660 a few days after UpperFilters was removed.  I am very tempted to restore my test machine to 3660, disable the network interface to prevent an upgrade and see if I can recreate the problem.  Do you think this would be helpful?

    Steve

     

     

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hi, All!
    Sorry for late response. Thank you for your patience and for reporting about these issues. The problem with UpperFilters registry value was fixed in last product update. We continue investigation of BSOD problem but we think that this may be due to the first problem with the registry.

    P.S. The actual version of the product can be found here: 
    Menu -> Get Support


    Thanks.

     

     

    Hi Vasily

     

    I really appreciate an official Kaspersky representative reaching out.  I’m sure you would agree the response time has been very slow.  I hope that whatever the cause of the slow response has been addressed.  I have many customers who are now afraid of Kaspersky software.  You may also notice there are others who voiced these same opinions in this forum thread.  I ‘d like to work together to gain back their trust.

     

    As you may see from my contributions in this thread, I have a HP ZBook G1 that had experienced the BSOD issue.  I reverted to a previous backup and have monitored the machine for many days.  I am hoping to assist with catching the BSOD bug.  As expected, the machine automatically upgraded to 5.0.0.3886(i).  I have enabled Event Logging with Maximum detail.  I’ve also created a reboot task for every 30 minutes.  It has rebooted without issue for 3 days.

     

    If there are any changes or suggestions to my test environment, I would appreciate your input

     

    Take care

     

    Steve Quinn

     

     

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Great work CF93.  Thanks for sharing your details.  Hopefully it will help others who land here and find this thread.  😎

     

    @pdwkand you have done a great work on this topic and it has helped me a lot too, thank you.

     

    For my tests, regarding the second setup of KART on my VM, despite the KART update notice, Windows log says that there was no update and KART is not updating yet.

     

     

    Thanks.  It feels good to help others save hair.  :-)

    Regarding your tests, it sounds like Kaspersky may have wisely paused the problematic update.

     

     

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Hi CF93

    Those are all good questions.  I don’t have a Windows 2016 to compare sorry.  Hopefully you have one.  It should not take too long to build a similar vintage 2016 VM to validate as a reference.

    Good luck

     

    Thanks for your good advice, I will try this and tell the result if no one can answer.

     

    No problemo.  This thread has been really quiet lately so it might be faster to build a VM.  Please post your results here so others can benefit :-)

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Made a little batch file

    1 Query the registry to see if UpperFiler is sad

    2 Pop open System32 so I can lazily inspect the size of CarRoot and DriverStore

    3 Pop open System Properties so I can if System Protection is sad

    4 Open Macrium if installed to query the last good backups and VSS state

     

    It’s not perfect but it saves me time.  YMMV

    Save below as whatever.bat and have fun

    reg query HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f} >UpperFiltersResult.txt & notepad UpperFiltersResult.txt
    pause

    explorer C:\Windows\System32
    pause

    sysdm.cpl
    pause

    cd C:\Program Files\Macrium\Reflect
    reflect
    pause

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    My current KART torture chamber is my personal HP ZBook. 

    I had restored a working Macrium image onto it that had KART 3409 already installed.

    I watched it upgrade in front of my eyes after about 4 hours of waiting to 3660

    UpperFilters is now toast.

     

    I had a customer who was in this exact situation.  Backups had failed for 4 to 5 days.

    I got called in for a BSOD and was first introduced to this mess.

    The last working backup had no UpperFilter.

    I made a backup of the machine in this BSOD state.

    I just checked now

    CatRoot/DriverStore are significantly smaller than his current working system.

     

    I’ll leave my Zbook on and watch if for days.  I have a funny feeling 3660 is still messed up and CatRoot and DriverStore will soon be gone.

     

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    WTF.  I just watched an upgrade to KART 3660 occur live and UpperFilter is now gone.  Now I await the deletion of CatRoot and DriverStore.

    I should be LiveStreaming this lol

    https://docs.google.com/document/d/1gDzDaWPk4s8L2eqP6qERVojFXUtevCaca2G7SmO0wK8/edit

    I’m glad I started uninstalling KART on my customer machines.  It almost seems intentional.

    First kill VSS so no backups can be made, then later render the system unbootable by deleting CatRoot and DriverStore. 

    Is this intentional or has the update mechanism been compromised by an external actor?
     

     

    Any connection between KART 5 (3660) and BSOD Critical Service Failed

    in Kaspersky Anti-Ransomware Tool

    Posted

    Just wanted to commend you, a fellow Canadian, on your notes and the things you tried. I read them all and it’s like re-living my past few weeks. (Including the attempts to disable the KART service via unchecking in msconfig and forcing the service’s registry entries. I even tried changing the Security Permissions on the registry keys).

    As you said, despite this system-stopping problem, we all learned some things and together were able to brainstorm some solutions. 

     

    Thanks PDWK.  Awesome to hear you too are Canadian.  I’ve learned a bunch.  I’m sure the Kaspersky lurkers have as well.  We wound up doing a bunch of their support, R&D and QA in this thread.

×
×
  • Create New...