Jump to content
WallyL

Kaspersky update floods network

Recommended Posts

Yesterday we updated versions on the admin kit.

 

Admin kit has been moved to 8.0.2134

 

Workstation version 6.0.1424

Server Version 6.0.1424

Network Agent 8.0.2134

 

Now every 2 hours on the nose we get full network outages, this is causing production level servers to go offline and pulling down my wan links.

THIS HAS TO BE FIXED.

85 clients

 

2 images uploaded, one showing a port on my network and the other a wan link with green.

We left it all night long, and it is updateing computers as it goes.

 

But why would it flood my network so bad as to take down operational servers. We cant even collect time card punches.

Our update schedule is not set to every 2 hours. Image attached.

post-315724-1289665477_thumb.png

post-315724-1289665581_thumb.png

post-315724-1289665700_thumb.png

Edited by WallyL

Share this post


Link to post

We had a slow down at about 10 am yesterday when I was working on the updates.

Then it started again at 2am.

Image attached.

 

We have now disabled all adminkit functions, will have to wait several hours to see what is happening.

post-315724-1289754577_thumb.png

Share this post


Link to post

I'm not liking the sounds of this. I'm finding lately a lot of companies are modifying their software for "better performance" but all they are doing is modifying their TCP\IP protocol to not play nice and use more of the available bandwidth. We use a file transfer application that was recently updated and now everytime one of our locations use it to transfer large graphic files between sites it slows their network to a crawl affecting all production on it. We've had to modify QOS on our firewalls to throttle the traffic from these clients. I wonder if Kaspersky is doing the same thing with this update.

Share this post


Link to post

Rob take a quick look at my graphs there.

 

Its like crazy, where it will physicaly flood my productions servers and stuff goes offline.

You can see it in the graphs.

 

I shut it all down, today graphs are clean and no phone calls.

Because I am going out of town tonight I dont have time to fix it properly. But we are talking about starting it all up again Friday and let it run all weekend again and see if it just needs time to sort itself out.

 

But you are right, it should have been tested. Its also odd that its every 2 hours on the button....

Share this post


Link to post
Rob take a quick look at my graphs there.

 

I looked them over and that's absolutely nuts. This is a big issue for us because we are waiting for the corporate version of the Kaspersky Mac client being released next week which requires this admin kit update. We found an issue with our current SAV installs on the Mac where it's corrupting our graphic files during scans so we've uninstalled it but this puts a damper on the 500+ clients we are looking to move to Kaspersky Mac.

Share this post


Link to post

Hi Everyone,

 

I just upgraded the bulk of my clients this morning, about 2,500. The upgrade took only about 1.5 hours and we did not see any network issues whatsoever.

 

Good luck,

Bruce

Share this post


Link to post

Bruce,

 

Did you do an upgrade of the admin kit itself, I installed over top, did not do a fresh install.

I am at a loss as to why its every 2 hours too.

 

 

watch your network and see how things go....I thought all was ok too for about 4 hours.....

Edited by WallyL

Share this post


Link to post
Bruce,

 

Did you do an upgrade of the admin kit itself, I installed over top, did not do a fresh install.

I am at a loss as to why its every 2 hours too.

watch your network and see how things go....I thought all was ok too for about 4 hours.....

 

 

I upgraded my largest adminkit base this morning. I installed over the top of a 8.0.2090 install. In fact, I just finished another adminkit in our TYO office and have not seen any problems. Where are you seeing the traffic? Is it on the AdminKit server or perhaps the clients connecting in?

Share this post


Link to post
I upgraded my largest adminkit base this morning. I installed over the top of a 8.0.2090 install. In fact, I just finished another adminkit in our TYO office and have not seen any problems. Where are you seeing the traffic? Is it on the AdminKit server or perhaps the clients connecting in?

 

 

I would appreciate any additional updates on this issue. I am slowly upgrading network agents and having my communications gurus keep an eye on network traffic... I have yet to see any noticeable changes at this time but would appreciate further details from those having issues.

 

kevgo.

Share this post


Link to post
I upgraded my largest adminkit base this morning. I installed over the top of a 8.0.2090 install. In fact, I just finished another adminkit in our TYO office and have not seen any problems. Where are you seeing the traffic? Is it on the AdminKit server or perhaps the clients connecting in?

 

It seems to be on the network in general, where it gets flooded as in the graph. The admin kit has been off for several days now and no issues.

 

Going to turn it back on late friday and let it sit all weekend....

Share this post


Link to post
It seems to be on the network in general, where it gets flooded as in the graph. The admin kit has been off for several days now and no issues.

 

Going to turn it back on late friday and let it sit all weekend....

 

Thanks for the update. I have 540 clients with the latest network agent and have yet to see anything out of the ordinary in terms of network traffic. If it makes a difference I do have several update agents defined.

 

kevgo.

Share this post


Link to post

Kevgo,

 

Can you explain "update agents defined", are you talking about groups and when they update?

 

I dont have anything like that setup, which might be the issue. We were talking about it at the IT meeting this week though.

Will play around a little more and see what I can do.

Share this post


Link to post
Kevgo,

 

Can you explain "update agents defined", are you talking about groups and when they update?

 

I dont have anything like that setup, which might be the issue. We were talking about it at the IT meeting this week though.

Will play around a little more and see what I can do.

 

I have several "groups" of users based upon the faculty (university environment) and geological location. For each group you can right click.. select properties.. Update Agents.. and add local update agents. This helped in reducing my server load. Rather than having 4,000 clients hit my admin server for updates, many clients would get their updates from the local update agents which reside on the same switch or in the same geographical location, much like a P2P strategy for updates. My understanding is the initial communication is with the Admin Kit Server.. which then refers clients to the update agents defined for the group. For 4,000 clients I have around 80 update agents. Alternatively setting up a slave Admit Kit server on a dedicated desktop in your office may be worthwhile. In my case keeping the majority of traffic from hitting my WAN links and/or firewalls is the goal.

 

More seasoned KAV veterans and or admins can chime in here.

 

kevgo

 

 

Share this post


Link to post
I have several "groups" of users based upon the faculty (university environment) and geological location. For each group you can right click.. select properties.. Update Agents.. and add local update agents. This helped in reducing my server load. Rather than having 4,000 clients hit my admin server for updates, many clients would get their updates from the local update agents which reside on the same switch or in the same geographical location, much like a P2P strategy for updates. My understanding is the initial communication is with the Admin Kit Server.. which then refers clients to the update agents defined for the group. For 4,000 clients I have around 80 update agents. Alternatively setting up a slave Admit Kit server on a dedicated desktop in your office may be worthwhile. In my case keeping the majority of traffic from hitting my WAN links and/or firewalls is the goal.

 

More seasoned KAV veterans and or admins can chime in here.

 

kevgo

 

Also..

 

I (with my limited knowledge) would also recommend setting up separate update tasks for each of your groups. Setup staggered schedules for each group which check for updates every 2 hours. If you have all your clients in one group you may wish to create separate ones. This allows me to have separate policies of student computer labs vs faculty staff machines. The same applies for servers vs workstations.. as well as provides granularity for reporting features. Just a thought..

 

kevgo

Share this post


Link to post

So,

 

I got around to working on Kaspersky, now that its sunday :)

 

Setup a single task for my servers only (8), huge spike in traffic. (anyone know how big the updates are each?)

 

I think there may be a database update on that server, so all the clients checking in are grabbing it. But there are NO TASKS to tell the clients to do this. (I have even deleted the admin kit task to download new updates)

 

 

So this server is dealing with 8 subnets across various branches, does that make any difference?

The admin kit servers is also a VMware server.

 

What I dont understand is why it floods my whole network, every single port on a 300 port switch, its just not right. Would it be brodcasting out updates to my other subnets causing the main subnet to go down?

It takes my wan links to its knees. (other subnets) They are not fast, just T1 lines. But with 5-8 computers at each one, and lets say updates are 5 megs each. 25-40 megs of data per location. Only 5 minutes ish per location. No big deal, but still does not explain the main branch getting flooded.

 

Thoughts?

Share this post


Link to post

So,

 

I got around to working on Kaspersky, now that its sunday :)

 

Setup a single task for my servers only (8), huge spike in traffic. (anyone know how big the updates are each?)

 

I think there may be a database update on that server, so all the clients checking in are grabbing it. But there are NO TASKS to tell the clients to do this. (I have even deleted the admin kit task to download new updates)

 

According to the documentation, the network agents on the clients default to checking for updates on the admin server and not directly on the Kaspersky ftp servers. If you have set up additional "update agent" servers, the admin servers will refer some of that traffic there.

 

The admin kit new updates task is for telling the admin kit to put get the updates from Kaspersky, not for sending them out to the clients (which is automatic, see above.

 

As for your 2 hour repeat cycle, I have noticed on regular clients that Kaspersky seems to release new updates on their public servers every 2 hours, so that could be the trigger.

 

 

What I dont understand is why it floods my whole network, every single port on a 300 port switch, its just not right. Would it be brodcasting out updates to my other subnets causing the main subnet to go down?

 

I have no idea why the network behavior you see is so abysmal (I am still struggling to install the admin server at all), but one thing you might check is name resolution for the DNS or NETBIOS name of the admin server. Name resolution problems in Windows networks can lead to broadcast storms.

 

One thing I would try in you situation is this:

 

Install WireShark or another packet dumper on a computer not running Kaspersky, plugged into one of the 290 ports that seem to receive unwanted traffic.

Tell it to display all the traffic on that computer, wait for the storm and then see what kind of noise it is getting flooded with.

 

It takes my wan links to its knees. (other subnets) They are not fast, just T1 lines. But with 5-8 computers at each one, and lets say updates are 5 megs each. 25-40 megs of data per location. Only 5 minutes ish per location. No big deal, but still does not explain the main branch getting flooded.

 

To reduce update traffic to branches with more than 2 clients on them, the use of "update agent" servers in each branch seems to be designed to make each branch download only one copy of the updates.

 

Share this post


Link to post

I Have been working on this all day, and watching what happens.. Here is an update since my last post.

 

All kaspersky default tasks were removed, updates, pushes, admin core updates etc.

 

Every single group was built a task to update the clients\servers at a specific time. 6:30pm onwards to midnight daily. missed tasks checkbox was NOT checked.

Network was super busy for a while, during the time clients all synced to the admin kit server. 24 hours of minimal network traffic, all ok.

 

Today i built a single task to get new updates from kaspersky labs, this is set to run once a day. I thought...what the hell, lets run it and see what happens. 21 seconds it was done.

Then all hell breaks loose, clocks offline and sql drops all over the place. IMAGE ATTACHED.

 

You can see at 8am yesterday was when i was playing around.

You can see that at 1 pm is when I played with the update task.

 

All 300 ports and my wan links went nuts. But it did die down quickly. THE FACT IS, that it killed my time clocks and my SQL server. I am sure that if the task went any longer I would have seen more issues.

 

WHY would an update cause this to happen?

Why would it push updates out to some of my clients? (only one of my wan offices is active now, image attached) THESE CLIENTS ARE SCHEDULED!!!! not do it right now!!

post-315724-1290458740_thumb.png

post-315724-1290458889_thumb.png

Share this post


Link to post

Out of curiosity what happens when you tell your clients to get updates from Kaspersky rather than admin kit? And have you tried using a traffic rule to limit bandwidth utilization?

 

Based upon what I have seen the behavior mentioned I can't see attributed to Kaspersky.. sounds more like a broadcast storm.

 

kevgo.

Edited by kevgo

Share this post


Link to post

Ok we were kicking this around....

 

2 of my subnets are NOT part of my WAN link, they are VPNs, this is one of the networks that get kaspersky updates, it is also one of the ones that is active in my last post.

All of my subnets are like this. (all controlled by my ISP on a managed wan layer 3 service)

10.10.10.x

10.10.20.x

10.10.30.x

 

except these other ones, they are 192.168.0.x and 192.168.85.x and 192.168.1.x and 192.168.2.x (these are the VPN ones)

 

 

 

Here is our thought, and this was touched upon by another member.

 

Inside kaspersky admin kit, you build IP ranges, and name them. There is also a check box here that says "enable subnet scanning"

 

On the inside of my network, i cannot resolve computernames to IP addresses for these 2 subnets because there is NO dns resolution or wins avaliable. (i could build one though)

E.G. ping computer18

 

SO...We are thinking that kaspersky is brodcasting for it, this takes some time but eventualy it hits my outside router and gets passed on the vpn.

 

Thoughts?

Edited by WallyL

Share this post


Link to post
Out of curiosity what happens when you tell your clients to get updates from Kaspersky rather than admin kit? And have you tried using a traffic rule to limit bandwidth utilization?

 

Based upon what I have seen the behavior mentioned I can't see attributed to Kaspersky.. sounds more like a broadcast storm.

 

kevgo.

 

Kevgo, I am totally with you here for sure....but it only seems to happen when the adminkit gets updates from the source.

 

All the clients were without an adminkit for almost a full week. I would assume that they went out and got their own updates after being disconected for such a long time.

We did actually test to make sure that they could go and get updates on the workstations without the admin kit and no issues.

 

We are going to trigger another update here now and see what happens.

Share this post


Link to post

Ok so i manualy triggered the update from source.

 

It in fact is the update from source that triggers these brodcast events.

Which is intersting.

I am going to now do a single group push at my local location to see what happens. Stay tuned!

Share this post


Link to post

I pushed out a small group of computers 4, no issues.

 

Second push group 30 computers no issues.

 

So it seems to just be the update from source task.

 

Will try it again in the morning and see how things are.

 

Need to let my tasks settle out tonight and check it out.

Share this post


Link to post

All my tasks have settled and its running great.

 

EXCEPT FOR:

 

When the admin kit gets updates from source it brodcast storms inside my network...no idea why....

Thoughts?

Share this post


Link to post

Hello,

 

Is your group update task configured to run right after the admin kit gets updates?

Share this post


Link to post
Hello,

 

Is your group update task configured to run right after the admin kit gets updates?

 

Nop, admin kit updates are now set manual, I dont want it to take down my network. Everything runs fine as long as there is no admin kit update task.

 

We are going to be doing some further testing today. Manual trigger with wire shark watching and try to pick it appart.

Share this post


Link to post

×
×
  • Create New...

Important Information

We use cookies to make your experience of our websites better. By using and further navigating this website you accept this. Detailed information about the use of cookies on this website is available by clicking on more information.