8 replies to this topic

[ibad]

Hi All,

I am confronting a very hairy issue on my server.

Basically, I have a HTML processing algo that finds and "adjusts" tables in an HTML document if they meet certain criteria w.r.t to size.

My worst case scenario documents can be up to 2 Mb with over 2 millions characters.

The algo, when run from a console app on my server (for worst case), takes about 2 seconds. However, when it is run from within the code of a website hosted on IIS, (called via .dll) it can take over 2 minutes. Commenting the method/algo fixes the delay. Yesterday I ran it again when the RAM and CPUs were quite free.... and it reacted very rapidly, without the normal delay.

Today it's slow as hell again. Another thing: RAM is about 85% used currently. Is this healthy or normal? We are running IIS, Tomcat, MySQL, SOLR and and Lucene and a bunch of other things on it with huge indexes etc. Also I have seen (on some occasions) all 16 cores spiking to 100%, though this is rare. Also sometimes the free "physical memory" drops to around 20 Mb, though normally this hovers about 250 MB.

Here are our specs:

16 cores 64-bit Xeon
Windows Server 2008 64bit
32 GB RAM

Does anyone have any idea what might be wrong? I would really appreciate the help!

Also.... if 15% of memory is free then there is still about 4GB of free RAM (though mostly reserved for in kernel momory I think? that's why I only get 250 Mb of free physical RAM? A lot is cached using the new Windows Cache mechanism... so should that not help?)...so there should not be a RAM shortage right?

Could it be that memory is heavily fragmented and the largest contiguous block just is not that big?

I am at my wit's end, and I admit that I am not a server admin (though I have been charged with helping in this problem because my code is causing it), so the more educational your responses the better! Assume I am a server/OS/resource-management newbie.

Thank you for any help!

[WingedPanther]

How much RAM / CPU are each of these applications using? Have you considered clustering servers?

[ibad]

Well, here is the thing, a lot of the memory is taken up by SuperFetch... so that should basically be like free memory right??? Should not matter if there is very little physical RAM left.

The other programs are taking up multiple Gigabytes of RAM. But again... here is the weird thing.. when I run the algo on the same input in an independent console app it barely takes 2 seconds to complete. But when called inside an IIS hosted website it can take 3-5 minutes to complete!

So...I do not know what the problem is... is it SuperFetch? An internal IIS memory issue? Also... what do you mean by clustering servers? It's just one lame method that is pretty serial in it's execution, and if MS's line abut SuperFetch is to be believed... I should have plenty of resources to run it.

Also... the web requests and other programs seem to be running fine with the exception of this method when called by web-site code. But do please explain more about clustering servers if you think that would help.

Best regards,
Ibad

P.S. I am guessing that the caching scheme taking up all the memory in Windows Server 2008 is SuperFetch.... is that correct?

[ibad]

Ok, it seems that SuperFetch is off. But for some reason there is still 15 GB is RAM that is cached and a very low amount of free physical RAM (most of the free RAM is in kernel memory I guess). Can someone explain to me what is going on?

[Alexander]

What we mean is, how much is your program taking up in physical RAM? If you have no physical RAM left, then the program will execute very slowly as you had experienced regardless of prefetching. The kernel shouldn't take too much memory, you should monitor what process uses what memory during the long execution.

[ibad]

Well, it's hard to tell how much my algo is taking since it is inside a asp.net website (.dll in the bin, being called by code-behind) and the website is being run by IIS. So...I can only tell you how much memory the overall webserver process it taking...not my website, or the particular algo. But normally, out of 32 Gb we only have 0-200 Mb physical RAM free. Supposedly only 85% of the RAM is used up, but most of the free RAM is in kernel memory. Also, SuperFetch is off, but I am guessing some other caching or prefetching mechanism is working, since 15 Gb are in "cached" physical memory.

But my server Admin says that the RAM situation is "fine" and the latest versions of Windows use caching a lot, so most of the data my program uses is already in cache, and the low levels of physical memory are typical of the latest versions of Windows. If he right? Especially considering that SuperFetch is Disabled?

[WingedPanther]

So you're saying IIS is consuming nearly all of the 32Gb of RAM? ouch.

[ibad]

No! Not IIS! Problem is there is supposedly 15Gb physical free RAM..... but it's almost all taken up by cache. IIS takes up about 1 Gig I think...SOLR and Tomcat and others take up multiple GBs. They all total about 15Gb. IIS is taking up about 2 Gb (thereare 2 instances of w3wp.exe each taking up about 1 Gb)

[ibad]

OK... It seems that memory may be scarce within the IIS calling environment. This is partially because of a LOH (Large Object Heap) memory leak occuring due to a lack of compacting by the GC in the LOH. So the memory used by by my algo goes up with ever operation on a very large string (2 Mb approx.). This is as far as I can tell.

Is there any way to get that memory back? Can I force it to compact the LOH?