Memory management - why doesn't available memory change instantaneously?

[CPhil]

Hi,

I have some pretty memory-hungry code and sometimes it overwhelms the machine it's running on. I'd like to stop it doing that, by getting it to wait until more memory becomes available.

<sys/sysinfo.h> isn't available in my local coding environment. Although it is available on the Unix machines I also use, it doesn't quite function as I'd like. Consequently, I wrote a bit of code to tell me (to the nearest 4bytes) how much space is available:

long FreeMem(void){
	
	typedef struct fmem{
		int i;	
		}fm;
	
	long L,inc;
	fm *j;
	int chk;
	
	inc=1000000000; L=inc;
	while (inc>=1){
		chk=0;
		while (chk==0){
			if ((j = malloc(sizeof(fm)*L))==NULL) {
				chk=1;
				L-=inc;
				inc=inc/10;
				if (L==0) L=inc;
				}
			else L+=inc;
			free(j);
			}
		}
		
	return(L*4);
	}

This code tries to allocate memory, beginning in gigabytes (first 1Gb then, if that's successful, 2Gb, etc.). Once it fails to get the amount of memory it needs, it starts again with the largest bit of memory it successfully allocated and reduces the increment by which it increases its requests (to 10^8, then 10^7 etc.). It continues until it has determined, to the size of an integer variable, the amount of memory it can allocate.

I have written a short program with a memory leak. It calls this function at the start (before it leaks any memory) and at the end (when it can no longer leak memory because there's nothing left to leak). My hope was that (1) the amount of memory leaked would approximate the amount that was available at the outset; and (2) that the amount of memory remaining at the end would be just less than the amount required for the final (failed) leak.

If I run the program in debug mode (using my local coding environment, Codewarrior IDE v.5), both of my expectations are fulfilled. However, if I run the code in normal mode (i.e. not debugging), the memory leaks (each of size 80Kb) fail whilst my FreeMem() function claims there's still > 100Kb of memory available. More bizarrely, when I run the code on a Unix machine, I can call FreeMem() after each leak, but the value returned doesn't change for many many iterations (i.e. the machine doesn't seem to "realise" immediately, that it has less memory available than it did a moment ago). This seems to be the same problem as when I use the <sys/sysinfo.h> library on the Unix machines: a deliberate memory leak doesn't result in an immediate change in the value returned by sys_info.freeram.

Does anyone know why I'm having these problems, and whether there's a better way to determine how much space is available on the host machine, in order that I don't overload it (and cause a segmentation error) by using malloc to sequester more memory than is available?

Thanks in advance for any suggestions.
Phil

[WingedPanther]

Not knowing what your actual code is doing, it's very hard to determine what might be happening. It seems like you need to isolate WHICH areas of code are behaving incorrectly. Given the complexities of memory management, it may be system/compiler dependent as well.

[dargueta]

If you run it on Windows, it will always say that there's 128KB available, which actually isn't. It's because Windows reserves 64KB before and after the program's addressible space to catch errant pointers; although it's not technically reserved, attempting to access it will result in an access violation. That's my theory, at least. I don't know anything about how Unix works.

EDIT: I just had an idea - I believe managed code has a process that crawls over memory and deallocates memory that isn't needed anymore. That might be it, or I might be shooting in the dark. Who knows?

[Aereshaa]

Hmm. I think that you should first verify whether the FreeMem function itself has no leaks. Second, why do you bother to create an fm struct type? It's plainly obvious and easy to prove that it will be the same size as an int. Therefore, why not use an int * it the first place?
Once you've done that: Do you know how malloc() works? It has a linked list of free chunks of memory in your program's heap. When you allocate, say 32 bytes, malloc finds a chunk of roughly the right size, and chops it into two, one for your pointer, the other goes back into your list. But since your code only has a limited amount of memory it's allowed to use, you can't allocate chunks above a certain size. In other words, your code can only allocate up to the amount of memory it's given, not the amount of memory in the computer. If you want more memory, you can ask for it using system calls, specifically sbrk().
EDIT:
One more thing: If you really need to allocate huge sections of memory, try the mmap() system call, using MAP_ANONYMOUS. It will allocate a massive chunk of memory, outside the heap. Try this for a whole gigabyte:

void * p = mmap(NULL,1024 *1024 * 1024,PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);

Change the size as you'd like. the OS won't bother you.

[CPhil]

Thanks to everyone for your replies.

To WingedPanther:
I'm sure it is system/compiler dependent. Nevertheless, I'm surprised no-one's noticed this using gcc on Unix. Basically, I'd be interested to see code that would run on Unix and would give an accurate picture of how much memory is available at any point in time. For example, a program with a single loop that 1) leaked memory and 2) showed how much RAM was available, would be interesting. I wrote a program to do just that (using <sys/sysinfo.h>) but the value for available RAM only changed occasionally - i.e. it didn't change each time I leaked memory. That led me to worry that the call to return the available RAM was inaccurate.

To dargueta:
I can imagine that a deallocation process could be involved, except that my code did eventually lead to an overflow and, therefore, to the malloc function returning a NULL value. Moreover, with the code I ran on Unix (see response to WingedPanther), the amount of available RAM would change occasionally - just not every time I leaked some memory.

To Aereshaa:
many thanks for the code - I'll see if it works for me. As for why I used a data structure, it was because I originally set the program up so that I could allocate memory to anything at all - it changed from an investigation of a specific problem I was having with a specific data structure, to a more general piece of code. Thanks for the explanation re. malloc() - very helpful. It suggests that the only way to determine whether malloc() will work, is to try it - just because available RAM > amount requested, presumably doesn't mean that malloc() will work.

Thanks again,
Phil