10 replies to this topic

[OthelloNYC]

Whenever I run the following code, it runs smoothly until the XML being serialized is over 120 megs in size, then I get an outofmemoryexception. "XmlDataSetData" is a dataset VB serialization object from an XSD, which I'm tthinking comes with a lot of overhead.
__________________________________________________________________
Public Sub WriteXml(ByRef XmlDataSet As XmlDataSetData, ByVal XmlFile As String)

If File.Exists(XmlFile) Then
File.Delete(XmlFile)
End If
Dim OutDocument As New XmlDocument
Dim Writer As New XmlTextWriter(XmlFile, Text.Encoding.Default)
Try
System.GC.Collect()
OutDocument.LoadXml(XmlDataSet.GetXml())
Catch ex As Exception
Throw New ArgumentException("Error loading Master Node: " + ex.Message + ex.StackTrace)
End Try
Try
If OutDocument.SelectSingleNode("//*/Master") Is Nothing Then

If OutDocument.SelectSingleNode("//Master") Is Nothing Then
Throw New ArgumentException("Master node not found in the data set.")
Else
OutDocument.SelectSingleNode("//Master").WriteTo(Writer)
Writer.Flush()
End If
Else
OutDocument.SelectSingleNode("//*/Master").WriteTo(Writer)
Writer.Flush()
End If
Catch ex As Exception
Throw New ArgumentException(ex.Message + " " + ex.StackTrace)
End Try
System.GC.Collect()
Writer.Close()
End Sub
__________________________________________________________________

I narrowed down the error to be coming from when I load the XML from the serialization data set object to the XmlDocument, but wqas wondering if anyone had a more efficient way of doing what I am doing here, memory-wise.:confused:

Thanks,
Stephen

[hoser2001]

Do you have to load from a dataset?

try xmlreader or xmltextreader which have a better performance than dataset:

maybe try looking into possible limitations of XmlDocument datatype, im not aware of it but perhaps it has a maximum size.

[OthelloNYC]

The dataset is not negotiable. However the problem is actually the system memory for the app itself (it's reaching 1 gig), and the exception is saying the system won't allow the application any more memory (the system is set to limit any process to 1 gig).

[hoser2001]

perhaps you can divide up the workload into different threads.

[OthelloNYC]

I tried that, what I might do is maybe make use of RPC for the file writing, but even that's iffy, since I'll be sending 126 megs to the remote procedure. Since the threads fall under the same process, the system is still saying no. I might mess with the memory settings and see what that bears.

The problem is that not using the dataset basically means redesigning a program I didn't write. I might end up having to do that, though, if I figure out that the dataset isn't being used for anything.

[hoser2001]

hmm, are you working with the .NET framework...?

if so what framework are you using?

[OthelloNYC]

It's the .net framework 1.1, which I'm told has memory issues with datasets.

[hoser2001]

I would take a look at trying to avoid that whole using 1 Gig of system memory.... (the system limits it for a reason) Is the dataset and the XML causing it to use so much? Or is it possibly something else?

[OthelloNYC]

hoser2001 said:

I would take a look at trying to avoid that whole using 1 Gig of system memory.... (the system limits it for a reason) Is the dataset and the XML causing it to use so much? Or is it possibly something else?

I am having difficulty auditing it, because I can't test with the dataset that causes this (the rest of the process files involved take up about 100 gigs of server space, and I have 30 on my whole machine), but I am looking for anything that might cause the overhead.

[hoser2001]

well, the methods you use are not much different then how I would do it, however I have never used VB so I can't say for sure, sorry I can't be any more help to you, but I'm stumped.

[OthelloNYC]

Oh me too. I thinkl the key is to see if the dataset is completely necessary and see if maybe offlining it and deleting it would help.