Jump to content


Check out our Community Blogs

Register and join over 40,000 other developers!


Recent Status Updates

View All Updates

Photo
- - - - -

OutOfMemoryException thrown when streaming an XML to a file

xml stream

  • Please log in to reply
10 replies to this topic

#1 OthelloNYC

OthelloNYC

    CC Newcomer

  • Just Joined
  • PipPip
  • 16 posts

Posted 25 July 2007 - 10:00 AM

Whenever I run the following code, it runs smoothly until the XML being serialized is over 120 megs in size, then I get an outofmemoryexception. "XmlDataSetData" is a dataset VB serialization object from an XSD, which I'm tthinking comes with a lot of overhead.
__________________________________________________________________
Public Sub WriteXml(ByRef XmlDataSet As XmlDataSetData, ByVal XmlFile As String)

If File.Exists(XmlFile) Then
File.Delete(XmlFile)
End If
Dim OutDocument As New XmlDocument
Dim Writer As New XmlTextWriter(XmlFile, Text.Encoding.Default)
Try
System.GC.Collect()
OutDocument.LoadXml(XmlDataSet.GetXml())
Catch ex As Exception
Throw New ArgumentException("Error loading Master Node: " + ex.Message + ex.StackTrace)
End Try
Try
If OutDocument.SelectSingleNode("//*/Master") Is Nothing Then

If OutDocument.SelectSingleNode("//Master") Is Nothing Then
Throw New ArgumentException("Master node not found in the data set.")
Else
OutDocument.SelectSingleNode("//Master").WriteTo(Writer)
Writer.Flush()
End If
Else
OutDocument.SelectSingleNode("//*/Master").WriteTo(Writer)
Writer.Flush()
End If
Catch ex As Exception
Throw New ArgumentException(ex.Message + " " + ex.StackTrace)
End Try
System.GC.Collect()
Writer.Close()
End Sub
__________________________________________________________________

I narrowed down the error to be coming from when I load the XML from the serialization data set object to the XmlDocument, but wqas wondering if anyone had a more efficient way of doing what I am doing here, memory-wise.:confused:

Thanks,
Stephen
  • 0

#2 hoser2001

hoser2001

    CC Addict

  • Just Joined
  • PipPipPipPipPip
  • 170 posts

Posted 31 July 2007 - 12:11 PM

Do you have to load from a dataset?

try xmlreader or xmltextreader which have a better performance than dataset:

maybe try looking into possible limitations of XmlDocument datatype, im not aware of it but perhaps it has a maximum size.
  • 0

#3 OthelloNYC

OthelloNYC

    CC Newcomer

  • Just Joined
  • PipPip
  • 16 posts

Posted 31 July 2007 - 01:34 PM

The dataset is not negotiable. However the problem is actually the system memory for the app itself (it's reaching 1 gig), and the exception is saying the system won't allow the application any more memory (the system is set to limit any process to 1 gig).
  • 0

#4 hoser2001

hoser2001

    CC Addict

  • Just Joined
  • PipPipPipPipPip
  • 170 posts

Posted 01 August 2007 - 04:11 AM

perhaps you can divide up the workload into different threads.
  • 0

#5 OthelloNYC

OthelloNYC

    CC Newcomer

  • Just Joined
  • PipPip
  • 16 posts

Posted 01 August 2007 - 06:46 AM

I tried that, what I might do is maybe make use of RPC for the file writing, but even that's iffy, since I'll be sending 126 megs to the remote procedure. Since the threads fall under the same process, the system is still saying no. I might mess with the memory settings and see what that bears.

The problem is that not using the dataset basically means redesigning a program I didn't write. I might end up having to do that, though, if I figure out that the dataset isn't being used for anything.
  • 0

#6 hoser2001

hoser2001

    CC Addict

  • Just Joined
  • PipPipPipPipPip
  • 170 posts

Posted 02 August 2007 - 04:02 AM

hmm, are you working with the .NET framework...?

if so what framework are you using?
  • 0

#7 OthelloNYC

OthelloNYC

    CC Newcomer

  • Just Joined
  • PipPip
  • 16 posts

Posted 02 August 2007 - 05:05 AM

It's the .net framework 1.1, which I'm told has memory issues with datasets.
  • 0

#8 hoser2001

hoser2001

    CC Addict

  • Just Joined
  • PipPipPipPipPip
  • 170 posts

Posted 03 August 2007 - 07:12 AM

I would take a look at trying to avoid that whole using 1 Gig of system memory.... (the system limits it for a reason) Is the dataset and the XML causing it to use so much? Or is it possibly something else?
  • 0

#9 OthelloNYC

OthelloNYC

    CC Newcomer

  • Just Joined
  • PipPip
  • 16 posts

Posted 03 August 2007 - 10:17 AM

I would take a look at trying to avoid that whole using 1 Gig of system memory.... (the system limits it for a reason) Is the dataset and the XML causing it to use so much? Or is it possibly something else?


I am having difficulty auditing it, because I can't test with the dataset that causes this (the rest of the process files involved take up about 100 gigs of server space, and I have 30 on my whole machine), but I am looking for anything that might cause the overhead.
  • 0

#10 hoser2001

hoser2001

    CC Addict

  • Just Joined
  • PipPipPipPipPip
  • 170 posts

Posted 03 August 2007 - 11:13 AM

well, the methods you use are not much different then how I would do it, however I have never used VB so I can't say for sure, sorry I can't be any more help to you, but I'm stumped.
  • 0

#11 OthelloNYC

OthelloNYC

    CC Newcomer

  • Just Joined
  • PipPip
  • 16 posts

Posted 03 August 2007 - 11:17 AM

Oh me too. I thinkl the key is to see if the dataset is completely necessary and see if maybe offlining it and deleting it would help.
  • 0





Also tagged with one or more of these keywords: xml, stream

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download