Using large XML data – 3Mbyte file

[monkeyplotter2Participant]

Hi folks,

[EDIT] – solved in post #13, I needed to add the #imports “1p7.xml” to the top of the file….

I’m new to using XML to parse data, so maybe I should be off learning how to parse data with XML prior to trying to do it….

I’m inclined to put specific data I want from an XML file thats been opened into specific arrays for manipulation.

Is there a way to break down a 3Mbyte XML file so its more digestible by Monkey2? I’ve tried to use my version of the tinyXML parser to achieve this goal however my 3MByte data file is a bit too chunky for the tinyXML parser ;(

I’ve written bespoke data parsers in Blitz3D and MonkeyX, however they also have issues with my 3MByte XML file. (Errm, they crash !) The good thing about this is it is giving me the impetus to get my teeth stuck into Monkey2.

Any help appreciated.

[DiffrenzyKeymaster]

Can you post the xml, or another xml file that crashes tinyXML in a similar way?

I would think this is a case of finding the error that causes the crash.

I am also very interested in having a super stable XML parser and just ported Monkey-X xml code to MX2 tinyXML without issues.

[codifiesParticipant]

definitely recommend you feed that xml file through an xml validator (theres a bunch of online ones)

personally I feel that size of file you’re starting to get to the practical limits for xml in terms of being a robust data encapsulation, on mangled character can bork the whole data set in difficult to find / fix way…

consider sqlite (you can always back up to SQL statements also a very portable way to convert the data)

if you really have found an issue with a *valid* xml and tinyxml you should report it, also you might consider seeing if you can run it through tinyxml2

[monkeyplotter2Participant]

codifies, thank you for your suggestions – I suspect it is valid (its a bespoke tcx file type, possibly its not ‘totally’ xml valid…).

I am going to try the XML validator you’ve suggested, tried the one on W3 schools and it has no errors. I appreciate your point about approaching the cut off for ‘usable’ files. I’ll attempt to create another 3MByte sized file for trialling purposes.

The XML in question is GPS data which has data saved every second, hi-resolution GPS data.

“I am also very interested in having a super stable XML parser and just ported Monkey-X xml code to MX2 tinyXML without issues. ” — good stuff.

[coconParticipant]

I would definitely be sure that 3MB is a reasonable size and that the TinyXML is solid.

Just out of curiosity if you parse the XML file with another programming language what will happen?
https://www.tutorialspoint.com/python/python_xml_processing.htm

P.S. I don’t know also if there is an encoding issue as well, is the file UTF8?

[monkeyplotter2Participant]

Hi, I’ve generated a (slightly) smaller file. its only 1.7Megs and W3C schools XML validates the file. Unfortunately this file (uploaded to the hyperlink below) seems a little bloaty still. I’ve never used python before and don’t have time before I go to work to try out your suggestion. Nevertheless – here’s the 1.7 meg tcx file in question:

 

hmmm, well, have uploaded my tcx file to my website, but it seems you cannot directly access a 1.7megs file of tcx file type. other stuff still links ok though. Too much brainache for first thing:

http://www.wishingwellsoftware.co.uk/powerup.png

[DiffrenzyKeymaster]

Can you zip the file?

Also: Sometimes renaming to .xml sometimes circumvents un-allowed filetypes download.

[monkeyplotter2Participant]

Hmm tried changing the extension to xml first, failed, thanks for the compression suggestion – wow, it gets it down to  a fraction of the size, only 64KBytes   Even uploaded it to these forums, which is sweet.

http://www.wishingwellsoftware.co.uk/2megs.rar

Yay, finally got it uploaded in a .rar file.  This is a slighlty smaller 1.8MegByte tcx / xml file – right – gotta get ready for that work thing.

Attachments:

1. 2megs.rar

[DiffrenzyKeymaster]

I’ve tested your file with banana/tinyxml2_test.monkey2 and it seems to load and display without any issues.

(macOS, latest monkey2 )

[monkeyplotter2Participant]

Ah, interesting. Trying now on my W7 box ….  Mac beats Windows on this occasion, even tried changing the tcx extension to xml, no dice:

”

Failed to parse embedded XML!
SDL_WINDOWEVENT_FOCUS_GAINED

Finished debugging app.

“

[Mark SiblyKeymaster]

Tested with tinyxml2_test and onemegs.xml file on Windows and it worked fine…

Are you loading the xml file properly? Check it’s not loading a null string is a good idea.

[DiffrenzyKeymaster]

@monkeyplotter: My guess is you are not getting the asset or name correct. Here is a complete working example:

Monkey

#Import "onemegs.xml" #Import "<std>" #Import "<mojo>" #import "<mojox>" #Import "<tinyxml2>" Using std.. Using mojo.. Using mojox.. Using tinyxml2.. Function Main() New AppInstance New MyWindow() App.Run() End Class MyWindow Extends Window Field _tree:TreeView Method New() Super.New("XML TreeView Test", 640, 480, WindowFlags.Resizable) Local xml := LoadString( "asset::onemegs.xml" ) Local doc := New XMLDocument() If doc.Parse(xml) <> XMLError.XML_SUCCESS Print "Failed to parse embedded XML!" Return Endif Local _tree := New TreeView _tree.RootNode.Text = "XML document" AddXMLNodeToTree(doc, _tree.RootNode) ContentView = _tree doc.Destroy() End Method AddXMLNodeToTree(xmlNode:XMLNode, parent:TreeView.Node) Local str := "" Local xmlElement := xmlNode.ToElement() If xmlElement str += "<" + xmlNode.Value() Local attrib := xmlElement.FirstAttribute() While attrib str += " " + attrib.Name() + "=~q" + attrib.Value() + "~q " attrib=attrib.NextAttribute() wend str += ">" Else str += xmlNode.Value() Endif Local treeNode:TreeView.Node If str treeNode = New TreeView.Node(str, parent) Endif Local xmlChild := xmlNode.FirstChild() While xmlChild If Not xmlChild.NoChildren() If treeNode Then parent = treeNode Endif AddXMLNodeToTree(xmlChild, parent) xmlChild = xmlChild.NextSibling() Wend End End

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94

#Import "onemegs.xml"   #Import "<std>" #Import "<mojo>" #import "<mojox>" #Import "<tinyxml2>" Using std.. Using mojo.. Using mojox.. Using tinyxml2.. Function Main() New AppInstance New MyWindow() App.Run() End Class MyWindow Extends Window Field _tree:TreeView Method New() Super.New("XML TreeView Test", 640, 480, WindowFlags.Resizable) Local xml := LoadString( "asset::onemegs.xml" ) Local doc := New XMLDocument() If doc.Parse(xml) <> XMLError.XML_SUCCESS Print "Failed to parse embedded XML!" Return Endif Local _tree := New TreeView _tree.RootNode.Text = "XML document" AddXMLNodeToTree(doc, _tree.RootNode) ContentView = _tree doc.Destroy() End Method AddXMLNodeToTree(xmlNode:XMLNode, parent:TreeView.Node) Local str := "" Local xmlElement := xmlNode.ToElement() If xmlElement str += "<" + xmlNode.Value() Local attrib := xmlElement.FirstAttribute() While attrib str += " " + attrib.Name() + "=~q" + attrib.Value() + "~q " attrib=attrib.NextAttribute() wend str += ">" Else str += xmlNode.Value() Endif Local treeNode:TreeView.Node If str treeNode = New TreeView.Node(str, parent) Endif Local xmlChild := xmlNode.FirstChild() While xmlChild If Not xmlChild.NoChildren() If treeNode Then parent = treeNode Endif AddXMLNodeToTree(xmlChild, parent) xmlChild = xmlChild.NextSibling() Wend End End

[monkeyplotter2Participant]

thanks for the help, I was failing to add the line ‘#imports “1p7.xml”‘ at the top of the code…. doh!

I only sussued this after wondering why there was a dream.xml copied to the Windows /assets folder but no 1p7.xml.

Solved, maybe I should stick to riding my bike

Attachments:

1. 

   1p7.png
   

[codifiesParticipant]

easy done I spent ages looking at

Monkey

[crayon-5cb9e11fb7d28792069582 inline="true" ] LoadPixmap("asset::smallhm.png" ...

1 2 3

[crayon-5cb9e11fb7d28792069582 inline="true" ] LoadPixmap("asset::smallhm.png" ...

[/crayon]

when it should have been

Monkey

[crayon-5cb9e11fb7d2e783551565 inline="true" ] LoadPixmap("assets::smallhm.png" ...

1 2 3

[crayon-5cb9e11fb7d2e783551565 inline="true" ] LoadPixmap("assets::smallhm.png" ...

[/crayon]

[Author]

Posts