Compress/Decompress data buffers

[PixelPaladinParticipant]

Is it possible to compress/decompress data from/into a data buffer?

I would like to do something like that:

Monkey

Local originalData := New DataBuffer(256) For Local i := 0 Until 256 originalData.PokeUByte(i, i) Next Local compressedData:DataBuffer = Compress(originalData) Local decompressedData:DataBuffer = Decompress(compressedData)

1 2 3 4 5 6 7 8 9

Local originalData := New DataBuffer(256)   For Local i := 0 Until 256     originalData.PokeUByte(i, i) Next   Local compressedData:DataBuffer = Compress(originalData)   Local decompressedData:DataBuffer = Decompress(compressedData)

[nerobotParticipant]

There is the miniz module.

And example of its usage here:

https://github.com/blitz-research/monkey2/blob/develop/modules/std/misc/zipfile.monkey2

[Mark SiblyKeymaster]

It’s probably time I added a zlib module – will do this soon so stay tuned for compress/decompress data buffer methods.

[Mark SiblyKeymaster]

New zlib module and data buffer compress/decompress methods have been pushed to develop branch.

[PixelPaladinParticipant]

Wow – that was fast!

I already tried it and it works like a charm:

Monkey

#Import "<std>" #Import "<zlib>" Function Main:Void() ' generate some random data Local originalData := New std.memory.DataBuffer(8000) For Local i:UInt = 0 Until originalData.Length originalData.PokeUByte(i, std.random.Rnd(5)) End ' get max compressed length (this can be more than original data size) Local maxCompressedLen := zlib.compressBound(originalData.Length) ' create buffer for compressed data Local compressedData:std.memory.DataBuffer = New std.memory.DataBuffer(maxCompressedLen) ' compress data Local compressedLen:ULong = maxCompressedLen If zlib.compress( compressedData.Data, Cast<zlib.z_uLong Ptr>(Varptr(compressedLen)), originalData.Data, originalData.Length ) <> zlib.Z_OK Then Print( "Compressing: Something went wrong!" ) Return End Print("Compressed data length: "+compressedLen+" Byte") ' create buffer for uncompressed data Local uncompressedData := New std.memory.DataBuffer(originalData.Length) ' uncompress data Local uncompressedLen:ULong = originalData.Length If zlib.uncompress(uncompressedData.Data, Cast<zlib.z_uLong Ptr>(Varptr(uncompressedLen)), compressedData.Data, compressedData.Length) <> zlib.Z_OK Then Print("Uncompressing: Something went wrong") Return End ' success! Print("Uncompressed all data successfully!") End

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

#Import "<std>" #Import "<zlib>"   Function Main:Void() ' generate some random data Local originalData := New std.memory.DataBuffer(8000) For Local i:UInt = 0 Until originalData.Length originalData.PokeUByte(i, std.random.Rnd(5)) End ' get max compressed length (this can be more than original data size) Local maxCompressedLen := zlib.compressBound(originalData.Length) ' create buffer for compressed data Local compressedData:std.memory.DataBuffer = New std.memory.DataBuffer(maxCompressedLen) ' compress data Local compressedLen:ULong = maxCompressedLen If zlib.compress( compressedData.Data,                   Cast<zlib.z_uLong Ptr>(Varptr(compressedLen)),                   originalData.Data,                   originalData.Length ) <> zlib.Z_OK Then Print( "Compressing: Something went wrong!" ) Return End Print("Compressed data length: "+compressedLen+" Byte") ' create buffer for uncompressed data Local uncompressedData := New std.memory.DataBuffer(originalData.Length) ' uncompress data Local uncompressedLen:ULong = originalData.Length If zlib.uncompress(uncompressedData.Data,                    Cast<zlib.z_uLong Ptr>(Varptr(uncompressedLen)),                    compressedData.Data,                    compressedData.Length) <> zlib.Z_OK Then Print("Uncompressing: Something went wrong") Return End ' success! Print("Uncompressed all data successfully!") End

[peterigzParticipant]

Is there anyway yet to load an image straight from a zip file without the need to extract it to disk first?

[Mark SiblyKeymaster]

See below for one possible solution. It loads a jpg image directly from a zip file.

It’s not very nice though, a simple ‘zip::’ stream would be sexier, or even some way to pass a DataStream to Image.Load. Will look into this…

Monkey

Namespace myapp #Import "<std>" #Import "<mojo>" #Import "yasuko.zip" Using std.. Using mojo.. Class MyWindow Extends Window Field image:Image Method New( title:String="Simple mojo app",width:Int=640,height:Int=480,flags:WindowFlags=Null ) Super.New( title,width,height,flags ) Local zip:=ZipFile.Open( "asset::yasuko.zip" ) For Local file:=Eachin zip.Files Print file Next Local data:=zip.ExtractData( "yasuko.jpg" ) Local path:="memory::("+ULong( data.Data )+","+data.Length+")" image=Image.Load( path ) zip.Close() End Method OnRender( canvas:Canvas ) Override App.RequestRender() canvas.DrawImage( image,0,0 ) canvas.DrawText( "Hello World!",Width/2,Height/2,.5,.5 ) End End Function Main() New AppInstance New MyWindow App.Run() End

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

Namespace myapp   #Import "<std>" #Import "<mojo>"   #Import "yasuko.zip"   Using std.. Using mojo..   Class MyWindow Extends Window Field image:Image   Method New( title:String="Simple mojo app",width:Int=640,height:Int=480,flags:WindowFlags=Null )   Super.New( title,width,height,flags ) Local zip:=ZipFile.Open( "asset::yasuko.zip" ) For Local file:=Eachin zip.Files Print file Next Local data:=zip.ExtractData( "yasuko.jpg" ) Local path:="memory::("+ULong( data.Data )+","+data.Length+")" image=Image.Load( path ) zip.Close() End   Method OnRender( canvas:Canvas ) Override App.RequestRender() canvas.DrawImage( image,0,0 ) canvas.DrawText( "Hello World!",Width/2,Height/2,.5,.5 ) End End   Function Main()   New AppInstance New MyWindow App.Run() End

[peterigzParticipant]

zip::stream sounds awesome if you can add that

Thanks for the example, didn’t know about memory:: looks like something I could use in the meantime.

It was really cool in Blitzmax the way you could pass either a string or object to the image loader and it would handle it, so would be a nice feature to have. I’m looking to update TimelineFX to be able to load .eff files (basically zip files) rather then the need to unpack them first.

[Mark SiblyKeymaster]

Ok, haven’t done zip:: stream yet but you can now use Stream.SharedPath to get a unique path to an already open stream. This means the zipfile stuff is a lot nicer:

Monkey

Local zip:=ZipFile.Open( "asset::yasuko.zip" ) Local data:=zip.ExtractData( "yasuko.jpg" ) Local stream:=New DataStream( data ) image=image.Load( stream.SharedPath ) stream.Close() zip.Close()

1 2 3 4 5 6 7

Local zip:=ZipFile.Open( "asset::yasuko.zip" ) Local data:=zip.ExtractData( "yasuko.jpg" ) Local stream:=New DataStream( data ) image=image.Load( stream.SharedPath ) stream.Close() zip.Close()

[peterigzParticipant]

That’s great Mark, thanks.

So if I wanted to save a zip would I compress stuff to a data buffer as above and then save the databuffer to disk, would that work?

[DiffrenzyKeymaster]

@peterigz:

I’m looking to update TimelineFX to be able to load .eff files (basically zip files) rather then the need to unpack them first.

That would be awesome, because the editor does not export unzipped, and it is a bit of a hazzle, when you are finetuning effects.

PS: When you update, please make it work with tinyxml2 out of the box, just a tlXmlElement will do it if you don’t wan’t to use tinyxml2.

[Mark SiblyKeymaster]

So if I wanted to save a zip

More functionality will need to be added to the ZipFile class before this can (easily) be done.

[Author]

Posts