Data Packer (read directly from stream)

[nerobotParticipant]

Hi guys.

I was playing with data packing – pack any resources like images into single pack-file and then read them directly from file stream.

This time I done with pack/unpack images.

How to pack files:

Monkey

Local packer:=New Packer( AppDir()+"data.bin" ) 'where to save packer.Pack( New String[]( "asset::blue.png","asset::red.png" ) ) 'what to save

1 2	Local packer:=New Packer( AppDir()+"data.bin" ) 'where to save packer.Pack( New String[]( "asset::blue.png","asset::red.png" ) ) 'what to save

How to read images:

Monkey

_img=packer.GetImage( "asset::red.png" )

1	_img=packer.GetImage( "asset::red.png" )

Packer produce two files – binary one and json one.

I decided to use 2 files to simplify process, to pack info into single file we must precalculate sizes, check this info size in bytes…

See attached demo.

Is it useful for anyone?

Attachments:

1. datapacker.zip

[EdzUpParticipant]

Nice it’s definitely something I’m going to look into for Star Rogue

I will go for the one data file approach that I started with MX1 this was easier to distribute and allowed for all types of media to be packed. However the reading from stream is something I will definitely look at as it will simplify the whole process if it can be used for meshes, textures etc.

[scurtyParticipant]

Wonderful! Very useful indeed.

[nerobotParticipant]

This solution depends on Stream.Seek( position:Int ) method, but not all streams are seek-able.

This time I know only that it works fine for desktops.

Also need to test very big pack file – about 500 mb and monitoring RAM usage.

I expect to see low RAM usage because it just jumps to resource offset in file stream and read only needed data part.

[nerobotParticipant]

Some ideas about using data packer.

I want to use it for release but don’t want to re-pack every time while developing.

My approach is to have 2 imports with different ‘configs’ for resource manager:

Monkey

#Import "ResDev" '#Import "ResRelease"

1 2	#Import "ResDev" '#Import "ResRelease"

We use one of them at a time – just comment/uncomment two lines.

Inside of them we have something like:

Monkey

' ResDev Namespace datapacker ' just import assets files #Import "assets/" Function InitAssets() ' do nothing End

1 2 3 4 5 6 7 8 9 10 11 12

' ResDev   Namespace datapacker   ' just import assets files   #Import "assets/"   Function InitAssets()   ' do nothing End

Monkey

' ResRelease Namespace datapacker ' import our packed files instead of all assets files #Import "packs/@/" ' Assets is a singleton class, you should use it for all loading stuff Function InitAssets() Assets.UsePack( "assets.bin" ) ' or do it another way End

1 2 3 4 5 6 7 8 9 10 11 12 13 14

' ResRelease   Namespace datapacker   ' import our packed files instead of all assets files #Import "packs/@/"   ' Assets is a singleton class, you should use it for all loading stuff   Function InitAssets()   Assets.UsePack( "assets.bin" ) ' or do it another way End

Don’t forget to call our InitAssets() in Main():

Monkey

Function Main() InitAssets() New AppInstance New MyWindow App.Run() End

1 2 3 4 5 6 7 8 9 10

Function Main() InitAssets() New AppInstance New MyWindow App.Run() End

Of course, we should use our Assets class for any loading stuff to get all discussed here benefits (easy swapping between loading-from-single-files and loading-from-packs).

Also, we can try to find single-files in assets folder if not found in pack-file.

Attachments:

1. 

   datapacker.png
   

[nerobotParticipant]

In addition to GUI app:

• there is a single output file now, info section with names / offsets / sizes placed alogn with binaries;
• addet GetText() method to extract textual files.

[AmonParticipant]

Now this, I would find very useful.

[nerobotParticipant]

I really like that monkey2 work with raw bytes to create resourses like images, strings, fonts.

I’m going to add font loading from pack file.

[Author]

Posts