Reading the string with databuffer

This topic contains 11 replies, has 2 voices, and was last updated by nerobot 8 months, 3 weeks ago.

Viewing 12 posts - 1 through 12 (of 12 total)

Author

Posts

July 22, 2018 at 2:49 pm #15108

Participant

I try to create some binary loader, but got stuck at the point of reading strings. Say that I know nothing about the size of the string, it might be anything of unknown length. So the point here is to let the reader scan the bytes and return the string by itself.

This code is only a sample for producing the binary file.

static void TestWriter(string filepath)
    {
      var file = new FileStream(filepath, FileMode.Create);
      var writer = new BinaryWriter(file);

      writer.Write(10);
      writer.Write("HELLO");
      writer.Write(10);
      writer.Write("MONKEYCODER");
      
      writer.Close();
    }

static void TestWriter(string filepath)

{

var file = new FileStream(filepath, FileMode.Create);

var writer = new BinaryWriter(file);

writer.Write(10);

writer.Write("HELLO");

writer.Write(10);

writer.Write("MONKEYCODER");

writer.Close();

}

July 23, 2018 at 1:17 pm #15117

cocon

Participant

... old code removed ...

1	... old code removed ...

after lots of testing i figured it out

Attachments:

readstring.png

July 23, 2018 at 1:28 pm #15119

cocon

Participant

Otherwise if you want to test the default peek string method you would see some problems:

Method ReadString2:String()
	Local s := Buffer.PeekString(Position)
	Position += s.Length
	DebugStop()
	Return s
End

Method ReadString2:String()

Local s := Buffer.PeekString(Position)

Position += s.Length

DebugStop()

Return s

End

July 23, 2018 at 3:29 pm #15120

nerobot

Participant

Otherwise if you want to test the default peek string method you would see some problems

Even if you use buffer.WriteString() ?

July 24, 2018 at 11:15 am #15131

cocon

Participant

I looked at it further today and I found differences on how CSharp + Monkey writes the data buffer. However I found a better way to read the string (which works both fine for the C# data and the Monkey data – hopefully would be universal). It seems that PeekCString does the work fine however there was some trickiness involved in figuring out the offsets and lengths on how the string is peeked and the pointer is advanced.

' databufferead.monkey2 '

#Import "<std>"
Using std.filesystem
Using std.memory

Class DataBufferReader
	Field Position:Int
	Field Buffer:DataBuffer
	
	Method New(file:String)
		Position = 0
		Buffer = DataBuffer.Load(file)
		DebugAssert(Buffer <> Null, "Could not open file: " + file)
	End
	
	Method ReadInt:Int()
		Position+=4
		Return Buffer.PeekInt(Position-4)
	End
	
	Method ReadString2:String()
		Local s := Buffer.PeekCString(Position)
		Position += s.Length - 1 ' ignore the null terminator [\0]
		s = s.Slice(0, s.Length - 1) ' discard the null terminator
		Return s
	End
End

Function Main()
	Local file   := AppDir() + "..\..\csharpdata.something"
'	Local file   := AppDir() + "..\..\monkeydata.something"
	Local reader := New DataBufferReader(file)
	
	Local int1 := reader.ReadInt()
	Local str1 := reader.ReadString2()
	Local int2 := reader.ReadInt()
	Local str2 := reader.ReadString2()
	
	DebugStop()
End

' databufferead.monkey2 '

#Import "<std>"

Using std.filesystem

Using std.memory

Class DataBufferReader

Field Position:Int

Field Buffer:DataBuffer

Method New(file:String)

Position = 0

Buffer = DataBuffer.Load(file)

DebugAssert(Buffer <> Null, "Could not open file: " + file)

End

Method ReadInt:Int()

Position+=4

Return Buffer.PeekInt(Position-4)

End

Method ReadString2:String()

Local s := Buffer.PeekCString(Position)

Position += s.Length - 1 ' ignore the null terminator [\0]

s = s.Slice(0, s.Length - 1) ' discard the null terminator

Return s

End

Function Main()

Local file := AppDir() + "..\..\csharpdata.something"

' Local file := AppDir() + "..\..\monkeydata.something"

Local reader := New DataBufferReader(file)

Local int1 := reader.ReadInt()

Local str1 := reader.ReadString2()

Local int2 := reader.ReadInt()

Local str2 := reader.ReadString2()

DebugStop()

End

' databufferwrite.monkey2 '

#Import "<std>"
Using monkey.debug
Using std.filesystem
Using std.memory

Class DataBufferWriter
	Field Position:Int
	Field Buffer:DataBuffer
	
	Method New(capacity:Int)
		Position = 0
		Buffer   = New DataBuffer(capacity)
	End
	
	Method WriteInt(i:Int)
		Buffer.PokeInt(Position, i)
		Position += 4
	End
	
	Method WriteString(s:String)
		Buffer.PokeString(Position, s)
		Position += s.Length
	End
	
	Method Save(path:String)
		Buffer.Resize(Buffer.Length)
		Buffer.Save(path)
	End
End

Function Main()
	Local buffer := New DataBufferWriter(100)
	buffer.WriteInt(10)
	buffer.WriteString("HELLO")
	buffer.WriteInt(100)
	buffer.WriteString("MONKEYCODER")
	buffer.Save(AppDir() + "..\..\monkeydata.something")
End

' databufferwrite.monkey2 '

#Import "<std>"

Using monkey.debug

Using std.filesystem

Using std.memory

Class DataBufferWriter

Field Position:Int

Field Buffer:DataBuffer

Method New(capacity:Int)

Position = 0

Buffer = New DataBuffer(capacity)

End

Method WriteInt(i:Int)

Buffer.PokeInt(Position, i)

Position += 4

End

Method WriteString(s:String)

Buffer.PokeString(Position, s)

Position += s.Length

End

Method Save(path:String)

Buffer.Resize(Buffer.Length)

Buffer.Save(path)

End

Function Main()

Local buffer := New DataBufferWriter(100)

buffer.WriteInt(10)

buffer.WriteString("HELLO")

buffer.WriteInt(100)

buffer.WriteString("MONKEYCODER")

buffer.Save(AppDir() + "..\..\monkeydata.something")

End

Attachments:

monkeywritesbinaryfile.png

July 24, 2018 at 2:41 pm #15137

nerobot

Participant

You poke string but peek cstring. Is it correct?
There is a CStringLength property of String type. Can be useful to set proper offset.

July 25, 2018 at 11:18 am #15146

cocon

Participant

Buffer.PeekCString helps a lot because it stops when it detects the 0 byte.
monkeydoc: Reads a null terminated CString from the databuffer.

However PeekString is designed differently.
monkeydoc: all bytes from offset until the end of the data buffer are read

I tried as well PeekCString to see if it works but it gives the same number (which is 7 ).

Print(s.Length)
Print(s.CStringLength)
DebugStop()

Print(s.Length)

Print(s.CStringLength)

DebugStop()

July 25, 2018 at 11:25 am #15147

nerobot

Participant

I tried as well PeekCString to see if it works but it gives the same number (which is 7 ).

It is true for latin locale, not for whole utf.

Look at this example with russian text:

Namespace myapp

#Import "<std>"

Using std..

Function Main()
	Print "Hello World".Length
	Print "Hello World".CStringLength
	Print "Привет Мир".Length
	Print "Привет Мир".CStringLength	
End

Namespace myapp

#Import "<std>"

Using std..

Function Main()

Print "Hello World".Length

Print "Hello World".CStringLength

Print "Привет Мир".Length

Print "Привет Мир".CStringLength

End

It prints: 11, 11, 10, 19.

July 25, 2018 at 2:22 pm #15152

cocon

Participant

Oh so this means that CStringLength is the raw string format, right?

Something else, I updated the code today but still it fails. For the sake of example, I coded the data generator again in C, this is as close as it gets to what the problem is. Monkey and C# might have been better design to avoid this confusion, but C is something different. I might have to do unicode conversion or bitshifting (null bytes might be bishifted to the other bytes).

#pragma warning(disable : 4996)

#include <stdio.h>

int main()
{
	int num;
	FILE *f;
	f = fopen("cdata.something", "w");
	
	fprintf(f, "%d", 10);
	fprintf(f, "%s", "HELLO");
	fprintf(f, "%d", 100);
	fprintf(f, "%s", "MONKEYCODER");

	fclose(f);

	return 0;
}

#pragma warning(disable : 4996)

#include <stdio.h>

int main()

{

int num;

FILE *f;

f = fopen("cdata.something", "w");

fprintf(f, "%d", 10);

fprintf(f, "%s", "HELLO");

fprintf(f, "%d", 100);

fprintf(f, "%s", "MONKEYCODER");

fclose(f);

return 0;

}

No more no less I captured the output in the screenshot.

Attachments:

cdata.png

July 25, 2018 at 2:26 pm #15154

nerobot

Participant

Maybe \0 is needed for strings? fprintf(f, "%s", "HELLO\0");
Why don’t you write such generator in mokey2?

July 26, 2018 at 11:13 am #15159

cocon

Participant

It seems that there are various conventions on how to write strings.
http://wiki.freepascal.org/Character_and_string_types#String

The STRING+NULLBYTE is the Ansi String (what monkey has)
The STR_LENGTH+STRING is the Pascal String (what the file format has)

The problem was that I had not notice what sort of strings the file format used. I got inconsistent results among Monkey+C#+C++ because of not knowing the convention. :-S

I can add simply a ReadPascalString method to the binary reader and close this.

July 26, 2018 at 12:09 pm #15160

nerobot

Participant

Java also use STR_LENGTH+STRING (afaik).

ReadPascalString name is not good.

Maybe

ReadString – for len+string
ReadZeroTerminatedString – for string+0

Would be easy to read and understand.

Author

Posts

Viewing 12 posts - 1 through 12 (of 12 total)

You must be logged in to reply to this topic.

Reading the string with databuffer

Attachments:

Attachments:

Attachments:

Archives

Recent Replies

What is this?