MegaLZ depacker - Printable Version

MegaLZ depacker - Printable Version

+- Boriel Basic Forum (https://forum.boriel.com)
+-- Forum: Compilers and Computer Languages (https://forum.boriel.com/forumdisplay.php?fid=12)
+--- Forum: ZX Basic Compiler (https://forum.boriel.com/forumdisplay.php?fid=11)
+---- Forum: How-To & Tutorials (https://forum.boriel.com/forumdisplay.php?fid=13)
+---- Thread: MegaLZ depacker (/showthread.php?tid=420)

MegaLZ depacker - britlion - 2012-01-29

I was using MegaLZ, and I figured it worth listing how to make it a SUB.

Trivially easy, actually - it's provided as ASM source. There's a windows binary program to compress data. Just call this like a mem copy, with source (packed data) and destination (for unpacked data).

I've used it to pack down screen data, and just unpacked it to the screen - it's slow enough to be visible, so do it with attributes set to invisible, and the user won't notice.

Code:
SUB megaLZDepack (source as uInteger, dest as uInteger)
ASM

LD E,(IX+6)
LD D,(IX+7)

;Z80 depacker for megalz V4 packed files   (C) fyrex^mhm

; DESCRIPTION:
;
; Depacker is fully relocatable, not self-modifying,
;it's length is 110 bytes starting from DEC40.
;Register usage: AF,AF',BC,DE,HL. Must be CALL'ed, return is done by RET.
;Provide extra stack location for store 2 bytes (1 word). Depacker does not
;disable or enable interrupts, as well as could be interrupted at any time
;(no f*cking wicked stack usage :).

; USAGE:
;
; - put depacker anywhere you want,
; - put starting address of packed block in HL,
; - put location where you want data to be depacked in DE,
;   (much like LDIR command, but without BC)
; - make CALL to depacker (DEC40).
; - enjoy! ;)

; PRECAUTIONS:
;
; Be very careful if packed and depacked blocks coincide somewhere in memory.
;Here are some advices:
;
; 1. put packed block to the highest addresses possible.
;     Best if last byte of packed block has address #FFFF.
;
; 2. Leave some gap between ends of packed and depacked block.
;     For example, last byte of depacked block at #FF00,
;     last byte of packed block at #FFFF.
;
; 3. Place nonpackable data to the end of block.
;
; 4. Always check whether depacking occurs OK and neither corrupts depacked data
;     nor hangs computer.
;

;DEC40

        LD      A,80h
        EX      AF,AF'
MS:     LDI
M0:      LD      BC,2FFh
M1:      EX      AF,AF'
M1X:     ADD     A,A
        JR      NZ,M2
        LD      A,(HL)
        INC     HL
        RLA
M2:      RL      C
        JR      NC,M1X
        EX      AF,AF'
        DJNZ    X2
        LD      A,2
        SRA     C
        JR      C,N1
        INC     A
        INC     C
        JR      Z,N2
        LD      BC,33Fh
        JR      M1

X2:      DJNZ    X3
        SRL     C
        JR      C,MS
        INC     B
        JR      M1
X6:
        ADD     A,C
N2:
        LD      BC,4FFh
        JR      M1
N1:
        INC     C
        JR      NZ,M4
        EX      AF,AF'
        INC     B
N5:      RR      C
        JP     C, END_DEC40
        RL      B
        ADD     A,A
        JR      NZ,N6
        LD      A,(HL)
        INC     HL
        RLA
N6:      JR      NC,N5
        EX      AF,AF'
        ADD     A,B
        LD      B,6
        JR      M1
X3:
        DJNZ    X4
        LD      A,1
        JR      M3
X4:      DJNZ    X5
        INC     C
        JR      NZ,M4
        LD      BC,51Fh
        JR      M1
X5:
        DJNZ    X6
        LD      B,C
M4:      LD      C,(HL)
        INC     HL
M3:      DEC     B
        PUSH    HL
        LD      L,C
        LD      H,B
        ADD     HL,DE
        LD      C,A
        LD      B,0
        LDIR
        POP     HL
        JR      M0
        
        
END_DEC40:

END ASM
END SUB

Re: MegaLZ depacker - boriel - 2012-01-29

Nice :!:
I'll convert it to fastcall and will pack it with the compiler.

Update: I see you load DE from (IX+n) but not HL? Probably it's working because the compiler uses HL to load addresses and it's already set. But this could crash under some circumnstances. There should be a LD HL, (IX+n) sequence to load HL too.

But better convert it to fascall:

Code:
SUB FASTCALL megaLZDepack (source as uInteger, dest as uInteger)
ASM
; HL already loaded (source)
; dest in the stack, before the return address
pop bc ; Remove return address
pop de ; DE <-- dest address ; NOTE: Remove LD D, (IX+n) and LD E, (IX+n) instructions
push bc ; Restores return address
; The remaing code is untouched:

;Z80 depacker for megalz V4 packed files   (C) fyrex^mhm

; DESCRIPTION:
;
; Depacker is fully relocatable, not self-modifying,
;it's length is 110 bytes starting from DEC40.
...

Re: MegaLZ depacker - britlion - 2012-01-29

I thought fastcall only worked with one parameter?

Still confused as to how those work!

Anyway, if you can make it fastcall, you can probably do ret c instead of jp c, end_of_subroutine; which makes it tighter. (it originally would have had that).

Re: MegaLZ depacker - boriel - 2012-01-30

britlion Wrote:I thought fastcall only worked with one parameter?

Still confused as to how those work!

Anyway, if you can make it fastcall, you can probably do ret c instead of jp c, end_of_subroutine; which makes it tighter. (it originally would have had that).

FASTCALL can be use with more than one parameter, but only the 1st one will be passed in registers. Remaining ones will be passed in the stack, and the programmer MUST pop them out before returning, or your program could crash, etc.

If you look back to my code, I pop out the 2nd parameter in DE. The problem here is that the return address (used by RET) is always pushed automatically by the CPU on TOP, so you must preserve it (pop it out, pop out your parameters, and push it in again before returning). So this is why I use POP BC, POP DE, PUSH BC sequence (BC = temporary storage here).

Re: MegaLZ depacker - LCD - 2012-02-01

THIS WILL be VERY USEFUL for me... But the problem is the gap between blocks of compressed blocks. I would prefer to have a parameter for length of packed or unpacked data.

Re: MegaLZ depacker - britlion - 2012-02-05

I don't think I understand what you mean there LCD. It sort of is what it is really. Takes compressed data from source and puts it as expanded data at destination. There's no data gap internal to either format.

My use for it at the moment is expanding banners to the first 2k of the screen from compressed version elsewhere.

I think a common use might be setting up a buffer for data, and expanding compressed data to the buffer for use, replacing the buffer contents as new data is required - such as the player moving to a new screen.

Re: MegaLZ depacker - boriel - 2012-02-05

britlion, does the FASTCALL and push pop sequence I put worked for you?

Re: MegaLZ depacker - LCD - 2012-02-06

britlion Wrote:I don't think I understand what you mean there LCD. It sort of is what it is really. Takes compressed data from source and puts it as expanded data at destination. There's no data gap internal to either format.

But how does the dapacker know, how much to depack?
"3. Place nonpackable data to the end of block."
means surely a gap (filled with rubbish), or did I understand it wrong and the size is encoded by compressor in packed data?

Edit: I now understand it!This note is for compressor only if we want to pack multiple data from same file. My fault!!!

Re: MegaLZ depacker - britlion - 2012-02-10

boriel Wrote:britlion, does the FASTCALL and push pop sequence I put worked for you?

No.
I still don't understand this, apparently.

e.g.: function mirror (dowedoit as uByte, number as uByte) as uByte

This gets dowedoit in the A register, and both dowedoit and number in the stack.

Stack seems to be:
return address
<some pair of bytes I can't work out>
first parameter pair
second parameter pair

However, if in the above case I do
pop hl
pop de
pop de
pop bc
push hl
ret

It still seems to behave very strangely.

I know that a first byte parameter is at IX+5 (and IX+4 if it's 16 bit)- so logically it's the return address at IX+0, and IX+1. What's at IX+2,IX+3?

Re: MegaLZ depacker - britlion - 2012-02-10

LCD Wrote:
britlion Wrote:I don't think I understand what you mean there LCD. It sort of is what it is really. Takes compressed data from source and puts it as expanded data at destination. There's no data gap internal to either format.
But how does the dapacker know, how much to depack?
"3. Place nonpackable data to the end of block."
means surely a gap (filled with rubbish), or did I understand it wrong and the size is encoded by compressor in packed data?

Edit: I now understand it!This note is for compressor only if we want to pack multiple data from same file. My fault!!!

Ah yes - LCD. A packed file knows its own length; it's not continuous. You can't unpack part of it without changing the decompression code, there.

Re: MegaLZ depacker - LCD - 2012-02-10

britlion Wrote:
LCD Wrote:
britlion Wrote:I don't think I understand what you mean there LCD. It sort of is what it is really. Takes compressed data from source and puts it as expanded data at destination. There's no data gap internal to either format.
But how does the dapacker know, how much to depack?
"3. Place nonpackable data to the end of block."
means surely a gap (filled with rubbish), or did I understand it wrong and the size is encoded by compressor in packed data?

Edit: I now understand it!This note is for compressor only if we want to pack multiple data from same file. My fault!!!

Ah yes - LCD. A packed file knows its own length; it's not continuous. You can't unpack part of it without changing the decompression code, there.

Cool, then it will be extremly useful in my next projects. Great!

Edit: If this is so easy, maybe Exonomizer 2.01 depacker could be a good idea. Metalbrain made a Z80 depacker for it. There are also Chrust and Bitbuster Extreme. No idea which is faster or has better compression.

Re: MegaLZ depacker - LCD - 2012-04-12

Okay, the Binary manager of BorIDE will have the ability to pack binaries (also imported from TAP) using MegaLZ. It is very effective. I tried to pack a previously packed screen, where PKLite saved 1 Kb, MegaLZ saved 2 Kb. So the screen packed from 6912 Bytes using Retro-X screen compressor (which will be used in BorIDE too) to 3500 Bytes was further packed by MegaLZ to 1500 Bytes.

Code:
JP     C, END_DEC40

IMHO should be replaced with

Code:
JR     C, END_DEC40

because all other Jumps in the source are relative, and this saves 1 Byte and is faster (okay, nobody will notice). For me every byte counts in the Dungeon Crawler project.

Re: MegaLZ depacker - slenkar - 2012-04-14

What kind of information can be compressed?

Graphics, arrays, sounds..?

[Image: xaVjy.jpg]

Re: MegaLZ depacker - LCD - 2012-04-14

slenkar Wrote:What kind of information can be compressed?

Graphics, arrays, sounds..?

Wow!
Excellent... Yes, I'm working hard to do so.

Any binary data can be compressed, but not arrays.
Natively: Graphics blocks and maps/screens (after adding map editor)

Re: MegaLZ depacker - LCD - 2013-03-07

Poke 23388 with current Bank
And this is the version which can decompress from other banks to a buffer to main memory:

Code:
' MegaLZ Unpacker by Britlion, Banking by LCD
SUB megaLZDepack(source as uInteger,dest as uInteger,bank as ubyte)
ASM

	LD E,(IX+6)
	LD D,(IX+7)
	di
	ld a,(ix+9)
	ld bc,32765
	out (c),a

;'Z80 depacker for megalz V4 packed files   (C) fyrex^mhm

;' DESCRIPTION:
;'
;' Depacker is fully relocatable, not self-modifying,
;'it's length is 110 bytes starting from DEC40.
;'Register usage: AF,AF',BC,DE,HL. Must be CALL'ed, return is done by RET.
;'Provide extra stack location for store 2 bytes (1 word). Depacker does not
;'disable or enable interrupts, as well as could be interrupted at any time
;'(no f*cking wicked stack usage :).

;' USAGE:
;'
;' - put depacker anywhere you want,
;' - put starting address of packed block in HL,
;' - put location where you want data to be depacked in DE,
;'   (much like LDIR command, but without BC)
;' - make CALL to depacker (DEC40).
;' - enjoy! ;)

;' PRECAUTIONS:
;'
;' Be very careful if packed and depacked blocks coincide somewhere in memory.
;'Here are some advices:
;'
;' 1. put packed block to the highest addresses possible.
;'     Best if last byte of packed block has address #FFFF.
;'
;' 2. Leave some gap between ends of packed and depacked block.
;'     For example, last byte of depacked block at #FF00,
;'     last byte of packed block at #FFFF.
;'
;' 3. Place nonpackable data to the end of block.
;'
;' 4. Always check whether depacking occurs OK and neither corrupts depacked data
;'     nor hangs computer.
;'

;'DEC40

        LD      A,80h
        EX      AF,AF'
MS:     LDI
M0:      LD      BC,2FFh
M1:      EX      AF,AF'
M1X:     ADD     A,A
        JR      NZ,M2
        LD      A,(HL)
        INC     HL
        RLA
M2:      RL      C
        JR      NC,M1X
        EX      AF,AF'
        DJNZ    X2
        LD      A,2
        SRA     C
        JR      C,N1
        INC     A
        INC     C
        JR      Z,N2
        LD      BC,33Fh
        JR      M1

X2:      DJNZ    X3
        SRL     C
        JR      C,MS
        INC     B
        JR      M1
X6:
        ADD     A,C
N2:
        LD      BC,4FFh
        JR      M1
N1:
        INC     C
        JR      NZ,M4
        EX      AF,AF'
        INC     B
N5:      RR      C
        JP     C, END_DEC40
        RL      B
        ADD     A,A
        JR      NZ,N6
        LD      A,(HL)
        INC     HL
        RLA
N6:      JR      NC,N5
        EX      AF,AF'
        ADD     A,B
        LD      B,6
        JR      M1
X3:
        DJNZ    X4
        LD      A,1
        JR      M3
X4:      DJNZ    X5
        INC     C
        JR      NZ,M4
        LD      BC,51Fh
        JR      M1
X5:
        DJNZ    X6
        LD      B,C
M4:      LD      C,(HL)
        INC     HL
M3:      DEC     B
        PUSH    HL
        LD      L,C
        LD      H,B
        ADD     HL,DE
        LD      C,A
        LD      B,0
        LDIR
        POP     HL
        JR      M0
       
       
END_DEC40:
	ld a,(23388)
	ld bc,32765
	out (c),a
	ei
END ASM
END SUB

megaLZDepack (32768,49152)