1. Forum
  2. FSXNet
  3. FSX BBS

  • MBSE compression transfers

    From deon@21:2/116 to Andrew Leary on Wed Jul 5 13:29:51 2023
    Hey Andrew,

    I've been playing with implementing GZ/BZ2 compression in clrghouz, which I've been testing with binkd.

    I've noticed that size value in M_FILE, is the size (of the file) before compression.

    - M_FIL list.txt 16384 1688519850 -1 BZ2

    ie: In this run, list.txt was full of zeros, which compresseses to 46 bytes.

    - Compressed 16384 bytes to 46 for list.txt, ratio 0.3%

    However, if list.txt was full of random binary data, the chance of it growing is actually high (and it did most times I tested).

    - Compressed 16384 bytes to 16867 for list.txt, ratio 102.9%

    I was wondering how MBSE handles it? Is the size element of M_FILE the size of the file, or the transfer (I assume the former, which is what the spec implies).

    The problem I'm trying to solve, is that there is no "end of file" marker being sent by the sender, and since compression is being used, I dont know how much data to receive before doing the uncompress.

    And if the file grows, I dont know how much bigger it will grow. (And the sender actually ends up waiting for the M_GOT.)

    I've worked around it, but curios how MBSE handles it.


    ...δεσ∩
    --- SBBSecho 3.20-Linux
    * Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)
  • From Andrew Leary@21:4/105 to deon on Wed Jul 5 10:36:36 2023
    Hello deon!

    05 Jul 23 13:29, you wrote to me:

    I've been playing with implementing GZ/BZ2 compression in clrghouz,
    which I've been testing with binkd.

    I've noticed that size value in M_FILE, is the size (of the file)
    before compression.

    Correct.

    - M_FIL list.txt 16384 1688519850 -1 BZ2

    ie: In this run, list.txt was full of zeros, which compresseses to 46 bytes.

    - Compressed 16384 bytes to 46 for list.txt, ratio 0.3%

    However, if list.txt was full of random binary data, the chance of it growing is actually high (and it did most times I tested).

    - Compressed 16384 bytes to 16867 for list.txt, ratio 102.9%

    I was wondering how MBSE handles it? Is the size element of M_FILE the size of the file, or the transfer (I assume the former, which is what
    the spec implies).

    It is the size of the original file. It's been a while since I went near that code in binkp.c, but I believe MBSE shuts off compression if it results in the data to send growing.

    The problem I'm trying to solve, is that there is no "end of file"
    marker being sent by the sender, and since compression is being used,
    I dont know how much data to receive before doing the uncompress.

    And if the file grows, I dont know how much bigger it will grow. (And
    the sender actually ends up waiting for the M_GOT.)

    I've worked around it, but curios how MBSE handles it.

    You're welcome to have a look at the code; it's available from https://sourceforge.net/projects/mbsebbs/ ...

    Andrew

    --- GoldED+/LNX 1.1.5-b20230304
    * Origin: Phoenix BBS * phoenix.bnbbbs.net (21:4/105)
  • From deon@21:2/116 to Andrew Leary on Thu Jul 13 13:30:58 2023
    Re: MBSE compression transfers
    By: Andrew Leary to deon on Wed Jul 05 2023 10:36 am

    Howdy,

    I was wondering how MBSE handles it? Is the size element of M_FILE the size of the file, or the transfer (I assume the former, which is what the spec implies).

    The problem I'm trying to solve, is that there is no "end of file" marker being sent by the sender, and since compression is being used,
    I dont know how much data to receive before doing the uncompress.

    You're welcome to have a look at the code; it's available from https://sourceforge.net/projects/mbsebbs/ ...

    Thought I'd ask, since its hard reading someone else's code :)

    How does MBSE handle receiving a compressed file?

    IE: The sender advertises the file with M_FILE - but the size is the pre-compressed size. The actual file might be smaller or larger than that.

    Since there is no "end of file marker" during the transfer, how does MBSE know it's finished receiving the last block of a compressed file? (Good thing TCP is a reliable transport.) IE: Do you continue uncompressing the received block of data until the resultant uncompressed size = the M_FILE advertised size?

    (In my case, Ive found that I cannot uncompress a BZ2 payload until I have all the data, and it seems "testing the de-compression" after each block of data is unnessarily unefficient.)

    I know compressed transfers are not that important, but I wanted to implement it for completeness.


    ...δεσ∩
    --- SBBSecho 3.20-Linux
    * Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)