Re: BHYVE SNAPSHOT image format proposal

Hi Corvin, 

> On 6 Jun 2023, at 15:59, Corvin Köhne <corvink@FreeBSD.org> wrote:
>> ...
> 
> We may have different version of the format from the same produce.
> IMHO, it makes sense to have a dedicated IDENT and VERSION field to
> easily figure out
> 
> 1) if the producer of the image is known
> 2) if we support that version of the producer
> 
> Even if you allocated a huge amount of free space, someone would need
> more. So, what do you think about this format:
> 
> +---------------------------------------------------------------------+
> | IDENT - 16 BYTES                                                    |
> +-------------------+-----------------------+-------------------------+
> | VERSION - 4 BYTES | NVLIST SIZE - 4 BYTES | NVLIST OFFSET - 8 BYTES |
> +-------------------+-----------------------+-------------------------+
> | POSSIBLE FREE SPACE (e.g. for custom data, alignment etc.)          |
> +---------------------------------------------------------------------+
> | NVLIST DATA                                                         |
> +---------------------------------------------------------------------+
> | POSSIBLE FREE SPACE (for whatever reason)                           |
> +---------------------------------------------------------------------+
> | SNAPSHOT DATA                                                       |
> +——————————————————————————————————+
> 

Note, simple string "BHYVE CHECKPOINT IMAGE” has 22 bytes. So 16 bytes seems
too small. 

So I would not to complicate a header first. 

I would rather describe ideas, conditions and then solutions:

Need to distinguish snapshot image file from other files. 
Solution: Header should have "magic id”.

Need a barrier for resuming if image "is not ours”. Idea is not to allow to resume images from other producers.

The reason to have it and use it instead of header versioning: 

Imagine that mainstream has its own implementation and company’s fork repo has its own implementation. How to ensure that the versions in an image file are ours and not somebody’s else?

Solution:  Header should have “Producer id” string.

Example: Snapshot image file has empty Producer string , but bhyve has current Producer as “MYCOMPANY”. Strings are not equal, resume must fail.

The Rule above does not restrict getting/decoding data from an image file. It should be possible to look at an image file and analyse internals, to get/decode values, etc.
Solution: Have additional option either in bhyve or bhyvectl to get into image file.

Following nvlist header data should have a short information about image file and its internals.
Solution: NV HEADER can have several sections: “config”, “kernel”, “devices”, “memory”, …

Versioning of NV HEADER. Idea is to have an information in advance whether it is possible to be resumed or not. In other words, before do resume, get information about ability to resume.

Solution: Each Section should have “version”  and “subversion”. While “version” is responsible for both  types of compatibility: backward and forward, “subversion” is for forward compatibility only.

Rules for check:
        If bh_version == version && bh_subversion >= subversion  then
                  Bhyve able to resume the Section
        Else
                  Bhyve cannot resume the Section
        Endif

Example 1: Section in image has “version=1", “subversion=5”,  Bhyve has “version=1", “subversion=6". That means, bhyve can resume the Section.

Example 2: The same image Section, but bhyve has “version=1", “subversion=4". Bhyve cannot resume the Section.

Example 3: The same image Section, but bhyve has “version=2", “subversion=5”. Bhyve cannot resume the Section.

Rules for increasing versions:
  -  If during code-change “backward” compatibility is broken, “version” should be increased and “subversion” is set to 0.
  -  If during code-change “forward” compatibility is broken, “subversion” should be increased.

Other versioning in HEADER is redundant. If something is changed in the format, “magic id” can be changed appropriately.
Solution: “magic id” should be stable and not changed for a long time.


As result I would suggest to give at least 32 bytes for "magic id” / ident and 32 bytes for “producer id”.

Format of entire image file can be:


+-----------------------------------------------------------------+
|               HEADER MAGIC ID     - 32 BYTES                    |
+-----------------------------------------------------------------+
|               HEADER PRODUCER ID  - 32 BYTES                    |
+———————————————----------------————————--—-----------------------+
|               NVLIST HEADER SIZE  -  4 BYTES                    | 
+-----------------------------------------------------------------+
|               NVLIST HEADER DATA (SECTIONS)                     |
+-----------------------------------------------------------------+
|                       SNAPSHOT DATA                             |
+-----------------------------------------------------------------+


MAGIC ID: should be hardcoded: "BHYVE CHECKPOINT IMAGE”.

PRODUCER ID: can be empty and supported by producer, i.e. reserved. 

NVLIST HEADER SIZE: has enough dimension, but in general size is less than 4KB

NVLIST HEADER DATA: Packed nvlist data, contains Sections:  “config”, “kernel”, “devices”, “memory”, … :

[config]
        offset = 0x1000 (4096)
        size = 0x1f6 (502)
        type = text
	vers = 1
	subvers = 5
[kernel]
        offset = 0x11f6 (4598)
        size = 0x19a7 (6567)
        type = nvlist
	vers = 1
	subvers = 0
[devices]
        offset = 0x2b9d (11165)
        size = 0x10145ba (16860602)
        type = nvlist
	vers = 2
	subvers = 1
[memory]
        offset = 0x1200000 (18874368)
        size = 0x3ce00000 (1021313024)
        type = pages
	vers = 1
	subvers = 0 


I hope I gained a whole understanding.
Thanks,
Vitaliy Gusev