From: Sergey Matveev Date: Thu, 8 May 2025 14:13:13 +0000 (+0300) Subject: Move from Texinfo to zettelkästen plaintext X-Git-Url: http://www.git.cypherpunks.su/?a=commitdiff_plain;h=67a7ce7a3e5e46c028502cae971a79c9beb6c87aee51e72728d5c11c0c8eed1e;p=keks.git Move from Texinfo to zettelkästen plaintext --- diff --git a/spec/.gitignore b/spec/.gitignore index bcf42ef..be62ab4 100644 --- a/spec/.gitignore +++ b/spec/.gitignore @@ -1,2 +1 @@ /spec.html/ -/spec.info diff --git a/spec/.zkignore b/spec/.zkignore new file mode 100644 index 0000000..d3294cb --- /dev/null +++ b/spec/.zkignore @@ -0,0 +1,2 @@ +^mk-html +^spec.html diff --git a/spec/ComparisonWithOtherCodecs b/spec/ComparisonWithOtherCodecs new file mode 100644 index 0000000..07a264a --- /dev/null +++ b/spec/ComparisonWithOtherCodecs @@ -0,0 +1,51 @@ +Are not there any satisfiable codecs? + +=> https://en.wikipedia.org/wiki/Distinguished_Encoding_Rules#DER_encoding DER +=> https://en.wikipedia.org/wiki/Distinguished_Encoding_Rules#CER_encoding CER +=> https://datatracker.ietf.org/doc/html/rfc1014 XDR +=> https://www.JSON.org/json-en.html JSON +=> https://bsonspec.org/ BSON +=> https://msgpack.org/ MessagePack +=> https://datatracker.ietf.org/doc/html/rfc8949 CBOR +=> https://datatracker.ietf.org/doc/html/draft-mcnally-deterministic-cbor-11 dCBOR +=> http://cr.yp.to/proto/netstrings.txt Netstrings +=> https://wiki.theory.org/BitTorrentSpecification#Bencoding Bencode +=> https://en.wikipedia.org/wiki/Canonical_S-expressions Canonical S-expression + + | Schemaless | Simple | Deterministic | Streamable | Compact +ASN.1 DER | N | N | Y | N | N +ASN.1 CER | N | N | Y | Y | N +XDR | N | Y | N | N | N +JSON | Y | N | N | Y | N +BSON | Y | Y | N | N | N +MessagePack | Y | Y | N | N | Y +CBOR | Y | N | N | Y | Y +dCBOR | Y | N | Y | N | Y +Netstrings | Y | Y | Y | N | ~ +Bencode | Y | Y | Y | Y | ~ +CSExp | Y | Y | Y | Y | ~ +KEKS | Y | Y | Y | Y | Y + + | BigStr | BinStr | UTF8Str | Int | BigInt | List | Struct | Time +ASN.1 DER | Y | Y | Y | Y | Y | Y | Y | Y +ASN.1 CER | Y | Y | Y | Y | Y | Y | Y | Y +XDR | N | Y | Y | Y | N | Y | Y | N +JSON | Y | N | Y | Y | Y | Y | Y | N +BSON | N | Y | Y | Y | N | Y | Y | Y +MessagePack | N | Y | Y | Y | N | Y | Y | N +CBOR | Y | Y | Y | Y | N | Y | Y | N +dCBOR | Y | Y | Y | Y | N | Y | Y | N +Netstrings | Y | Y | N | N | N | N | N | N +Bencode | Y | Y | N | Y | Y | Y | Y | N +CSExp | Y | Y | N | N | N | Y | N | N +KEKS | Y | Y | Y | Y | Y | Y | Y | Y + +Note about CBOR: + +* Hardly you will find wide range of CBOR libraries supporting strict + validation of deterministically encoded CBOR structures. +* Tagged string/integer can not be taken as a viable first-class + bigint/datetime data support, because many decoders do not support + tags and won't be able to interpret/validate them. +* Non-string map keys very complicates representation process for + dynamically types languages. diff --git a/spec/INSTALL b/spec/INSTALL new file mode 100644 index 0000000..cfd68cd --- /dev/null +++ b/spec/INSTALL @@ -0,0 +1,17 @@ +Currently there are draft versions of the codec written on C, Go, +Python and Tcl. + +You can obtain development source code with: + git clone git://git.cypherpunks.su/keks.git + +You can also use following URLs instead: +anongit@master.git.stargrave.org:cypherpunks.su/keks.git +anongit@slave.git.stargrave.org:cypherpunks.su/keks.git +anongit@master.git.cypherpunks.su:cypherpunks.su/keks.git +anongit@slave.git.cypherpunks.su:cypherpunks.su/keks.git +git://git.stargrave.org/keks.git +git://y.git.stargrave.org/keks.git +git://y.git.cypherpunks.su/keks.git + +Also there is Yggdrasil accessible address: http://y.www.keks.cypherpunks.su/ +=> https://yggdrasil-network.github.io/ Yggdrasil diff --git a/spec/THANKS b/spec/THANKS new file mode 100644 index 0000000..3be247d --- /dev/null +++ b/spec/THANKS @@ -0,0 +1,3 @@ +* Sergey Mayorov for his valuable consultation and suggestions. +* Anton Rudenko for the initial unittests of the Python and + Go codec implementation. diff --git a/spec/cm/authcrypt.texi b/spec/cm/authcrypt.texi deleted file mode 100644 index f6de895..0000000 --- a/spec/cm/authcrypt.texi +++ /dev/null @@ -1,21 +0,0 @@ -@node Authcrypt -@cindex authenticated public-key encryption -@nodedescription Authenticated public-key encryption -@subsection Authenticated public-key encryption - -Public-key based @ref{KEM, KEMs} provides sender authentication -@strong{only} if @code{/kem/*/from} field is specified. It should -contain public key's @code{/load/v/id}, but may be equal to 256-bit -zeros, to explicitly specify that sender's public key is used, but it is -anonymous and hidden. It is not specified how recipient should find -corresponding sender's key that way -- implementation/protocol specific. - -Optional @code{/pubs} is a list public keys, which may be used to supply -sender's public key(s). Public keys may be encrypted, to hide the actual -deanonymisation contents. - -It is @strong{highly} recommended to use multi-recipient safe DEM when -encrypting to multiple recipients. For example -@code{@ref{dem-xchapoly-krmr, dem-xchapoly-krmr}} instead of -@code{@ref{dem-xchapoly-krkc, dem-xchapoly-krkc}}, but unfortunately -with the price of more expensive double pass authentication scheme. diff --git a/spec/cm/dem-kuznechik-ctr-hmac-kr.texi b/spec/cm/dem-kuznechik-ctr-hmac-kr.texi deleted file mode 100644 index b9e2fb0..0000000 --- a/spec/cm/dem-kuznechik-ctr-hmac-kr.texi +++ /dev/null @@ -1,35 +0,0 @@ -@node dem-kuznechik-ctr-hmac-kr -@cindex dem-kuznechik-ctr-hmac-kr -@nodedescription Kuznechik-CTR-HMAC with key ratcheting DEM -@subsubsection Kuznechik-CTR-HMAC with key ratcheting DEM - -@code{cm/encrypted}'s @code{/dem/a} equals to "kuznechik-ctr-hmac-kr". - -CEK is 64 bytes long. -Data is split on 128 KiB chunks, each of which is encrypted the following way: - -@verbatim -H = Streebog-512 -CK0 = CEK -CKi = HKDF-Extract(H, salt="", ikm=CK{i-1}) -Kenc = HKDF-Expand(H, prk=CKi, info="cm/encrypted/kuznechik-ctr-hmac-kr/enc") -IV = HKDF-Expand(H, prk=CKi, info="cm/encrypted/kuznechik-ctr-hmac-kr/iv", len=8) -Kauth || KauthTail = HKDF-Expand(H, prk=CKi, info="cm/encrypted/kuznechik-ctr-hmac-kr/auth") -KauthTail = HKDF-Expand(H, prk=CKi, info="cm/encrypted/kuznechik-ctr-hmac-kr/authTail") -CT = Kuznechik-CTR(key=Kenc, ctr=IV, data=chunk) -CT || HMAC(Streebog-256, key={Kauth|KauthTail}, data=CT) -@end verbatim - -@code{KauthTail} is used only in the last chunk to explicitly signal -that it is the last one. - -@code{/payload}'s chunk length equals to 128KiB+32 bytes. - -HKDF is KDF algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc5869.html, RFC 5869}. -HMAC is MAC algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc2104.html, RFC 2104}. -Kuznechik is GOST R 34.12-2015 encryption algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc7801.html, RFC 7801}. -Streebog is GOST R 34.11-2012 hashing algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc6986.html, RFC 6986}. diff --git a/spec/cm/dem-xchapoly-krkc.texi b/spec/cm/dem-xchapoly-krkc.texi deleted file mode 100644 index ae3e520..0000000 --- a/spec/cm/dem-xchapoly-krkc.texi +++ /dev/null @@ -1,36 +0,0 @@ -@node dem-xchapoly-krkc -@cindex dem-xchapoly-krkc -@nodedescription XChaCha20-Poly1305 with key ratcheting and key commitment DEM -@subsubsection XChaCha20-Poly1305 with key ratcheting and key commitment DEM - -@code{cm/encrypted}'s @code{/dem/a} equals to "xchapoly-krkc". - -CEK is 64 bytes long. -Data is split on 128 KiB chunks, each of which is encrypted the following way: - -@verbatim -H = BLAKE2b -CK0 = CEK -CKi = HKDF-Extract(H, salt="", ikm=CK{i-1}) -KEY = HKDF-Expand(H, prk=CKi, info="cm/encrypted/xchapoly-krkc/key") -IV = HKDF-Expand(H, prk=CKi, info="cm/encrypted/xchapoly-krkc/iv", len=24) -if last chunk { IV[23] |= 0x01 } else { IV[23] &= 0xFE } -CIPHERTEXT || TAG = XChaCha20-Poly1305(key=KEY, ad="", nonce=IV, data=chunk) -COMMITMENT = BLAKE2b-256(KEY || IV || TAG) -CIPHERTEXT || TAG || COMMITMENT -@end verbatim - -Chaining key (CK) advances with every chunk. 256-bit encryption key and -randomised 192-bit nonce (initialisation vector) are derived from it. - -Nonce's lowest bit is set only if this is the last chunk we encrypting. - -@code{/payload}'s chunk length equals to 128KiB+16+32 bytes. - -HKDF is KDF algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc5869.html, RFC 5869}. -@url{https://www.blake2.net/, BLAKE2b} is hashing algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc7693.html, RFC 7693}. -@url{https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-xchacha, XChaCha20-Poly1305} -is an authenticated encryption algorithm, extended nonce version of ChaCha20-Poly1305, -@url{https://datatracker.ietf.org/doc/html/rfc8439.html, RFC 8439}. diff --git a/spec/cm/dem-xchapoly-krmr.texi b/spec/cm/dem-xchapoly-krmr.texi deleted file mode 100644 index 1b133c0..0000000 --- a/spec/cm/dem-xchapoly-krmr.texi +++ /dev/null @@ -1,42 +0,0 @@ -@node dem-xchapoly-krmr -@cindex dem-xchapoly-krmr -@nodedescription XChaCha20-Poly1305 with key ratcheting and multi-recipient DEM -@subsubsection XChaCha20-Poly1305 with key ratcheting and multi-recipient DEM - -@code{cm/encrypted}'s @code{/dem/a} equals to "xchapoly-krmr". - -CEK consists of common 64 bytes long part equal in all KEMs (@code{CEK} itself), -and 64 bytes long per-KEM/per-recipient random MAC key (@code{prMAC}). - -Data is split on 128 KiB chunks, each of which is encrypted the following way: - -@verbatim -H = BLAKE2b -CK0, prMACx0 = CEK || prMACx -CKi = HKDF-Extract(H, salt="", ikm=CK{i-1}) -KEY = HKDF-Expand(H, prk=CKi, info="cm/encrypted/xchapoly-krmr/key") -IV = HKDF-Expand(H, prk=CKi, info="cm/encrypted/xchapoly-krmr/iv", len=24) -if last chunk { IV[23] |= 0x01 } else { IV[23] &= 0xFE } -CIPHERTEXT || TAG = XChaCha20-Poly1305(key=KEY, ad="", nonce=IV, data=chunk) -prMACxi = HKDF-Extract(H, salt="", ikm=prMACx{i-1}) -MACx = BLAKE2b-256-MAC(key=prMACxi, H(CIPHERTEXT || TAG)) -CIPHERTEXT || TAG || MACx || MAC{x+1} [|| MAC{x+2} ...] -@end verbatim - -Chaining key (CK) and per-recipient MAC key advance with every chunk. -256-bit encryption key and randomised 192-bit nonce (initialisation -vector) are derived from chaining key. - -Nonce's lowest bit is set only if this is the last chunk we encrypting. - -MACs are ordered the same way as KEMs in the list. - -@code{/payload}'s chunk length equals to 128KiB+16+32*recipients bytes. - -HKDF is KDF algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc5869.html, RFC 5869}. -@url{https://www.blake2.net/, BLAKE2b} is hashing algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc7693.html, RFC 7693}. -@url{https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-xchacha, XChaCha20-Poly1305} -is an authenticated encryption algorithm, extended nonce version of ChaCha20-Poly1305, -@url{https://datatracker.ietf.org/doc/html/rfc8439.html, RFC 8439}. diff --git a/spec/cm/dem/kuznechik-ctr-hmac-kr b/spec/cm/dem/kuznechik-ctr-hmac-kr new file mode 100644 index 0000000..7ce03f2 --- /dev/null +++ b/spec/cm/dem/kuznechik-ctr-hmac-kr @@ -0,0 +1,26 @@ +Kuznechik-CTR-HMAC with key ratcheting DEM. +[cm/encrypted/]'s "/dem/a" equals to "kuznechik-ctr-hmac-kr". +CEK is 64 bytes long. +Data is split on 128 KiB chunks, each of which is encrypted the following way: + + H = Streebog-512 + CK0 = CEK + CKi = HKDF-Extract(H, salt="", ikm=CK{i-1}) + Kenc = HKDF-Expand(H, prk=CKi, info="cm/encrypted/kuznechik-ctr-hmac-kr/enc") + IV = HKDF-Expand(H, prk=CKi, len=8, info="cm/encrypted/kuznechik-ctr-hmac-kr/iv") + Kauth || KauthTail = HKDF-Expand(H, prk=CKi, + info="cm/encrypted/kuznechik-ctr-hmac-kr/auth") + KauthTail = HKDF-Expand(H, prk=CKi, + info="cm/encrypted/kuznechik-ctr-hmac-kr/authTail") + CT = Kuznechik-CTR(key=Kenc, ctr=IV, data=chunk) + CT || HMAC(Streebog-256, key={Kauth|KauthTail}, data=CT) + +KauthTail is used only in the last chunk to explicitly signal +that it is the last one. + +"/payload"'s chunk length equals to 128KiB+32 bytes. + +=> https://datatracker.ietf.org/doc/html/rfc5869.html RFC 5869, HKDF +=> https://datatracker.ietf.org/doc/html/rfc2104.html RFC 2104, HMAC +=> https://datatracker.ietf.org/doc/html/rfc7801.html RFC 7801, Kuznechik, GOST R 34.12-2015 encryption algorithm +=> https://datatracker.ietf.org/doc/html/rfc6986.html RFC 6986, Streebog, GOST R 34.11-2012 hashing algorithm diff --git a/spec/cm/dem/xchapoly-krkc b/spec/cm/dem/xchapoly-krkc new file mode 100644 index 0000000..f883a9e --- /dev/null +++ b/spec/cm/dem/xchapoly-krkc @@ -0,0 +1,27 @@ +XChaCha20-Poly1305 with key ratcheting and key commitment DEM. +[cm/encrypted/]'s "/dem/a" equals to "xchapoly-krkc". +CEK is 64 bytes long. +Data is split on 128 KiB chunks, each of which is encrypted the following way: + + H = BLAKE2b + CK0 = CEK + CKi = HKDF-Extract(H, salt="", ikm=CK{i-1}) + KEY = HKDF-Expand(H, prk=CKi, info="cm/encrypted/xchapoly-krkc/key") + IV = HKDF-Expand(H, prk=CKi, info="cm/encrypted/xchapoly-krkc/iv", len=24) + if last chunk { IV[23] |= 0x01 } else { IV[23] &= 0xFE } + CIPHERTEXT || TAG = XChaCha20-Poly1305(key=KEY, ad="", nonce=IV, data=chunk) + COMMITMENT = BLAKE2b-256(KEY || IV || TAG) + CIPHERTEXT || TAG || COMMITMENT + +Chaining key (CK) advances with every chunk. 256-bit encryption key and +randomised 192-bit nonce (initialisation vector) are derived from it. + +Nonce's lowest bit is set only if this is the last chunk we encrypting. + +"/payload"'s chunk length equals to 128KiB+16+32 bytes. + +=> https://datatracker.ietf.org/doc/html/rfc5869.html RFC 5869, HKDF +=> https://www.blake2.net/ BLAKE2b is hashing algorithm +=> https://datatracker.ietf.org/doc/html/rfc7693.html, RFC 7693, same +=> https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-xchacha, XChaCha20-Poly1305 AEAD +=> https://datatracker.ietf.org/doc/html/rfc8439.html, RFC 8439 diff --git a/spec/cm/dem/xchapoly-krmr b/spec/cm/dem/xchapoly-krmr new file mode 100644 index 0000000..e81d02c --- /dev/null +++ b/spec/cm/dem/xchapoly-krmr @@ -0,0 +1,32 @@ +XChaCha20-Poly1305 with key ratcheting and multi-recipient DEM. +[cm/encrypted/]'s "/dem/a" equals to "xchapoly-krmr". +CEK consists of common 64 bytes long part equal in all KEMs (@code{CEK} itself), +and 64 bytes long per-KEM/per-recipient random MAC key (@code{prMAC}). +Data is split on 128 KiB chunks, each of which is encrypted the following way: + + H = BLAKE2b + CK0, prMACx0 = CEK || prMACx + CKi = HKDF-Extract(H, salt="", ikm=CK{i-1}) + KEY = HKDF-Expand(H, prk=CKi, info="cm/encrypted/xchapoly-krmr/key") + IV = HKDF-Expand(H, prk=CKi, info="cm/encrypted/xchapoly-krmr/iv", len=24) + if last chunk { IV[23] |= 0x01 } else { IV[23] &= 0xFE } + CIPHERTEXT || TAG = XChaCha20-Poly1305(key=KEY, ad="", nonce=IV, data=chunk) + prMACxi = HKDF-Extract(H, salt="", ikm=prMACx{i-1}) + MACx = BLAKE2b-256-MAC(key=prMACxi, H(CIPHERTEXT || TAG)) + CIPHERTEXT || TAG || MACx || MAC{x+1} [|| MAC{x+2} ...] + +Chaining key (CK) and per-recipient MAC (prMAC) key advance with every +chunk. 256-bit encryption key and randomised 192-bit nonce +(initialisation vector) are derived from chaining key. + +Nonce's lowest bit is set only if this is the last chunk we encrypting. + +MACs are ordered the same way as KEMs in the list. + +"/payload"'s chunk length equals to 128KiB+16+32*recipients bytes. + +=> https://datatracker.ietf.org/doc/html/rfc5869.html RFC 5869, HKDF +=> https://www.blake2.net/ BLAKE2b is hashing algorithm +=> https://datatracker.ietf.org/doc/html/rfc7693.html, RFC 7693, same +=> https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-xchacha, XChaCha20-Poly1305 AEAD +=> https://datatracker.ietf.org/doc/html/rfc8439.html, RFC 8439 diff --git a/spec/cm/encrypted.texi b/spec/cm/encrypted.texi deleted file mode 100644 index 465cc76..0000000 --- a/spec/cm/encrypted.texi +++ /dev/null @@ -1,79 +0,0 @@ -@node cm-encrypted -@nodedescription Encrypted messages -@cindex cm/encrypted -@section cm/encrypted format - -Encrypted container, some kind of analogue to ASN.1-based -@url{https://datatracker.ietf.org/doc/html/rfc5652, CMS} EnvelopedData, -@url{https://librepgp.org/, LibrePGP} or -@url{https://age-encryption.org/, age}. - -@itemize -@item Ability to use multiple recipients -@item Either passphrase- or public-key based KEMs -@item Hybrid PQ/T KEMs -@item Ability to authenticate the sender -@item Optionally anonymous recipients and sender -@item Streaming friendly -@item Ability to parallelise encryption/decryption procedures -@item Current DEM schemes do explicit key commitment -@item Current DEM schemes use key ratcheting and rotation -@item Ability to safely encrypt to multiple recipients -@end itemize - -Stored in a file, it should begin with "cm/encrypted" @ref{MAGIC, magic}. - -@verbatiminclude ../tcl/schemas/encrypted.tcl - -@code{/payload} contains the ciphertext. It is encrypted with random -"content encryption key" (CEK) with an algorithm specified in -@code{/dem/a} (data encapsulation mechanism). @code{/dem} may contain -additional fields supplementing the decryption process, like -initialisation vector. - -If @code{/payload} is absent, then ciphertext is provided by other -means, for example just by following the @code{cm/encrypted} structure. -It is recommended to encode it as a BLOB, which chunk's length depends -on DEM algorithm. - -CEK is encapsulated in @code{/kem/*/cek} entries (key encapsulation -mechanisms), using @code{/kem/*/a} algorithm. - -If KEM uses public-key based cryptography, then recipient's @ref{cm-pub, -public key}(s) must be provided. Optional @code{/kem/*/to}, public key's -fingerprint, may provide a hint to quickly search for the key on the -recipient's side. - -Optional @code{/id} is used in KEMs for domain separation and envelope -identification. UUIDv4 is recommended. If absent, then null UUID is used -in KDF. - -@include cm/authcrypt.texi - -@node Key wrapping -@cindex key wrapping -@nodedescription Key wrapping mechanisms -@subsection Key wrapping mechanisms - -@include cm/keywrap-xchapoly.texi -@include cm/keywrap-kexp15.texi - -@node DEM -@cindex DEM -@nodedescription Data encapsulation mechanisms -@subsection Data encapsulation mechanisms - -@include cm/dem-xchapoly-krkc.texi -@include cm/dem-xchapoly-krmr.texi -@include cm/dem-kuznechik-ctr-hmac-kr.texi - -@node KEM -@cindex KEM -@nodedescription Key encapsulation mechanisms -@subsection Key encapsulation mechanisms - -@include cm/kem-balloon-blake2b-hkdf.texi -@include cm/kem-gost3410-hkdf.texi -@include cm/kem-sntrup4591761-x25519-hkdf-blake2b.texi -@include cm/kem-mceliece6960119-x25519-hkdf-shake256.texi -@include cm/kem-pbkdf2.texi diff --git a/spec/cm/encrypted/authcrypt b/spec/cm/encrypted/authcrypt new file mode 100644 index 0000000..309a501 --- /dev/null +++ b/spec/cm/encrypted/authcrypt @@ -0,0 +1,15 @@ +Public-key based [cm/kem/]s provides sender authentication +*only* if "/kem/*/from" field is specified. It should contain public +key's "/load/v/id", but may be equal to 256-bit zeros, to explicitly +specify that sender's public key is used, but it is anonymous and +hidden. It is not specified how recipient should find corresponding +sender's key that way -- implementation/protocol specific. + +Optional "/pubs" is a list public keys, which may be used to supply +sender's public key(s). Public keys may be encrypted, to hide the actual +deanonymisation contents. + +It is *highly* recommended to use multi-recipient safe DEM when +encrypting to multiple recipients. For example [cm/dem/xchapoly-krmr] +instead of [cm/dem/xchapoly-krkc], but unfortunately with the price of +more expensive double pass authentication scheme. diff --git a/spec/cm/encrypted/index b/spec/cm/encrypted/index new file mode 100644 index 0000000..7a4c148 --- /dev/null +++ b/spec/cm/encrypted/index @@ -0,0 +1,45 @@ +Encrypted container, some kind of analogue to ASN.1-based CMS +EnvelopedData, LibrePGP or age. +=> https://datatracker.ietf.org/doc/html/rfc5652 CMS +=> https://librepgp.org/ LibrePGP +=> https://age-encryption.org/ age + +* Ability to use multiple recipients +* Either passphrase- or public-key based KEMs +* Hybrid PQ/T KEMs +* Ability to authenticate the sender +* Optionally anonymous recipients and sender +* Streaming friendly +* Ability to parallelise encryption/decryption procedures +* Current DEM schemes do explicit key commitment +* Current DEM schemes use key ratcheting and rotation +* Ability to safely encrypt to multiple recipients + +Stored in a file, it should begin with "cm/encrypted" [encoding/MAGIC]. +[schemas/encrypted.tcl] + +"/payload" contains the ciphertext. It is encrypted with random "content +encryption key" (CEK) with an algorithm specified in "/dem/a" (data +encapsulation mechanism). "/dem" may contain additional fields +supplementing the decryption process, like initialisation vector. + +If "/payload" is absent, then ciphertext is provided by other means, for +example just by following the "cm/encrypted" structure. It is recommended +to encode it as a BLOB, which chunk's length depends on DEM algorithm. + +CEK is encapsulated in "/kem/*/cek" entries (key encapsulation +mechanisms), using "/kem/*/a" algorithm. + +If KEM uses public-key based cryptography, then recipient's +[cm/pub/]lic key(s) must be provided. Optional "/kem/*/to", public key's +fingerprint, may provide a hint to quickly search for the key on the +recipient's side. + +Optional "/id" is used in KEMs for domain separation and envelope +identification. UUIDv4 is recommended. +If absent, then null UUID is used in KDF. + +[cm/encrypted/authcrypt] -- authenticated public-key encryption +[cm/keywrap/] -- key wrapping mechanisms +[cm/dem/] -- data encapsulation mechanisms +[cm/kem/] -- key encapsulation mechanisms diff --git a/spec/cm/hashed.texi b/spec/cm/hashed.texi deleted file mode 100644 index a17d286..0000000 --- a/spec/cm/hashed.texi +++ /dev/null @@ -1,123 +0,0 @@ -@node cm-hashed -@nodedescription Hashed messages -@cindex cm/hashed -@section cm/hashed format - -Integrity protected container, analogue to ASN.1-based -@url{https://datatracker.ietf.org/doc/html/rfc5652, CMS} DigestedData. - -Stored in a file, it should begin with "cm/hashed" @ref{MAGIC, magic}. - -@verbatiminclude ../tcl/schemas/hashed.tcl - -@code{/a} tells what algorithms will be used to hash the data. - -@code{/t} tells the type of the data inside. - -@code{/hash} contains the hash values for all corresponding @code{/a} -algorithms. - -@node Merkle hashing -@cindex Merkle tree -@cindex Merkle-tree hashing -@nodedescription Merkle-tree based hashing -@subsection Merkle-tree based hashing - - Merkle trees are very convenient way to parallelise data hashing. - @url{https://datatracker.ietf.org/doc/html/rfc9162, RFC 9162} is used as - a base for all Merkle-tree based hashers. - - By default 128KiB chunks are used. - -@node cm-hashed-blake2b -@cindex cm-hashed-blake2b -@nodedescription cm/hashed with BLAKE2b -@subsection cm/hashed with BLAKE2b - - BLAKE2b with - 512-bit output has @code{blake2b} algorithm identifier. - - 256-bit output has @code{blake2b256} algorithm identifier. - - @url{https://www.blake2.net/, BLAKE2b} is hashing algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc7693.html, RFC 7693}. - -@node cm-hashed-blake2b-merkle -@cindex cm-hashed-blake2b-merkle -@nodedescription cm/hashed with BLAKE2b in Merkle-tree mode -@subsection cm/hashed with BLAKE2b in Merkle-tree mode - - BLAKE2b-512 is initialised with either "LEAF" or "NODE" keys, - instead of using @code{0x00}/@code{0x01} constants prepending to the - data. Although BLAKE2 has ability to set tree-hashing parameters on - its own, many implementations do not provide necessary API for that. - - @code{blake2b-merkle} algorithm identifier is used. - -@node cm-hashed-blake3 -@cindex cm-hashed-blake3 -@nodedescription cm/hashed with BLAKE3 -@subsection cm/hashed with BLAKE3 - - @url{https://github.com/BLAKE3-team/BLAKE3/, BLAKE3} with fixed - 256-bit output has @code{blake3} algorithm identifier. - -@node cm-hashed-shake -@cindex cm-hashed-shake -@nodedescription cm/hashed with SHAKE -@subsection cm/hashed with SHAKE - - @url{https://keccak.team/, SHAKE} XOF function with fixed - 256 (SHAKE128) or 512 (SHAKE256) bit output. - - @code{shake128}, @code{shake256} algorithm identifiers are used. - -@node cm-hashed-shake-merkle -@cindex cm-hashed-shake-merkle -@nodedescription cm/hashed with SHAKE in Merkle-tree mode -@subsection cm/hashed with SHAKE in Merkle-tree mode - - cSHAKE with either "LEAF" or "NODE" personalisation strings are used - instead of @code{0x00}/@code{0x01} constants prepending to the - hashed data. - - @code{shake128-merkle}, @code{shake256-merkle} algorithm identifiers - are used. - -@node cm-hashed-skein512 -@cindex cm-hashed-skein512 -@nodedescription cm/hashed with Skein-512 -@subsection cm/hashed with Skein-512 - - 512-bit @url{https://www.schneier.com/academic/skein/, Skein-512} hash. - - @code{skein512} algorithm identifier is used. - -@node cm-hashed-gost3411 -@cindex cm-hashed-gost3411 -@nodedescription cm/hashed with GOST R 34.11-2012 -@subsection cm/hashed with GOST R 34.11-2012 - - Streebog must be big-endian serialised. - - @code{streebog256}, @code{streebog512} algorithm identifiers are used. - - Streebog is GOST R 34.11-2012 hashing algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc6986.html, RFC 6986}. - -@node cm-hashed-gost3411-merkle -@cindex cm-hashed-gost3411-merkle -@nodedescription cm/hashed with GOST R 34.11-2012 in Merkle tree mode -@subsection cm/hashed with GOST R 34.11-2012 in Merkle tree mode - - @code{streebog256-merkle}, @code{streebog512-merkle} algorithm - identifiers are used. - -@node cm-hashed-xxh3-128 -@cindex cm-hashed-xxh3-128 -@nodedescription cm/hashed with XXH3-128 -@subsection cm/hashed with XXH3-128 - - 128-bit @url{https://xxhash.com/, XXH3} hash must be big-endian encoded. - - @code{xxh3-128} algorithm identifier is used. diff --git a/spec/cm/hashed/Merkle b/spec/cm/hashed/Merkle new file mode 100644 index 0000000..5cbbafe --- /dev/null +++ b/spec/cm/hashed/Merkle @@ -0,0 +1,4 @@ +Merkle trees are very convenient way to parallelise data hashing. +RFC 9162 is used as a base for all Merkle-tree based hashers. +=> https://datatracker.ietf.org/doc/html/rfc9162 RFC 9162 +By default 128KiB chunks are used. diff --git a/spec/cm/hashed/blake2b b/spec/cm/hashed/blake2b new file mode 100644 index 0000000..1c9db7f --- /dev/null +++ b/spec/cm/hashed/blake2b @@ -0,0 +1,5 @@ +[cm/hashed/] with BLAKE2b. +BLAKE2b with 512-bit output has "blake2b" algorithm identifier. +256-bit output has "blake2b256" algorithm identifier. +=> https://www.blake2.net/, BLAKE2b +=> https://datatracker.ietf.org/doc/html/rfc7693.html RFC 7693, same diff --git a/spec/cm/hashed/blake2b-merkle b/spec/cm/hashed/blake2b-merkle new file mode 100644 index 0000000..dcaee2e --- /dev/null +++ b/spec/cm/hashed/blake2b-merkle @@ -0,0 +1,8 @@ +[cm/hashed/] with BLAKE2b in [cm/hashed/Merkle]-tree mode. + +BLAKE2b-512 is initialised with either "LEAF" or "NODE" keys, +instead of using "0x00"/"0x01" constants prepending to the +data. Although BLAKE2 has ability to set tree-hashing parameters on +its own, many implementations do not provide necessary API for that. + +"blake2b-merkle" algorithm identifier is used. diff --git a/spec/cm/hashed/blake3 b/spec/cm/hashed/blake3 new file mode 100644 index 0000000..80b44bd --- /dev/null +++ b/spec/cm/hashed/blake3 @@ -0,0 +1,3 @@ +[cm/hashed/] with BLAKE3. +BLAKE3 with fixed 256-bit output has "blake3" algorithm identifier. +=> https://github.com/BLAKE3-team/BLAKE3/ BLAKE3 diff --git a/spec/cm/hashed/index b/spec/cm/hashed/index new file mode 100644 index 0000000..109b436 --- /dev/null +++ b/spec/cm/hashed/index @@ -0,0 +1,10 @@ +Integrity protected container, analogue to ASN.1-based CMS DigestedData. +=> https://datatracker.ietf.org/doc/html/rfc5652 CMS + +Stored in a file, it should begin with "cm/hashed" [encoding/MAGIC]. +[schemas/hashed.tcl] +"/a" tells what algorithms will be used to hash the data. +"/t" tells the type of the data inside. +"/hash" contains the hash values for all corresponding "/a" algorithms. + +do-backs diff --git a/spec/cm/hashed/shake b/spec/cm/hashed/shake new file mode 100644 index 0000000..f82f1c0 --- /dev/null +++ b/spec/cm/hashed/shake @@ -0,0 +1,4 @@ +[cm/hashed/] with SHAKE. +SHAKE XOF function with fixed 256 (SHAKE128) or 512 (SHAKE256) bit output. +"shake128", "shake256" algorithm identifiers are used. +=> https://keccak.team/ SHAKE diff --git a/spec/cm/hashed/shake-merkle b/spec/cm/hashed/shake-merkle new file mode 100644 index 0000000..86d5f4f --- /dev/null +++ b/spec/cm/hashed/shake-merkle @@ -0,0 +1,4 @@ +[cm/hashed/] with SHAKE in [cm/hashed/Merkle]-tree mode. +cSHAKE with either "LEAF" or "NODE" personalisation strings are used +instead of "0x00"/"0x01" constants prepending to the hashed data. +"shake128-merkle", "shake256-merkle" algorithm identifiers are used. diff --git a/spec/cm/hashed/skein512 b/spec/cm/hashed/skein512 new file mode 100644 index 0000000..feaf9db --- /dev/null +++ b/spec/cm/hashed/skein512 @@ -0,0 +1,3 @@ +[cm/hashed/] with Skein-512. +"skein512" algorithm identifier is used. +=> https://www.schneier.com/academic/skein/ Skein-512 diff --git a/spec/cm/hashed/streebog b/spec/cm/hashed/streebog new file mode 100644 index 0000000..5fb3a05 --- /dev/null +++ b/spec/cm/hashed/streebog @@ -0,0 +1,4 @@ +[cm/hashed/] with GOST R 34.11-2012. +Streebog must be big-endian serialised. +"streebog256", "streebog512" algorithm identifiers are used. +=> https://datatracker.ietf.org/doc/html/rfc6986.html RFC 6986 diff --git a/spec/cm/hashed/streebog-merkle b/spec/cm/hashed/streebog-merkle new file mode 100644 index 0000000..7ca9283 --- /dev/null +++ b/spec/cm/hashed/streebog-merkle @@ -0,0 +1,2 @@ +[cm/hashed/] with GOST R 34.11-2012 in [cm/hashed/Merkle]-tree mode. +"streebog256-merkle", "streebog512-merkle" algorithm identifiers are used. diff --git a/spec/cm/hashed/xxh3-128 b/spec/cm/hashed/xxh3-128 new file mode 100644 index 0000000..1ebce9e --- /dev/null +++ b/spec/cm/hashed/xxh3-128 @@ -0,0 +1,4 @@ +[cm/hashed/] with XXH3-128. +128-bit XXH3 hash must be big-endian encoded. +"xxh3-128" algorithm identifier is used. +=> https://xxhash.com/ XXH3 diff --git a/spec/cm/index b/spec/cm/index new file mode 100644 index 0000000..fb91010 --- /dev/null +++ b/spec/cm/index @@ -0,0 +1,10 @@ +Cryptographic messages + +Here are some suggested formats for use with cryptographic messages. +They are written in [schema/tcl] format. + +Private keys: [cm/prv/] +Signed data: [cm/signed/] +Public keys: [cm/pub/] +Hashed data: [cm/hashed/] +Encrypted data: [cm/encrypted/] diff --git a/spec/cm/index.texi b/spec/cm/index.texi deleted file mode 100644 index 424099a..0000000 --- a/spec/cm/index.texi +++ /dev/null @@ -1,14 +0,0 @@ -@node CM -@cindex CM -@cindex cryptographic messages -@nodedescription Cryptographic messages -@unnumbered Cryptographic messages - -Here are some suggested formats for use with cryptographic messages. -They are written in @ref{SchemaTcl, KEKS/Schema} format. - -@include cm/prv.texi -@include cm/signed.texi -@include cm/pub.texi -@include cm/hashed.texi -@include cm/encrypted.texi diff --git a/spec/cm/kem-balloon-blake2b-hkdf.texi b/spec/cm/kem-balloon-blake2b-hkdf.texi deleted file mode 100644 index c6e8c50..0000000 --- a/spec/cm/kem-balloon-blake2b-hkdf.texi +++ /dev/null @@ -1,23 +0,0 @@ -@node kem-balloon-blake2b-hkdf -@cindex kem-balloon-blake2b-hkdf -@nodedescription Balloon-BLAKE2b+HKDF KEM -@subsubsection Balloon-BLAKE2b+HKDF KEM - -@verbatiminclude ../tcl/schemas/kem-balloon-blake2b-hkdf.tcl - -@url{https://crypto.stanford.edu/balloon/, Balloon} memory-hardened -password hasher must be used with BLAKE2b hash. - -@verbatim -H = BLAKE2b -KEK = HKDF-Expand(H, - prk=balloon(H, passphrase, /kem/salt, s, t, p), - info="cm/encrypted/balloon-blake2b-hkdf" || /id) -@end verbatim - -@code{/kem/*/cek} is wrapped with @ref{keywrap-xchapoly} mechanism. - -HKDF is KDF algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc5869.html, RFC 5869}. -@url{https://www.blake2.net/, BLAKE2b} is hashing algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc7693.html, RFC 7693}. diff --git a/spec/cm/kem-gost3410-hkdf.texi b/spec/cm/kem-gost3410-hkdf.texi deleted file mode 100644 index d8cbaa3..0000000 --- a/spec/cm/kem-gost3410-hkdf.texi +++ /dev/null @@ -1,28 +0,0 @@ -@node kem-gost3410-hkdf -@cindex kem-gost3410-hkdf -@nodedescription GOST R 34.10+HKDF KEM -@subsubsection GOST R 34.10+HKDF KEM - -@verbatiminclude ../tcl/schemas/kem-gost3410-hkdf.tcl - -GOST R 34.10-2012 VKO parameter set A/C ("gost3410-256A", "gost3410-512C") -must be used for DH operation, with UKM taken from the structure. VKO's -output is 512- or 1024-bit @code{BE(X)||BE(Y)} point, used in HKDF below: - -@verbatim -H = Streebog-512 -DH(sk, pk) = GOSTR3410-VKO(prv=sk, pub=pk, ukm=UKM) -PRK = HKDF-Extract(H, salt="", ikm=DH(e, s)) -if specified(sender): - PRK = HKDF-Extract(H, salt=PRK, ikm=DH(s, s)) -KEK = HKDF-Expand(H, prk=PRK, info="cm/encrypted/gost3410-hkdf" || /id) -@end verbatim - -@code{/kem/*/cek} is wrapped with @ref{keywrap-kexp15} mechanism. - -HKDF is KDF algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc5869.html, RFC 5869}. -Streebog is GOST R 34.11-2012 hashing algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc6986.html, RFC 6986}. -GOST R 34.10-2012 is signing/key-aggreement algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc7091.html, RFC 7091}. diff --git a/spec/cm/kem-mceliece6960119-x25519-hkdf-shake256.texi b/spec/cm/kem-mceliece6960119-x25519-hkdf-shake256.texi deleted file mode 100644 index c57c860..0000000 --- a/spec/cm/kem-mceliece6960119-x25519-hkdf-shake256.texi +++ /dev/null @@ -1,46 +0,0 @@ -@node kem-mceliece6960119-x25519-hkdf-shake256 -@cindex kem-mceliece6960119-x25519-hkdf-shake256 -@nodedescription Classic McEliece 6960-119+X25519+HKDF-SHAKE256 KEM -@subsubsection Classic McEliece 6960-119+X25519+HKDF-SHAKE256 KEM - -@verbatiminclude ../tcl/schemas/kem-with-encap.tcl - -@code{/kem/*/a} equals to "mceliece6960119-x25519-hkdf-shake256". -Recipient public key with -@ref{cm-pub-mceliece6960119-x25519, @code{mceliece6960119-x25519}} -algorithm must be used. It should have "kem" key usage set. - -Recipient's map @code{/kem/*/encap} field is a concatenation of -194 bytes of Classic McEliece 6960-119 ciphertext, containing -ephemeral key, with 32 bytes of ephemeral X25519 public key. - -Recipient performs X25519 and Classic McEliece computations to -derive/decapsulate two 32-byte shared keys. Then it combines -them to get the KEK decryption key of the CEK. - -@verbatim -H = SHAKE256 -PRK = HKDF-Extract(H, salt="", ikm= - mceliece6960119-shared-key || es-x25519-shared-key || - H(mceliece6960119-sender-ciphertext || e-x25519-sender-public-key) || - H(mceliece6960119-recipient-public-key || s-x25519-recipient-public-key)) -if specified(sender): - PRK = HKDF-Extract(H, salt=PRK, ikm= - ss-x25519-shared-key || - s-x25519-sender-public-key || - s-x25519-recipient-public-key) -KEK = HKDF-Expand(H, prk=PRK, - info="cm/encrypted/mceliece6960119-x25519-hkdf-shake256" || /salt) -@end verbatim - -@code{/kem/*/cek} is wrapped with @ref{keywrap-xchapoly} mechanism. - -HKDF is KDF algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc5869.html, RFC 5869}. -@url{https://keccak.team/, SHAKE} is a XOF function. -KEM combiner nearly fully resembles -@url{https://datatracker.ietf.org/doc/draft-josefsson-chempat/, Chempat}. - -If sender/recipient's public key structure contains -@code{/load/v/prehash} field, then it could be used as already -calculated values of SHAKE256 calls of PRK. diff --git a/spec/cm/kem-pbkdf2.texi b/spec/cm/kem-pbkdf2.texi deleted file mode 100644 index 5ec6d5e..0000000 --- a/spec/cm/kem-pbkdf2.texi +++ /dev/null @@ -1,13 +0,0 @@ -@node kem-pbkdf2 -@cindex kem-pbkdf2 -@nodedescription PBKDF2 KEM -@subsubsection PBKDF2 KEM - -@verbatiminclude ../tcl/schemas/kem-pbkdf2.tcl - -PBKDF2 is @url{https://datatracker.ietf.org/doc/html/rfc2898, RFC 2898} -algorithm. Key length equal to key wrapping algorithm requirements. - -Key wrapping algorithm may be one of: -@ref{keywrap-xchapoly, @code{xchapoly}}, -@ref{keywrap-kexp15, @code{kexp15}}, depending on the hash chosen. diff --git a/spec/cm/kem-sntrup4591761-x25519-hkdf-blake2b.texi b/spec/cm/kem-sntrup4591761-x25519-hkdf-blake2b.texi deleted file mode 100644 index b5434d6..0000000 --- a/spec/cm/kem-sntrup4591761-x25519-hkdf-blake2b.texi +++ /dev/null @@ -1,38 +0,0 @@ -@node kem-sntrup4591761-x25519-hkdf-blake2b -@cindex kem-sntrup4591761-x25519-hkdf-blake2b -@nodedescription SNTRUP4591761+X25519+HKDF-BLAKE2b KEM -@subsubsection SNTRUP4591761+X25519+HKDF-BLAKE2b KEM - -@verbatiminclude ../tcl/schemas/kem-with-encap.tcl - -@code{/kem/*/a} equals to "sntrup4591761-x25519-hkdf-blake2b". -Recipient public key with @ref{cm-pub-sntrup4591761-x25519, -@code{sntrup4591761-x25519}} algorithm must be used. It should have -"kem" key usage set. - -Recipient's map @code{/kem/*/encap} field is a concatenation of 1047 -bytes of Streamlined NTRU Prime 4591^761's ciphertext, containing -ephemeral key, with 32 bytes of ephemeral X25519 public key. - -Recipient performs X25519 and SNTRUP computations to derive/decapsulate -two 32-byte shared keys. Then it combines them to get the KEK decryption -key of the CEK. - -@verbatim -H = BLAKE2b -PRK = HKDF-Extract(H, salt="", ikm= - sntrup4591761-shared-key || es-x25519-shared-key || - H(sntrup4591761-sender-ciphertext || e-x25519-sender-public-key) || - H(sntrup4591761-recipient-public-key || s-x25519-recipient-public-key)) -if specified(sender): - PRK = HKDF-Extract(H, salt=PRK, ikm= - ss-x25519-shared-key || - s-x25519-sender-public-key || - s-x25519-recipient-public-key) -KEK = HKDF-Expand(H, prk=PRK, info="cm/encrypted/sntrup4591761-x25519-hkdf-blake2b" || /id) -@end verbatim - -@code{/kem/*/cek} is wrapped with @ref{keywrap-xchapoly} mechanism. - -KEM combiner nearly fully resembles -@url{https://datatracker.ietf.org/doc/draft-josefsson-chempat/, Chempat}. diff --git a/spec/cm/kem/balloon-blake2b-hkdf b/spec/cm/kem/balloon-blake2b-hkdf new file mode 100644 index 0000000..eb565fd --- /dev/null +++ b/spec/cm/kem/balloon-blake2b-hkdf @@ -0,0 +1,15 @@ +Balloon-BLAKE2b+HKDF KEM. +[schemas/kem-balloon-blake2b-hkdf.tcl] +Balloon memory-hardened password hasher must be used with BLAKE2b hash. +=> https://crypto.stanford.edu/balloon/ Balloon + + H = BLAKE2b + KEK = HKDF-Expand(H, + prk=balloon(H, passphrase, /kem/salt, s, t, p), + info="cm/encrypted/balloon-blake2b-hkdf" || /id) + +"/kem/*/cek" is wrapped with [cm/keywrap/xchapoly] mechanism. + +=> https://datatracker.ietf.org/doc/html/rfc5869.html RFC 5869, HKDF +=> https://www.blake2.net/ BLAKE2b is hashing algorithm +=> https://datatracker.ietf.org/doc/html/rfc7693.html, RFC 7693, same diff --git a/spec/cm/kem/gost3410-hkdf b/spec/cm/kem/gost3410-hkdf new file mode 100644 index 0000000..92923dd --- /dev/null +++ b/spec/cm/kem/gost3410-hkdf @@ -0,0 +1,18 @@ +GOST R 34.10+HKDF KEM. +[schemas/kem-gost3410-hkdf.tcl] +GOST R 34.10-2012 VKO parameter set A/C ("gost3410-256A", "gost3410-512C") +must be used for DH operation, with UKM taken from the structure. VKO's +output is 512- or 1024-bit "BE(X)||BE(Y)" point, used in HKDF below: + + H = Streebog-512 + DH(sk, pk) = GOSTR3410-VKO(prv=sk, pub=pk, ukm=UKM) + PRK = HKDF-Extract(H, salt="", ikm=DH(e, s)) + if specified(sender): + PRK = HKDF-Extract(H, salt=PRK, ikm=DH(s, s)) + KEK = HKDF-Expand(H, prk=PRK, info="cm/encrypted/gost3410-hkdf" || /id) + +"/kem/*/cek" is wrapped with [cm/keywrap/kexp15] mechanism. + +=> https://datatracker.ietf.org/doc/html/rfc5869.html RFC 5869, HKDF +=> https://datatracker.ietf.org/doc/html/rfc6986.html RFC 6986, Streebog, GOST R 34.11-2012 hashing algorithm +=> https://datatracker.ietf.org/doc/html/rfc7091.html RFC 7091, GOST R 34.10-2012 is signing/key-aggreement algorithm diff --git a/spec/cm/kem/mceliece6960119-x25519-hkdf-shake256 b/spec/cm/kem/mceliece6960119-x25519-hkdf-shake256 new file mode 100644 index 0000000..9c260f1 --- /dev/null +++ b/spec/cm/kem/mceliece6960119-x25519-hkdf-shake256 @@ -0,0 +1,37 @@ +Classic McEliece 6960-119+X25519+HKDF-SHAKE256 KEM. +[schemas/kem-with-encap.tcl] +"/kem/*/a" equals to "mceliece6960119-x25519-hkdf-shake256". +Recipient public key with [cm/pub/mceliece6960119-x25519] +algorithm must be used. It should have "kem" key usage set. + +Recipient's map "/kem/*/encap" field is a concatenation of +194 bytes of Classic McEliece 6960-119 ciphertext, containing +ephemeral key, with 32 bytes of ephemeral X25519 public key. + +Recipient performs X25519 and Classic McEliece computations to +derive/decapsulate two 32-byte shared keys. Then it combines +them to get the KEK decryption key of the CEK. + + H = SHAKE256 + PRK = HKDF-Extract(H, salt="", ikm= + mceliece6960119-shared-key || es-x25519-shared-key || + H(mceliece6960119-sender-ciphertext || e-x25519-sender-public-key) || + H(mceliece6960119-recipient-public-key || s-x25519-recipient-public-key)) + if specified(sender): + PRK = HKDF-Extract(H, salt=PRK, ikm= + ss-x25519-shared-key || + s-x25519-sender-public-key || + s-x25519-recipient-public-key) + KEK = HKDF-Expand(H, prk=PRK, + info="cm/encrypted/mceliece6960119-x25519-hkdf-shake256" || /salt) + +"/kem/*/cek" is wrapped with [cm/keywrap/xchapoly] mechanism. + +=> https://datatracker.ietf.org/doc/html/rfc5869.html RFC 5869, HKDF +=> https://keccak.team/ SHAKE XOF function +KEM combiner nearly fully resembles: +=> https://datatracker.ietf.org/doc/draft-josefsson-chempat/ Chempat + +If sender/recipient's public key structure contains +"/load/v/prehash" field, then it could be used as already +calculated values of SHAKE256 calls of PRK. diff --git a/spec/cm/kem/pbkdf2 b/spec/cm/kem/pbkdf2 new file mode 100644 index 0000000..ad8d62e --- /dev/null +++ b/spec/cm/kem/pbkdf2 @@ -0,0 +1,9 @@ +PBKDF2 KEM. +[schemas/kem-pbkdf2.tcl] +PBKDF2 is RFC 2898 algorithm. +Key length equal to key wrapping algorithm requirements. +=> https://datatracker.ietf.org/doc/html/rfc2898 RFC 2898 + +Key wrapping algorithm may be one of: +[cm/keywrap/xchapoly], [cm/keywrap/kexp15], +depending on the hash chosen. diff --git a/spec/cm/kem/sntrup4591761-x25519-hkdf-blake2b b/spec/cm/kem/sntrup4591761-x25519-hkdf-blake2b new file mode 100644 index 0000000..e219c82 --- /dev/null +++ b/spec/cm/kem/sntrup4591761-x25519-hkdf-blake2b @@ -0,0 +1,31 @@ +SNTRUP4591761+X25519+HKDF-BLAKE2b KEM. +[schemas/kem-with-encap.tcl] +"/kem/*/a" equals to "sntrup4591761-x25519-hkdf-blake2b". +Recipient public key with [cm/pub/sntrup4591761-x25519] +algorithm must be used. It should have "kem" key usage set. + +Recipient's map "/kem/*/encap" field is a concatenation of 1047 +bytes of Streamlined NTRU Prime 4591^761's ciphertext, containing +ephemeral key, with 32 bytes of ephemeral X25519 public key. + +Recipient performs X25519 and SNTRUP computations to derive/decapsulate +two 32-byte shared keys. Then it combines them to get the KEK decryption +key of the CEK. + + H = BLAKE2b + PRK = HKDF-Extract(H, salt="", ikm= + sntrup4591761-shared-key || es-x25519-shared-key || + H(sntrup4591761-sender-ciphertext || e-x25519-sender-public-key) || + H(sntrup4591761-recipient-public-key || s-x25519-recipient-public-key)) + if specified(sender): + PRK = HKDF-Extract(H, salt=PRK, ikm= + ss-x25519-shared-key || + s-x25519-sender-public-key || + s-x25519-recipient-public-key) + KEK = HKDF-Expand(H, prk=PRK, + info="cm/encrypted/sntrup4591761-x25519-hkdf-blake2b" || /id) + +"/kem/*/cek" is wrapped with [cm/keywrap/xchapoly] mechanism. + +KEM combiner nearly fully resembles: +=> https://datatracker.ietf.org/doc/draft-josefsson-chempat/ Chempat diff --git a/spec/cm/keywrap-kexp15.texi b/spec/cm/keywrap-kexp15.texi deleted file mode 100644 index 6e27c98..0000000 --- a/spec/cm/keywrap-kexp15.texi +++ /dev/null @@ -1,17 +0,0 @@ -@node keywrap-kexp15 -@cindex keywrap-kexp15 -@nodedescription KExp15 key wrapping mechanism -@subsubsection KExp15 key wrapping mechanism - -KExp15 (Р 1323565.1.017) key wrapping mechanism uses GOST -cryptography algorithms. KEK is 32+8+32=72 bytes long. - -@verbatim -Kenc || IV || Kauth = KEK -KExp15(Kenc, Kauth, IV, CEK) = Kuznechik-CTR( - Kenc, CEK || Kuznechik-CMAC(Kauth, IV || CEK), IV=IV) -@end verbatim - -Kuznechik is GOST R 34.12-2015 encryption algorithm, -@url{https://datatracker.ietf.org/doc/html/rfc7801.html, RFC 7801}. -CMAC, OMAC1 is described in GOST R 34.13-2015. diff --git a/spec/cm/keywrap-xchapoly.texi b/spec/cm/keywrap-xchapoly.texi deleted file mode 100644 index 331740d..0000000 --- a/spec/cm/keywrap-xchapoly.texi +++ /dev/null @@ -1,13 +0,0 @@ -@node keywrap-xchapoly -@cindex keywrap-xchapoly -@nodedescription XChaCha20-Poly1305 key wrapping mechanism -@subsubsection XChaCha20-Poly1305 key wrapping mechanism - -Key is encrypted using XChaCha20-Poly1305 algorithm. -Random 192-bit nonce is prepended to the ciphertext. -KEK has 256-bit length. - -@verbatim -NONCE = random(24 bytes) -NONCE || XChaCha20-Poly1305(key=KEK, ad="", nonce=NONCE, data=CEK) -@end verbatim diff --git a/spec/cm/keywrap/kexp15 b/spec/cm/keywrap/kexp15 new file mode 100644 index 0000000..3f25302 --- /dev/null +++ b/spec/cm/keywrap/kexp15 @@ -0,0 +1,10 @@ +KExp15 (Р 1323565.1.017) key wrapping mechanism uses GOST +cryptography algorithms. KEK is 32+8+32=72 bytes long. + + Kenc || IV || Kauth = KEK + KExp15(Kenc, Kauth, IV, CEK) = Kuznechik-CTR( + Kenc, CEK || Kuznechik-CMAC(Kauth, IV || CEK), IV=IV) + +Kuznechik is GOST R 34.12-2015 encryption algorithm, RFC 7801. +=> https://datatracker.ietf.org/doc/html/rfc7801.html RFC 7801 +CMAC (OMAC1) is described in GOST R 34.13-2015. diff --git a/spec/cm/keywrap/xchapoly b/spec/cm/keywrap/xchapoly new file mode 100644 index 0000000..ca8f69a --- /dev/null +++ b/spec/cm/keywrap/xchapoly @@ -0,0 +1,7 @@ +XChaCha20-Poly1305 key wrapping mechanism. +Key is encrypted using XChaCha20-Poly1305 algorithm. +Random 192-bit nonce is prepended to the ciphertext. +KEK has 256-bit length. + + NONCE = random(24 bytes) + NONCE || XChaCha20-Poly1305(key=KEK, ad="", nonce=NONCE, data=CEK) diff --git a/spec/cm/prv.texi b/spec/cm/prv.texi deleted file mode 100644 index 4b25e17..0000000 --- a/spec/cm/prv.texi +++ /dev/null @@ -1,64 +0,0 @@ -@node cm-prv -@nodedescription Private keys -@cindex cm/prv -@section cm/prv format - -Private key container. - -@verbatiminclude ../tcl/schemas/av.tcl - -Stored in a file, it should begin with "cm/prv" @ref{MAGIC, magic}. - -@node cm-prv-gost3410 -@cindex cm-prv-gost3410 -@nodedescription cm/prv with GOST R 34.10-2012 -@subsection cm/prv with GOST R 34.10-2012 - - Big-endian private key representation must be used. - - Following algorithm identifiers are used: - @code{gost3410-256A}, @code{gost3410-512C}. - -@node cm-prv-ed25519-blake2b -@cindex cm-prv-ed25519-blake2b -@nodedescription cm/prv with Ed25519-BLAKE2b -@subsection cm/prv with Ed25519-BLAKE2b - - 32-byte Ed25519 private key is used, as described in - @url{https://datatracker.ietf.org/doc/html/rfc8032, EdDSA} RFC. - In many libraries it is called "seed". - - @code{ed25519-blake2b} algorithm identifier is used, however actually no - hash is involved in private key storage. - -@node cm-prv-sntrup4591761-x25519 -@cindex cm-prv-sntrup4591761-x25519 -@nodedescription cm/prv with SNTRUP4591761+X25519 -@subsection cm/prv with SNTRUP4591761+X25519 - - Concatenation of Streamlined NTRU Prime 4591^761's 1600-byte private key - and X25519's 32-byte one. - - @code{sntrup4591761-x25519} algorithm identifier is used. - -@node cm-prv-mceliece6960119-x25519 -@cindex cm-prv-mceliece6960119-x25519 -@nodedescription cm/prv with Classic McEliece 6960-119+X25519 -@subsection cm/prv with Classic McEliece 6960-119+X25519 - - Concatenation of Classic McEliece 6960-119 13948-byte private key - and X25519's 32-byte one. - - @code{mceliece6960119-x25519} algorithm identifier is used. - -@node cm-prv-sphincs+-shake-256f -@cindex cm-prv-sphincs+-shake-256f -@nodedescription cm/prv with SPHINCS+-SHAKE256-256f -@subsection cm/prv with SPHINCS+-SHAKE256-256f - - @url{https://sphincs.org/, SPHINCS+} with - @url{https://keccak.team/, SHAKE256} hash, - 255-bit security level, fast variant and simple parameters. - Value is concatenation of private and public keys (128+64 bytes). - - Algorithm identifier for the public key: @code{sphincs+-shake-256f}. diff --git a/spec/cm/prv/ed25519-blake2b b/spec/cm/prv/ed25519-blake2b new file mode 100644 index 0000000..e839d8c --- /dev/null +++ b/spec/cm/prv/ed25519-blake2b @@ -0,0 +1,6 @@ +[cm/prv/] with Ed25519-BLAKE2b. +32-byte Ed25519 private key is used, as described in EdDSA RFC. +In many libraries it is called "seed". +=> https://datatracker.ietf.org/doc/html/rfc8032 EdDSA +"ed25519-blake2b" algorithm identifier is used, however actually no +hash is involved in private key storage. diff --git a/spec/cm/prv/gost3410 b/spec/cm/prv/gost3410 new file mode 100644 index 0000000..5026c53 --- /dev/null +++ b/spec/cm/prv/gost3410 @@ -0,0 +1,6 @@ +[cm/prv/] with GOST R 34.10-2012. +GOST R 34.10-2012 is signing/key-aggreement algorithm. +=> https://datatracker.ietf.org/doc/html/rfc7091.html RFC 7091, GOST R 34.10-2012 +Big-endian private key representation must be used. +Only twisted Edwards curves are used. +Following algorithm identifiers are used: "gost3410-256A", "gost3410-512C". diff --git a/spec/cm/prv/index b/spec/cm/prv/index new file mode 100644 index 0000000..38c6714 --- /dev/null +++ b/spec/cm/prv/index @@ -0,0 +1,5 @@ +Private key container. +[schemas/av.tcl] +Stored in a file, it should begin with "cm/prv" [encoding/MAGIC]. + +do-backs diff --git a/spec/cm/prv/mceliece6960119-x25519 b/spec/cm/prv/mceliece6960119-x25519 new file mode 100644 index 0000000..947e55e --- /dev/null +++ b/spec/cm/prv/mceliece6960119-x25519 @@ -0,0 +1,4 @@ +[cm/prv/] with Classic McEliece 6960-119+X25519. +Concatenation of Classic McEliece 6960-119 13948-byte private key +and X25519's 32-byte one. +"mceliece6960119-x25519" algorithm identifier is used. diff --git a/spec/cm/prv/sntrup4591761-x25519 b/spec/cm/prv/sntrup4591761-x25519 new file mode 100644 index 0000000..34cfdd1 --- /dev/null +++ b/spec/cm/prv/sntrup4591761-x25519 @@ -0,0 +1,4 @@ +[cm/prv/] with SNTRUP4591761+X25519. +Concatenation of Streamlined NTRU Prime 4591^761's 1600-byte private key +and X25519's 32-byte one. +"sntrup4591761-x25519" algorithm identifier is used. diff --git a/spec/cm/prv/sphincs+-shake-256f b/spec/cm/prv/sphincs+-shake-256f new file mode 100644 index 0000000..2ead934 --- /dev/null +++ b/spec/cm/prv/sphincs+-shake-256f @@ -0,0 +1,6 @@ +[cm/prv/] with SPHINCS+-SHAKE256-256f. +255-bit security level, fast variant and simple parameters. +=> https://sphincs.org/ SPHINCS+ +=> https://keccak.team/ SHAKE256 +Value is concatenation of private and public keys (128+64 bytes). +Algorithm identifier for the public key: "sphincs+-shake-256f". diff --git a/spec/cm/pub.texi b/spec/cm/pub.texi deleted file mode 100644 index 7d5296f..0000000 --- a/spec/cm/pub.texi +++ /dev/null @@ -1,205 +0,0 @@ -@node cm-pub -@nodedescription Public keys -@cindex cm/pub -@section cm/pub format - -Public key is the @code{@ref{cm-signed, cm/signed}} structure. - -Stored in a file, it should begin with "cm/pub" @ref{MAGIC, magic}. - -Its @code{/load/t} equals to @code{pub}. -@code{/load/v} contains @code{cm-pub-load}: - -@verbatiminclude ../tcl/schemas/pub-load.tcl - -@table @code - -@item sub -Subject is a map of arbitrary strings. Currently no constraints on what -fields must be present. Each application and usage context defines it on -his own. But you may mimic X.509's subject with keys like "CN", "C", "O" -and similar ones. - -@item pub -Public key container itself may contain multiple public keys. - -That is @strong{solely} intended for tasks requiring more than single -key usage. For example @url{http://www.nncpgo.org, NNCP} uses one -X25519 for (DH) encryption, one curve25519 for online authentication -and one ed25519 for signing purposes. All those three keys are used -together. That public key's key usage field must contain something like -"nncp". - -If your keypair is intended for general purposes like signing of -arbitrary data, then single public key @strong{should} be used, with a -key usage like "sig". - -@item id - -Public key(s)'s fingerprint @strong{should} be generated as 256-bit hash -hash of the encoded @code{pub} field. If not stated otherwise for -specific algorithm. - -@item ku -Intended public key(s) usage. -Application-specific example with multiple public keys is described -above. It @strong{must} be absent if empty. - -@item crit -Optional critical (in terms of X.509) extensions. Non-critical -ones may be placed outside that map, directly in @code{cm-pub-load}. -It @strong{must} be absent if empty. Values are extension specific. - -@end table - -@code{cm/signed}'s @code{sig-tbs} @strong{must} contain additional fields: - -@verbatiminclude ../tcl/schemas/pub-sig-tbs.tcl - -@table @code - -@item sid -Signing public key's fingerprint. - -@item cid -Certification unique identifier. UUIDv7 is a good choice. But it may be -UUIDv4, or any desired method of generation. - -@item exp -Certification expiration period. It @strong{must} contain TAI64 -datetime (no nanoseconds). - -@end table - -Example minimal certified public key may look like: - -@verbatim -MAGIC cm/pub -MAP { - load {MAP { - t {STR pub} - v {MAP { - id {BIN "6aee..."} - pub {LIST { - {MAP { - a {STR ed25519-blake2b} - v {BIN "c1bf..."} - }} - }} - sub {MAP { - N {STR test} - }} - }} - }} - sigs {LIST { - {MAP { - tbs {MAP { - cid {HEXLET 01963308-1033-75a7-bfb6-7d3ab3db6d63} - exp {LIST { - {TAI64 "2025-04-14 06:41:28"} - {TAI64 "2026-04-14 06:41:28"} - }} - sid {BIN "0087..."} - }} - sign {MAP { - a {STR ed25519-blake2b} - v {BIN "7450..."} - }} - }} - }} -} -@end verbatim - -@node cm-pub-gost3410 -@cindex cm-pub-gost3410 -@nodedescription cm/pub with GOST R 34.10-2012 -@subsection cm/pub with GOST R 34.10-2012 - - GOST R 34.10-2012 must be used with Streebog (GOST R 34.11-2012) - hash function. Its digest must be big-endian serialised. Public key - must be in @code{BE(X)||BE(Y)} format. - - Algorithm identifiers for the public key: @code{gost3410-256A}, - @code{gost3410-512C}. - - Public key's fingerprint should be calculated using big-endian - Streebog-256 hash. - - Streebog is GOST R 34.11-2012 hashing algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc6986.html, RFC 6986}. - GOST R 34.10-2012 is signing/key-aggreement algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc7091.html, RFC 7091}. - -@node cm-pub-ed25519-blake2b -@cindex cm-pub-ed25519-blake2b -@nodedescription cm/pub with Ed25519-BLAKE2b -@subsection cm/pub with Ed25519-BLAKE2b - - Same calculation and serialisation rules must be used as with - @code{@ref{cm-signed-ed25519-blake2b}}. - - Public key's fingerprint should be calculated using BLAKE2b hash - with 256 bit output length specified. - - Algorithm identifier for the public key: @code{ed25519-blake2b}. - - @url{https://cr.yp.to/ecdh.html, X2559} is key-agreement algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc7748.html, RFC 7748}. - @url{https://www.blake2.net/, BLAKE2b} is hashing algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc7693.html, RFC 7693}. - -@node cm-pub-sntrup4591761-x25519 -@cindex cm-pub-sntrup4591761-x25519 -@nodedescription cm/pub with SNTRUP4591761+X25519 -@subsection cm/pub with SNTRUP4591761+X25519 - - Combined Streamlined NTRU Prime 4591^761 and X25519 public keys are - used for KEM purposes, so should have "kem" key usage set. - - Its algorithm identifier is @code{sntrup4591761-x25519}. Its public - key value is a concatenation of 1218-byte SNTRUP4591761 public key - and 32-byte X25519 one. - - Public key's fingerprint should be calculated using BLAKE2b hash - with 256 bit output length specified. - - @url{https://cr.yp.to/ecdh.html, X2559} is key-agreement algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc7748.html, RFC 7748}. - @url{https://ntruprime.cr.yp.to/, Streamlined NTRU Prime} is KEM algorithm. - -@node cm-pub-mceliece6960119-x25519 -@cindex cm-pub-mceliece6960119-x25519 -@nodedescription cm/pub with Classic McEliece 6960-119+X25519 -@subsection cm/pub with Classic McEliece 6960-119+X25519 - - Combined Classic McEliece 6960-119 and X25519 public keys are used - for KEM purposes, so should have "kem" key usage set. - - Its algorithm identifier is @code{mceliece6960119-x25519}. Its - public key value is a concatenation of 1047319-byte - @code{mceliece6960119} public key and 32-byte X25519 one. - - Public key's fingerprint should be calculated using SHAKE128. - - @url{https://cr.yp.to/ecdh.html, X2559} is key-agreement algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc7748.html, RFC 7748}. - @url{https://classic.mceliece.org/, Classic McEliece} is KEM algorithm. - @url{https://keccak.team/, SHAKE} is a XOF function. - - Optional @code{/load/v/prehash} field can contain the SHAKE256 hash - of the concatenated public keys in @code{/load/v/pub/0}, that could - save resources during @ref{kem-mceliece6960119-x25519-hkdf-shake256} - KDF calculations. - -@node cm-pub-sphincs+-shake-256f -@cindex cm-pub-sphincs+-shake-256f -@nodedescription cm/pub with SPHINCS+-SHAKE256-256f -@subsection cm/pub with SPHINCS+-SHAKE256-256f - - @url{https://sphincs.org/, SPHINCS+} with - @url{https://keccak.team/, SHAKE256} hash, - 255-bit security level, fast variant and simple parameters. - - @code{sphincs+-shake-256f} algorithm identifier is used. - - Public key's fingerprint should be calculated using SHAKE128. diff --git a/spec/cm/pub/ed25519-blake2b b/spec/cm/pub/ed25519-blake2b new file mode 100644 index 0000000..a4996f6 --- /dev/null +++ b/spec/cm/pub/ed25519-blake2b @@ -0,0 +1,14 @@ +[cm/pub/] with Ed25519-BLAKE2b. + +Same calculation and serialisation rules must be used as with +[cm/signed/ed25519-blake2b]. + +Public key's fingerprint should be calculated using BLAKE2b hash +with 256 bit output length specified. + +Algorithm identifier for the public key: "ed25519-blake2b". + +=> https://cr.yp.to/ecdh.html X2559, key-agreement algorithm +=> https://datatracker.ietf.org/doc/html/rfc7748.html RFC 7748, same +=> https://www.blake2.net/ BLAKE2b hashing algorithm +=> https://datatracker.ietf.org/doc/html/rfc7693.html RFC 7693, same diff --git a/spec/cm/pub/gost3410 b/spec/cm/pub/gost3410 new file mode 100644 index 0000000..da2aa94 --- /dev/null +++ b/spec/cm/pub/gost3410 @@ -0,0 +1,14 @@ +[cm/pub/] with GOST R 34.10-2012. + +GOST R 34.10-2012 must be used with Streebog (GOST R 34.11-2012) +hash function. Its digest must be big-endian serialised. Public key +must be in "BE(X)||BE(Y)" format. + +Algorithm identifiers for the public key: "gost3410-256A", "gost3410-512C". + +Public key's fingerprint should be calculated using big-endian Streebog-256 hash. + +Streebog is GOST R 34.11-2012 hashing algorithm. +GOST R 34.10-2012 is signing/key-aggreement algorithm. +=> https://datatracker.ietf.org/doc/html/rfc6986.html RFC 6986, GOST R 34.11-2012 +=> https://datatracker.ietf.org/doc/html/rfc7091.html RFC 7091, GOST R 34.10-2012 diff --git a/spec/cm/pub/index b/spec/cm/pub/index new file mode 100644 index 0000000..ec9bb51 --- /dev/null +++ b/spec/cm/pub/index @@ -0,0 +1,81 @@ +Public key is the [cm/signed/] structure. +Stored in a file, it should begin with "cm/pub" [encoding/MAGIC]. + +Its "/load/t" equals to "pub". "/load/v" contains "cm/pub/load": +[schemas/pub-load.tcl] + +sub: + Subject is a map of arbitrary strings. Currently no constraints on + what fields must be present. Each application and usage context + defines it on his own. But you may mimic X.509's subject with keys + like "CN", "C", "O" and similar ones. +pub: + Public key container itself may contain multiple public keys. + That is *solely* intended for tasks requiring more than single + key usage. For example NNCP uses one X25519 for (DH) encryption, one + curve25519 for online authentication and one ed25519 for signing + purposes. All those three keys are used together. That public key's + key usage field must contain something like "nncp". + => http://www.nncpgo.org NNCP + If your keypair is intended for general purposes like signing of + arbitrary data, then single public key @strong{should} be used, with + a key usage like "sig". +id: + Public key(s)'s fingerprint *should* be generated as 256-bit + hash hash of the encoded "pub" field. If not stated otherwise + for specific algorithm. +ku: + Intended public key(s) usage. + Application-specific example with multiple public keys is described + above. It *must* be absent if empty. +crit: + Optional critical (in terms of X.509) extensions. Non-critical + ones may be placed outside that map, directly in cm/pub/load. + It *must* be absent if empty. Values are extension specific. + +[cm/signed/]'s "sig-tbs" *must* contain additional fields: [schemas/pub-sig-tbs.tcl] + +sid: Signing public key's fingerprint. +cid: Certification unique identifier. UUIDv7 is a good choice. + But it may be UUIDv4, or any desired method of generation. +exp: Certification expiration period. + It *must* contain TAI64 datetime (no nanoseconds). + +Example minimal certified public key may look like: + + MAGIC cm/pub + MAP { + load {MAP { + t {STR pub} + v {MAP { + id {BIN "6aee..."} + pub {LIST { + {MAP { + a {STR ed25519-blake2b} + v {BIN "c1bf..."} + }} + }} + sub {MAP { + N {STR test} + }} + }} + }} + sigs {LIST { + {MAP { + tbs {MAP { + cid {HEXLET 01963308-1033-75a7-bfb6-7d3ab3db6d63} + exp {LIST { + {TAI64 "2025-04-14 06:41:28"} + {TAI64 "2026-04-14 06:41:28"} + }} + sid {BIN "0087..."} + }} + sign {MAP { + a {STR ed25519-blake2b} + v {BIN "7450..."} + }} + }} + }} + } + +do-backs diff --git a/spec/cm/pub/mceliece6960119-x25519 b/spec/cm/pub/mceliece6960119-x25519 new file mode 100644 index 0000000..94ca0f6 --- /dev/null +++ b/spec/cm/pub/mceliece6960119-x25519 @@ -0,0 +1,20 @@ +[cm/pub/] with Classic McEliece 6960-119+X25519. + +Combined Classic McEliece 6960-119 and X25519 public keys are used +for KEM purposes, so should have "kem" key usage set. + +Its algorithm identifier is "mceliece6960119-x25519". Its +public key value is a concatenation of 1047319-byte +mceliece6960119 public key and 32-byte X25519 one. + +Public key's fingerprint should be calculated using SHAKE128. + +=> https://cr.yp.to/ecdh.html, X2559 key-agreement algorithm +=> https://datatracker.ietf.org/doc/html/rfc7748.html RFC 7748, same +=> https://classic.mceliece.org/ Classic McEliece KEM algorithm +=> https://keccak.team/ SHAKE XOF function + +Optional "/load/v/prehash" field can contain the SHAKE256 hash +of the concatenated public keys in "/load/v/pub/0", that could +save resources during [cm/kem/mceliece6960119-x25519-hkdf-shake256] +KDF calculations. diff --git a/spec/cm/pub/sntrup4591761-x25519 b/spec/cm/pub/sntrup4591761-x25519 new file mode 100644 index 0000000..40247b2 --- /dev/null +++ b/spec/cm/pub/sntrup4591761-x25519 @@ -0,0 +1,15 @@ +[cm/pub/] with SNTRUP4591761+X25519. + +Combined Streamlined NTRU Prime 4591^761 and X25519 public keys are +used for KEM purposes, so should have "kem" key usage set. + +Its algorithm identifier is "sntrup4591761-x25519". Its public +key value is a concatenation of 1218-byte SNTRUP4591761 public key +and 32-byte X25519 one. + +Public key's fingerprint should be calculated using BLAKE2b hash +with 256 bit output length specified. + +=> https://cr.yp.to/ecdh.html X2559, key-agreement algorithm +=> https://datatracker.ietf.org/doc/html/rfc7748.html RFC 7748, same +=> https://ntruprime.cr.yp.to/ Streamlined NTRU Prime KEM algorithm diff --git a/spec/cm/pub/sphincs+-shake-256f b/spec/cm/pub/sphincs+-shake-256f new file mode 100644 index 0000000..9d05458 --- /dev/null +++ b/spec/cm/pub/sphincs+-shake-256f @@ -0,0 +1,6 @@ +[cm/pub/] with SPHINCS+-SHAKE256-256f. +255-bit security level, fast variant and simple parameters. +=> https://sphincs.org/ SPHINCS+ +=> https://keccak.team/ SHAKE256 +"sphincs+-shake-256f" algorithm identifier is used. +Public key's fingerprint should be calculated using SHAKE128. diff --git a/spec/cm/signed.texi b/spec/cm/signed.texi deleted file mode 100644 index da70afd..0000000 --- a/spec/cm/signed.texi +++ /dev/null @@ -1,142 +0,0 @@ -@node cm-signed -@nodedescription Signed messages -@cindex cm/signed -@section cm/signed format - -Signed container, some kind of analogue to ASN.1-based -@url{https://datatracker.ietf.org/doc/html/rfc5652, CMS} SignedData. - -@itemize -@item Ability to embed the data in the signed container -@item Ability to create detached signature -@item Ability to use non-prehashed signature of the embedded data, - potentially gaining more security -@item Prehashed format is streaming friendly -@item You can use @ref{Merkle hashing} mode to parallelise calculations -@item Ability to attach arbitrary additional data -@item Ability to store multiple signatures -@end itemize - -Stored in a file, it should begin with "cm/signed" @ref{MAGIC, magic}, -unless it is a @ref{cm-pub, public key}. - -@verbatiminclude ../tcl/schemas/av.tcl -@verbatiminclude ../tcl/schemas/fpr.tcl -@verbatiminclude ../tcl/schemas/signed.tcl - -Signature is created by signing the: - -@verbatim -[detached-data] || /load || sig-tbs -@end verbatim - -If no @code{/load/v} is provided, then the data is detached from the -@code{cm/signed} structure itself and it is fed into hasher before that -structure. You can provide it any way you wish, but for keeping that -detached data closely to the @code{cm/signed}, you should use the -following approach: - -@verbatim -cm-signed-prehash || BLOB(detached-data) || cm-signed -@end verbatim - -@verbatiminclude ../tcl/schemas/signed-prehash.tcl - -With @code{cm-signed-prehash} you initialise your hashers used during -signing process and feed BLOB's contents (not the encoded BLOB itself!) -into the them. - -@code{/sigs/*/tbs/when} is optional signing time. - -Additional values that must be protected (covered by signature) are -placed in @code{/sigs/*/tbs} map. Non-protected (informational) fields -are placed outside it. - -@code{/pubs} are optionally provided @ref{cm-pub, public keys} to -help creating the whole verification chain. They are placed outside -@code{/sigs}, because some of them may be shared among signers. - -If signed data is also intended to be @ref{cm-encrypted, encrypted}, -then @code{/sigs/*/tbs/encrypted-to} should be set to corresponding -recipient's public key fingerprint(s). - -@node cm-signed-gost3410 -@cindex cm-signed-gost3410 -@nodedescription cm/signed with GOST R 34.10-2012 -@subsection cm/signed with GOST R 34.10-2012 - - GOST R 34.10-2012 must be used with Streebog (GOST R 34.11-2012) - hash function. Its digest must be big-endian serialised. Signature - is in @code{BE(R)||BE(S)} format. - - Algorithm identifiers for the signature: @code{gost3410-256A}, - @code{gost3410-512C}. - - Streebog is GOST R 34.11-2012 hashing algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc6986.html, RFC 6986}. - GOST R 34.10-2012 is signing/key-aggreement algorithm, - @url{https://datatracker.ietf.org/doc/html/rfc7091.html, RFC 7091}. - -@node cm-signed-gost3410-merkle -@cindex cm-signed-gost3410-merkle -@nodedescription cm/signed with GOST R 34.10-2012 with Merkle-tree hashing -@subsection cm/signed with GOST R 34.10-2012 with Merkle-tree hashing - - @ref{cm-hashed-gost3411-merkle} Merkle-tree hashing is used. - - Algorithm identifiers for the signature: @code{gost3410-256A-merkle}, - @code{gost3410-512C-merkle}. - -@node cm-signed-ed25519-blake2b -@cindex cm-signed-ed25519-blake2b -@cindex cm-signed-ed25519ph-blake2b -@nodedescription cm/signed with Ed25519-BLAKE2b -@subsection cm/signed with Ed25519-BLAKE2b - - @url{https://datatracker.ietf.org/doc/html/rfc8032, EdDSA} with - Edwards25519 is used similarly as in RFC 8032. But BLAKE2b is used - instead of SHA2-512 hash. - - Strict @url{https://zips.z.cash/zip-0215, ZIP-0215} validation rules - should be used while verifying the signature. - - PureEdDSA @strong{must} be used when no detached data exists and - @code{ed25519-blake2b} algorithm identifier is used for signature. - - HashEdDSA @strong{must} be used otherwise, using BLAKE2b-512 as a - hash, using @code{ed25519ph-blake2b} algorithm identifier for - signature. - -@node cm-signed-ed25519ph-blake2b-merkle -@cindex cm-signed-ed25519ph-blake2b-merkle -@nodedescription cm/signed with Ed25519-BLAKE2b with Merkle-tree hashing -@subsection cm/signed with Ed25519-BLAKE2b with Merkle-tree hashing - - @ref{cm-hashed-blake2b-merkle} Merkle-tree hashing is used. - HashEdDSA mode is used with @code{ed25519ph-blake2b-merkle} - algorithm identifier for signature. - -@node cm-signed-sphincs+-shake-256f -@cindex cm-signed-sphincs+-shake-256f -@cindex cm-signed-sphincs+-shake-256f-ph -@nodedescription cm/signed with SPHINCS+-SHAKE256-256f -@subsection cm/signed with SPHINCS+-SHAKE256-256f - - @url{https://sphincs.org/, SPHINCS+} with - @url{https://keccak.team/, SHAKE256} hash, - 255-bit security level, fast variant, - simple parameters and deterministic signatures. - - @code{sphincs+-shake-256f} algorithm identifier - must be used for the signature in pure signing mode. - @code{sphincs+-shake-256f-ph} is used in prehash mode. - -@node cm-signed-sphincs+-shake-256f-merkle -@cindex cm-signed-sphincs+-shake-256f-merkle -@nodedescription cm-signed-sphincs+-shake-256f with Merkle-tree hashing -@subsection cm-signed-sphincs+-shake-256f with Merkle-tree hashing - - @ref{cm-hashed-shake-merkle, shake256-merkle} Merkle-tree hashing is used. - - @code{sphincs+-shake-256f-merkle} algorithm - identifier must be used for the signature. diff --git a/spec/cm/signed/ed25519-blake2b b/spec/cm/signed/ed25519-blake2b new file mode 100644 index 0000000..8ecc964 --- /dev/null +++ b/spec/cm/signed/ed25519-blake2b @@ -0,0 +1,15 @@ +[cm/signed/] with Ed25519-BLAKE2b. + +EdDSA with Edwards25519 is used similarly as in RFC 8032. +But BLAKE2b is used instead of SHA2-512 hash. +Strict ZIP-0215 validation rules should be used while verifying the signature. + +=> https://datatracker.ietf.org/doc/html/rfc8032 RFC 8032, EdDSA +=> https://zips.z.cash/zip-0215 ZIP-0215 + +PureEdDSA *must" be used when no detached data exists and +"ed25519-blake2b" algorithm identifier is used for signature. + +HashEdDSA *must" be used otherwise, using BLAKE2b-512 as a +hash, using "ed25519ph-blake2b" algorithm identifier for +signature. diff --git a/spec/cm/signed/ed25519ph-blake2b-merkle b/spec/cm/signed/ed25519ph-blake2b-merkle new file mode 100644 index 0000000..db38e95 --- /dev/null +++ b/spec/cm/signed/ed25519ph-blake2b-merkle @@ -0,0 +1,4 @@ +[cm/signed/] with Ed25519-BLAKE2b with Merkle-tree hashing. +[cm/hashed/blake2b-merkle] hashing is used. +HashEdDSA mode is used with "ed25519ph-blake2b-merkle" +algorithm identifier for signature. diff --git a/spec/cm/signed/gost3410 b/spec/cm/signed/gost3410 new file mode 100644 index 0000000..0ab49bc --- /dev/null +++ b/spec/cm/signed/gost3410 @@ -0,0 +1,12 @@ +[cm/signed/] with GOST R 34.10-2012. + +GOST R 34.10-2012 must be used with Streebog (GOST R 34.11-2012) +hash function. Its digest must be big-endian serialised. Signature +is in "BE(R)||BE(S)" format. + +Algorithm identifiers for the signature: "gost3410-256A", "gost3410-512C". + +Streebog is GOST R 34.11-2012 hashing algorithm. +GOST R 34.10-2012 is signing/key-aggreement algorithm. +=> https://datatracker.ietf.org/doc/html/rfc6986.html RFC 6986, GOST R 34.11-2012 +=> https://datatracker.ietf.org/doc/html/rfc7091.html RFC 7091, GOST R 34.10-2012 diff --git a/spec/cm/signed/gost3410-merkle b/spec/cm/signed/gost3410-merkle new file mode 100644 index 0000000..afd506b --- /dev/null +++ b/spec/cm/signed/gost3410-merkle @@ -0,0 +1,4 @@ +[cm/signed/] with GOST R 34.10-2012 with Merkle-tree hashing. +[cm/hashed/streebog-merkle] hashing is used. +Algorithm identifiers for the signature: +"gost3410-256A-merkle", "gost3410-512C-merkle". diff --git a/spec/cm/signed/index b/spec/cm/signed/index new file mode 100644 index 0000000..ce658fe --- /dev/null +++ b/spec/cm/signed/index @@ -0,0 +1,49 @@ +Signed container, some kind of analogue to ASN.1-based CMS SignedData. +=> https://datatracker.ietf.org/doc/html/rfc5652 CMS + +* Ability to embed the data in the signed container +* Ability to create detached signature +* Ability to use non-prehashed signature of the embedded data, potentially + gaining more security +* Prehashed format is streaming friendly +* You can use [cm/hashed/Merkle] hashing mode to parallelise calculations +* Ability to attach arbitrary additional data +* Ability to store multiple signatures + +Stored in a file, it should begin with "cm/signed" [encoding/MAGIC], +unless it is a [cm/pub/]lic key. + +[schemas/av.tcl] [schemas/fpr.tcl] [schemas/signed.tcl] + +Signature is created by signing the: + + [detached-data] || /load || sig-tbs + +If no "/load/v" is provided, then the data is detached from the +"cm/signed" structure itself and it is fed into hasher before that +structure. You can provide it any way you wish, but for keeping that +detached data closely to the "cm/signed", you should use the +following approach: + + cm/signed/prehash || BLOB(detached-data) || cm/signed + +[schemas/signed-prehash.tcl] + +With "cm/signed/prehash" you initialise your hashers used during signing +process and feed BLOB's contents (not the encoded BLOB itself!) into the them. + +"/sigs/*/tbs/when" is optional signing time. + +Additional values that must be protected (covered by signature) are +placed in "/sigs/*/tbs" map. Non-protected (informational) fields +are placed outside it. + +"/pubs" are optionally provided [cm/pub/]lic keys to help creating the +whole verification chain. They are placed outside "/sigs", because some +of them may be shared among signers. + +If signed data is also intended to be [cm/encrypted/], then +"/sigs/*/tbs/encrypted-to" should be set to corresponding recipient's +public key fingerprint(s). + +do-backs diff --git a/spec/cm/signed/sphincs+-shake-256f b/spec/cm/signed/sphincs+-shake-256f new file mode 100644 index 0000000..d51768a --- /dev/null +++ b/spec/cm/signed/sphincs+-shake-256f @@ -0,0 +1,8 @@ +[cm/signed/] with SPHINCS+-SHAKE256-256f. +255-bit security level, fast variant, +simple parameters and deterministic signatures. +=> https://sphincs.org/ SPHINCS+ +=> https://keccak.team/ SHAKE256 +"sphincs+-shake-256f" algorithm identifier +must be used for the signature in pure signing mode. +"sphincs+-shake-256f-ph" is used in prehash mode. diff --git a/spec/cm/signed/sphincs+-shake-256f-merkle b/spec/cm/signed/sphincs+-shake-256f-merkle new file mode 100644 index 0000000..86813dc --- /dev/null +++ b/spec/cm/signed/sphincs+-shake-256f-merkle @@ -0,0 +1,3 @@ +[cm/signed/] with SPHINCS+-SHAKE256-256f with Merkle-tree hashing. +[cm/hashed/shake-merkle] SHAKE256 Merkle-tree hashing is used. +"sphincs+-shake-256f-merkle" algorithm identifier must be used for the signature. diff --git a/spec/comparison.texi b/spec/comparison.texi deleted file mode 100644 index fe7539c..0000000 --- a/spec/comparison.texi +++ /dev/null @@ -1,75 +0,0 @@ -Are not there any satisfiable codecs? - -@multitable @columnfractions .30 .05 .05 .05 .05 .05 - -@headitem @tab Schemaless @tab Simple @tab Deterministic @tab Streamable @tab Compact - -@item ASN.1 @url{https://en.wikipedia.org/wiki/Distinguished_Encoding_Rules#DER_encoding, DER} @tab - N @tab @strong{N} @tab Y @tab N @tab N -@item ASN.1 @url{https://en.wikipedia.org/wiki/Distinguished_Encoding_Rules#CER_encoding, CER} @tab - N @tab @strong{N} @tab Y @tab Y @tab N -@item @url{https://datatracker.ietf.org/doc/html/rfc1014, XDR} @tab - N @tab Y @tab N @tab N @tab N -@item @url{https://www.JSON.org/json-en.html, JSON} @tab - Y @tab N @tab N @tab Y @tab N -@item @url{https://bsonspec.org/, BSON} @tab - Y @tab Y @tab N @tab N @tab N -@item @url{https://msgpack.org/, MessagePack} @tab - Y @tab Y @tab N @tab N @tab Y -@item @url{https://datatracker.ietf.org/doc/html/rfc8949, CBOR} @tab - Y @tab N @tab N @tab Y @tab Y -@item @url{https://datatracker.ietf.org/doc/html/draft-mcnally-deterministic-cbor-11, dCBOR} @tab - Y @tab @strong{N} @tab Y @tab N @tab Y -@item @url{http://cr.yp.to/proto/netstrings.txt, Netstrings} @tab - Y @tab Y @tab Y @tab N @tab ~ -@item @url{https://wiki.theory.org/BitTorrentSpecification#Bencoding, Bencode} @tab - Y @tab Y @tab Y @tab Y @tab ~ -@item @url{https://en.wikipedia.org/wiki/Canonical_S-expressions, Canonical S-expression} @tab - Y @tab Y @tab Y @tab Y @tab ~ -@item KEKS @tab - Y @tab Y @tab Y @tab Y @tab Y - -@end multitable - -@multitable @columnfractions .20 .05 .05 .05 .05 .05 .05 .05 .05 - -@headitem @tab Big str @tab Bin str @tab Human str @tab Ints @tab Bigints @tab Lists @tab Structures @tab Datetime - -@item ASN.1 DER @tab - Y @tab Y @tab Y @tab Y @tab Y @tab Y @tab Y @tab Y -@item ASN.1 CER @tab - Y @tab Y @tab Y @tab Y @tab Y @tab Y @tab Y @tab Y -@item XDR @tab - N @tab Y @tab Y @tab Y @tab N @tab Y @tab Y @tab N -@item JSON @tab - Y @tab N @tab Y @tab Y @tab Y @tab Y @tab Y @tab N -@item BSON @tab - N @tab Y @tab Y @tab Y @tab N @tab Y @tab Y @tab Y -@item MessagePack @tab - N @tab Y @tab Y @tab Y @tab N @tab Y @tab Y @tab N -@item CBOR @tab - Y @tab Y @tab Y @tab Y @tab N @tab Y @tab Y @tab N -@item dCBOR @tab - Y @tab Y @tab Y @tab Y @tab N @tab Y @tab Y @tab N -@item Netstrings @tab - Y @tab Y @tab N @tab N @tab N @tab N @tab N @tab N -@item Bencode @tab - Y @tab Y @tab N @tab Y @tab Y @tab Y @tab Y @tab N -@item CSExp @tab - Y @tab Y @tab N @tab N @tab N @tab Y @tab N @tab N -@item KEKS @tab - Y @tab Y @tab Y @tab Y @tab Y @tab Y @tab Y @tab Y - -@end multitable - -Note about CBOR: - -@itemize -@item Hardly you will find wide range of CBOR libraries supporting -strict validation of deterministically encoded CBOR structures. -@item Tagged string/integer can not be taken as a viable first-class -bigint/datetime data support, because many decoders do not support tags -and won't be able to interpret/validate them. -@item Non-string map keys very complicates representation process for -dynamically types languages. -@end itemize diff --git a/spec/encoding/BLOB b/spec/encoding/BLOB new file mode 100644 index 0000000..80bdfe3 --- /dev/null +++ b/spec/encoding/BLOB @@ -0,0 +1,21 @@ +BLOB (binary large object) allows you to transfer binary data in chunks, +in a streaming way, when data may not fit in memory. + +64-bit big-endian integer follows the BLOB tag, setting the following +chunks payload size (+1). Then come one or more binary [encoding/String] +with the chunk-length payload. All of them, except for the last +one, must have fixed chunk length payload. Last terminating string's +payload must be shorter. + +Data format definition must specify exact chunk size expected to be +used, if it needs deterministic encoding. + + BLOB chunk-len [BIN(len=chunk-len) || ...] BIN(len https://datatracker.ietf.org/doc/html/rfc9562 UUID + +Application is left responsible for UUID validation. + +Simplest decoder can safely replace HEXLET's tag with 0x90 and decode it +as ordinary 16-byte binary string. + +Example representations: + +Nil UUID | 04 00000000000000000000000000000000 +Max UUID | 04 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF +0e875e3f-d385-49eb-87b4-be42d641c367 | 04 0E875E3FD38549EB87B4BE42D641C367 +2001:db8::1234 IPv6 | 04 20010db8000000000000000000001234 diff --git a/spec/encoding/INT b/spec/encoding/INT new file mode 100644 index 0000000..2a813eb --- /dev/null +++ b/spec/encoding/INT @@ -0,0 +1,27 @@ +Integers are encoded as ordinary binary [encoding/String], that has +prepended byte indicating is it positive or negative integer. + +Negative integers store their absolute value the same way as positive +integers. After decoding, their value is subtracted from -1. Negative +value encoded as 0x02 means: -1 - 0x02 => -3. + +Shortest possible form *must* be used, that means no leading zero +byte. 0 and -1 values are empty strings, so even they won't have +leading zero. + +Example representations: + +0 | 0C 80 +1 | 0C 81 01 +10 | 0C 81 0A +100 | 0C 01 64 +65536 | 0C 83 010000 +1000000000000 | 0C 85 E8D4A51000 +18446744073709551615 | 0C 88 FFFFFFFFFFFFFFFF +18446744073709551616 | 0C 89 010000000000000000 +-18446744073709551616 | 0D 88 FFFFFFFFFFFFFFFF +-18446744073709551617 | 0D 89 010000000000000000 +-1 | 0D 80 +-10 | 0D 81 09 +-100 | 0D 81 63 +-65536 | 0D 82 FFFF diff --git a/spec/encoding/LIST b/spec/encoding/LIST new file mode 100644 index 0000000..259df19 --- /dev/null +++ b/spec/encoding/LIST @@ -0,0 +1,9 @@ +LIST contains a concatenation of items of arbitrary type, terminated by +EOC atom. + + LIST [ITEM0 || ITEM1 || ...] EOC + +Example representations: + +LIST[] | 08 00 +LIST[INT(123) FALSE] | 08 0C817B 02 00 diff --git a/spec/encoding/MAGIC b/spec/encoding/MAGIC new file mode 100644 index 0000000..53f55d9 --- /dev/null +++ b/spec/encoding/MAGIC @@ -0,0 +1,12 @@ +MAGIC is an atom holding magic number/string aimed to identify the data +following it. It is a concatenation of ASCII("KEKS") and 12-byte magic, +padded with zeros if necessary. Zero bytes should not be used in it, +except for trailing ones. + +It is intended to be prepended to the KEKS-encoded data in files. + +Example representations: + +MAGIC(cm/pub) | 4B454B53 636D2F707562 000000000000 +MAGIC(cm/signed) | 4B454B53 636D2F7369676E6564 000000 +MAGIC(cm/encrypted) | 4B454B53 636D2F656E63727970746564 diff --git a/spec/encoding/MAP b/spec/encoding/MAP new file mode 100644 index 0000000..1f6e5a0 --- /dev/null +++ b/spec/encoding/MAP @@ -0,0 +1,15 @@ +MAP contains concatenation of string(key)-value pairs. +Keys *must* be non-empty, unique and length-first bytewise ascending ordered. + + MAP [STR(KEY0) || ITEM0 || STR(KEY1) || ITEM1 || ... ] EOC + +Hint: Encoding code for known format can be ordered itself to emit +values in an already properly sorted way. + +SET is emulated by using MAPs with NIL values. That gives only 1-byte +overhead for each element, but reuses already existing code. + +Example representations: + +MAP[foo: LIST["bar"]] | 09 C3666F6F 08 C3626172 00 00 +SET[sig, dh] | 09 C26468 01 C3736967 01 00 diff --git a/spec/encoding/PRIM b/spec/encoding/PRIM new file mode 100644 index 0000000..cc7edd0 --- /dev/null +++ b/spec/encoding/PRIM @@ -0,0 +1,8 @@ +Primitive types like NIL (None, Null), boolean FALSE and TRUE are +encoded just as a single tag. + +Example representations: + +NIL | 01 +FALSE | 02 +TRUE | 03 diff --git a/spec/encoding/String b/spec/encoding/String new file mode 100644 index 0000000..51c14d1 --- /dev/null +++ b/spec/encoding/String @@ -0,0 +1,32 @@ +There are two kinds of strings: binary (STR) and +UTF-8 (human-readable ones, STR). + +Most significant tag's bit is set for them. Seventh bit tells is it +UTF-8 string, binary otherwise. Next six bits contain the length of +the string. + + len + +------+ + / \ + 1 U L L L L L L + ^ + \-is it UTF-8? + +If length value equals to: +0-60 => Use as is. +61 => 61 + next 8-bits value. +62 => 62 + 255 + next big-endian 16-bits value. +63 => 63 + 255 + 65535 + next big-endian 64-bits value. + +String's length *must* be encoded in shortest possible form. + +UTF-8 strings *must* be valid UTF-8 sequences, except that null +byte *is not* allowed. That should be normalized Unicode string. + +Example representations: + +0-byte binary string => 80 +4-byte binary string 0x01 0x02 0x03 0x04 => 84 01 02 03 04 +64-byte binary string with 0x41 => BD 03 41 41 ... 41 +UTF-8 string "привет мир" ("hello world" in russian) => + D3 D0 BF D1 80 D0 B8 D0 B2 D0 B5 D1 82 20 D0 BC D0 B8 D1 80 diff --git a/spec/encoding/TAI64 b/spec/encoding/TAI64 new file mode 100644 index 0000000..0a9754c --- /dev/null +++ b/spec/encoding/TAI64 @@ -0,0 +1,29 @@ +Datetime is represented in, so called, external TAI64 format. +TAI stands for Temps Atomique International, the current international +real time standard. Unlike UTC, it takes leap seconds into account, +making it monotonous. + +=> http://cr.yp.to/libtai/tai64.html TAI64 +=> http://cr.yp.to/proto/utctai.html TAI + +You can convert TAI to UTC by subtracting number of leap seconds. + +TAI64N format adds 32-bit big-endian number of nanoseconds +(up to 999999999) count. +TAI64NA adds another 32-big big-endian part of attoseconds count. + +Shortest form *must* be used: if number of nanoseconds equals to zero, +then use TAI64 format. + +Example representations: + +1970-01-01 00:00:00 TAI | 18 4000000000000000 +1970-01-01 00:00:00 UTC | 18 400000000000000A +1969-12-31 23:59:59 TAI | 18 3FFFFFFFFFFFFFFF +1992-06-02 08:07:09 TAI | 18 400000002A2B2C2D +1992-06-02 08:06:43 UTC | 18 400000002A2B2C2D +1997-10-03 18:15:19 TAI | 18 4000000034353637 +1997-10-03 18:14:48 UTC | 18 4000000034353637 +2016-12-31 23:59:59 UTC | 18 40000000586846A3 +2017-01-01 00:00:00 UTC | 18 40000000586846A5 +2024-11-20 12:19:08 921772500ns UTC | 19 40000000673DD3E136F121D4 diff --git a/spec/encoding/blob.texi b/spec/encoding/blob.texi deleted file mode 100644 index bedc724..0000000 --- a/spec/encoding/blob.texi +++ /dev/null @@ -1,30 +0,0 @@ -@node BLOB -@cindex BLOB -@cindex chunk -@section BLOB - -BLOB (binary large object) allows you to transfer binary data in chunks, -in a streaming way, when data may not fit in memory. - -64-bit big-endian integer follows the BLOB tag, setting the following -chunks payload size (+1). Then come one or more @ref{Strings, BIN} -strings with the chunk-length payload. All of them, except for the last -one, must have fixed chunk length payload. Last terminating string's -payload must be shorter. - -Data format definition must specify exact chunk size expected to be -used, if it needs deterministic encoding. - -@verbatim -BLOB chunk-len [BIN(len=chunk-len) || ...] BIN(len -3}. - -Shortest possible form @strong{must} be used, that means no leading zero -byte. 0 and -1 values are empty strings, so even they won't have leading -zero. - -Example representations: - -@multitable @columnfractions .5 .5 - -@item 0 @tab @code{0C 80} -@item 1 @tab @code{0C 81 01} -@item 10 @tab @code{0C 81 0A} -@item 100 @tab @code{0C 01 64} -@item 65536 @tab @code{0C 83 010000} -@item 1000000000000 @tab @code{0C 85 E8D4A51000} -@item 18446744073709551615 @tab @code{0C 88 FFFFFFFFFFFFFFFF} -@item 18446744073709551616 @tab @code{0C 89 010000000000000000} -@item -18446744073709551616 @tab @code{0D 88 FFFFFFFFFFFFFFFF} -@item -18446744073709551617 @tab @code{0D 89 010000000000000000} -@item -1 @tab @code{0D 80} -@item -10 @tab @code{0D 81 09} -@item -100 @tab @code{0D 81 63} -@item -65536 @tab @code{0D 82 FFFF} - -@end multitable diff --git a/spec/encoding/list.texi b/spec/encoding/list.texi deleted file mode 100644 index 432f8e2..0000000 --- a/spec/encoding/list.texi +++ /dev/null @@ -1,20 +0,0 @@ -@node LIST -@cindex LIST -@cindex EOC -@section LIST - -LIST contains a concatenation of items of arbitrary type, terminated by -EOC atom. - -@verbatim -LIST [ITEM0 || ITEM1 || ...] EOC -@end verbatim - -Example representations: - -@multitable @columnfractions .5 .5 - -@item LIST[] @tab @code{08 00} -@item LIST[INT(123) FALSE] @tab @code{08 0C817B 02 00} - -@end multitable diff --git a/spec/encoding/magic.texi b/spec/encoding/magic.texi deleted file mode 100644 index f46b941..0000000 --- a/spec/encoding/magic.texi +++ /dev/null @@ -1,18 +0,0 @@ -@node MAGIC -@cindex MAGIC -@section MAGIC - -MAGIC is an atom holding magic number/string aimed to identify the data -following it. It is a concatenation of ASCII("KEKS") and 12-byte magic, -padded with zeros if necessary. Zero bytes should not be used in it, -except for trailing ones. - -It is intended to be prepended to the KEKS-encoded data in files. - -@multitable @columnfractions .5 .5 - -@item MAGIC(cm/pub) @tab @code{4B454B53 636D2F707562 000000000000} -@item MAGIC(cm/signed) @tab @code{4B454B53 636D2F7369676E6564 000000} -@item MAGIC(cm/encrypted) @tab @code{4B454B53 636D2F656E63727970746564} - -@end multitable diff --git a/spec/encoding/map.texi b/spec/encoding/map.texi deleted file mode 100644 index ebfdf45..0000000 --- a/spec/encoding/map.texi +++ /dev/null @@ -1,27 +0,0 @@ -@node MAP -@cindex MAP -@section MAP - -MAP contains concatenation of @ref{Strings, STR(key)}-value pairs. Keys -@strong{must} be non-empty, unique and length-first bytewise ascending ordered. - -@verbatim -MAP [STR(KEY0) || ITEM0 || STR(KEY1) || ITEM1 || ... ] EOC -@end verbatim - -Hint: Encoding code for known format can be ordered itself to emit -values in an already properly sorted way. - -@cindex SET -SET is emulated by using MAPs with NIL values. That gives only 1-byte -overhead for each element, but reuses already existing code. - -Example representations: - -@multitable @columnfractions .5 .5 - -@item MAP[foo: LIST["bar"]] @tab @code{09 C3666F6F 08 C3626172 00 00} -@item SET[sig, dh] @tab @code{09 C26468 01 C3736967 01 00} - -@end multitable - diff --git a/spec/encoding/prim.texi b/spec/encoding/prim.texi deleted file mode 100644 index 76c3ab0..0000000 --- a/spec/encoding/prim.texi +++ /dev/null @@ -1,18 +0,0 @@ -@node Primitives -@cindex NIL -@cindex FALSE -@cindex TRUE -@section Primitives - -Very primitive types like NIL (None, Null), boolean FALSE and TRUE are -encoded just as a single tag. - -Example representations: - -@multitable @columnfractions .5 .5 - -@item NIL @tab @code{01} -@item FALSE @tab @code{02} -@item TRUE @tab @code{03} - -@end multitable diff --git a/spec/encoding/str.texi b/spec/encoding/str.texi deleted file mode 100644 index e2f5316..0000000 --- a/spec/encoding/str.texi +++ /dev/null @@ -1,48 +0,0 @@ -@node Strings -@cindex BIN -@cindex STR -@section Strings - -There are two kinds of strings: binary and UTF-8 (human-readable ones). -Most significant tag's bit is set for them. Seventh bit tells is it -UTF-8 string, binary otherwise. Next six bits contain the length of the -string. - -@verbatim - len - +------+ - / \ -1 U L L L L L L - ^ - +-is it UTF-8? -@end verbatim - -If length value equals to: - -@table @asis -@item 0-60 - Use as is. -@item 61 - 61 plus next 8-bits value. -@item 62 - 62 plus 255 plus next big-endian 16-bits value. -@item 63 - 63 plus 255 plus 65535 plus next big-endian 64-bits value. -@end table - -String's length @strong{must} be encoded in shortest possible form. - -UTF-8 strings @strong{must} be valid UTF-8 sequences, except that null -byte @strong{is not} allowed. That should be normalized Unicode string. - -Example representations: - -@multitable @columnfractions .5 .5 - -@item 0-byte binary string @tab @code{80} -@item 4-byte binary string @code{0x01 0x02 0x03 0x04} @tab @code{84 01 02 03 04} -@item 64-byte binary string with 0x41 @tab @code{BD 03 41 41 .. 41} -@item UTF-8 string "привет мир" ("hello world" in russian) @tab - @code{D3 D0 BF D1 80 D0 B8 D0 B2 D0 B5 D1 82 20 D0 BC D0 B8 D1 80} - -@end multitable diff --git a/spec/encoding/table.texi b/spec/encoding/table.texi deleted file mode 100644 index 04509dd..0000000 --- a/spec/encoding/table.texi +++ /dev/null @@ -1,264 +0,0 @@ -@node Encoding table -@section Full encoding table - -@multitable {dec} {hex} {12345678} {len} {float256} -@headitem dec @tab hex @tab bin @tab vlen @tab description - -@item 000 @tab 00 @tab @code{00000000} @tab 0 @tab @ref{LIST, EOC} -@item 001 @tab 01 @tab @code{00000001} @tab 0 @tab @ref{Primitives, NIL} -@item 002 @tab 02 @tab @code{00000010} @tab 0 @tab @ref{Primitives, FALSE} -@item 003 @tab 03 @tab @code{00000011} @tab 0 @tab @ref{Primitives, TRUE} -@item 004 @tab 04 @tab @code{00000100} @tab 16 @tab @ref{HEXLET} -@item 005 @tab 05 @tab @code{00000101} @tab 0 @tab -@item 006 @tab 06 @tab @code{00000110} @tab 0 @tab -@item 007 @tab 07 @tab @code{00000111} @tab 0 @tab -@item 008 @tab 08 @tab @code{00001000} @tab 0 @tab @ref{LIST} -@item 009 @tab 09 @tab @code{00001001} @tab 0 @tab @ref{MAP} -@item 010 @tab 0A @tab @code{00001010} @tab 0 @tab -@item 011 @tab 0B @tab @code{00001011} @tab 8+~ @tab @ref{BLOB} -@item 012 @tab 0C @tab @code{00001100} @tab 1+~ @tab @ref{Integers, +INT} -@item 013 @tab 0D @tab @code{00001101} @tab 1+~ @tab @ref{Integers, -INT} -@item 014 @tab 0E @tab @code{00001110} @tab 0 @tab -@item 015 @tab 0F @tab @code{00001111} @tab 0 @tab -@item 016 @tab 10 @tab @code{00010000} @tab 2 @tab @ref{Floats, FLOAT16} -@item 017 @tab 11 @tab @code{00010001} @tab 4 @tab @ref{Floats, FLOAT32} -@item 018 @tab 12 @tab @code{00010010} @tab 8 @tab @ref{Floats, FLOAT64} -@item 019 @tab 13 @tab @code{00010011} @tab 16 @tab @ref{Floats, FLOAT128} -@item 020 @tab 14 @tab @code{00010100} @tab 32 @tab @ref{Floats, FLOAT256} -@item 021 @tab 15 @tab @code{00010101} @tab 0 @tab -@item 022 @tab 16 @tab @code{00010110} @tab 0 @tab -@item 023 @tab 17 @tab @code{00010111} @tab 0 @tab -@item 024 @tab 18 @tab @code{00011000} @tab 8 @tab @ref{TAI64, TAI64} -@item 025 @tab 19 @tab @code{00011001} @tab 12 @tab @ref{TAI64, TAI64N} -@item 026 @tab 1A @tab @code{00011010} @tab 16 @tab @ref{TAI64, TAI64NA} -@item 027 @tab 1B @tab @code{00011011} @tab 0 @tab -@item 028 @tab 1C @tab @code{00011100} @tab 0 @tab -@item 029 @tab 1D @tab @code{00011101} @tab 0 @tab -@item 030 @tab 1E @tab @code{00011110} @tab 0 @tab -@item 031 @tab 1F @tab @code{00011111} @tab 0 @tab -@item 032 @tab 20 @tab @code{00100000} @tab 0 @tab -@item 033 @tab 21 @tab @code{00100001} @tab 0 @tab -@item 034 @tab 22 @tab @code{00100010} @tab 0 @tab -@item 035 @tab 23 @tab @code{00100011} @tab 0 @tab -@item 036 @tab 24 @tab @code{00100100} @tab 0 @tab -@item 037 @tab 25 @tab @code{00100101} @tab 0 @tab -@item 038 @tab 26 @tab @code{00100110} @tab 0 @tab -@item 039 @tab 27 @tab @code{00100111} @tab 0 @tab -@item 040 @tab 28 @tab @code{00101000} @tab 0 @tab -@item 041 @tab 29 @tab @code{00101001} @tab 0 @tab -@item 042 @tab 2A @tab @code{00101010} @tab 0 @tab -@item 043 @tab 2B @tab @code{00101011} @tab 0 @tab -@item 044 @tab 2C @tab @code{00101100} @tab 0 @tab -@item 045 @tab 2D @tab @code{00101101} @tab 0 @tab -@item 046 @tab 2E @tab @code{00101110} @tab 0 @tab -@item 047 @tab 2F @tab @code{00101111} @tab 0 @tab -@item 048 @tab 30 @tab @code{00110000} @tab 0 @tab -@item 049 @tab 31 @tab @code{00110001} @tab 0 @tab -@item 050 @tab 32 @tab @code{00110010} @tab 0 @tab -@item 051 @tab 33 @tab @code{00110011} @tab 0 @tab -@item 052 @tab 34 @tab @code{00110100} @tab 0 @tab -@item 053 @tab 35 @tab @code{00110101} @tab 0 @tab -@item 054 @tab 36 @tab @code{00110110} @tab 0 @tab -@item 055 @tab 37 @tab @code{00110111} @tab 0 @tab -@item 056 @tab 38 @tab @code{00111000} @tab 0 @tab -@item 057 @tab 39 @tab @code{00111001} @tab 0 @tab -@item 058 @tab 3A @tab @code{00111010} @tab 0 @tab -@item 059 @tab 3B @tab @code{00111011} @tab 0 @tab -@item 060 @tab 3C @tab @code{00111100} @tab 0 @tab -@item 061 @tab 3D @tab @code{00111101} @tab 0 @tab -@item 062 @tab 3E @tab @code{00111110} @tab 0 @tab -@item 063 @tab 3F @tab @code{00111111} @tab 0 @tab -@item 064 @tab 40 @tab @code{01000000} @tab 0 @tab -@item 065 @tab 41 @tab @code{01000001} @tab 0 @tab -@item 066 @tab 42 @tab @code{01000010} @tab 0 @tab -@item 067 @tab 43 @tab @code{01000011} @tab 0 @tab -@item 068 @tab 44 @tab @code{01000100} @tab 0 @tab -@item 069 @tab 45 @tab @code{01000101} @tab 0 @tab -@item 070 @tab 46 @tab @code{01000110} @tab 0 @tab -@item 071 @tab 47 @tab @code{01000111} @tab 0 @tab -@item 072 @tab 48 @tab @code{01001000} @tab 0 @tab -@item 073 @tab 49 @tab @code{01001001} @tab 0 @tab -@item 074 @tab 4A @tab @code{01001010} @tab 0 @tab -@item 075 @tab 4B @tab @code{01001011} @tab 15 @tab @ref{MAGIC} -@item 076 @tab 4C @tab @code{01001100} @tab 0 @tab -@item 077 @tab 4D @tab @code{01001101} @tab 0 @tab -@item 078 @tab 4E @tab @code{01001110} @tab 0 @tab -@item 079 @tab 4F @tab @code{01001111} @tab 0 @tab -@item 080 @tab 50 @tab @code{01010000} @tab 0 @tab -@item 081 @tab 51 @tab @code{01010001} @tab 0 @tab -@item 082 @tab 52 @tab @code{01010010} @tab 0 @tab -@item 083 @tab 53 @tab @code{01010011} @tab 0 @tab -@item 084 @tab 54 @tab @code{01010100} @tab 0 @tab -@item 085 @tab 55 @tab @code{01010101} @tab 0 @tab -@item 086 @tab 56 @tab @code{01010110} @tab 0 @tab -@item 087 @tab 57 @tab @code{01010111} @tab 0 @tab -@item 088 @tab 58 @tab @code{01011000} @tab 0 @tab -@item 089 @tab 59 @tab @code{01011001} @tab 0 @tab -@item 090 @tab 5A @tab @code{01011010} @tab 0 @tab -@item 091 @tab 5B @tab @code{01011011} @tab 0 @tab -@item 092 @tab 5C @tab @code{01011100} @tab 0 @tab -@item 093 @tab 5D @tab @code{01011101} @tab 0 @tab -@item 094 @tab 5E @tab @code{01011110} @tab 0 @tab -@item 095 @tab 5F @tab @code{01011111} @tab 0 @tab -@item 096 @tab 60 @tab @code{01100000} @tab 0 @tab -@item 097 @tab 61 @tab @code{01100001} @tab 0 @tab -@item 098 @tab 62 @tab @code{01100010} @tab 0 @tab -@item 099 @tab 63 @tab @code{01100011} @tab 0 @tab -@item 100 @tab 64 @tab @code{01100100} @tab 0 @tab -@item 101 @tab 65 @tab @code{01100101} @tab 0 @tab -@item 102 @tab 66 @tab @code{01100110} @tab 0 @tab -@item 103 @tab 67 @tab @code{01100111} @tab 0 @tab -@item 104 @tab 68 @tab @code{01101000} @tab 0 @tab -@item 105 @tab 69 @tab @code{01101001} @tab 0 @tab -@item 106 @tab 6A @tab @code{01101010} @tab 0 @tab -@item 107 @tab 6B @tab @code{01101011} @tab 0 @tab -@item 108 @tab 6C @tab @code{01101100} @tab 0 @tab -@item 109 @tab 6D @tab @code{01101101} @tab 0 @tab -@item 110 @tab 6E @tab @code{01101110} @tab 0 @tab -@item 111 @tab 6F @tab @code{01101111} @tab 0 @tab -@item 112 @tab 70 @tab @code{01110000} @tab 0 @tab -@item 113 @tab 71 @tab @code{01110001} @tab 0 @tab -@item 114 @tab 72 @tab @code{01110010} @tab 0 @tab -@item 115 @tab 73 @tab @code{01110011} @tab 0 @tab -@item 116 @tab 74 @tab @code{01110100} @tab 0 @tab -@item 117 @tab 75 @tab @code{01110101} @tab 0 @tab -@item 118 @tab 76 @tab @code{01110110} @tab 0 @tab -@item 119 @tab 77 @tab @code{01110111} @tab 0 @tab -@item 120 @tab 78 @tab @code{01111000} @tab 0 @tab -@item 121 @tab 79 @tab @code{01111001} @tab 0 @tab -@item 122 @tab 7A @tab @code{01111010} @tab 0 @tab -@item 123 @tab 7B @tab @code{01111011} @tab 0 @tab -@item 124 @tab 7C @tab @code{01111100} @tab 0 @tab -@item 125 @tab 7D @tab @code{01111101} @tab 0 @tab -@item 126 @tab 7E @tab @code{01111110} @tab 0 @tab -@item 127 @tab 7F @tab @code{01111111} @tab 0 @tab -@item 128 @tab 80 @tab @code{10000000} @tab 0 @tab @ref{Strings, BIN(len=0)} -@item 129 @tab 81 @tab @code{10000001} @tab 1 @tab @ref{Strings, BIN(len=1)} -@item 130 @tab 82 @tab @code{10000010} @tab 2 @tab @ref{Strings, BIN(len=2)} -@item 131 @tab 83 @tab @code{10000011} @tab 3 @tab @ref{Strings, BIN(len=3)} -@item 132 @tab 84 @tab @code{10000100} @tab 4 @tab @ref{Strings, BIN(len=4)} -@item 133 @tab 85 @tab @code{10000101} @tab 5 @tab @ref{Strings, BIN(len=5)} -@item 134 @tab 86 @tab @code{10000110} @tab 6 @tab @ref{Strings, BIN(len=6)} -@item 135 @tab 87 @tab @code{10000111} @tab 7 @tab @ref{Strings, BIN(len=7)} -@item 136 @tab 88 @tab @code{10001000} @tab 8 @tab @ref{Strings, BIN(len=8)} -@item 137 @tab 89 @tab @code{10001001} @tab 9 @tab @ref{Strings, BIN(len=9)} -@item 138 @tab 8A @tab @code{10001010} @tab 10 @tab @ref{Strings, BIN(len=10)} -@item 139 @tab 8B @tab @code{10001011} @tab 11 @tab @ref{Strings, BIN(len=11)} -@item 140 @tab 8C @tab @code{10001100} @tab 12 @tab @ref{Strings, BIN(len=12)} -@item 141 @tab 8D @tab @code{10001101} @tab 13 @tab @ref{Strings, BIN(len=13)} -@item 142 @tab 8E @tab @code{10001110} @tab 14 @tab @ref{Strings, BIN(len=14)} -@item 143 @tab 8F @tab @code{10001111} @tab 15 @tab @ref{Strings, BIN(len=15)} -@item 144 @tab 90 @tab @code{10010000} @tab 16 @tab @ref{Strings, BIN(len=16)} -@item 145 @tab 91 @tab @code{10010001} @tab 17 @tab @ref{Strings, BIN(len=17)} -@item 146 @tab 92 @tab @code{10010010} @tab 18 @tab @ref{Strings, BIN(len=18)} -@item 147 @tab 93 @tab @code{10010011} @tab 19 @tab @ref{Strings, BIN(len=19)} -@item 148 @tab 94 @tab @code{10010100} @tab 20 @tab @ref{Strings, BIN(len=20)} -@item 149 @tab 95 @tab @code{10010101} @tab 21 @tab @ref{Strings, BIN(len=21)} -@item 150 @tab 96 @tab @code{10010110} @tab 22 @tab @ref{Strings, BIN(len=22)} -@item 151 @tab 97 @tab @code{10010111} @tab 23 @tab @ref{Strings, BIN(len=23)} -@item 152 @tab 98 @tab @code{10011000} @tab 24 @tab @ref{Strings, BIN(len=24)} -@item 153 @tab 99 @tab @code{10011001} @tab 25 @tab @ref{Strings, BIN(len=25)} -@item 154 @tab 9A @tab @code{10011010} @tab 26 @tab @ref{Strings, BIN(len=26)} -@item 155 @tab 9B @tab @code{10011011} @tab 27 @tab @ref{Strings, BIN(len=27)} -@item 156 @tab 9C @tab @code{10011100} @tab 28 @tab @ref{Strings, BIN(len=28)} -@item 157 @tab 9D @tab @code{10011101} @tab 29 @tab @ref{Strings, BIN(len=29)} -@item 158 @tab 9E @tab @code{10011110} @tab 30 @tab @ref{Strings, BIN(len=30)} -@item 159 @tab 9F @tab @code{10011111} @tab 31 @tab @ref{Strings, BIN(len=31)} -@item 160 @tab A0 @tab @code{10100000} @tab 32 @tab @ref{Strings, BIN(len=32)} -@item 161 @tab A1 @tab @code{10100001} @tab 33 @tab @ref{Strings, BIN(len=33)} -@item 162 @tab A2 @tab @code{10100010} @tab 34 @tab @ref{Strings, BIN(len=34)} -@item 163 @tab A3 @tab @code{10100011} @tab 35 @tab @ref{Strings, BIN(len=35)} -@item 164 @tab A4 @tab @code{10100100} @tab 36 @tab @ref{Strings, BIN(len=36)} -@item 165 @tab A5 @tab @code{10100101} @tab 37 @tab @ref{Strings, BIN(len=37)} -@item 166 @tab A6 @tab @code{10100110} @tab 38 @tab @ref{Strings, BIN(len=38)} -@item 167 @tab A7 @tab @code{10100111} @tab 39 @tab @ref{Strings, BIN(len=39)} -@item 168 @tab A8 @tab @code{10101000} @tab 40 @tab @ref{Strings, BIN(len=40)} -@item 169 @tab A9 @tab @code{10101001} @tab 41 @tab @ref{Strings, BIN(len=41)} -@item 170 @tab AA @tab @code{10101010} @tab 42 @tab @ref{Strings, BIN(len=42)} -@item 171 @tab AB @tab @code{10101011} @tab 43 @tab @ref{Strings, BIN(len=43)} -@item 172 @tab AC @tab @code{10101100} @tab 44 @tab @ref{Strings, BIN(len=44)} -@item 173 @tab AD @tab @code{10101101} @tab 45 @tab @ref{Strings, BIN(len=45)} -@item 174 @tab AE @tab @code{10101110} @tab 46 @tab @ref{Strings, BIN(len=46)} -@item 175 @tab AF @tab @code{10101111} @tab 47 @tab @ref{Strings, BIN(len=47)} -@item 176 @tab B0 @tab @code{10110000} @tab 48 @tab @ref{Strings, BIN(len=48)} -@item 177 @tab B1 @tab @code{10110001} @tab 49 @tab @ref{Strings, BIN(len=49)} -@item 178 @tab B2 @tab @code{10110010} @tab 50 @tab @ref{Strings, BIN(len=50)} -@item 179 @tab B3 @tab @code{10110011} @tab 51 @tab @ref{Strings, BIN(len=51)} -@item 180 @tab B4 @tab @code{10110100} @tab 52 @tab @ref{Strings, BIN(len=52)} -@item 181 @tab B5 @tab @code{10110101} @tab 53 @tab @ref{Strings, BIN(len=53)} -@item 182 @tab B6 @tab @code{10110110} @tab 54 @tab @ref{Strings, BIN(len=54)} -@item 183 @tab B7 @tab @code{10110111} @tab 55 @tab @ref{Strings, BIN(len=55)} -@item 184 @tab B8 @tab @code{10111000} @tab 56 @tab @ref{Strings, BIN(len=56)} -@item 185 @tab B9 @tab @code{10111001} @tab 57 @tab @ref{Strings, BIN(len=57)} -@item 186 @tab BA @tab @code{10111010} @tab 58 @tab @ref{Strings, BIN(len=58)} -@item 187 @tab BB @tab @code{10111011} @tab 59 @tab @ref{Strings, BIN(len=59)} -@item 188 @tab BC @tab @code{10111100} @tab 60 @tab @ref{Strings, BIN(len=60)} -@item 189 @tab BD @tab @code{10111101} @tab 1+~ @tab @ref{Strings, BIN(len of 8b)} -@item 190 @tab BE @tab @code{10111110} @tab 2+~ @tab @ref{Strings, BIN(len of 16b)} -@item 191 @tab BF @tab @code{10111111} @tab 8+~ @tab @ref{Strings, BIN(len of 64b)} -@item 192 @tab C0 @tab @code{11000000} @tab 0 @tab @ref{Strings, STR(len=0)} -@item 193 @tab C1 @tab @code{11000001} @tab 1 @tab @ref{Strings, STR(len=1)} -@item 194 @tab C2 @tab @code{11000010} @tab 2 @tab @ref{Strings, STR(len=2)} -@item 195 @tab C3 @tab @code{11000011} @tab 3 @tab @ref{Strings, STR(len=3)} -@item 196 @tab C4 @tab @code{11000100} @tab 4 @tab @ref{Strings, STR(len=4)} -@item 197 @tab C5 @tab @code{11000101} @tab 5 @tab @ref{Strings, STR(len=5)} -@item 198 @tab C6 @tab @code{11000110} @tab 6 @tab @ref{Strings, STR(len=6)} -@item 199 @tab C7 @tab @code{11000111} @tab 7 @tab @ref{Strings, STR(len=7)} -@item 200 @tab C8 @tab @code{11001000} @tab 8 @tab @ref{Strings, STR(len=8)} -@item 201 @tab C9 @tab @code{11001001} @tab 9 @tab @ref{Strings, STR(len=9)} -@item 202 @tab CA @tab @code{11001010} @tab 10 @tab @ref{Strings, STR(len=10)} -@item 203 @tab CB @tab @code{11001011} @tab 11 @tab @ref{Strings, STR(len=11)} -@item 204 @tab CC @tab @code{11001100} @tab 12 @tab @ref{Strings, STR(len=12)} -@item 205 @tab CD @tab @code{11001101} @tab 13 @tab @ref{Strings, STR(len=13)} -@item 206 @tab CE @tab @code{11001110} @tab 14 @tab @ref{Strings, STR(len=14)} -@item 207 @tab CF @tab @code{11001111} @tab 15 @tab @ref{Strings, STR(len=15)} -@item 208 @tab D0 @tab @code{11010000} @tab 16 @tab @ref{Strings, STR(len=16)} -@item 209 @tab D1 @tab @code{11010001} @tab 17 @tab @ref{Strings, STR(len=17)} -@item 210 @tab D2 @tab @code{11010010} @tab 18 @tab @ref{Strings, STR(len=18)} -@item 211 @tab D3 @tab @code{11010011} @tab 19 @tab @ref{Strings, STR(len=19)} -@item 212 @tab D4 @tab @code{11010100} @tab 20 @tab @ref{Strings, STR(len=20)} -@item 213 @tab D5 @tab @code{11010101} @tab 21 @tab @ref{Strings, STR(len=21)} -@item 214 @tab D6 @tab @code{11010110} @tab 22 @tab @ref{Strings, STR(len=22)} -@item 215 @tab D7 @tab @code{11010111} @tab 23 @tab @ref{Strings, STR(len=23)} -@item 216 @tab D8 @tab @code{11011000} @tab 24 @tab @ref{Strings, STR(len=24)} -@item 217 @tab D9 @tab @code{11011001} @tab 25 @tab @ref{Strings, STR(len=25)} -@item 218 @tab DA @tab @code{11011010} @tab 26 @tab @ref{Strings, STR(len=26)} -@item 219 @tab DB @tab @code{11011011} @tab 27 @tab @ref{Strings, STR(len=27)} -@item 220 @tab DC @tab @code{11011100} @tab 28 @tab @ref{Strings, STR(len=28)} -@item 221 @tab DD @tab @code{11011101} @tab 29 @tab @ref{Strings, STR(len=29)} -@item 222 @tab DE @tab @code{11011110} @tab 30 @tab @ref{Strings, STR(len=30)} -@item 223 @tab DF @tab @code{11011111} @tab 31 @tab @ref{Strings, STR(len=31)} -@item 224 @tab E0 @tab @code{11100000} @tab 32 @tab @ref{Strings, STR(len=32)} -@item 225 @tab E1 @tab @code{11100001} @tab 33 @tab @ref{Strings, STR(len=33)} -@item 226 @tab E2 @tab @code{11100010} @tab 34 @tab @ref{Strings, STR(len=34)} -@item 227 @tab E3 @tab @code{11100011} @tab 35 @tab @ref{Strings, STR(len=35)} -@item 228 @tab E4 @tab @code{11100100} @tab 36 @tab @ref{Strings, STR(len=36)} -@item 229 @tab E5 @tab @code{11100101} @tab 37 @tab @ref{Strings, STR(len=37)} -@item 230 @tab E6 @tab @code{11100110} @tab 38 @tab @ref{Strings, STR(len=38)} -@item 231 @tab E7 @tab @code{11100111} @tab 39 @tab @ref{Strings, STR(len=39)} -@item 232 @tab E8 @tab @code{11101000} @tab 40 @tab @ref{Strings, STR(len=40)} -@item 233 @tab E9 @tab @code{11101001} @tab 41 @tab @ref{Strings, STR(len=41)} -@item 234 @tab EA @tab @code{11101010} @tab 42 @tab @ref{Strings, STR(len=42)} -@item 235 @tab EB @tab @code{11101011} @tab 43 @tab @ref{Strings, STR(len=43)} -@item 236 @tab EC @tab @code{11101100} @tab 44 @tab @ref{Strings, STR(len=44)} -@item 237 @tab ED @tab @code{11101101} @tab 45 @tab @ref{Strings, STR(len=45)} -@item 238 @tab EE @tab @code{11101110} @tab 46 @tab @ref{Strings, STR(len=46)} -@item 239 @tab EF @tab @code{11101111} @tab 47 @tab @ref{Strings, STR(len=47)} -@item 240 @tab F0 @tab @code{11110000} @tab 48 @tab @ref{Strings, STR(len=48)} -@item 241 @tab F1 @tab @code{11110001} @tab 49 @tab @ref{Strings, STR(len=49)} -@item 242 @tab F2 @tab @code{11110010} @tab 50 @tab @ref{Strings, STR(len=50)} -@item 243 @tab F3 @tab @code{11110011} @tab 51 @tab @ref{Strings, STR(len=51)} -@item 244 @tab F4 @tab @code{11110100} @tab 52 @tab @ref{Strings, STR(len=52)} -@item 245 @tab F5 @tab @code{11110101} @tab 53 @tab @ref{Strings, STR(len=53)} -@item 246 @tab F6 @tab @code{11110110} @tab 54 @tab @ref{Strings, STR(len=54)} -@item 247 @tab F7 @tab @code{11110111} @tab 55 @tab @ref{Strings, STR(len=55)} -@item 248 @tab F8 @tab @code{11111000} @tab 56 @tab @ref{Strings, STR(len=56)} -@item 249 @tab F9 @tab @code{11111001} @tab 57 @tab @ref{Strings, STR(len=57)} -@item 250 @tab FA @tab @code{11111010} @tab 58 @tab @ref{Strings, STR(len=58)} -@item 251 @tab FB @tab @code{11111011} @tab 59 @tab @ref{Strings, STR(len=59)} -@item 252 @tab FC @tab @code{11111100} @tab 60 @tab @ref{Strings, STR(len=60)} -@item 253 @tab FD @tab @code{11111101} @tab 1+~ @tab @ref{Strings, STR(len of 8b)} -@item 254 @tab FE @tab @code{11111110} @tab 2+~ @tab @ref{Strings, STR(len of 16b)} -@item 255 @tab FF @tab @code{11111111} @tab 8+~ @tab @ref{Strings, STR(len of 64b)} - -@end multitable diff --git a/spec/encoding/tai64.texi b/spec/encoding/tai64.texi deleted file mode 100644 index de69d16..0000000 --- a/spec/encoding/tai64.texi +++ /dev/null @@ -1,37 +0,0 @@ -@node TAI64 -@cindex TAI64 -@cindex TAI64N -@cindex TAI64NA -@section TAI64 - -Datetime is represented in, so called, external -@url{http://cr.yp.to/libtai/tai64.html, TAI64} -format. @url{http://cr.yp.to/proto/utctai.html, TAI} stands for Temps Atomique -International, the current international real time standard. Unlike UTC, it -takes leap seconds into account, making it monotonous. - -You can convert TAI to UTC by subtracting number of leap seconds. - -TAI64N format adds 32-bit big-endian number of nanoseconds (in up to -999999999) count. TAI64NA adds another 32-big big-endian part of -attoseconds count. - -Shortest form @strong{must} be used: if number of nanoseconds -equals to zero, then use TAI64 format. - -Example representations: - -@multitable @columnfractions .5 .5 - -@item 1970-01-01 00:00:00 TAI @tab @code{18 4000000000000000} -@item 1970-01-01 00:00:00 UTC @tab @code{18 400000000000000A} -@item 1969-12-31 23:59:59 TAI @tab @code{18 3FFFFFFFFFFFFFFF} -@item 1992-06-02 08:07:09 TAI @tab @code{18 400000002A2B2C2D} -@item 1992-06-02 08:06:43 UTC @tab @code{18 400000002A2B2C2D} -@item 1997-10-03 18:15:19 TAI @tab @code{18 4000000034353637} -@item 1997-10-03 18:14:48 UTC @tab @code{18 4000000034353637} -@item 2016-12-31 23:59:59 UTC @tab @code{18 40000000586846A3} -@item 2017-01-01 00:00:00 UTC @tab @code{18 40000000586846A5} -@item 2024-11-20 12:19:08 921772500ns UTC @tab @code{19 40000000673DD3E136F121D4} - -@end multitable diff --git a/spec/index b/spec/index new file mode 100644 index 0000000..bd8096b --- /dev/null +++ b/spec/index @@ -0,0 +1,38 @@ +KEKS is compact, deterministic, concise and streaming binary +serialisation format. It is aimed to be lightweight in terms of CPU, +memory, storage and codec implementation size usage. It supports wide +range of data types, making it able to transparently replace JSON. + +KEKS means: kompakt, entschlossen, knapp, strömend. + +But why!? Because there is no satisfiable codec for all set of +requirements below. + +* It *must* be schema-less format. Schema-aware ones have their definite + valuable advantages, but also a complication drawbacks and + non-friendliness to humans. +* Its encoder/decoder *must* be very compact and small in terms of code + and branches amount, to reduce attack surface on the codec itself. +* It *must* support enough data types for being able at least to replace + JSON transparently. +* Its encoding *must* be deterministic -- there must be only a single + representation of the structured data, allowing its usage in + cryptography-related contexts. +* Its encoder *should* be streaming-friendly, making encoder + simpler and allowing memory-constrained systems workability. +* Its encoding *should* be reasonably compact, to be friendly to + storage space constrained systems. +* It *should* be frugal to CPU usage for both performance/memory + constrained and high data volume applications. +* It *should* differentiate binary and human-readable strings. +* It *would* be nice to have human-editable intermediate representation. + +include [ComparisonWithOtherCodecs] + +[INSTALL] +[encoding/] +[schema/] -- structure validation against schemas +[cm/] -- cryptographic messages +[THANKS] + +Copyright © 2024-2025 Sergey Matveev diff --git a/spec/index.texi b/spec/index.texi deleted file mode 100644 index 91d41ef..0000000 --- a/spec/index.texi +++ /dev/null @@ -1,67 +0,0 @@ -\input texinfo -@settitle KEKS - -@copying -Copyright @copyright{} 2024-2025 @email{stargrave@@stargrave.org, Sergey Matveev} -@end copying - -@firstparagraphindent insert - -@node Top -@top KEKS - -KEKS is compact, deterministic, concise and streaming binary -serialisation format. It is aimed to be lightweight in terms of CPU, -memory, storage and codec implementation size usage. It supports wide -range of data types, making it able to transparently replace JSON. - -KEKS means: kompakt, entschlossen, knapp, strömend. - -But why!? Because there is no satisfiable codec for all set of -requirements below. - -@itemize -@item - It @strong{must} be schema-less format. Schema-aware ones have - their definite valuable advantages, but also a complication drawbacks - and non-friendliness to humans. -@item - Its encoder/decoder @strong{must} be very compact and small in terms of - code and branches amount, to reduce attack surface on the codec itself. -@item - It @strong{must} support enough data types for being able at - least to replace JSON transparently. -@item - Its encoding @strong{must} be deterministic -- there must be only a - single representation of the structured data, allowing its usage in - cryptography-related contexts. -@item - Its encoder @strong{should} be streaming-friendly, making encoder - simpler and allowing memory-constrained systems workability. -@item - Its encoding @strong{should} be reasonably compact, to be friendly to - storage space constrained systems. -@item - It @strong{should} be frugal to CPU usage for both performance/memory - constrained and high data volume applications. -@item - It @strong{should} differentiate binary and human-readable strings. -@item - It @strong{would} be nice to have human-editable intermediate representation. -@end itemize - -@include comparison.texi - -@insertcopying - -@include install.texi -@include encoding/index.texi -@include schema/index.texi -@include cm/index.texi -@include thanks.texi - -@node Concepts Index -@unnumbered Concepts Index -@printindex cp - -@bye diff --git a/spec/install.texi b/spec/install.texi deleted file mode 100644 index dab251a..0000000 --- a/spec/install.texi +++ /dev/null @@ -1,20 +0,0 @@ -@node Install -@unnumbered Install - -Currently there are draft versions of the codec written on C, Go, -Python and Tcl. - -@cindex git -You can obtain development source code with -@command{git clone git://git.cypherpunks.su/keks.git}. -You can also use -@code{anongit@@master.git.stargrave.org:cypherpunks.su/keks.git}, -@code{anongit@@slave.git.stargrave.org:cypherpunks.su/keks.git}, -@code{anongit@@master.git.cypherpunks.su:cypherpunks.su/keks.git}, -@code{anongit@@slave.git.cypherpunks.su:cypherpunks.su/keks.git}, -@url{git://git.stargrave.org/keks.git}, -@url{git://y.git.stargrave.org/keks.git}, -@url{git://y.git.cypherpunks.su/keks.git} URLs instead. - -Also there is @url{https://yggdrasil-network.github.io/, Yggdrasil} -accessible address: @url{http://y.www.keks.cypherpunks.su/}. diff --git a/spec/mk-html b/spec/mk-html index 3df0d37..44ea4ce 100755 --- a/spec/mk-html +++ b/spec/mk-html @@ -1,13 +1,8 @@ #!/bin/sh -e +# git://git.stargrave.org/zk.git is used html=spec.html -makeinfo --html \ - --set-customization-variable SECTION_NAME_IN_TITLE=1 \ - --set-customization-variable TREE_TRANSFORMATIONS=complete_tree_nodes_menus \ - --set-customization-variable ASCII_PUNCTUATION=1 \ - --set-customization-variable NO_CSS=1 \ - --set-customization-variable FORMAT_MENU=menu \ - --set-customization-variable DATE_IN_HEADER=1 \ - --output $html index.texi +ZK_DO_LINKS=0 ZK_DO_BACKS=0 zk htmls $html +perl -i -npe 's#^.*$#<title>KEKS#' $html/index.html find $html -type d -exec chmod 755 {} + find $html -type f -exec chmod 644 {} + diff --git a/spec/mk-info b/spec/mk-info deleted file mode 100755 index e762f16..0000000 --- a/spec/mk-info +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/sh -e - -makeinfo \ - --set-customization-variable SECTION_NAME_IN_TITLE=1 \ - --set-customization-variable TREE_TRANSFORMATIONS=complete_tree_nodes_menus \ - --set-customization-variable ASCII_PUNCTUATION=1 \ - --output spec.info index.texi diff --git a/spec/schema/cmds b/spec/schema/cmds new file mode 100644 index 0000000..259bd41 --- /dev/null +++ b/spec/schema/cmds @@ -0,0 +1,135 @@ +Data structure validation commands are grouped in so-called map of +schemas. Map's key is schema name. Schema's value is a list of commands. +Each command is a list of string-encoded words (with several exceptions). +First element of the command's list is a command name. Possible +following elements are command-specific. + +Here is full list of structure validation commands, that should be +generated from higher level schema descriptions. Here comes human +readable command name, its string byte-code and possible arguments. + +TAKE | "." | args: k + Take/choose the value of the "k" key in the map, if "k" is a string. + If "k" is integer, then choose the k-th value in a list. + If "k" equals to ".", then choose the element you are currently in. + Command never fails, but key can be non-existent. + +EXISTS | "E" + Assure that chosen element exists. + +!EXISTS | "!E" + Assure that chosen element does not exist. + +EACH | "*" + Execute the next command against every element of the chosen + (if it exists) list, or every value of the map. + +TYPE | "T" | args: T0 [T1 ...] + Check that chosen (if it exists) element's type is in (T0, T1 ...) + set. Possible types: BIN, BLOB, BOOL, HEXLET, INT, LIST, MAGIC, MAP, + NIL, STR, TAI. + +GT | ">" | args: n + Check that chosen (if it exists) integer value is greater than "n". + If chosen value is either list or map, then check their length. + If the value is a string, then check its length. + +LT | "<" | args: n + Same as ">", but check that value is less than "n". + +SCHEMA | "S" | args: s + Check chosen (if it exists) element against schema named "s". + +TIMEPREC | "TP" | args: p + Check that chosen (if it exists) element, of time type, has value of + maximal specified time precision. "p" is integer with following + possible values: + 0: only full seconds allowed, no parts; + 3: only up to milliseconds; + 6: only up to microseconds; + 9: only up to nanoseconds; + 12: only up to picoseconds; + 15: only up to femtoseconds; + 18: up to attoseconds; + +UTC | "UTC" + Check that chosen (if it exists) element, of time type, + can be converted to UTC. + +EQ | "=" | args: v + Check that chosen (if it exists) element's value equals to binary + string "v". + +For example let's check "our" structure, described in CDDL as: + + ai = text .gt 0 + fpr = bytes .size 32 + our = {a: ai, v: bytes/text, fpr: fpr, ?comment: text} + +"a", "v", "fpr" fields are required ones. "v" has two allowable types. +"comment" is optional, but typed. And "fpr" has fixed length. +Corresponding schema can be: + + {"our": [ + [".", "a"], + ["E"], + [".", "a"], + ["T", "STR"], + [".", "a"], + [">", 0], + + [".", "v"], + ["E"], + [".", "v"], + ["T", "BIN", "STR"], + + [".", "fpr"], + ["E"], + [".", "fpr"], + ["T", "BIN"], + [".", "fpr"], + [">", 31], + [".", "fpr"], + ["<", 33], + + [".", "comment"], + ["T", "STR"], + ]} + +Here is example with multiple schemas: + + latitude = -90..90 + longitude = -180..180 + where = [latitude, longitude] + wheres = [+ where] + + { + "where": [ + [".", "."], + ["T", "LIST"], + [".", "."], + [">", 1], + [".", "."], + ["<", 3], + [".", "."], + ["*"], + [".", "INT"], + [".", 0], + [">", -91], + [".", 0], + ["<", 91], + [".", 1], + [">", -181], + [".", 1], + ["<", 181], + ], + "wheres": [ + [".", "."], + ["T", "LIST"], + [".", "."], + [">", 0], + [".", "."], + ["*"], + ["S", "where"], + ], + } diff --git a/spec/schema/cmds.texi b/spec/schema/cmds.texi deleted file mode 100644 index 6013bab..0000000 --- a/spec/schema/cmds.texi +++ /dev/null @@ -1,153 +0,0 @@ -@node SchemaCmds -@cindex schema commands -@nodedescription Schema commands -@section Schema commands - -Data structure validation commands are grouped in so-called map of -schemas. Map's key is schema name. Schema's value is a list of commands. -Each command is a list of string-encoded words (with several exceptions). -First element of the command's list is a command name. Possible -following elements are command-specific. - -Here is full list of structure validation commands, that should be -generated from higher level schema descriptions. - -@multitable @columnfractions .20 .10 .20 .50 -@headitem name @tab encoded @tab args @tab description - -@item TAKE @tab @code{.} @tab k @tab -Take/choose the value of the @code{k} key in the map, if @code{k} is a -string. If @code{k} is integer, then choose the k-th value in a list. If -@code{k} equals to @code{.}, then choose the element you are currently -in. Command never fails, but key can be non-existent. - -@item EXISTS @tab @code{E} @tab @tab -Assure that chosen element exists. - -@item !EXISTS @tab @code{!E} @tab @tab -Assure that chosen element does not exist. - -@item EACH @tab @code{*} @tab @tab -Execute the next command against every element of the chosen (if it -exists) list, or every value of the map. - -@item TYPE @tab @code{T} @tab T0 [T1 ...] @tab -Check that chosen (if it exists) element's type is in (T0, T1 ...) set. -Possible types: BIN, BLOB, BOOL, HEXLET, INT, LIST, MAGIC, MAP, NIL, STR, TAI. - -@item GT @tab @code{>} @tab n @tab -Check that chosen (if it exists) integer value is greater than @code{n}. -If chosen value is either list or map, then check their length. -If the value is a string, then check its length. - -@item LT @tab @code{<} @tab n @tab -Same as @code{>}, but check that value is less than @code{n}. - -@item SCHEMA @tab @code{S} @tab s @tab -Check chosen (if it exists) element against schema named @code{s}. - -@item TIMEPREC @tab @code{TP} @tab p @tab -Check that chosen (if it exists) element, of time type, has value of -maximal specified time precision. @code{p} is integer with following -possible values: - @itemize - @item 0 -- only full seconds allowed, no parts; - @item 3 -- only up to milliseconds; - @item 6 -- only up to microseconds; - @item 9 -- only up to nanoseconds; - @item 12 -- only up to picoseconds; - @item 15 -- only up to femtoseconds; - @item 18 -- up to attoseconds; - @end itemize - -@item UTC @tab @code{UTC} @tab @tab -Check that chosen (if it exists) element, of time type, can be converted -to UTC. - -@item EQ @tab @code{=} @tab v @tab -Check that chosen (if it exists) element's value equals to binary string -@code{v}. - -@end multitable - -For example let's check "our" structure, described in CDDL as: - -@verbatim -ai = text .gt 0 -fpr = bytes .size 32 -our = {a: ai, v: bytes/text, fpr: fpr, ?comment: text} -@end verbatim - -@code{a}, @code{v}, @code{fpr} fields are required ones. @code{v} has -two allowable types. @code{comment} is optional, but typed. And -@code{fpr} has fixed length. Corresponding schema can be: - -@verbatim -{"our": [ - [".", "a"], - ["E"], - [".", "a"], - ["T", "STR"], - [".", "a"], - [">", 0], - - [".", "v"], - ["E"], - [".", "v"], - ["T", "BIN", "STR"], - - [".", "fpr"], - ["E"], - [".", "fpr"], - ["T", "BIN"], - [".", "fpr"], - [">", 31], - [".", "fpr"], - ["<", 33], - - [".", "comment"], - ["T", "STR"], -]} -@end verbatim - -Here is example with multiple schemas: - -@verbatim -latitude = -90..90 -longitude = -180..180 -where = [latitude, longitude] -wheres = [+ where] -@end verbatim - -@verbatim -{ - "where": [ - [".", "."], - ["T", "LIST"], - [".", "."], - [">", 1], - [".", "."], - ["<", 3], - [".", "."], - ["*"], - [".", "INT"], - [".", 0], - [">", -91], - [".", 0], - ["<", 91], - [".", 1], - [">", -181], - [".", 1], - ["<", 181], - ], - "wheres": [ - [".", "."], - ["T", "LIST"], - [".", "."], - [">", 0], - [".", "."], - ["*"], - ["S", "where"], - ], -} -@end verbatim diff --git a/spec/schema/index.texi b/spec/schema/index similarity index 75% rename from spec/schema/index.texi rename to spec/schema/index index 3f515f8..75e914f 100644 --- a/spec/schema/index.texi +++ b/spec/schema/index @@ -1,10 +1,3 @@ -@node Schema -@cindex schema -@cindex structure validation -@cindex data schema -@nodedescription Structure validation against schemas -@unnumbered Schemas - Although KEKS can be decoded without any schema definition/specification, data structures are likely to be checked against some kind of the schema. Here is suggestion (not a requirement!) to use relatively simple data @@ -25,10 +18,5 @@ conveniently parsed even in C-language. Let's use KEKS format itself for the serialised validation steps! And generate them from higher level language/code, convenient for humans. -@verbatim -Tcl-schema -> keks-encode(validation-commands) - validate(keks-decode(validation-commands), keks-decode(data)) -@end verbatim - -@include schema/cmds.texi -@include schema/tcl.texi + cmds = KEKS-Encode([schema/tcl] -> [schema/cmds]) + validate(KEKS-Decode(cmds), KEKS-Decode(data)) diff --git a/spec/schema/tcl b/spec/schema/tcl new file mode 100644 index 0000000..a2f37cb --- /dev/null +++ b/spec/schema/tcl @@ -0,0 +1,59 @@ +Validation commands are pretty low-level and are inconvenient to write +by hand, at least because of huge quantity of TAKEs. +tcl/schema.tcl utility gives ability to convert much more nicer schemas +written on Tcl language to the KEKS-encoded commands. We call that +Tcl-written schemas KEKS/Schema. + +Example with "our" structure ([schema/cmds]) can be written as: + + ai {{field . {str} >0}} + fpr {{field . {bin} len=32}} + our { + {field a {with ai}} + {field v {bin str}} + {field fpr {with fpr}} + {field comment {str} optional} + } + +and [cm/pub/] as: +[schemas/pub.tcl] +[schemas/fpr.tcl] +[schemas/pub-load.tcl] +[schemas/pub-sig-tbs.tcl] + +schema.tcl calls "schemas {s0 cmds0 s1 cmds1 ...}" +commands to produce an encoded map with "cmds*" commands for +"s*" schemas. There is "field" command that helps creation of +commands related to the field. + +Its first argument is either field's name in the map, or list's index or +dot, meaning the self-structure itself. + +Second argument is a list of allowable types, written in lowercase. If +that list consists of "with S", then SCHEMA command will be called +instead of TYPE checking. If list consists of "set", then it is +checked to be a MAP with EACH value of NIL. + +All other arguments are optional. + +By default, if no "optional" argument is specified, then explicit +EXISTS check is called for the field. If "!exists" argument is +specified, then it is explicitly checked to be non-existent and +you can specify empty list of types in second argument. + +">n" and "0" assures that either list/map or strings are not +empty. "len=n" checks the exact length. "=v" checks that given +element has specified string/binary value (use "len=" for integers). + +"prec=p" issues TIMEPREC command, but instead of specifying the raw +integer values, you choose one of: s, ms, us, ns, ps, fs, as. "utc" +issues UTC command. + +"of s" argument issues checking of EACH element of the list or map +against the specified schema, or against specified type if "s" is a +known type. + +"schema-include filename.tcl" command used instead of "field" allows +inclusion of the specified file with the path relative to given schema +file. diff --git a/spec/schema/tcl.texi b/spec/schema/tcl.texi deleted file mode 100644 index 886a1b7..0000000 --- a/spec/schema/tcl.texi +++ /dev/null @@ -1,68 +0,0 @@ -@node SchemaTcl -@cindex Tcl schema -@cindex KEKS/Schema -@nodedescription Tcl-written schemas -@section Tcl schema - -Validation commands are pretty low-level and are inconvenient to write -by hand, at least because of huge quantity of TAKEs. -@command{tcl/schema.tcl} utility gives ability to convert much more -nicer schemas written on Tcl language to the KEKS-encoded commands. We -call that Tcl-written schemas KEKS/Schema. - -Example with "our" structure can be written as: - -@verbatim -ai {{field . {str} >0}} -fpr {{field . {bin} len=32}} -our { - {field a {with ai}} - {field v {bin str}} - {field fpr {with fpr}} - {field comment {str} optional} -} -@end verbatim - -and @ref{cm-pub, cm/pub} as: - -@verbatiminclude ../tcl/schemas/pub.tcl -@verbatiminclude ../tcl/schemas/fpr.tcl -@verbatiminclude ../tcl/schemas/pub-load.tcl -@verbatiminclude ../tcl/schemas/pub-sig-tbs.tcl - -@command{schema.tcl} calls @code{schemas@{s0 cmds0 s1 cmds1 ...@}} -commands to produce an encoded map with @code{cmds*} commands for -@code{s*} schemas. There is @code{field} command that helps creation of -commands related to the field. - -Its first argument is either field's name in the map, or list's index or -dot, meaning the self-structure itself. - -Second argument is a list of allowable types, written in lowercase. If -that list consists of @code{with S}, then SCHEMA command will be called -instead of TYPE checking. If list consists of @code{set}, then it is -checked to be a MAP with EACH value of NIL. - -All other arguments are optional. - -By default, if no @code{optional} argument is specified, then explicit -EXISTS check is called for the field. If @code{!exists} argument is -specified, then it is explicitly checked to be non-existent and you can -specify empty list of types in second argument. - -@code{>n} and @code{0} assures that either list/map or strings are not -empty. @code{len=n} checks the exact length. @code{=v} checks that given -element has specified string/binary value (use @code{len=} for integers). - -@code{prec=p} issues TIMEPREC command, but instead of specifying the raw -integer values, you choose one of: s, ms, us, ns, ps, fs, as. @code{utc} -issues UTC command. - -@code{of s} argument issues checking of EACH element of the list or map -against the specified schema, or against specified type if @code{s} is a -known type. - -@code{schema-include filename.tcl} command used instead of @code{field} -allows inclusion of the specified file with the path relative to given -schema file. diff --git a/spec/schemas b/spec/schemas new file mode 120000 index 0000000..fd03250 --- /dev/null +++ b/spec/schemas @@ -0,0 +1 @@ +../tcl/schemas \ No newline at end of file diff --git a/spec/thanks.texi b/spec/thanks.texi deleted file mode 100644 index 3184088..0000000 --- a/spec/thanks.texi +++ /dev/null @@ -1,11 +0,0 @@ -@node Thanks -@cindex thanks -@unnumbered Thanks - -@itemize - -@item Sergey Mayorov for his valuable consultation and suggestions. -@item Anton Rudenko for the initial unittests of the Python and Go codec - implementation. - -@end itemize