Cramming scrapscript into msgpack

msgpack is a lovely little serialization format. As a
JSON replacement, it saves bandwidth while preserving native language features
(e.g. tuples, records, objects, dates).

By leveraging msgpack, every scrapscript
program/expression can be crammed into “flat” scraps.

Because flat scraps encode a full programming language, you can send unambiguous
programs/queries across a wire:

"Joy" |> (name -> ok { name })

-- EXPR       0xc7201f
-- "Joy"      0xa34a6f79
-- name=name  0x81a46e616d65c704786e616d65
-- REC        0xc17b7d
-- OK         0xc10006
-- name       0xc704786e616d65
-- ->         0xc12d3e
-- |>         0xc17c3e

…and represent its evaluated result:

ok { name = "Joy" }

-- EXPR        0xc7101f
-- name="Joy"  0x81a46e616d65a34a6f79
-- REC         0xc17b7d
-- OK          0xc10006

This first-class serialization format confers non-obvious benefits:

• a compact format affords a canonical format for hashing
• when computation is cheaper than bandwidth, you can send programs instead of
  sending data
• both code & data can share the same tooling for compression, analysis,
  optimization, etc.
• cheaper to store programs on disk (or KV store) and load into memory
• it’s a simple/convenient cross-platform compilation target
• with types, encoders/decoders can be magically inferred/generated/typechecked
  at compile-time
• msgpack is supported by 50+ languages/environments; all
  msgpack messages are already scraps, and scrap-specific extensions can be
  incrementally implemented in each host language

Literals

Expressions

Flat scrap expressions are written in
reverse polish notation:

Within the expression, prepend each 2-byte operator with 0xc1.

Here are many of the normal computer operations you know and love:

Fancier operators are also available:

To keep things tiny, we also cram common expressions into the operator space:

References

Flat scraps distinguish between
scope variables (76),
native functions (72), and
expression hashes (24):

Structures

A scrapscript list corresponds to a plain msgpack array:

The other structures are represented as
maps:

Because maps are ambiguously typed, use special operators to clarify the
intended structure:

Evaluation

Astute observers may have noticed some strange operators like
where and
await:

To build a basic interpreter, push values on a stack and perform operations
in-place. Bind function arguments and environment variables into a scoped map
that can be copied from.

And someday soon, we’ll all be able to do irresponsible stuff like this:

// javascript
const fib = scrap.decode(await fetch("yard.scrap.land/tom/fib"));
console.log(fib(40)); // 102334155

// javascript
import fib from "yard.scrap.land/tom/fib";
console.log(fib(40)); // 102334155

// golang
import (
  "fmt"
  "yard.scrap.land/tom"
)
func main() {
  fmt.Println(tom.Fib(40)) // 102334155
}

# python
async with aiohttp.ClientSession() as sess:
    async with sess.get("yard.scrap.land/tom/fib") as res:
        fib = scrap.decode(await res.read())
        print(fib(40)) # 102334155

// javascript
const fib = n => {
  let phi = (1 + Math.sqrt(5)) / 2;
  let asymp = Math.pow(phi, n) / Math.sqrt(5);
  return Math.round(asymp);
};
await fetch("api.example.com", {
  headers: { "Content-Type": "application/scrap" },
  method: "POST",
  body: scrap.encode(fib),
});