Rename PUFFS to WUFFS, part 10 of N

This commit is contained in:
Nigel Tao committed 2017-11-30 18:15:36 +11:00
1 parent 8e3b5cce5e
commit b9fd20bfe4
6 files changed
+141 -141

No files matched your search

+54 -54
View File
@@ -1,16 +1,16 @@
# Wrangling Untrusted File Formats Safely
([Formerly known as
PUFFS](https://groups.google.com/d/topic/puffslang/cSrH-s7UqwA/discussion):
Puffs](https://groups.google.com/d/topic/puffslang/cSrH-s7UqwA/discussion):
Parsing Untrusted File Formats Safely).
Puffs is a domain-specific language and library for parsing untrusted file
Wuffs is a domain-specific language and library for parsing untrusted file
formats safely. Examples of such file formats include images, audio, video,
fonts and compressed archives.
Unlike the C programming language, Puffs is safe with respect to buffer
Unlike the C programming language, Wuffs is safe with respect to buffer
overflows, integer arithmetic overflows and null pointer dereferences. The key
difference between Puffs and other memory-safe languages is that all such
difference between Wuffs and other memory-safe languages is that all such
checks are done at compile time, not at run time. *If it compiles, it is safe*,
with respect to those three bug classes.
@@ -20,37 +20,37 @@ are used. This includes very large C/C++ products, such as popular web browsers
and operating systems (using that term to include desktop and mobile user
interfaces, not just the kernel).
The trade-off in aiming for both safety and speed is that Puffs programs take
The trade-off in aiming for both safety and speed is that Wuffs programs take
longer for a programmer to write, as they have to explicitly annotate their
programs with proofs of safety. A statement like `x += 1` unsurprisingly means
to increment the variable `x` by `1`. However, in Puffs, such a statement is a
to increment the variable `x` by `1`. However, in Wuffs, such a statement is a
compile time error unless the compiler can also prove that `x` is not the
maximal value of `x`'s type (e.g. `x` is not `255` if `x` is a `u8`), as the
increment would otherwise overflow. Similarly, an integer arithmetic expression
like `x / y` is a compile time error unless the compiler can also prove that
`y` is not zero.
Puffs is not a general purpose programming language. While technically
possible, it is unlikely that a Puffs compiler would be worth writing in Puffs.
Wuffs is not a general purpose programming language. While technically
possible, it is unlikely that a Wuffs compiler would be worth writing in Wuffs.
## What Does Puffs Code Look Like?
## What Does Wuffs Code Look Like?
The [`std/gif/decode_lzw.puffs`](./std/gif/decode_lzw.puffs) file is a good
The [`std/gif/decode_lzw.wuffs`](./std/gif/decode_lzw.wuffs) file is a good
example. See the "Poking Around" section below for more guidance.
## What Does Compile Time Checking Look Like?
For example, making this one-line edit to the GIF codec leads to a compile time
error. `puffs gen` fails to generate the C code, i.e. fails to compile
(transpile) the Puffs code to C code:
error. `wuffs gen` fails to generate the C code, i.e. fails to compile
(transpile) the Wuffs code to C code:
```diff
diff --git a/std/gif/decode_lzw.puffs b/std/gif/decode_lzw.puffs
diff --git a/std/gif/decode_lzw.wuffs b/std/gif/decode_lzw.wuffs
index f878c5e..f10dcee 100644
--- a/std/gif/decode_lzw.puffs
+++ b/std/gif/decode_lzw.puffs
--- a/std/gif/decode_lzw.wuffs
+++ b/std/gif/decode_lzw.wuffs
@@ -98,7 +98,7 @@ pub func lzw_decoder.decode?(dst ptr buf1, src ptr buf1, src_final bool)() {
in.dst.write?(x:s)
@@ -62,9 +62,9 @@ index f878c5e..f10dcee 100644
```
```bash
$ puffs gen std/gif
$ wuffs gen std/gif
check: expression "(c + 1) as u8" bounds [1..256] is not within bounds [0..255] at
/home/n/go/src/github.com/google/wuffs/std/gif/decode_lzw.puffs:101. Facts:
/home/n/go/src/github.com/google/wuffs/std/gif/decode_lzw.wuffs:101. Facts:
n_bits < 8
c < 256
this.stack[s] == (c as u8)
@@ -75,10 +75,10 @@ In comparison, this two-line edit will compile (but the "does it decode GIF
correctly" tests then fail):
```diff
diff --git a/std/gif/decode_lzw.puffs b/std/gif/decode_lzw.puffs
diff --git a/std/gif/decode_lzw.wuffs b/std/gif/decode_lzw.wuffs
index f878c5e..b43443d 100644
--- a/std/gif/decode_lzw.puffs
+++ b/std/gif/decode_lzw.puffs
--- a/std/gif/decode_lzw.wuffs
+++ b/std/gif/decode_lzw.wuffs
@@ -97,8 +97,8 @@ pub func lzw_decoder.decode?(dst ptr buf1, src ptr buf1, src_final bool)() {
// type checking, bounds checking and code generation for it).
in.dst.write?(x:s)
@@ -92,10 +92,10 @@ index f878c5e..b43443d 100644
```
```bash
$ puffs gen std/gif
$ wuffs gen std/gif
gen wrote: /home/n/go/src/github.com/google/wuffs/gen/c/gif.c
gen unchanged: /home/n/go/src/github.com/google/wuffs/gen/h/gif.h
$ puffs test std/gif
$ wuffs test std/gif
gen unchanged: /home/n/go/src/github.com/google/wuffs/gen/c/gif.c
gen unchanged: /home/n/go/src/github.com/google/wuffs/gen/h/gif.h
test: /home/n/go/src/github.com/google/wuffs/test/c/gif
@@ -125,8 +125,8 @@ excerpts of got (above) versus want (below):
000020: eded 8f91 9191 9090 9090 9090 9191 9191 ................
000030: 9191 9191 9191 9193 93f0 f0f0 f1f1 f2f2 ................
puffs-test-c: some tests failed
puffs test: some tests failed
wuffs-test-c: some tests failed
wuffs test: some tests failed
```
# Background
@@ -186,15 +186,15 @@ The run time cost is small, measured in nanoseconds. But if an image decoding
library has to eat this cost per pixel, and you have a megapixel image, then
nanoseconds become milliseconds, and milliseconds can matter.
In comparison, in Puffs, all bounds checks and arithmetic overflow checks
In comparison, in Wuffs, all bounds checks and arithmetic overflow checks
happen at compile time, with zero run time overhead.
# Getting Started
Puffs code (that is proved safe via explicit assertions) is compiled to C code
Wuffs code (that is proved safe via explicit assertions) is compiled to C code
(with those assertions removed) - it is transpiled. If you are a C/C++
programmer and just want to *use* the C edition of the Puffs standard library,
programmer and just want to *use* the C edition of the Wuffs standard library,
then clone the repository and look at the files in the `gen/c` and `gen/h`
directories. No other software tools are required and there are no library
dependencies, other than C standard library concepts like `<stdint.h>`'s
@@ -213,38 +213,38 @@ gzip or zlib, roughly speaking) without necessarily processing PNG images.
## Getting Deeper
If you want to modify the Puffs standard library, or compile your own Puffs
If you want to modify the Wuffs standard library, or compile your own Wuffs
code, you will need to do a little more work, and will have to install at least
the Go toolchain in order to build the Puffs tools. To run the test suite, you
the Go toolchain in order to build the Wuffs tools. To run the test suite, you
might also have to install C compilers like clang and gcc, as well as C
libraries (and their .h files) like libjpeg and libpng, as some tests compare
that Puffs produces exactly the same output as these other libraries.
that Wuffs produces exactly the same output as these other libraries.
Running `go get -v github.com/google/wuffs/cmd/...` will download and install
the Puffs tools. Change `get` to `install` to re-install those programs without
the Wuffs tools. Change `get` to `install` to re-install those programs without
downloading, e.g. after you've modified their source code, or after a manually
issued `git pull`. The Puffs tools that you'll most often use are `puffsfmt`
(analogous to `clang-format`, `gofmt` or `rustfmt`) and `puffs` (roughly
issued `git pull`. The Wuffs tools that you'll most often use are `wuffsfmt`
(analogous to `clang-format`, `gofmt` or `rustfmt`) and `wuffs` (roughly
analogous to `make`, `go` or `cargo`).
You should now be able to run `puffs test`. If all goes well, you should see
You should now be able to run `wuffs test`. If all goes well, you should see
some output containing the word "PASS" multiple times.
## Poking Around
Feel free to edit the `std/gif/decode_lzw.puffs` file, which implements the GIF
LZW decoder. After editing, run `puffs gen std/gif` or `puffs test std/gif` to
re-generate the C edition of the Puffs standard library's GIF codec, and
Feel free to edit the `std/gif/decode_lzw.wuffs` file, which implements the GIF
LZW decoder. After editing, run `wuffs gen std/gif` or `wuffs test std/gif` to
re-generate the C edition of the Wuffs standard library's GIF codec, and
optionally run its tests.
Try deleting an assert statement and re-running `puffs gen`. The result should
Try deleting an assert statement and re-running `wuffs gen`. The result should
be syntactically valid, but a compile error, as some bounds checks can no
longer be proven.
Find the line `var bits u32`, which declares the bits variable and initializes
it to zero. Try adding `bits -= 1` on a new line of code after it. Again,
`puffs gen` should fail, as the computation can underflow.
`wuffs gen` should fail, as the computation can underflow.
Similarly, replacing the line `var n_bits u32` with `var n_bits u32 = 10`
should fail, as an `n_bits < 8` assertion, a pre-condition, a few lines further
@@ -254,8 +254,8 @@ Similarly, changing the `4095` in `var prev_code u32[..4095]` either higher or
lower should fail.
Try adding `assert false` at various places, which should obviously fail, but
should also cause `puffs gen` to print what facts the compiler can prove at
that point. This can be useful when debugging why Puffs can't prove something
should also cause `wuffs gen` to print what facts the compiler can prove at
that point. This can be useful when debugging why Wuffs can't prove something
you think it should be able to.
@@ -265,14 +265,14 @@ If you've changed any of the tools (i.e. changed any `.go` code), re-run `go
install -v github.com/google/wuffs/cmd/...` and `go test
github.com/google/wuffs/lang/...`.
If you've changed any of the libraries (i.e. changed any `.puffs` code), run
`puffs test` or, ideally, `puffs test -mimic` to also check that Puffs' output
If you've changed any of the libraries (i.e. changed any `.wuffs` code), run
`wuffs test` or, ideally, `wuffs test -mimic` to also check that Wuffs' output
mimics (i.e. exactly matches) other libraries' output, such as giflib for GIF,
libpng for PNG, etc.
If your library change is an optimization, run `puffs bench` or `puffs bench
If your library change is an optimization, run `wuffs bench` or `wuffs bench
-mimic` both before and after your change to quantify the improvement. The
mimic benchmark numbers should't change if you're only changing `.puffs` code,
mimic benchmark numbers should't change if you're only changing `.wuffs` code,
but seeing zero change in those numbers is a sanity check on any unrelated
system variance, such as software updates or virus checkers running in the
background.
@@ -280,30 +280,30 @@ background.
## Directory Layout
- `lang` holds the Go libraries that implement the Puffs language: tokenizer,
AST, parser, renderer, etc. The Puffs tools are written in Go, but as
mentioned above, Puffs transpiles to C code, and Go is not necessarily
involved if all you want is to use the C edition of Puffs.
- `cmd` holds Puffs' command line tools, also written in Go.
- `std` holds the Puffs standard library's code. The initial focus is on
- `lang` holds the Go libraries that implement the Wuffs language: tokenizer,
AST, parser, renderer, etc. The Wuffs tools are written in Go, but as
mentioned above, Wuffs transpiles to C code, and Go is not necessarily
involved if all you want is to use the C edition of Wuffs.
- `cmd` holds Wuffs' command line tools, also written in Go.
- `std` holds the Wuffs standard library's code. The initial focus is on
popular image codecs: BMP, GIF, JPEG, PNG, TIFF and WEBP.
- `gen` holds the transpiled editions of that standard library. The initial
focus is generating C code. Later on, the repository might include generated
Go and Rust code.
- `test` holds the tests for the Puffs standard library.
- `test` holds the tests for the Wuffs standard library.
- `script` holds miscellaneous utility programs.
- `doc` holds documentation.
- `example` holds example programs.
For a guide on how various things work together, the "99ff8e2 Let fields have
default values" commit is an example of adding new Puffs syntax and threading
default values" commit is an example of adding new Wuffs syntax and threading
that all the way through to C code generation and testing.
# Documentation
- [Wuffs the Language](./doc/wuffs-the-language.md)
- Puffs the Library (TODO)
- Wuffs the Library (TODO)
- [Related Work](./doc/related-work.md)
- [Roadmap](./doc/roadmap.md)
+36 -36
View File
@@ -1,9 +1,9 @@
# Benchmarks
*Preliminary* `puffs bench -mimic` summarized throughput numbers for various
*Preliminary* `wuffs bench -mimic` summarized throughput numbers for various
codecs are below. Higher is better.
"Mimic" tests check that Puffs' output mimics (i.e. exactly matches) other
"Mimic" tests check that Wuffs' output mimics (i.e. exactly matches) other
libraries' output. "Mimic" benchmarks give the numbers for those other
libraries, as shipped with the OS. These were measured on a Debian Testing
system as of November 2017, which means these compiler versions:
@@ -27,7 +27,7 @@ The benchmark programs aim to be runnable "out of the box" without any
configuration or installation. For example, to run the `std/flate` benchmarks:
git clone https://github.com/google/wuffs.git
cd puffs/test/c/std
cd wuffs/test/c/std
gcc -O3 flate.c
./a.out -bench
rm a.out
@@ -42,24 +42,24 @@ code change. To install it, first install Go, then run `go get
golang.org/x/perf/cmd/benchstat`.
## puffs bench
## wuffs bench
As mentioned above, individual benchmark programs can be run manually. However,
the canonical way to run the benchmarks (across multiple compilers and multiple
packages like GIF and PNG) is to use the `puffs` command line tool, as it will
also re-generate (transpile) the C code whenever you edit the \*.puffs code.
Running `go install -v github.com/google/wuffs/cmd/...` will install the Puffs
packages like GIF and PNG) is to use the `wuffs` command line tool, as it will
also re-generate (transpile) the C code whenever you edit the \*.wuffs code.
Running `go install -v github.com/google/wuffs/cmd/...` will install the Wuffs
tools. After that, you can say
puffs bench
wuffs bench
or
puffs bench -mimic std/flate
wuffs bench -mimic std/flate
or
puffs bench -ccompilers=gcc -reps=3 -focus=Benchmarkpuffs_gif_lzw std/gif
wuffs bench -ccompilers=gcc -reps=3 -focus=Benchmarkwuffs_gif_lzw std/gif
## CPU Scaling
@@ -82,21 +82,21 @@ output.
name speed
puffs_adler32_10k/clang 2.42GB/s ± 3%
puffs_adler32_100k/clang 2.42GB/s ± 3%
puffs_flate_decode_1k/clang 133MB/s ± 3%
puffs_flate_decode_10k/clang 199MB/s ± 3%
puffs_flate_decode_100k/clang 229MB/s ± 4%
puffs_zlib_decode_10k/clang 185MB/s ± 1%
puffs_zlib_decode_100k/clang 210MB/s ± 3%
wuffs_adler32_10k/clang 2.42GB/s ± 3%
wuffs_adler32_100k/clang 2.42GB/s ± 3%
wuffs_flate_decode_1k/clang 133MB/s ± 3%
wuffs_flate_decode_10k/clang 199MB/s ± 3%
wuffs_flate_decode_100k/clang 229MB/s ± 4%
wuffs_zlib_decode_10k/clang 185MB/s ± 1%
wuffs_zlib_decode_100k/clang 210MB/s ± 3%
puffs_adler32_10k/gcc 3.22GB/s ± 1%
puffs_adler32_100k/gcc 3.23GB/s ± 0%
puffs_flate_decode_1k/gcc 152MB/s ± 1%
puffs_flate_decode_10k/gcc 250MB/s ± 1%
puffs_flate_decode_100k/gcc 298MB/s ± 1%
puffs_zlib_decode_10k/gcc 238MB/s ± 2%
puffs_zlib_decode_100k/gcc 270MB/s ± 2%
wuffs_adler32_10k/gcc 3.22GB/s ± 1%
wuffs_adler32_100k/gcc 3.23GB/s ± 0%
wuffs_flate_decode_1k/gcc 152MB/s ± 1%
wuffs_flate_decode_10k/gcc 250MB/s ± 1%
wuffs_flate_decode_100k/gcc 298MB/s ± 1%
wuffs_zlib_decode_10k/gcc 238MB/s ± 2%
wuffs_zlib_decode_100k/gcc 270MB/s ± 2%
mimic_adler32_10k 3.00GB/s ± 1%
mimic_adler32_100k 2.91GB/s ± 2%
@@ -116,15 +116,15 @@ is 1 byte per pixel.
name speed
puffs_gif_decode_1k/clang 346MB/s ± 1%
puffs_gif_decode_10k/clang 137MB/s ± 0%
puffs_gif_decode_100k/clang 118MB/s ± 0%
puffs_gif_decode_1000k/clang 120MB/s ± 0%
wuffs_gif_decode_1k/clang 346MB/s ± 1%
wuffs_gif_decode_10k/clang 137MB/s ± 0%
wuffs_gif_decode_100k/clang 118MB/s ± 0%
wuffs_gif_decode_1000k/clang 120MB/s ± 0%
puffs_gif_decode_1k/gcc 399MB/s ± 1%
puffs_gif_decode_10k/gcc 141MB/s ± 0%
puffs_gif_decode_100k/gcc 128MB/s ± 0%
puffs_gif_decode_1000k/gcc 131MB/s ± 0%
wuffs_gif_decode_1k/gcc 399MB/s ± 1%
wuffs_gif_decode_10k/gcc 141MB/s ± 0%
wuffs_gif_decode_100k/gcc 128MB/s ± 0%
wuffs_gif_decode_1000k/gcc 131MB/s ± 0%
mimic_gif_decode_1k 147MB/s ± 0%
mimic_gif_decode_10k 90.7MB/s ± 0%
@@ -134,10 +134,10 @@ is 1 byte per pixel.
TODO: investigate why gcc 4.8 (Ubuntu Trusty) seems to generate faster code
than gcc 7.2 (Debian Testing):
puffs_gif_decode_1k/gcc 411MB/s ± 2%
puffs_gif_decode_10k/gcc 162MB/s ± 0%
puffs_gif_decode_100k/gcc 138MB/s ± 0%
puffs_gif_decode_1000k/gcc 141MB/s ± 0%
wuffs_gif_decode_1k/gcc 411MB/s ± 2%
wuffs_gif_decode_10k/gcc 162MB/s ± 0%
wuffs_gif_decode_100k/gcc 138MB/s ± 0%
wuffs_gif_decode_1000k/gcc 141MB/s ± 0%
---
+6 -6
View File
@@ -1,25 +1,25 @@
# Binary Size
*Preliminary* measurements of `puffs genlib` libraries' binary size on x86_64
*Preliminary* measurements of `wuffs genlib` libraries' binary size on x86_64
are below. Lower is better.
clang-dynamic:
-rwxr-xr-x 1 nigeltao eng 38352 Nov 9 22:59 libpuffs.so
-rwxr-xr-x 1 nigeltao eng 38352 Nov 9 22:59 libwuffs.so
-rw-r--r-- 1 nigeltao eng 22624 Nov 9 22:59 std-flate.lo
-rw-r--r-- 1 nigeltao eng 9456 Nov 9 22:59 std-gif.lo
clang-static:
-rw-r----- 1 nigeltao eng 32966 Nov 9 22:59 libpuffs.a
-rw-r----- 1 nigeltao eng 32966 Nov 9 22:59 libwuffs.a
-rw-r--r-- 1 nigeltao eng 22648 Nov 9 22:59 std-flate.o
-rw-r--r-- 1 nigeltao eng 9480 Nov 9 22:59 std-gif.o
gcc-dynamic:
-rwxr-xr-x 1 nigeltao eng 42504 Nov 9 22:59 libpuffs.so
-rwxr-xr-x 1 nigeltao eng 42504 Nov 9 22:59 libwuffs.so
-rw-r--r-- 1 nigeltao eng 24776 Nov 9 22:59 std-flate.lo
-rw-r--r-- 1 nigeltao eng 13520 Nov 9 22:59 std-gif.lo
gcc-static:
-rw-r----- 1 nigeltao eng 39102 Nov 9 22:59 libpuffs.a
-rw-r----- 1 nigeltao eng 39102 Nov 9 22:59 libwuffs.a
-rw-r--r-- 1 nigeltao eng 24728 Nov 9 22:59 std-flate.o
-rw-r--r-- 1 nigeltao eng 13536 Nov 9 22:59 std-gif.o
@@ -29,7 +29,7 @@ are below. Lower is better.
Below are some standard C libraries shipped as part of Debian Testing as of
November 2017. The numbers aren't directly comparable, as these libraries have
a richer API, especially in providing an encoder and not just a decoder. Still,
it is a reference point for e.g. Puffs flate vs libz and Puffs gif vs libgif.
it is a reference point for e.g. Wuffs flate vs libz and Wuffs gif vs libgif.
dynamic:
-rw-r--r-- 1 root root 105088 Jan 29 2017 /lib/x86_64-linux-gnu/libz.so.1.2.8
+1 -1
View File
@@ -1,7 +1,7 @@
# Compatibility
TODO: get a decently large corpus of test images from a web crawl. Ensure that
Puffs produces 100% identical decodings to the suite of giflib, libjpeg,
Wuffs produces 100% identical decodings to the suite of giflib, libjpeg,
libpng, etc.
TODO: also trawl through Go's bug tracker for "this image failed to load".
+27 -27
View File
@@ -1,34 +1,34 @@
# Related Work
Puffs is an industrial project, not an academic project, and this Related Work
Wuffs is an industrial project, not an academic project, and this Related Work
section is not as extensive or as rigorous as would be found in an academic
thesis or journal article. The underlying goal is usefulness, not novelty. For
Puffs' users, it'd perfectly fine to have multiple implementations of a good
Wuffs' users, it'd perfectly fine to have multiple implementations of a good
idea, even if the later ones wouldn't earn a Ph. D.
A number of other projects are listed below. Puffs is not exactly like any of
A number of other projects are listed below. Wuffs is not exactly like any of
them, for one reason or another. We're not claiming that "use a state of the
art theorem prover" or "write it in Rust" are unworkable approaches. They're
just different approaches, with different trade-offs.
For example, Puffs is primarily concerned about *safety* and *performance*.
For example, Wuffs is primarily concerned about *safety* and *performance*.
Formal *correctness*, such as being able to mathematically express that "when
this function returns, that array's elements will be sorted in increasing
order", is less of a concern for Puffs the Language than for other projects.
order", is less of a concern for Wuffs the Language than for other projects.
This is especially as higher level concerns like "correctly implements the JPEG
specification" are less amenable to formalization than foundational concepts
like searching and sorting.
For Puffs the Library, confidence about correctness is instead based on
For Wuffs the Library, confidence about correctness is instead based on
testing, both unit testing and for e.g. media codecs, corpora of test files. A
small corpus is included in the Puffs source code repository. Other, much
small corpus is included in the Wuffs source code repository. Other, much
larger corpora (e.g. based on a sample of a web crawl), are not included for
download size and copyright reasons.
## Static Checkers
Puffs is a language by itself, integrated with a compiler, not something
Wuffs is a language by itself, integrated with a compiler, not something
embedded in the comments of another language's program, supported by a separate
program checker, as the bounds check elimination affects the generated code.
@@ -37,18 +37,18 @@ similar goal of safety, but aims to be backwards compatible with C, including
the idiomatic ways in which C code plays fast and loose with pointers and
arrays, and implicitly inserting run time checks while maintaining existing
data layout (e.g. the sizeof a struct type) for interoperability with other
systems. This is a much harder problem than Puffs' approach of a new, simple,
systems. This is a much harder problem than Wuffs' approach of a new, simple,
domain-specific programming language.
Extended Static Checker for Java (ESC/Java) and its successor
[OpenJML](http://www.openjml.org/), which obviously target the Java programming
language, similarly has to analyze a language that is more complicated that
Puffs. Java is also not commonly used for writing e.g. low level image codecs,
Wuffs. Java is also not commonly used for writing e.g. low level image codecs,
as the language lacks unsigned integer types, it is garbage collected and
idiomatic code often allocates.
Similarly, [Whiley](http://whiley.org/) is a programming language with extended
static checking. In contrast to Puffs, its `int` type uses unbounded
static checking. In contrast to Wuffs, its `int` type uses unbounded
arithmetic, instead of e.g. 32-bit twos-complement representation. That can be
easier for theorem provers to reason about, but this can have a performance
impact when such integers are used in inner loops. In theory, a compiler could
@@ -57,7 +57,7 @@ potentially 'big integer', but even so, it may improve performance to
explicitly use a `uint32` here even when a `uint8` would do.
[Spark ADA](http://libre.adacore.com/tools/spark-gpl-edition/) targets a subset
of the Ada programming language, which again is more complicated than Puffs. It
of the Ada programming language, which again is more complicated than Wuffs. It
also allows for pluggable theorem provers such as Z3, similar to
[Dafny](http://research.microsoft.com/dafny). This can increase programmer
productivity in that it can take less effort to prove more programs safe, but
@@ -67,19 +67,19 @@ trusted computing base. It's also not obvious up front what effect different
theorem provers, and their different heuristics, will have on compile times or
proof times.
In constrast, Puffs' proof system is fast (and dumb) instead of smart (and
In constrast, Wuffs' proof system is fast (and dumb) instead of smart (and
slow). For example, [Vale (Verified Assembly Language for
Everest)](https://github.com/project-everest/vale) is a promising approach that
uses Dafny to verify implementations of cryptographic algorithms. However,
["Vale: Verifying High-Performance Cryptographic Assembly
Code"](http://www.andrew.cmu.edu/user/bparno/papers/vale.pdf) reports in Table
1 that verification times are measured in *minutes*. Puffs aims for
1 that verification times are measured in *minutes*. Wuffs aims for
*sub-second* compilation times, including proofs of (memory) safety. Figure 15
in that paper suggests that Vale's verification times can grow exponentially in
some cases, and also goes on to say, "Dafny/Z3 fails to verify the AES key
inversion for 6 unrolled iterations and 9 unrolled iterations, indicating that
SMT solvers like Z3 are still occasionally unpredictable". Of course, Vale is
also tackling a much harder problem than Puffs: proving correctness, not just
also tackling a much harder problem than Wuffs: proving correctness, not just
safety.
Once again, we're not claiming that these other approaches are unworkable, or
@@ -91,17 +91,17 @@ not useful, just different, with different trade-offs.
Simpler languages are easier to prove things about. Macros, inheritance,
closures, generics, operator overloading, goto's and built-in container types
are all useful features, but as mentioned in the top-level
[README](../README.md), Puffs is not a general purpose programming language.
[README](../README.md), Wuffs is not a general purpose programming language.
Instead, its focus is on compile-time provable memory safety, with C-like
performance, for CPU-intensive file format decoders.
Nonetheless, Puffs is an imperative language, not a functional language,
Nonetheless, Wuffs is an imperative language, not a functional language,
despite the long history of functional languages and provability. Inner loop
performance usage matters, and it is easier to match the optimization
techniques of incumbent C libraries with a C-like language than with a
functional language. Compared to something like
[F\*](https://www.fstar-lang.org/), [Idris](https://www.idris-lang.org/) or
[Liquid Haskell](https://ucsd-progsys.github.io/liquidhaskell-blog/), Puffs has
[Liquid Haskell](https://ucsd-progsys.github.io/liquidhaskell-blog/), Wuffs has
no garbage collector overhead, as it has no garbage collector.
Memory usage also matters, again considering per-pixel costs can be multiplied
@@ -110,8 +110,8 @@ u8 values (or one u32), not as four naturally-sized-for-the-CPU integers or
four 'big integers'.
[F\* / KreMLin](https://github.com/FStarLang/kremlin) is a subset of F\* that
is similar in some sense to Puffs, in that it transpiles to C and cares about
both safety and performance. Unlike Puffs, it is a functional language (with
is similar in some sense to Wuffs, in that it transpiles to C and cares about
both safety and performance. Unlike Wuffs, it is a functional language (with
dependent types), not an imperative one (with a simpler type system), and uses
a sophisticated theorem prover like Z3, with the same trade-offs as discussed
above for Spark ADA and Dafny. It also tackles a more complicated problem, in
@@ -122,7 +122,7 @@ Low-Level Programming Embedded in F\*"](https://arxiv.org/pdf/1703.00053.pdf).
[Cryptol](https://github.com/GaloisInc/cryptol) is another project that, like
Everest (including F\* / KreMLin and its HACL\* sub-project, and Dafny / Vale),
focuses on cryptographic algorithms, rather than Puffs' focus on file formats,
focuses on cryptographic algorithms, rather than Wuffs' focus on file formats,
and also relies on a sophisticated theorem prover like Z3.
Once again, we're not claiming that these other approaches are unworkable, or
@@ -131,9 +131,9 @@ not useful, just different, with different trade-offs.
## Why Not Rust?
Puffs transpiles to standard C99 code with no dependencies, and should run
Wuffs transpiles to standard C99 code with no dependencies, and should run
anywhere that has a C99 compliant C compiler. An existing C/C++ project, such
as the Chromium web browser, can choose to replace e.g. libpng with Puffs PNG,
as the Chromium web browser, can choose to replace e.g. libpng with Wuffs PNG,
without needing any additional toolchains. Sure, languages like D and Rust have
great binary-level interop with C code, and Mozilla are reporting progress with
parsing media formats in Rust, but it's still an operational hurdle to grow a
@@ -141,18 +141,18 @@ project's build process and to assess build times and binary sizes.
[Nom](https://github.com/Geal/nom) is a parser combinator library in Rust,
described in ["Writing parsers like it is
2017"](http://spw17.langsec.org/papers/chifflier-parsing-in-2017.pdf). Puffs
differs from nom by itself, in that Puffs is an end to end implementation, not
2017"](http://spw17.langsec.org/papers/chifflier-parsing-in-2017.pdf). Wuffs
differs from nom by itself, in that Wuffs is an end to end implementation, not
limited to that part of a file format that is easily expressible as a formal
grammar. In particular, it also handles entropy encodings such as LZW (for
GIF), ZLIB (for PNG) and Huffman (for JPEG, TODO). Puffs differs from nom
GIF), ZLIB (for PNG) and Huffman (for JPEG, TODO). Wuffs differs from nom
combined with other Rust code (e.g. a Rust LZW implementation) in that bounds
and overflow checks not just ubiquitous but also completely eliminated at
compile time.
[Kaitai Struct](http://kaitai.io/) is in a similar space, generating safe
parsers for multiple target programming languages from one declarative
specification. Again, Puffs differs in that it is a complete (and performant)
specification. Again, Wuffs differs in that it is a complete (and performant)
end to end implementation, not just for the structured parts of a file format.
Repeating a point in the previous paragraph, the difficulty in decoding the GIF
format isn't in the regularly-expressible part of the format, it's in the LZW
+17 -17
View File
@@ -1,12 +1,12 @@
# Puffs the Language
# Wuffs the Language
Puffs is an imperative, C-like language and much of it should look familiar to
Wuffs is an imperative, C-like language and much of it should look familiar to
somebody versed in any one of C, C++, D, Go, Java, JavaScript, Objective-C,
Rust, Swift, etc. Accordingly, this section is not an exhaustive specification,
merely a rough guide of where Puffs differs.
merely a rough guide of where Wuffs differs.
Like Go, [semi-colons can be omitted](https://golang.org/ref/spec#Semicolons),
and `puffsfmt` will remove them. Similarly, the body of an `if` or `while` must
and `wuffsfmt` will remove them. Similarly, the body of an `if` or `while` must
be enclosed by curly `{}`s. There is no 'dangling else' ambiguity.
@@ -68,7 +68,7 @@ associative: `(a + b) + c` and `a + (b + c)` are equivalent, and can be written
as `a + b + c`.
The logical operators, `&&` and `||` and `!` in C, are written as `and` and
`or` and `not` in Puffs.
`or` and `not` in Wuffs.
TODO: ignore-overflow ops, equivalent to Swift's `&+`.
@@ -84,7 +84,7 @@ array of unsigned 32-bit integers. `ptr` here means a non-null pointer. Use
Integer types can also be refined: `var x u32[10..20]` defines a variable x
that is stored as 4 bytes (32 bits) and can be combined arithmetically (e.g.
added, compared) with other `u32`s, but whose value must be between 10 and 20
inclusive. The syntax is reminiscent of Pascal's subranges, but in Puffs, a
inclusive. The syntax is reminiscent of Pascal's subranges, but in Wuffs, a
subsequent assignment like `x = 21` or even `x = y + z` is a compile time error
unless the right hand side can be proven to be within range.
@@ -97,7 +97,7 @@ i8[-7..]` means that `y` is between -7 and +127. `var z u32[..]` is equivalent
to `var z u32`.
Refinement bounds must be constant expressions. `var x u32[..2+3]` is valid,
but `var x u32; var y u32[..x]` is not. Puffs does not have dependent types.
but `var x u32; var y u32[..x]` is not. Wuffs does not have dependent types.
Relationships such as `y <= x` are expressible as assertions (see below), but
not by the type system.
@@ -114,7 +114,7 @@ its methods may be coroutines. (See below).
Function signatures read from left to right: `func max(x i32, y i32)(z i32)` is
a function that takes two `i32`s and returns one `i32`. Two pairs of
parentheses are required: a function that in other languages would return
`void` in Puffs returns the empty struct `()`, also known as the unit type.
`void` in Wuffs returns the empty struct `()`, also known as the unit type.
When calling a function, each argument must be named. It is `m = max(x:10,
y:20)` and not `m = max(10, 20)`.
@@ -208,7 +208,7 @@ facts that include both the explicit assertion `x < y` and the implicit
assertion `z == 3`, if none of `x`, `y` and `z` alias another (e.g. they are
all local variables).
Puffs has two forms of non-sequential control flow: `if` branches (including
Wuffs has two forms of non-sequential control flow: `if` branches (including
`if`, `else if`, `else if` chains) and `while` loops.
For an `if` statement, such as `if b { etc0 } else { etc1 }`, the condition `b`
@@ -237,14 +237,14 @@ and assumed after it.
## Proofs
Puffs' assertion and bounds checking system is a proof checker, not a full SMT
Wuffs' assertion and bounds checking system is a proof checker, not a full SMT
solver or automated theorem prover. It verifies explicit annotations instead of
the more open-ended task of searching for implicit proof steps. This involves
more explicit work by the programmer, but compile times matter, so Puffs is
more explicit work by the programmer, but compile times matter, so Wuffs is
fast (and dumb) instead of smart (and slow).
Nonetheless, the Puffs syntax is regular (and unlike C++, does not require a
symbol table to parse), so it should be straightforward to transform Puffs code
Nonetheless, the Wuffs syntax is regular (and unlike C++, does not require a
symbol table to parse), so it should be straightforward to transform Wuffs code
to and from file formats used by more sophisticated proof engines.
Some rules are applied automatically by the proof checker. For example, if `x
@@ -253,15 +253,15 @@ type system) or dynamic constraint (an asserted fact), then the checker knows
that the expression `x + y` is bounded above by `10 + 5` and therefore will not
overflow a `u8` (but would overflow a `u8[..12]`).
TODO: rigorously specify these automatic rules, when we have written more Puffs
TODO: rigorously specify these automatic rules, when we have written more Wuffs
code and thus have more experience on what rules are needed to implement
multiple, real world image codecs.
Other rules are built in to the proof checker but are not applied automatically
(see "fast... instead of smart" above). Such rules have double-quote enclosed
names that look a little like mathematical statements. They are axiomatic, in
that these rules are assumed, not proved, by the Puffs toolchain. They are
typically at a higher level than e.g. Peano axioms, as Puffs emphasizes
that these rules are assumed, not proved, by the Wuffs toolchain. They are
typically at a higher level than e.g. Peano axioms, as Wuffs emphasizes
practicality over theoretical minimalism. As they are axiomatic, they endeavour
to only encode 'obvious' mathematical rules. For example, the rule named `"a <
b: a < c; c <= b"` is one expression of transitivity: the assertion `a < b` is
@@ -283,7 +283,7 @@ TODO: specify these built-in `via` rules, again after more experience.
Labeled `break` and `continue` statements enable jumping out of loops that
aren't the most deeply nested. The syntax for labels is `while:label` instead
of Java's `label:while`, as the former is slightly easier to parse, and Puffs
of Java's `label:while`, as the former is slightly easier to parse, and Wuffs
does not otherwise use labels for switch cases or goto targets.
TODO: describe the built in `buf1` and `buf2` types: 1- and 2-dimensional