mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-18 08:32:11 +01:00
Rename PUFFS to WUFFS, part 10 of N
This commit is contained in:
1 parent
8e3b5cce5e
commit
b9fd20bfe4
6 files changed
+141
-141
No files matched your search
@@ -1,16 +1,16 @@
|
||||
# Wrangling Untrusted File Formats Safely
|
||||
|
||||
([Formerly known as
|
||||
PUFFS](https://groups.google.com/d/topic/puffslang/cSrH-s7UqwA/discussion):
|
||||
Puffs](https://groups.google.com/d/topic/puffslang/cSrH-s7UqwA/discussion):
|
||||
Parsing Untrusted File Formats Safely).
|
||||
|
||||
Puffs is a domain-specific language and library for parsing untrusted file
|
||||
Wuffs is a domain-specific language and library for parsing untrusted file
|
||||
formats safely. Examples of such file formats include images, audio, video,
|
||||
fonts and compressed archives.
|
||||
|
||||
Unlike the C programming language, Puffs is safe with respect to buffer
|
||||
Unlike the C programming language, Wuffs is safe with respect to buffer
|
||||
overflows, integer arithmetic overflows and null pointer dereferences. The key
|
||||
difference between Puffs and other memory-safe languages is that all such
|
||||
difference between Wuffs and other memory-safe languages is that all such
|
||||
checks are done at compile time, not at run time. *If it compiles, it is safe*,
|
||||
with respect to those three bug classes.
|
||||
|
||||
@@ -20,37 +20,37 @@ are used. This includes very large C/C++ products, such as popular web browsers
|
||||
and operating systems (using that term to include desktop and mobile user
|
||||
interfaces, not just the kernel).
|
||||
|
||||
The trade-off in aiming for both safety and speed is that Puffs programs take
|
||||
The trade-off in aiming for both safety and speed is that Wuffs programs take
|
||||
longer for a programmer to write, as they have to explicitly annotate their
|
||||
programs with proofs of safety. A statement like `x += 1` unsurprisingly means
|
||||
to increment the variable `x` by `1`. However, in Puffs, such a statement is a
|
||||
to increment the variable `x` by `1`. However, in Wuffs, such a statement is a
|
||||
compile time error unless the compiler can also prove that `x` is not the
|
||||
maximal value of `x`'s type (e.g. `x` is not `255` if `x` is a `u8`), as the
|
||||
increment would otherwise overflow. Similarly, an integer arithmetic expression
|
||||
like `x / y` is a compile time error unless the compiler can also prove that
|
||||
`y` is not zero.
|
||||
|
||||
Puffs is not a general purpose programming language. While technically
|
||||
possible, it is unlikely that a Puffs compiler would be worth writing in Puffs.
|
||||
Wuffs is not a general purpose programming language. While technically
|
||||
possible, it is unlikely that a Wuffs compiler would be worth writing in Wuffs.
|
||||
|
||||
|
||||
## What Does Puffs Code Look Like?
|
||||
## What Does Wuffs Code Look Like?
|
||||
|
||||
The [`std/gif/decode_lzw.puffs`](./std/gif/decode_lzw.puffs) file is a good
|
||||
The [`std/gif/decode_lzw.wuffs`](./std/gif/decode_lzw.wuffs) file is a good
|
||||
example. See the "Poking Around" section below for more guidance.
|
||||
|
||||
|
||||
## What Does Compile Time Checking Look Like?
|
||||
|
||||
For example, making this one-line edit to the GIF codec leads to a compile time
|
||||
error. `puffs gen` fails to generate the C code, i.e. fails to compile
|
||||
(transpile) the Puffs code to C code:
|
||||
error. `wuffs gen` fails to generate the C code, i.e. fails to compile
|
||||
(transpile) the Wuffs code to C code:
|
||||
|
||||
```diff
|
||||
diff --git a/std/gif/decode_lzw.puffs b/std/gif/decode_lzw.puffs
|
||||
diff --git a/std/gif/decode_lzw.wuffs b/std/gif/decode_lzw.wuffs
|
||||
index f878c5e..f10dcee 100644
|
||||
--- a/std/gif/decode_lzw.puffs
|
||||
+++ b/std/gif/decode_lzw.puffs
|
||||
--- a/std/gif/decode_lzw.wuffs
|
||||
+++ b/std/gif/decode_lzw.wuffs
|
||||
@@ -98,7 +98,7 @@ pub func lzw_decoder.decode?(dst ptr buf1, src ptr buf1, src_final bool)() {
|
||||
in.dst.write?(x:s)
|
||||
|
||||
@@ -62,9 +62,9 @@ index f878c5e..f10dcee 100644
|
||||
```
|
||||
|
||||
```bash
|
||||
$ puffs gen std/gif
|
||||
$ wuffs gen std/gif
|
||||
check: expression "(c + 1) as u8" bounds [1..256] is not within bounds [0..255] at
|
||||
/home/n/go/src/github.com/google/wuffs/std/gif/decode_lzw.puffs:101. Facts:
|
||||
/home/n/go/src/github.com/google/wuffs/std/gif/decode_lzw.wuffs:101. Facts:
|
||||
n_bits < 8
|
||||
c < 256
|
||||
this.stack[s] == (c as u8)
|
||||
@@ -75,10 +75,10 @@ In comparison, this two-line edit will compile (but the "does it decode GIF
|
||||
correctly" tests then fail):
|
||||
|
||||
```diff
|
||||
diff --git a/std/gif/decode_lzw.puffs b/std/gif/decode_lzw.puffs
|
||||
diff --git a/std/gif/decode_lzw.wuffs b/std/gif/decode_lzw.wuffs
|
||||
index f878c5e..b43443d 100644
|
||||
--- a/std/gif/decode_lzw.puffs
|
||||
+++ b/std/gif/decode_lzw.puffs
|
||||
--- a/std/gif/decode_lzw.wuffs
|
||||
+++ b/std/gif/decode_lzw.wuffs
|
||||
@@ -97,8 +97,8 @@ pub func lzw_decoder.decode?(dst ptr buf1, src ptr buf1, src_final bool)() {
|
||||
// type checking, bounds checking and code generation for it).
|
||||
in.dst.write?(x:s)
|
||||
@@ -92,10 +92,10 @@ index f878c5e..b43443d 100644
|
||||
```
|
||||
|
||||
```bash
|
||||
$ puffs gen std/gif
|
||||
$ wuffs gen std/gif
|
||||
gen wrote: /home/n/go/src/github.com/google/wuffs/gen/c/gif.c
|
||||
gen unchanged: /home/n/go/src/github.com/google/wuffs/gen/h/gif.h
|
||||
$ puffs test std/gif
|
||||
$ wuffs test std/gif
|
||||
gen unchanged: /home/n/go/src/github.com/google/wuffs/gen/c/gif.c
|
||||
gen unchanged: /home/n/go/src/github.com/google/wuffs/gen/h/gif.h
|
||||
test: /home/n/go/src/github.com/google/wuffs/test/c/gif
|
||||
@@ -125,8 +125,8 @@ excerpts of got (above) versus want (below):
|
||||
000020: eded 8f91 9191 9090 9090 9090 9191 9191 ................
|
||||
000030: 9191 9191 9191 9193 93f0 f0f0 f1f1 f2f2 ................
|
||||
|
||||
puffs-test-c: some tests failed
|
||||
puffs test: some tests failed
|
||||
wuffs-test-c: some tests failed
|
||||
wuffs test: some tests failed
|
||||
```
|
||||
|
||||
# Background
|
||||
@@ -186,15 +186,15 @@ The run time cost is small, measured in nanoseconds. But if an image decoding
|
||||
library has to eat this cost per pixel, and you have a megapixel image, then
|
||||
nanoseconds become milliseconds, and milliseconds can matter.
|
||||
|
||||
In comparison, in Puffs, all bounds checks and arithmetic overflow checks
|
||||
In comparison, in Wuffs, all bounds checks and arithmetic overflow checks
|
||||
happen at compile time, with zero run time overhead.
|
||||
|
||||
|
||||
# Getting Started
|
||||
|
||||
Puffs code (that is proved safe via explicit assertions) is compiled to C code
|
||||
Wuffs code (that is proved safe via explicit assertions) is compiled to C code
|
||||
(with those assertions removed) - it is transpiled. If you are a C/C++
|
||||
programmer and just want to *use* the C edition of the Puffs standard library,
|
||||
programmer and just want to *use* the C edition of the Wuffs standard library,
|
||||
then clone the repository and look at the files in the `gen/c` and `gen/h`
|
||||
directories. No other software tools are required and there are no library
|
||||
dependencies, other than C standard library concepts like `<stdint.h>`'s
|
||||
@@ -213,38 +213,38 @@ gzip or zlib, roughly speaking) without necessarily processing PNG images.
|
||||
|
||||
## Getting Deeper
|
||||
|
||||
If you want to modify the Puffs standard library, or compile your own Puffs
|
||||
If you want to modify the Wuffs standard library, or compile your own Wuffs
|
||||
code, you will need to do a little more work, and will have to install at least
|
||||
the Go toolchain in order to build the Puffs tools. To run the test suite, you
|
||||
the Go toolchain in order to build the Wuffs tools. To run the test suite, you
|
||||
might also have to install C compilers like clang and gcc, as well as C
|
||||
libraries (and their .h files) like libjpeg and libpng, as some tests compare
|
||||
that Puffs produces exactly the same output as these other libraries.
|
||||
that Wuffs produces exactly the same output as these other libraries.
|
||||
|
||||
Running `go get -v github.com/google/wuffs/cmd/...` will download and install
|
||||
the Puffs tools. Change `get` to `install` to re-install those programs without
|
||||
the Wuffs tools. Change `get` to `install` to re-install those programs without
|
||||
downloading, e.g. after you've modified their source code, or after a manually
|
||||
issued `git pull`. The Puffs tools that you'll most often use are `puffsfmt`
|
||||
(analogous to `clang-format`, `gofmt` or `rustfmt`) and `puffs` (roughly
|
||||
issued `git pull`. The Wuffs tools that you'll most often use are `wuffsfmt`
|
||||
(analogous to `clang-format`, `gofmt` or `rustfmt`) and `wuffs` (roughly
|
||||
analogous to `make`, `go` or `cargo`).
|
||||
|
||||
You should now be able to run `puffs test`. If all goes well, you should see
|
||||
You should now be able to run `wuffs test`. If all goes well, you should see
|
||||
some output containing the word "PASS" multiple times.
|
||||
|
||||
|
||||
## Poking Around
|
||||
|
||||
Feel free to edit the `std/gif/decode_lzw.puffs` file, which implements the GIF
|
||||
LZW decoder. After editing, run `puffs gen std/gif` or `puffs test std/gif` to
|
||||
re-generate the C edition of the Puffs standard library's GIF codec, and
|
||||
Feel free to edit the `std/gif/decode_lzw.wuffs` file, which implements the GIF
|
||||
LZW decoder. After editing, run `wuffs gen std/gif` or `wuffs test std/gif` to
|
||||
re-generate the C edition of the Wuffs standard library's GIF codec, and
|
||||
optionally run its tests.
|
||||
|
||||
Try deleting an assert statement and re-running `puffs gen`. The result should
|
||||
Try deleting an assert statement and re-running `wuffs gen`. The result should
|
||||
be syntactically valid, but a compile error, as some bounds checks can no
|
||||
longer be proven.
|
||||
|
||||
Find the line `var bits u32`, which declares the bits variable and initializes
|
||||
it to zero. Try adding `bits -= 1` on a new line of code after it. Again,
|
||||
`puffs gen` should fail, as the computation can underflow.
|
||||
`wuffs gen` should fail, as the computation can underflow.
|
||||
|
||||
Similarly, replacing the line `var n_bits u32` with `var n_bits u32 = 10`
|
||||
should fail, as an `n_bits < 8` assertion, a pre-condition, a few lines further
|
||||
@@ -254,8 +254,8 @@ Similarly, changing the `4095` in `var prev_code u32[..4095]` either higher or
|
||||
lower should fail.
|
||||
|
||||
Try adding `assert false` at various places, which should obviously fail, but
|
||||
should also cause `puffs gen` to print what facts the compiler can prove at
|
||||
that point. This can be useful when debugging why Puffs can't prove something
|
||||
should also cause `wuffs gen` to print what facts the compiler can prove at
|
||||
that point. This can be useful when debugging why Wuffs can't prove something
|
||||
you think it should be able to.
|
||||
|
||||
|
||||
@@ -265,14 +265,14 @@ If you've changed any of the tools (i.e. changed any `.go` code), re-run `go
|
||||
install -v github.com/google/wuffs/cmd/...` and `go test
|
||||
github.com/google/wuffs/lang/...`.
|
||||
|
||||
If you've changed any of the libraries (i.e. changed any `.puffs` code), run
|
||||
`puffs test` or, ideally, `puffs test -mimic` to also check that Puffs' output
|
||||
If you've changed any of the libraries (i.e. changed any `.wuffs` code), run
|
||||
`wuffs test` or, ideally, `wuffs test -mimic` to also check that Wuffs' output
|
||||
mimics (i.e. exactly matches) other libraries' output, such as giflib for GIF,
|
||||
libpng for PNG, etc.
|
||||
|
||||
If your library change is an optimization, run `puffs bench` or `puffs bench
|
||||
If your library change is an optimization, run `wuffs bench` or `wuffs bench
|
||||
-mimic` both before and after your change to quantify the improvement. The
|
||||
mimic benchmark numbers should't change if you're only changing `.puffs` code,
|
||||
mimic benchmark numbers should't change if you're only changing `.wuffs` code,
|
||||
but seeing zero change in those numbers is a sanity check on any unrelated
|
||||
system variance, such as software updates or virus checkers running in the
|
||||
background.
|
||||
@@ -280,30 +280,30 @@ background.
|
||||
|
||||
## Directory Layout
|
||||
|
||||
- `lang` holds the Go libraries that implement the Puffs language: tokenizer,
|
||||
AST, parser, renderer, etc. The Puffs tools are written in Go, but as
|
||||
mentioned above, Puffs transpiles to C code, and Go is not necessarily
|
||||
involved if all you want is to use the C edition of Puffs.
|
||||
- `cmd` holds Puffs' command line tools, also written in Go.
|
||||
- `std` holds the Puffs standard library's code. The initial focus is on
|
||||
- `lang` holds the Go libraries that implement the Wuffs language: tokenizer,
|
||||
AST, parser, renderer, etc. The Wuffs tools are written in Go, but as
|
||||
mentioned above, Wuffs transpiles to C code, and Go is not necessarily
|
||||
involved if all you want is to use the C edition of Wuffs.
|
||||
- `cmd` holds Wuffs' command line tools, also written in Go.
|
||||
- `std` holds the Wuffs standard library's code. The initial focus is on
|
||||
popular image codecs: BMP, GIF, JPEG, PNG, TIFF and WEBP.
|
||||
- `gen` holds the transpiled editions of that standard library. The initial
|
||||
focus is generating C code. Later on, the repository might include generated
|
||||
Go and Rust code.
|
||||
- `test` holds the tests for the Puffs standard library.
|
||||
- `test` holds the tests for the Wuffs standard library.
|
||||
- `script` holds miscellaneous utility programs.
|
||||
- `doc` holds documentation.
|
||||
- `example` holds example programs.
|
||||
|
||||
For a guide on how various things work together, the "99ff8e2 Let fields have
|
||||
default values" commit is an example of adding new Puffs syntax and threading
|
||||
default values" commit is an example of adding new Wuffs syntax and threading
|
||||
that all the way through to C code generation and testing.
|
||||
|
||||
|
||||
# Documentation
|
||||
|
||||
- [Wuffs the Language](./doc/wuffs-the-language.md)
|
||||
- Puffs the Library (TODO)
|
||||
- Wuffs the Library (TODO)
|
||||
- [Related Work](./doc/related-work.md)
|
||||
- [Roadmap](./doc/roadmap.md)
|
||||
|
||||
|
||||
+36
-36
@@ -1,9 +1,9 @@
|
||||
# Benchmarks
|
||||
|
||||
*Preliminary* `puffs bench -mimic` summarized throughput numbers for various
|
||||
*Preliminary* `wuffs bench -mimic` summarized throughput numbers for various
|
||||
codecs are below. Higher is better.
|
||||
|
||||
"Mimic" tests check that Puffs' output mimics (i.e. exactly matches) other
|
||||
"Mimic" tests check that Wuffs' output mimics (i.e. exactly matches) other
|
||||
libraries' output. "Mimic" benchmarks give the numbers for those other
|
||||
libraries, as shipped with the OS. These were measured on a Debian Testing
|
||||
system as of November 2017, which means these compiler versions:
|
||||
@@ -27,7 +27,7 @@ The benchmark programs aim to be runnable "out of the box" without any
|
||||
configuration or installation. For example, to run the `std/flate` benchmarks:
|
||||
|
||||
git clone https://github.com/google/wuffs.git
|
||||
cd puffs/test/c/std
|
||||
cd wuffs/test/c/std
|
||||
gcc -O3 flate.c
|
||||
./a.out -bench
|
||||
rm a.out
|
||||
@@ -42,24 +42,24 @@ code change. To install it, first install Go, then run `go get
|
||||
golang.org/x/perf/cmd/benchstat`.
|
||||
|
||||
|
||||
## puffs bench
|
||||
## wuffs bench
|
||||
|
||||
As mentioned above, individual benchmark programs can be run manually. However,
|
||||
the canonical way to run the benchmarks (across multiple compilers and multiple
|
||||
packages like GIF and PNG) is to use the `puffs` command line tool, as it will
|
||||
also re-generate (transpile) the C code whenever you edit the \*.puffs code.
|
||||
Running `go install -v github.com/google/wuffs/cmd/...` will install the Puffs
|
||||
packages like GIF and PNG) is to use the `wuffs` command line tool, as it will
|
||||
also re-generate (transpile) the C code whenever you edit the \*.wuffs code.
|
||||
Running `go install -v github.com/google/wuffs/cmd/...` will install the Wuffs
|
||||
tools. After that, you can say
|
||||
|
||||
puffs bench
|
||||
wuffs bench
|
||||
|
||||
or
|
||||
|
||||
puffs bench -mimic std/flate
|
||||
wuffs bench -mimic std/flate
|
||||
|
||||
or
|
||||
|
||||
puffs bench -ccompilers=gcc -reps=3 -focus=Benchmarkpuffs_gif_lzw std/gif
|
||||
wuffs bench -ccompilers=gcc -reps=3 -focus=Benchmarkwuffs_gif_lzw std/gif
|
||||
|
||||
|
||||
## CPU Scaling
|
||||
@@ -82,21 +82,21 @@ output.
|
||||
|
||||
name speed
|
||||
|
||||
puffs_adler32_10k/clang 2.42GB/s ± 3%
|
||||
puffs_adler32_100k/clang 2.42GB/s ± 3%
|
||||
puffs_flate_decode_1k/clang 133MB/s ± 3%
|
||||
puffs_flate_decode_10k/clang 199MB/s ± 3%
|
||||
puffs_flate_decode_100k/clang 229MB/s ± 4%
|
||||
puffs_zlib_decode_10k/clang 185MB/s ± 1%
|
||||
puffs_zlib_decode_100k/clang 210MB/s ± 3%
|
||||
wuffs_adler32_10k/clang 2.42GB/s ± 3%
|
||||
wuffs_adler32_100k/clang 2.42GB/s ± 3%
|
||||
wuffs_flate_decode_1k/clang 133MB/s ± 3%
|
||||
wuffs_flate_decode_10k/clang 199MB/s ± 3%
|
||||
wuffs_flate_decode_100k/clang 229MB/s ± 4%
|
||||
wuffs_zlib_decode_10k/clang 185MB/s ± 1%
|
||||
wuffs_zlib_decode_100k/clang 210MB/s ± 3%
|
||||
|
||||
puffs_adler32_10k/gcc 3.22GB/s ± 1%
|
||||
puffs_adler32_100k/gcc 3.23GB/s ± 0%
|
||||
puffs_flate_decode_1k/gcc 152MB/s ± 1%
|
||||
puffs_flate_decode_10k/gcc 250MB/s ± 1%
|
||||
puffs_flate_decode_100k/gcc 298MB/s ± 1%
|
||||
puffs_zlib_decode_10k/gcc 238MB/s ± 2%
|
||||
puffs_zlib_decode_100k/gcc 270MB/s ± 2%
|
||||
wuffs_adler32_10k/gcc 3.22GB/s ± 1%
|
||||
wuffs_adler32_100k/gcc 3.23GB/s ± 0%
|
||||
wuffs_flate_decode_1k/gcc 152MB/s ± 1%
|
||||
wuffs_flate_decode_10k/gcc 250MB/s ± 1%
|
||||
wuffs_flate_decode_100k/gcc 298MB/s ± 1%
|
||||
wuffs_zlib_decode_10k/gcc 238MB/s ± 2%
|
||||
wuffs_zlib_decode_100k/gcc 270MB/s ± 2%
|
||||
|
||||
mimic_adler32_10k 3.00GB/s ± 1%
|
||||
mimic_adler32_100k 2.91GB/s ± 2%
|
||||
@@ -116,15 +116,15 @@ is 1 byte per pixel.
|
||||
|
||||
name speed
|
||||
|
||||
puffs_gif_decode_1k/clang 346MB/s ± 1%
|
||||
puffs_gif_decode_10k/clang 137MB/s ± 0%
|
||||
puffs_gif_decode_100k/clang 118MB/s ± 0%
|
||||
puffs_gif_decode_1000k/clang 120MB/s ± 0%
|
||||
wuffs_gif_decode_1k/clang 346MB/s ± 1%
|
||||
wuffs_gif_decode_10k/clang 137MB/s ± 0%
|
||||
wuffs_gif_decode_100k/clang 118MB/s ± 0%
|
||||
wuffs_gif_decode_1000k/clang 120MB/s ± 0%
|
||||
|
||||
puffs_gif_decode_1k/gcc 399MB/s ± 1%
|
||||
puffs_gif_decode_10k/gcc 141MB/s ± 0%
|
||||
puffs_gif_decode_100k/gcc 128MB/s ± 0%
|
||||
puffs_gif_decode_1000k/gcc 131MB/s ± 0%
|
||||
wuffs_gif_decode_1k/gcc 399MB/s ± 1%
|
||||
wuffs_gif_decode_10k/gcc 141MB/s ± 0%
|
||||
wuffs_gif_decode_100k/gcc 128MB/s ± 0%
|
||||
wuffs_gif_decode_1000k/gcc 131MB/s ± 0%
|
||||
|
||||
mimic_gif_decode_1k 147MB/s ± 0%
|
||||
mimic_gif_decode_10k 90.7MB/s ± 0%
|
||||
@@ -134,10 +134,10 @@ is 1 byte per pixel.
|
||||
TODO: investigate why gcc 4.8 (Ubuntu Trusty) seems to generate faster code
|
||||
than gcc 7.2 (Debian Testing):
|
||||
|
||||
puffs_gif_decode_1k/gcc 411MB/s ± 2%
|
||||
puffs_gif_decode_10k/gcc 162MB/s ± 0%
|
||||
puffs_gif_decode_100k/gcc 138MB/s ± 0%
|
||||
puffs_gif_decode_1000k/gcc 141MB/s ± 0%
|
||||
wuffs_gif_decode_1k/gcc 411MB/s ± 2%
|
||||
wuffs_gif_decode_10k/gcc 162MB/s ± 0%
|
||||
wuffs_gif_decode_100k/gcc 138MB/s ± 0%
|
||||
wuffs_gif_decode_1000k/gcc 141MB/s ± 0%
|
||||
|
||||
|
||||
---
|
||||
|
||||
+6
-6
@@ -1,25 +1,25 @@
|
||||
# Binary Size
|
||||
|
||||
*Preliminary* measurements of `puffs genlib` libraries' binary size on x86_64
|
||||
*Preliminary* measurements of `wuffs genlib` libraries' binary size on x86_64
|
||||
are below. Lower is better.
|
||||
|
||||
clang-dynamic:
|
||||
-rwxr-xr-x 1 nigeltao eng 38352 Nov 9 22:59 libpuffs.so
|
||||
-rwxr-xr-x 1 nigeltao eng 38352 Nov 9 22:59 libwuffs.so
|
||||
-rw-r--r-- 1 nigeltao eng 22624 Nov 9 22:59 std-flate.lo
|
||||
-rw-r--r-- 1 nigeltao eng 9456 Nov 9 22:59 std-gif.lo
|
||||
|
||||
clang-static:
|
||||
-rw-r----- 1 nigeltao eng 32966 Nov 9 22:59 libpuffs.a
|
||||
-rw-r----- 1 nigeltao eng 32966 Nov 9 22:59 libwuffs.a
|
||||
-rw-r--r-- 1 nigeltao eng 22648 Nov 9 22:59 std-flate.o
|
||||
-rw-r--r-- 1 nigeltao eng 9480 Nov 9 22:59 std-gif.o
|
||||
|
||||
gcc-dynamic:
|
||||
-rwxr-xr-x 1 nigeltao eng 42504 Nov 9 22:59 libpuffs.so
|
||||
-rwxr-xr-x 1 nigeltao eng 42504 Nov 9 22:59 libwuffs.so
|
||||
-rw-r--r-- 1 nigeltao eng 24776 Nov 9 22:59 std-flate.lo
|
||||
-rw-r--r-- 1 nigeltao eng 13520 Nov 9 22:59 std-gif.lo
|
||||
|
||||
gcc-static:
|
||||
-rw-r----- 1 nigeltao eng 39102 Nov 9 22:59 libpuffs.a
|
||||
-rw-r----- 1 nigeltao eng 39102 Nov 9 22:59 libwuffs.a
|
||||
-rw-r--r-- 1 nigeltao eng 24728 Nov 9 22:59 std-flate.o
|
||||
-rw-r--r-- 1 nigeltao eng 13536 Nov 9 22:59 std-gif.o
|
||||
|
||||
@@ -29,7 +29,7 @@ are below. Lower is better.
|
||||
Below are some standard C libraries shipped as part of Debian Testing as of
|
||||
November 2017. The numbers aren't directly comparable, as these libraries have
|
||||
a richer API, especially in providing an encoder and not just a decoder. Still,
|
||||
it is a reference point for e.g. Puffs flate vs libz and Puffs gif vs libgif.
|
||||
it is a reference point for e.g. Wuffs flate vs libz and Wuffs gif vs libgif.
|
||||
|
||||
dynamic:
|
||||
-rw-r--r-- 1 root root 105088 Jan 29 2017 /lib/x86_64-linux-gnu/libz.so.1.2.8
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Compatibility
|
||||
|
||||
TODO: get a decently large corpus of test images from a web crawl. Ensure that
|
||||
Puffs produces 100% identical decodings to the suite of giflib, libjpeg,
|
||||
Wuffs produces 100% identical decodings to the suite of giflib, libjpeg,
|
||||
libpng, etc.
|
||||
|
||||
TODO: also trawl through Go's bug tracker for "this image failed to load".
|
||||
|
||||
+27
-27
@@ -1,34 +1,34 @@
|
||||
# Related Work
|
||||
|
||||
Puffs is an industrial project, not an academic project, and this Related Work
|
||||
Wuffs is an industrial project, not an academic project, and this Related Work
|
||||
section is not as extensive or as rigorous as would be found in an academic
|
||||
thesis or journal article. The underlying goal is usefulness, not novelty. For
|
||||
Puffs' users, it'd perfectly fine to have multiple implementations of a good
|
||||
Wuffs' users, it'd perfectly fine to have multiple implementations of a good
|
||||
idea, even if the later ones wouldn't earn a Ph. D.
|
||||
|
||||
A number of other projects are listed below. Puffs is not exactly like any of
|
||||
A number of other projects are listed below. Wuffs is not exactly like any of
|
||||
them, for one reason or another. We're not claiming that "use a state of the
|
||||
art theorem prover" or "write it in Rust" are unworkable approaches. They're
|
||||
just different approaches, with different trade-offs.
|
||||
|
||||
For example, Puffs is primarily concerned about *safety* and *performance*.
|
||||
For example, Wuffs is primarily concerned about *safety* and *performance*.
|
||||
Formal *correctness*, such as being able to mathematically express that "when
|
||||
this function returns, that array's elements will be sorted in increasing
|
||||
order", is less of a concern for Puffs the Language than for other projects.
|
||||
order", is less of a concern for Wuffs the Language than for other projects.
|
||||
This is especially as higher level concerns like "correctly implements the JPEG
|
||||
specification" are less amenable to formalization than foundational concepts
|
||||
like searching and sorting.
|
||||
|
||||
For Puffs the Library, confidence about correctness is instead based on
|
||||
For Wuffs the Library, confidence about correctness is instead based on
|
||||
testing, both unit testing and for e.g. media codecs, corpora of test files. A
|
||||
small corpus is included in the Puffs source code repository. Other, much
|
||||
small corpus is included in the Wuffs source code repository. Other, much
|
||||
larger corpora (e.g. based on a sample of a web crawl), are not included for
|
||||
download size and copyright reasons.
|
||||
|
||||
|
||||
## Static Checkers
|
||||
|
||||
Puffs is a language by itself, integrated with a compiler, not something
|
||||
Wuffs is a language by itself, integrated with a compiler, not something
|
||||
embedded in the comments of another language's program, supported by a separate
|
||||
program checker, as the bounds check elimination affects the generated code.
|
||||
|
||||
@@ -37,18 +37,18 @@ similar goal of safety, but aims to be backwards compatible with C, including
|
||||
the idiomatic ways in which C code plays fast and loose with pointers and
|
||||
arrays, and implicitly inserting run time checks while maintaining existing
|
||||
data layout (e.g. the sizeof a struct type) for interoperability with other
|
||||
systems. This is a much harder problem than Puffs' approach of a new, simple,
|
||||
systems. This is a much harder problem than Wuffs' approach of a new, simple,
|
||||
domain-specific programming language.
|
||||
|
||||
Extended Static Checker for Java (ESC/Java) and its successor
|
||||
[OpenJML](http://www.openjml.org/), which obviously target the Java programming
|
||||
language, similarly has to analyze a language that is more complicated that
|
||||
Puffs. Java is also not commonly used for writing e.g. low level image codecs,
|
||||
Wuffs. Java is also not commonly used for writing e.g. low level image codecs,
|
||||
as the language lacks unsigned integer types, it is garbage collected and
|
||||
idiomatic code often allocates.
|
||||
|
||||
Similarly, [Whiley](http://whiley.org/) is a programming language with extended
|
||||
static checking. In contrast to Puffs, its `int` type uses unbounded
|
||||
static checking. In contrast to Wuffs, its `int` type uses unbounded
|
||||
arithmetic, instead of e.g. 32-bit twos-complement representation. That can be
|
||||
easier for theorem provers to reason about, but this can have a performance
|
||||
impact when such integers are used in inner loops. In theory, a compiler could
|
||||
@@ -57,7 +57,7 @@ potentially 'big integer', but even so, it may improve performance to
|
||||
explicitly use a `uint32` here even when a `uint8` would do.
|
||||
|
||||
[Spark ADA](http://libre.adacore.com/tools/spark-gpl-edition/) targets a subset
|
||||
of the Ada programming language, which again is more complicated than Puffs. It
|
||||
of the Ada programming language, which again is more complicated than Wuffs. It
|
||||
also allows for pluggable theorem provers such as Z3, similar to
|
||||
[Dafny](http://research.microsoft.com/dafny). This can increase programmer
|
||||
productivity in that it can take less effort to prove more programs safe, but
|
||||
@@ -67,19 +67,19 @@ trusted computing base. It's also not obvious up front what effect different
|
||||
theorem provers, and their different heuristics, will have on compile times or
|
||||
proof times.
|
||||
|
||||
In constrast, Puffs' proof system is fast (and dumb) instead of smart (and
|
||||
In constrast, Wuffs' proof system is fast (and dumb) instead of smart (and
|
||||
slow). For example, [Vale (Verified Assembly Language for
|
||||
Everest)](https://github.com/project-everest/vale) is a promising approach that
|
||||
uses Dafny to verify implementations of cryptographic algorithms. However,
|
||||
["Vale: Verifying High-Performance Cryptographic Assembly
|
||||
Code"](http://www.andrew.cmu.edu/user/bparno/papers/vale.pdf) reports in Table
|
||||
1 that verification times are measured in *minutes*. Puffs aims for
|
||||
1 that verification times are measured in *minutes*. Wuffs aims for
|
||||
*sub-second* compilation times, including proofs of (memory) safety. Figure 15
|
||||
in that paper suggests that Vale's verification times can grow exponentially in
|
||||
some cases, and also goes on to say, "Dafny/Z3 fails to verify the AES key
|
||||
inversion for 6 unrolled iterations and 9 unrolled iterations, indicating that
|
||||
SMT solvers like Z3 are still occasionally unpredictable". Of course, Vale is
|
||||
also tackling a much harder problem than Puffs: proving correctness, not just
|
||||
also tackling a much harder problem than Wuffs: proving correctness, not just
|
||||
safety.
|
||||
|
||||
Once again, we're not claiming that these other approaches are unworkable, or
|
||||
@@ -91,17 +91,17 @@ not useful, just different, with different trade-offs.
|
||||
Simpler languages are easier to prove things about. Macros, inheritance,
|
||||
closures, generics, operator overloading, goto's and built-in container types
|
||||
are all useful features, but as mentioned in the top-level
|
||||
[README](../README.md), Puffs is not a general purpose programming language.
|
||||
[README](../README.md), Wuffs is not a general purpose programming language.
|
||||
Instead, its focus is on compile-time provable memory safety, with C-like
|
||||
performance, for CPU-intensive file format decoders.
|
||||
|
||||
Nonetheless, Puffs is an imperative language, not a functional language,
|
||||
Nonetheless, Wuffs is an imperative language, not a functional language,
|
||||
despite the long history of functional languages and provability. Inner loop
|
||||
performance usage matters, and it is easier to match the optimization
|
||||
techniques of incumbent C libraries with a C-like language than with a
|
||||
functional language. Compared to something like
|
||||
[F\*](https://www.fstar-lang.org/), [Idris](https://www.idris-lang.org/) or
|
||||
[Liquid Haskell](https://ucsd-progsys.github.io/liquidhaskell-blog/), Puffs has
|
||||
[Liquid Haskell](https://ucsd-progsys.github.io/liquidhaskell-blog/), Wuffs has
|
||||
no garbage collector overhead, as it has no garbage collector.
|
||||
|
||||
Memory usage also matters, again considering per-pixel costs can be multiplied
|
||||
@@ -110,8 +110,8 @@ u8 values (or one u32), not as four naturally-sized-for-the-CPU integers or
|
||||
four 'big integers'.
|
||||
|
||||
[F\* / KreMLin](https://github.com/FStarLang/kremlin) is a subset of F\* that
|
||||
is similar in some sense to Puffs, in that it transpiles to C and cares about
|
||||
both safety and performance. Unlike Puffs, it is a functional language (with
|
||||
is similar in some sense to Wuffs, in that it transpiles to C and cares about
|
||||
both safety and performance. Unlike Wuffs, it is a functional language (with
|
||||
dependent types), not an imperative one (with a simpler type system), and uses
|
||||
a sophisticated theorem prover like Z3, with the same trade-offs as discussed
|
||||
above for Spark ADA and Dafny. It also tackles a more complicated problem, in
|
||||
@@ -122,7 +122,7 @@ Low-Level Programming Embedded in F\*"](https://arxiv.org/pdf/1703.00053.pdf).
|
||||
|
||||
[Cryptol](https://github.com/GaloisInc/cryptol) is another project that, like
|
||||
Everest (including F\* / KreMLin and its HACL\* sub-project, and Dafny / Vale),
|
||||
focuses on cryptographic algorithms, rather than Puffs' focus on file formats,
|
||||
focuses on cryptographic algorithms, rather than Wuffs' focus on file formats,
|
||||
and also relies on a sophisticated theorem prover like Z3.
|
||||
|
||||
Once again, we're not claiming that these other approaches are unworkable, or
|
||||
@@ -131,9 +131,9 @@ not useful, just different, with different trade-offs.
|
||||
|
||||
## Why Not Rust?
|
||||
|
||||
Puffs transpiles to standard C99 code with no dependencies, and should run
|
||||
Wuffs transpiles to standard C99 code with no dependencies, and should run
|
||||
anywhere that has a C99 compliant C compiler. An existing C/C++ project, such
|
||||
as the Chromium web browser, can choose to replace e.g. libpng with Puffs PNG,
|
||||
as the Chromium web browser, can choose to replace e.g. libpng with Wuffs PNG,
|
||||
without needing any additional toolchains. Sure, languages like D and Rust have
|
||||
great binary-level interop with C code, and Mozilla are reporting progress with
|
||||
parsing media formats in Rust, but it's still an operational hurdle to grow a
|
||||
@@ -141,18 +141,18 @@ project's build process and to assess build times and binary sizes.
|
||||
|
||||
[Nom](https://github.com/Geal/nom) is a parser combinator library in Rust,
|
||||
described in ["Writing parsers like it is
|
||||
2017"](http://spw17.langsec.org/papers/chifflier-parsing-in-2017.pdf). Puffs
|
||||
differs from nom by itself, in that Puffs is an end to end implementation, not
|
||||
2017"](http://spw17.langsec.org/papers/chifflier-parsing-in-2017.pdf). Wuffs
|
||||
differs from nom by itself, in that Wuffs is an end to end implementation, not
|
||||
limited to that part of a file format that is easily expressible as a formal
|
||||
grammar. In particular, it also handles entropy encodings such as LZW (for
|
||||
GIF), ZLIB (for PNG) and Huffman (for JPEG, TODO). Puffs differs from nom
|
||||
GIF), ZLIB (for PNG) and Huffman (for JPEG, TODO). Wuffs differs from nom
|
||||
combined with other Rust code (e.g. a Rust LZW implementation) in that bounds
|
||||
and overflow checks not just ubiquitous but also completely eliminated at
|
||||
compile time.
|
||||
|
||||
[Kaitai Struct](http://kaitai.io/) is in a similar space, generating safe
|
||||
parsers for multiple target programming languages from one declarative
|
||||
specification. Again, Puffs differs in that it is a complete (and performant)
|
||||
specification. Again, Wuffs differs in that it is a complete (and performant)
|
||||
end to end implementation, not just for the structured parts of a file format.
|
||||
Repeating a point in the previous paragraph, the difficulty in decoding the GIF
|
||||
format isn't in the regularly-expressible part of the format, it's in the LZW
|
||||
|
||||
+17
-17
@@ -1,12 +1,12 @@
|
||||
# Puffs the Language
|
||||
# Wuffs the Language
|
||||
|
||||
Puffs is an imperative, C-like language and much of it should look familiar to
|
||||
Wuffs is an imperative, C-like language and much of it should look familiar to
|
||||
somebody versed in any one of C, C++, D, Go, Java, JavaScript, Objective-C,
|
||||
Rust, Swift, etc. Accordingly, this section is not an exhaustive specification,
|
||||
merely a rough guide of where Puffs differs.
|
||||
merely a rough guide of where Wuffs differs.
|
||||
|
||||
Like Go, [semi-colons can be omitted](https://golang.org/ref/spec#Semicolons),
|
||||
and `puffsfmt` will remove them. Similarly, the body of an `if` or `while` must
|
||||
and `wuffsfmt` will remove them. Similarly, the body of an `if` or `while` must
|
||||
be enclosed by curly `{}`s. There is no 'dangling else' ambiguity.
|
||||
|
||||
|
||||
@@ -68,7 +68,7 @@ associative: `(a + b) + c` and `a + (b + c)` are equivalent, and can be written
|
||||
as `a + b + c`.
|
||||
|
||||
The logical operators, `&&` and `||` and `!` in C, are written as `and` and
|
||||
`or` and `not` in Puffs.
|
||||
`or` and `not` in Wuffs.
|
||||
|
||||
TODO: ignore-overflow ops, equivalent to Swift's `&+`.
|
||||
|
||||
@@ -84,7 +84,7 @@ array of unsigned 32-bit integers. `ptr` here means a non-null pointer. Use
|
||||
Integer types can also be refined: `var x u32[10..20]` defines a variable x
|
||||
that is stored as 4 bytes (32 bits) and can be combined arithmetically (e.g.
|
||||
added, compared) with other `u32`s, but whose value must be between 10 and 20
|
||||
inclusive. The syntax is reminiscent of Pascal's subranges, but in Puffs, a
|
||||
inclusive. The syntax is reminiscent of Pascal's subranges, but in Wuffs, a
|
||||
subsequent assignment like `x = 21` or even `x = y + z` is a compile time error
|
||||
unless the right hand side can be proven to be within range.
|
||||
|
||||
@@ -97,7 +97,7 @@ i8[-7..]` means that `y` is between -7 and +127. `var z u32[..]` is equivalent
|
||||
to `var z u32`.
|
||||
|
||||
Refinement bounds must be constant expressions. `var x u32[..2+3]` is valid,
|
||||
but `var x u32; var y u32[..x]` is not. Puffs does not have dependent types.
|
||||
but `var x u32; var y u32[..x]` is not. Wuffs does not have dependent types.
|
||||
Relationships such as `y <= x` are expressible as assertions (see below), but
|
||||
not by the type system.
|
||||
|
||||
@@ -114,7 +114,7 @@ its methods may be coroutines. (See below).
|
||||
Function signatures read from left to right: `func max(x i32, y i32)(z i32)` is
|
||||
a function that takes two `i32`s and returns one `i32`. Two pairs of
|
||||
parentheses are required: a function that in other languages would return
|
||||
`void` in Puffs returns the empty struct `()`, also known as the unit type.
|
||||
`void` in Wuffs returns the empty struct `()`, also known as the unit type.
|
||||
|
||||
When calling a function, each argument must be named. It is `m = max(x:10,
|
||||
y:20)` and not `m = max(10, 20)`.
|
||||
@@ -208,7 +208,7 @@ facts that include both the explicit assertion `x < y` and the implicit
|
||||
assertion `z == 3`, if none of `x`, `y` and `z` alias another (e.g. they are
|
||||
all local variables).
|
||||
|
||||
Puffs has two forms of non-sequential control flow: `if` branches (including
|
||||
Wuffs has two forms of non-sequential control flow: `if` branches (including
|
||||
`if`, `else if`, `else if` chains) and `while` loops.
|
||||
|
||||
For an `if` statement, such as `if b { etc0 } else { etc1 }`, the condition `b`
|
||||
@@ -237,14 +237,14 @@ and assumed after it.
|
||||
|
||||
## Proofs
|
||||
|
||||
Puffs' assertion and bounds checking system is a proof checker, not a full SMT
|
||||
Wuffs' assertion and bounds checking system is a proof checker, not a full SMT
|
||||
solver or automated theorem prover. It verifies explicit annotations instead of
|
||||
the more open-ended task of searching for implicit proof steps. This involves
|
||||
more explicit work by the programmer, but compile times matter, so Puffs is
|
||||
more explicit work by the programmer, but compile times matter, so Wuffs is
|
||||
fast (and dumb) instead of smart (and slow).
|
||||
|
||||
Nonetheless, the Puffs syntax is regular (and unlike C++, does not require a
|
||||
symbol table to parse), so it should be straightforward to transform Puffs code
|
||||
Nonetheless, the Wuffs syntax is regular (and unlike C++, does not require a
|
||||
symbol table to parse), so it should be straightforward to transform Wuffs code
|
||||
to and from file formats used by more sophisticated proof engines.
|
||||
|
||||
Some rules are applied automatically by the proof checker. For example, if `x
|
||||
@@ -253,15 +253,15 @@ type system) or dynamic constraint (an asserted fact), then the checker knows
|
||||
that the expression `x + y` is bounded above by `10 + 5` and therefore will not
|
||||
overflow a `u8` (but would overflow a `u8[..12]`).
|
||||
|
||||
TODO: rigorously specify these automatic rules, when we have written more Puffs
|
||||
TODO: rigorously specify these automatic rules, when we have written more Wuffs
|
||||
code and thus have more experience on what rules are needed to implement
|
||||
multiple, real world image codecs.
|
||||
|
||||
Other rules are built in to the proof checker but are not applied automatically
|
||||
(see "fast... instead of smart" above). Such rules have double-quote enclosed
|
||||
names that look a little like mathematical statements. They are axiomatic, in
|
||||
that these rules are assumed, not proved, by the Puffs toolchain. They are
|
||||
typically at a higher level than e.g. Peano axioms, as Puffs emphasizes
|
||||
that these rules are assumed, not proved, by the Wuffs toolchain. They are
|
||||
typically at a higher level than e.g. Peano axioms, as Wuffs emphasizes
|
||||
practicality over theoretical minimalism. As they are axiomatic, they endeavour
|
||||
to only encode 'obvious' mathematical rules. For example, the rule named `"a <
|
||||
b: a < c; c <= b"` is one expression of transitivity: the assertion `a < b` is
|
||||
@@ -283,7 +283,7 @@ TODO: specify these built-in `via` rules, again after more experience.
|
||||
|
||||
Labeled `break` and `continue` statements enable jumping out of loops that
|
||||
aren't the most deeply nested. The syntax for labels is `while:label` instead
|
||||
of Java's `label:while`, as the former is slightly easier to parse, and Puffs
|
||||
of Java's `label:while`, as the former is slightly easier to parse, and Wuffs
|
||||
does not otherwise use labels for switch cases or goto targets.
|
||||
|
||||
TODO: describe the built in `buf1` and `buf2` types: 1- and 2-dimensional
|
||||
|
||||
Reference in new issue
Block a user