mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-17 16:32:11 +01:00
287 lines
12 KiB
Markdown
287 lines
12 KiB
Markdown
Wuffs is a **memory-safe programming language** (and a **standard library**
|
|
written in that language) for **Wrangling Untrusted File Formats Safely**.
|
|
Wrangling includes parsing, decoding and encoding. Example file formats include
|
|
images, audio, video, fonts and compressed archives.
|
|
|
|
It is [**"ridiculously
|
|
fast"**](https://twitter.com/richgel999/status/1481027198530248714).
|
|
|
|

|
|
|
|
Per its [benchmarks](/doc/benchmarks.md) and other linked-to blog posts:
|
|
|
|
- It can decode bzip2 **[1.3x faster than `/usr/bin/bzcat`
|
|
(C)](https://nigeltao.github.io/blog/2022/wuffs-bzip2-decoder.html)**.
|
|
- It can decode deflate up to **1.4x faster than zlib-the-library (C)**.
|
|
- It can decode GIF **2x-6x faster than "giflib" (C), "image/gif" (Go) and
|
|
"gif" (Rust)**.
|
|
- It can decode PNG **[1.2x-2.7x faster than "libpng" (C), "image/png" (Go) and
|
|
"png"
|
|
(Rust)](https://nigeltao.github.io/blog/2021/fastest-safest-png-decoder.html)**.
|
|
|
|
|
|
## Goals and Non-Goals
|
|
|
|
Wuffs' goal is to produce software libraries that are as safe as Go or Rust,
|
|
roughly speaking, but as fast as C, and that can be used anywhere C libraries
|
|
are used. This includes very large C/C++ projects, such as popular web browsers
|
|
and operating systems (using that term to include desktop and mobile user
|
|
interfaces, not just the kernel).
|
|
|
|
[Wuffs the Library](/doc/wuffs-the-library.md) is [available](/release/c) as
|
|
transpiled C code. Other C/C++ projects can **use that library without
|
|
requiring the [Wuffs the Language](/doc/wuffs-the-language.md) toolchain**.
|
|
Those projects can use Wuffs the Library like using any other third party C
|
|
library. It's just not hand-written C.
|
|
|
|
However, unlike hand-written C, Wuffs the Language is safe with respect to
|
|
buffer overflows, integer arithmetic overflows and null pointer dereferences. A
|
|
key difference between Wuffs and other memory-safe languages is that **all such
|
|
checks are done at compile time, not at run time**. If it compiles, it is safe,
|
|
with respect to those three bug classes.
|
|
|
|
The trade-off in aiming for both safety and speed is that Wuffs programs take
|
|
longer for a programmer to write, as they have to **explicitly annotate their
|
|
programs with proofs of safety**. A statement like `x += 1` unsurprisingly
|
|
means to increment the variable `x` by `1`. However, in Wuffs, such a statement
|
|
is a compile time error unless the compiler can also prove that `x` is not the
|
|
maximal value of `x`'s type (e.g. `x` is not `255` if `x` is a `base.u8`), as
|
|
the increment would otherwise overflow. Similarly, an integer arithmetic
|
|
expression like `x / y` is a compile time error unless the compiler can also
|
|
prove that `y` is not zero.
|
|
|
|
|
|
## Hermeticity
|
|
|
|
Wuffs is not a general purpose programming language. **It is for writing
|
|
libraries, not programs**. Wuffs code is [hermetic](/doc/note/hermeticity.md)
|
|
and can only compute (e.g. convert "compressed bytes" to "decompressed bytes").
|
|
**It cannot make any syscalls** (e.g. it has no ambient authority to read your
|
|
files), implying that it cannot allocate or free memory (and is therefore
|
|
trivially safe against things like memory leaks, use-after-frees and
|
|
double-frees).
|
|
|
|
It produces [Sans I/O style](https://sans-io.readthedocs.io/) libraries (but C
|
|
libraries, not Python), meaning that they are agnostic to ['function
|
|
colors'](https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/).
|
|
They can be combined with synchronous or asynchronous I/O, as the library
|
|
caller (not library implementation) is responsible for the actual I/O.
|
|
|
|
The idea isn't to write your whole program in Wuffs, **only the parts that are
|
|
both performance-conscious and security-conscious**. For example, while
|
|
technically possible, it is unlikely that a Wuffs compiler would be worth
|
|
writing entirely in Wuffs.
|
|
|
|
|
|
## What Does Wuffs Code Look Like?
|
|
|
|
The [`/std/lzw/decode_lzw.wuffs`](/std/lzw/decode_lzw.wuffs) file is a good
|
|
example. The [Wuffs the Language](/doc/wuffs-the-language.md) document has more
|
|
information on how it differs from other languages in the C family.
|
|
|
|
|
|
## What Does Compile Time Checking Look Like?
|
|
|
|
For example, making this one-line edit to the LZW codec leads to a compile time
|
|
error. `wuffs gen` fails to generate the C code, i.e. fails to compile
|
|
(transpile) the Wuffs code to C code:
|
|
|
|
```diff
|
|
diff --git a/std/lzw/decode_lzw.wuffs b/std/lzw/decode_lzw.wuffs
|
|
index f878c5e..f10dcee 100644
|
|
--- a/std/lzw/decode_lzw.wuffs
|
|
+++ b/std/lzw/decode_lzw.wuffs
|
|
@@ -98,7 +98,7 @@ pub func lzw_decoder.decode?(dst ptr buf1, src ptr buf1, src_final bool)() {
|
|
in.dst.write?(x:s)
|
|
|
|
if use_save_code {
|
|
- this.suffixes[save_code] = c as u8
|
|
+ this.suffixes[save_code] = (c + 1) as u8
|
|
this.prefixes[save_code] = prev_code as u16
|
|
}
|
|
```
|
|
|
|
```
|
|
$ wuffs gen std/gif
|
|
check: expression "(c + 1) as u8" bounds [1 ..= 256] is not within bounds [0 ..= 255] at
|
|
/home/n/go/src/github.com/google/wuffs/std/lzw/decode_lzw.wuffs:101. Facts:
|
|
n_bits < 8
|
|
c < 256
|
|
this.stack[s] == (c as u8)
|
|
use_save_code
|
|
```
|
|
|
|
In comparison, this two-line edit will compile (but the "does it decode GIF
|
|
correctly" tests then fail):
|
|
|
|
```diff
|
|
diff --git a/std/lzw/decode_lzw.wuffs b/std/lzw/decode_lzw.wuffs
|
|
index f878c5e..b43443d 100644
|
|
--- a/std/lzw/decode_lzw.wuffs
|
|
+++ b/std/lzw/decode_lzw.wuffs
|
|
@@ -97,8 +97,8 @@ pub func lzw_decoder.decode?(dst ptr buf1, src ptr buf1, src_final bool)() {
|
|
// type checking, bounds checking and code generation for it).
|
|
in.dst.write?(x:s)
|
|
|
|
- if use_save_code {
|
|
- this.suffixes[save_code] = c as u8
|
|
+ if use_save_code and (c < 200) {
|
|
+ this.suffixes[save_code] = (c + 1) as u8
|
|
this.prefixes[save_code] = prev_code as u16
|
|
}
|
|
```
|
|
|
|
```
|
|
$ wuffs gen std/gif
|
|
gen wrote: /home/n/go/src/github.com/google/wuffs/gen/c/gif.c
|
|
gen unchanged: /home/n/go/src/github.com/google/wuffs/gen/h/gif.h
|
|
$ wuffs test std/gif
|
|
gen unchanged: /home/n/go/src/github.com/google/wuffs/gen/c/gif.c
|
|
gen unchanged: /home/n/go/src/github.com/google/wuffs/gen/h/gif.h
|
|
test: /home/n/go/src/github.com/google/wuffs/test/c/gif
|
|
gif/basic.c clang PASS (8 tests run)
|
|
gif/basic.c gcc PASS (8 tests run)
|
|
gif/gif.c clang FAIL test_lzw_decode: bufs1_equal: wi: got 19311, want 19200.
|
|
contents differ at byte 3 (in hex: 0x000003):
|
|
000000: dcdc dc00 00d9 f5f9 f6df dc5f 393a 3a3a ..........._9:::
|
|
000010: 3a3b 618e c8e4 e4e4 e5e4 e600 00e4 bbbb :;a.............
|
|
000020: eded 8f91 9191 9090 9090 9190 9192 9192 ................
|
|
000030: 9191 9292 9191 9293 93f0 f0f0 f1f1 f2f2 ................
|
|
excerpts of got (above) versus want (below):
|
|
000000: dcdc dcdc dcd9 f5f9 f6df dc5f 393a 3a3a ..........._9:::
|
|
000010: 3a3a 618e c8e4 e4e4 e5e4 e6e4 e4e4 bbbb ::a.............
|
|
000020: eded 8f91 9191 9090 9090 9090 9191 9191 ................
|
|
000030: 9191 9191 9191 9193 93f0 f0f0 f1f1 f2f2 ................
|
|
|
|
gif/gif.c gcc FAIL test_lzw_decode: bufs1_equal: wi: got 19311, want 19200.
|
|
contents differ at byte 3 (in hex: 0x000003):
|
|
000000: dcdc dc00 00d9 f5f9 f6df dc5f 393a 3a3a ..........._9:::
|
|
000010: 3a3b 618e c8e4 e4e4 e5e4 e600 00e4 bbbb :;a.............
|
|
000020: eded 8f91 9191 9090 9090 9190 9192 9192 ................
|
|
000030: 9191 9292 9191 9293 93f0 f0f0 f1f1 f2f2 ................
|
|
excerpts of got (above) versus want (below):
|
|
000000: dcdc dcdc dcd9 f5f9 f6df dc5f 393a 3a3a ..........._9:::
|
|
000010: 3a3a 618e c8e4 e4e4 e5e4 e6e4 e4e4 bbbb ::a.............
|
|
000020: eded 8f91 9191 9090 9090 9090 9191 9191 ................
|
|
000030: 9191 9191 9191 9193 93f0 f0f0 f1f1 f2f2 ................
|
|
|
|
wuffs-test-c: some tests failed
|
|
wuffs test: some tests failed
|
|
```
|
|
|
|
# Directory Layout
|
|
|
|
- `lang` holds the Go libraries that implement Wuffs the Language: tokenizer,
|
|
AST, parser, renderer, etc. The Wuffs tools are written in Go, but as
|
|
mentioned above, Wuffs transpiles to C code, and Go is not necessarily
|
|
involved if all you want is to use the C edition of Wuffs.
|
|
- `lib` holds other Go libraries, not specific to Wuffs the Language per se.
|
|
- `internal` holds internal implementation details, as per Go's [internal
|
|
packages](https://golang.org/s/go14internal) convention.
|
|
- `cmd` holds Wuffs the Language' command line tools, also written in Go.
|
|
- `std` holds Wuffs the Library's code.
|
|
- `release` holds the releases (e.g. in their C form) of Wuffs the Library.
|
|
- `test` holds the regular tests for Wuffs the Library.
|
|
- `fuzz` holds the fuzz tests for Wuffs the Library.
|
|
- `script` holds miscellaneous utility programs.
|
|
- `doc` holds documentation.
|
|
- `example` holds example programs for Wuffs the Library.
|
|
- `hello-wuffs-c` holds an example program for Wuffs the Language.
|
|
|
|
|
|
# Building
|
|
|
|
See the [BUILD](/BUILD.md) instructions.
|
|
|
|
|
|
# Documentation
|
|
|
|
- [Getting Started](/doc/getting-started.md). **Start here** if you want to
|
|
play but aren't sure how (and [BUILD](/BUILD.md) doesn't help).
|
|
- [Background](/doc/background.md).
|
|
- [Benchmarks](/doc/benchmarks.md).
|
|
- [Binary Size](/doc/binary-size.md).
|
|
- [Changelog](/doc/changelog.md).
|
|
- [Glossary](/doc/glossary.md).
|
|
- [Related Work](/doc/related-work.md).
|
|
- [Roadmap](/doc/roadmap.md).
|
|
- [Wuffs the Language](/doc/wuffs-the-language.md) overview.
|
|
- [Wuffs the Library](/doc/wuffs-the-library.md) overview and see also [API
|
|
categories](/doc/std).
|
|
|
|
The [Note](/doc/note) directory also contains various short articles.
|
|
|
|
|
|
# Non-C/C++ Languages
|
|
|
|
- [dev0x13/pywuffs](https://github.com/dev0x13/pywuffs) holds Python bindings
|
|
for Wuffs the Library.
|
|
- Bindings for Go, Rust and other languages are tracked as [issue
|
|
#38](https://github.com/google/wuffs/issues/38).
|
|
|
|
|
|
# Status
|
|
|
|
Version 0.3 (April 2023) is the latest stable version. Stable means that
|
|
its API won't change any further, but being a "version 0.x" means that:
|
|
|
|
- It will not have long term support.
|
|
- Newer versions make no promises about compatibility.
|
|
|
|
The compiler undoubtedly has bugs. Assertion checking needs more rigor,
|
|
especially around side effects and aliasing, and being sufficiently well
|
|
specified to allow alternative implementations. Lots of detail needs work, but
|
|
the broad brushstrokes are there.
|
|
|
|
Nonetheless, Wuffs' GIF decoder has shipped in the Google Chrome web browser
|
|
[since June
|
|
2021](https://chromium-review.googlesource.com/c/chromium/src/+/2940044)
|
|
(milestone M93). See also the ["ridiculously
|
|
fast"](https://twitter.com/richgel999/status/1481027198530248714) tweet already
|
|
mentioned above.
|
|
|
|
|
|
# Discussion
|
|
|
|
The mailing list is at
|
|
[https://groups.google.com/forum/#!forum/wuffs](https://groups.google.com/forum/#!forum/wuffs).
|
|
|
|
|
|
# Contributing
|
|
|
|
The [CONTRIBUTING.md](/CONTRIBUTING.md) file contains instructions on how to
|
|
file the Contributor License Agreement before sending any pull requests (PRs).
|
|
Of course, if you're new to the project, it's usually best to discuss any
|
|
proposals and reach consensus before sending your first PR.
|
|
|
|
Source code is [auto-formatted](/doc/note/auto-formatting.md).
|
|
|
|
|
|
# License
|
|
|
|
This software is distributed under the terms of both the MIT license and the
|
|
Apache License (Version 2.0).
|
|
|
|
See LICENSE for details.
|
|
|
|
|
|
# Disclaimer
|
|
|
|
This is not an official Google product, it is just code that happens to be
|
|
owned by Google.
|
|
|
|
|
|
# Mascot
|
|
|
|
Tony is an arse-kicking wombat who loves playing
|
|
[full-forward](https://en.wikipedia.org/wiki/Full-forward) and hates buffer
|
|
overflows.
|
|
|
|

|
|
|
|
|
|
---
|
|
|
|
Updated on November 2023.
|