diff --git a/README.md b/README.md index 1cb78a9c..3bbd9e3b 100644 --- a/README.md +++ b/README.md @@ -297,96 +297,14 @@ that all the way through to C code generation and testing. - [Puffs the Language](./doc/puffs-the-language.md) - Puffs the Library (TODO) +- [Related Work](./doc/related-work.md) +- [Roadmap](./doc/roadmap.md) Measurements: - [Benchmarks](./doc/benchmarks.md) - [Binary Size](./doc/binary-size.md) - - -# Related Work - - -## Static Checkers - -Puffs is a language by itself, integrated with a compiler, not something -embedded in the comments of another language's program, supported by a separate -program checker, as the bounds check elimination affects the generated code. - -[Checked C](https://www.microsoft.com/en-us/research/project/checked-c/) has a -similar goal of safety, but aims to be backwards compatible with C, including -the idiomatic ways in which C code plays fast and loose with pointers and -arrays, and implicitly inserting run time checks while maintaining existing -data layout (e.g. the sizeof a struct type) for interoperability with other -systems. This is a much harder problem than Puffs' approach of a new, simple, -domain-specific programming language. - -Extended Static Checker for Java (ESC/Java) and its successor -[OpenJML](http://www.openjml.org/), which obviously target the Java programming -language, similarly has to analyze a language that is more complicated that -Puffs. Java is also not commonly used for writing e.g. low level image codecs, -as the language lacks unsigned integer types, it is garbage collected and -idiomatic code often allocates. - -[Spark ADA](http://libre.adacore.com/tools/spark-gpl-edition/) targets a subset -of the Ada programming language, which again is more complicated than Puffs. It -also allows for pluggable theorem provers such as Z3. This can increase -programmer productivity in that it can take less effort to prove more programs -safe, but it also affects reproducibility: when some programs that are provable -in one programmer's configuration are unprovable in another. It's also not -obvious up front what effect different theorem provers, and their different -heuristics, will have on compile times or proof times. In constrast, Puffs' -proof system is fast (and dumb) instead of smart (and slow). - -We're not claiming that "use a state of the art theorem prover" is an -unworkable approach. It's just a different approach, with different trade-offs. - - -## Why a New Language? - -Simpler languages are easier to prove things about. Macros, inheritance, -closures, generics, operator overloading, goto's and built-in container types -are all useful features, but as mentioned at the top, Puffs is not a general -purpose programming language. Instead, its focus is on compile-time provable -memory safety, with C-like performance, for CPU-intensive file format decoders. - -Nonetheless, Puffs is an imperative language, not a functional language, -despite the long history of functional languages and provability. Inner loop -performance usage matters, and it is easier to match the optimization -techniques of incumbent C libraries with a C-like language than with a -functional language. Compared to something like -[Liquid Haskell](https://ucsd-progsys.github.io/liquidhaskell-blog/), Puffs -also has no garbage collector overhead, as it has no garbage collector. - -Memory usage also matters, again considering per-pixel costs can be multiplied -by millions of pixels. It is important to represent e.g. an RGBA pixel as four -u8 values (or one u32), not as four naturally-sized-for-the-CPU integers or -four big precision integers. - - -## Why Not Rust? - -Puffs transpiles to standard C99 code with no dependencies, and should run -anywhere that has a C99 compliant C compiler. An existing C/C++ project, such -as the Chromium web browser, can choose to replace e.g. libpng with Puffs PNG, -without needing any additional toolchains. Sure, languages like D and Rust have -great binary-level interop with C code, and Mozilla are reporting progress with -parsing media formats in Rust, but it's still a non-zero operational hurdle to -grow a project's build process and to assess build times and binary sizes. - -Again, we're not claiming that "write it in Rust" is an unworkable approach. -It's just a different approach, with different trade-offs. - -[Nom](https://github.com/Geal/nom) is a -[promising](http://spw17.langsec.org/papers/chifflier-parsing-in-2017.pdf) -parser combinator library, in Rust. Puffs differs from nom by itself, in that -Puffs is an end to end implementation, not limited to that part of a file -format that is easily expressible as a formal grammar. In particular, it also -handles entropy encodings such as LZW (for GIF), ZLIB (for PNG, TODO) and -Huffman (for JPEG, TODO). Puffs differs from nom combined with other Rust code -(e.g. a Rust LZW implementation) in that bounds and overflow checks not just -ubiquitous but also completely eliminated at compile time. Once again, we're -not claiming that nom or Rust are unworkable, just different. +- [Compatibility](./doc/compatibility.md) # Status @@ -399,55 +317,6 @@ alternative implementations. Lots of detail needs work, but the broad brushstrokes are there. -## Compatibility - -TODO: get a decently large corpus of test images from a web crawl. Ensure that -Puffs produces 100% identical decodings to the suite of giflib, libjpeg, -libpng, etc. - -TODO: also trawl through Go's bug tracker for "this image failed to load". - - -## Roadmap - -Short term: - -- Do lots of TODOs. -- Implement coroutines, for streaming decoding. -- Finish the GIF decoder. -- Write the PNG decoder. -- Write the JPEG decoder. -- Write the PNG encoder. -- Design an image API that works with all these, including animated GIF. -- Write some example programs, maybe cpng a la cjpeg and cwebp. -- Add compatibility tests, benchmarks, code size comparison. -- Release as open source. - -Medium term: - -- Write the BMP/ICO decoder. -- Write the TIFF decoder. -- Write the WEBP decoder. -- Write the JPEG encoder. -- Get to 100% compatibility, on a large web crawl corpus, with libpng etc. -- Optimize the codecs; benchmark comparably to libpng etc. Might require SIMD. -- Ensure code size compares favorably to libpng etc. -- Fuzz it, both for crashers and for different output to libpng etc. - -Long term: - -- Ship with Google Chrome: safer code, smaller binaries, no regressions. -- Ship a version 1.0: stabilize the language and library APIs. -- Write a language spec. -- Generate Go code. -- Generate Rust code. - -Very long term: - -- Other file formats? Fonts? PDF? Audio? Video? -- Other image operations? Image scaling that's both safe and fast? - - # Discussion TODO: set up a mailing list. diff --git a/doc/compatibility.md b/doc/compatibility.md new file mode 100644 index 00000000..0306076c --- /dev/null +++ b/doc/compatibility.md @@ -0,0 +1,12 @@ +# Compatibility + +TODO: get a decently large corpus of test images from a web crawl. Ensure that +Puffs produces 100% identical decodings to the suite of giflib, libjpeg, +libpng, etc. + +TODO: also trawl through Go's bug tracker for "this image failed to load". + + +--- + +Updated on August 2017. diff --git a/doc/puffs-the-language.md b/doc/puffs-the-language.md index 447955e1..5bb1be54 100644 --- a/doc/puffs-the-language.md +++ b/doc/puffs-the-language.md @@ -290,3 +290,8 @@ does not otherwise use labels for switch cases or goto targets. TODO: describe the built in `buf1` and `buf2` types: 1- and 2-dimensional buffers of bytes, such as an I/O stream or a table of pixel data. + + +--- + +Updated on August 2017. diff --git a/doc/related-work.md b/doc/related-work.md new file mode 100644 index 00000000..fcb1d7b2 --- /dev/null +++ b/doc/related-work.md @@ -0,0 +1,88 @@ +# Related Work + + +## Static Checkers + +Puffs is a language by itself, integrated with a compiler, not something +embedded in the comments of another language's program, supported by a separate +program checker, as the bounds check elimination affects the generated code. + +[Checked C](https://www.microsoft.com/en-us/research/project/checked-c/) has a +similar goal of safety, but aims to be backwards compatible with C, including +the idiomatic ways in which C code plays fast and loose with pointers and +arrays, and implicitly inserting run time checks while maintaining existing +data layout (e.g. the sizeof a struct type) for interoperability with other +systems. This is a much harder problem than Puffs' approach of a new, simple, +domain-specific programming language. + +Extended Static Checker for Java (ESC/Java) and its successor +[OpenJML](http://www.openjml.org/), which obviously target the Java programming +language, similarly has to analyze a language that is more complicated that +Puffs. Java is also not commonly used for writing e.g. low level image codecs, +as the language lacks unsigned integer types, it is garbage collected and +idiomatic code often allocates. + +[Spark ADA](http://libre.adacore.com/tools/spark-gpl-edition/) targets a subset +of the Ada programming language, which again is more complicated than Puffs. It +also allows for pluggable theorem provers such as Z3. This can increase +programmer productivity in that it can take less effort to prove more programs +safe, but it also affects reproducibility: when some programs that are provable +in one programmer's configuration are unprovable in another. It's also not +obvious up front what effect different theorem provers, and their different +heuristics, will have on compile times or proof times. In constrast, Puffs' +proof system is fast (and dumb) instead of smart (and slow). + +We're not claiming that "use a state of the art theorem prover" is an +unworkable approach. It's just a different approach, with different trade-offs. + + +## Why a New Language? + +Simpler languages are easier to prove things about. Macros, inheritance, +closures, generics, operator overloading, goto's and built-in container types +are all useful features, but as mentioned at the top, Puffs is not a general +purpose programming language. Instead, its focus is on compile-time provable +memory safety, with C-like performance, for CPU-intensive file format decoders. + +Nonetheless, Puffs is an imperative language, not a functional language, +despite the long history of functional languages and provability. Inner loop +performance usage matters, and it is easier to match the optimization +techniques of incumbent C libraries with a C-like language than with a +functional language. Compared to something like +[Liquid Haskell](https://ucsd-progsys.github.io/liquidhaskell-blog/), Puffs +also has no garbage collector overhead, as it has no garbage collector. + +Memory usage also matters, again considering per-pixel costs can be multiplied +by millions of pixels. It is important to represent e.g. an RGBA pixel as four +u8 values (or one u32), not as four naturally-sized-for-the-CPU integers or +four big precision integers. + + +## Why Not Rust? + +Puffs transpiles to standard C99 code with no dependencies, and should run +anywhere that has a C99 compliant C compiler. An existing C/C++ project, such +as the Chromium web browser, can choose to replace e.g. libpng with Puffs PNG, +without needing any additional toolchains. Sure, languages like D and Rust have +great binary-level interop with C code, and Mozilla are reporting progress with +parsing media formats in Rust, but it's still a non-zero operational hurdle to +grow a project's build process and to assess build times and binary sizes. + +Again, we're not claiming that "write it in Rust" is an unworkable approach. +It's just a different approach, with different trade-offs. + +[Nom](https://github.com/Geal/nom) is a +[promising](http://spw17.langsec.org/papers/chifflier-parsing-in-2017.pdf) +parser combinator library, in Rust. Puffs differs from nom by itself, in that +Puffs is an end to end implementation, not limited to that part of a file +format that is easily expressible as a formal grammar. In particular, it also +handles entropy encodings such as LZW (for GIF), ZLIB (for PNG, TODO) and +Huffman (for JPEG, TODO). Puffs differs from nom combined with other Rust code +(e.g. a Rust LZW implementation) in that bounds and overflow checks not just +ubiquitous but also completely eliminated at compile time. Once again, we're +not claiming that nom or Rust are unworkable, just different. + + +--- + +Updated on August 2017. diff --git a/doc/roadmap.md b/doc/roadmap.md new file mode 100644 index 00000000..f168ae66 --- /dev/null +++ b/doc/roadmap.md @@ -0,0 +1,43 @@ +# Roadmap + +Short term: + +- Do lots of TODOs. +- Implement coroutines, for streaming decoding. +- Finish the GIF decoder. +- Write the PNG decoder. +- Write the JPEG decoder. +- Write the PNG encoder. +- Design an image API that works with all these, including animated GIF. +- Write some example programs, maybe cpng a la cjpeg and cwebp. +- Add compatibility tests, benchmarks, code size comparison. +- Release as open source. + +Medium term: + +- Write the BMP/ICO decoder. +- Write the TIFF decoder. +- Write the WEBP decoder. +- Write the JPEG encoder. +- Get to 100% compatibility, on a large web crawl corpus, with libpng etc. +- Optimize the codecs; benchmark comparably to libpng etc. Might require SIMD. +- Ensure code size compares favorably to libpng etc. +- Fuzz it, both for crashers and for different output to libpng etc. + +Long term: + +- Ship with Google Chrome: safer code, smaller binaries, no regressions. +- Ship a version 1.0: stabilize the language and library APIs. +- Write a language spec. +- Generate Go code. +- Generate Rust code. + +Very long term: + +- Other file formats? Fonts? PDF? Audio? Video? +- Other image operations? Image scaling that's both safe and fast? + + +--- + +Updated on August 2017.