Have JSON line-comment tokens exclude the "\n"

Before this commit, line-comment token chains included the final "\n"
the way block-comments included the final "*/". The "\n" was mandatory.

However, when parsing with both QUIRK_ALLOW_COMMENT_LINE and
QUIRK_ALLOW_TRAILING_FILLER, input like "123//xyz" (without a "\n"
before the EOF) was rejected before and accepted after this commit.

So that line-comments are consistent, in that they either always or
never include the final "\n", they now never include the final "\n".

An input fragment (parsed with QUIRK_ALLOW_COMMENT_LINE) like
"//abc\n\t\t" was, before this commit, parsed as a 6 byte comment token
and a 2 byte other-filler token. That split is now 5+3.
This commit is contained in:
Nigel Tao committed 2021-07-20 14:17:09 +10:00
1 parent 40ffa334ee
commit 77f855206d
6 files changed
+41 -50

No files matched your search

+11 -6
View File
@@ -1530,7 +1530,16 @@ pri func decoder.decode_comment?(dst: base.token_writer, src: base.io_reader) {
pre args.dst.length() > 0,
{
if args.src.length() <= 0 {
if length > 0 {
if args.src.is_closed() {
args.dst.write_simple_token_fast!(
value_major: 0,
value_minor: (base.TOKEN__VBC__FILLER << 21) |
base.TOKEN__VBD__FILLER__COMMENT_LINE,
continued: 0,
length: length)
this.comment_type = 2
return ok
} else if length > 0 {
args.dst.write_simple_token_fast!(
value_major: 0,
value_minor: (base.TOKEN__VBC__FILLER << 21) |
@@ -1538,9 +1547,6 @@ pri func decoder.decode_comment?(dst: base.token_writer, src: base.io_reader) {
continued: 1,
length: length)
}
if args.src.is_closed() {
return "#bad input"
}
yield? base."$short read"
length = 0
continue.comment_line
@@ -1548,13 +1554,12 @@ pri func decoder.decode_comment?(dst: base.token_writer, src: base.io_reader) {
c = args.src.peek_u8()
if c == '\n' {
args.src.skip_u32_fast!(actual: 1, worst_case: 1)
args.dst.write_simple_token_fast!(
value_major: 0,
value_minor: (base.TOKEN__VBC__FILLER << 21) |
base.TOKEN__VBD__FILLER__COMMENT_LINE,
continued: 0,
length: length + 1)
length: length)
this.comment_type = 2
return ok
}
+7 -5
View File
@@ -109,15 +109,17 @@ pub const QUIRK_ALLOW_COMMENT_BLOCK : base.u32 = 0x4909_9400 | 0x0B
// When this quirk is enabled, "// C/C++ style line comments\n" are accepted
// anywhere whitespace would be. See also QUIRK_ALLOW_TRAILING_FILLER.
//
// A line comment may not omit the ending "\n", even if there is no input
// afterwards (i.e. the prospective line comment ends with the end-of-file).
// Line comments are terminated by a '\n' byte. '\r' bytes are irrelevant. When
// combined with QUIRK_ALLOW_TRAILING_FILLER, a line comment at the end of the
// file may omit the final '\n' byte.
//
// They produce WUFFS_BASE__TOKEN__VBD__FILLER__COMMENT_LINE tokens. The token
// chain's source bytes includes the starting "//" and the ending "\n".
// chain's source bytes includes the starting "//" but always excludes the
// ending "\n", regardless of whether it met the end of the file instead.
//
// Even if the line comments are on consecutive lines, each line comment is a
// separate token chain. There may be whitespace tokens between one line
// comment's ending "\n" and the next one's starting "//".
// separate token chain. Between them is at least one "\n" and maybe other
// whitespace, which are not part of a line comment token chain.
//
// To avoid ambiguity (as comments can contain new lines), this quirk cannot be
// combined with QUIRK_EXPECT_TRAILING_NEW_LINE_OR_EOF.