refine `@io.Reader` and `@io.Writer` interface

[Guest0x0]

Current @io.Reader and @io.Writer interface has several problems. This issue discuss how refined interface should look like in the future. The discussion is divided into separated topics:

Binary v.s. Text

Currently, @io.Reader/@io.Writer are essentially binary byte stream. While they do provide some text based API, those API are merely UTF-8 wrappers over the underlying binary stream. This approach is not ideal. Different streams have different understanding of text. Here are a few notable examples:

• On Windows, console handles are inherently text based (ReadConsoleW/WriteConsoleW). Binary API (ReadFile/WriteFile) convert UTF-16 text to current console code page. This conversion is lossy, and the resulting binary stream is usually not UTF-8. So the "UTF-8 over binary" approach for text stream does not apply here.
  • stdio channels are even more challenging than console along, because stdio channels may or may not be a console. For non-console channels such as pipes, the primitive is binary based, not text based
• Conceptually, HTTP message body is essentially a binary stream. However, in practice, most HTTP message payload are text (HTML, JSON etc.), while binary payload also exist (file download). For HTTP message body with text semantic, there is a canonical encoding (determined by Content-Type in the header)
• Regular files may be text or binary as well, but the proportion of binary case is arguably higher than HTTP message body. Theoretically there is no canonical text encoding for regular files, especially on Windows. However it is also a common choice to make UTF-8 as the only first-class encoding here

These examples post different challenges on interface design:

• the Windows console example show that the primitive IO operation on an object is not necessarily binary based
• the HTTP example shows that surface API ergonomic should be considered separately from the essence of a stream. In terms of implementation, HTTP message body should be a binary stream with text wrappers. However, in terms of surface API, we should make text based operation as convenient as possible for HTTP message body
• the regular file example shows that whatever solution we come up for HTTP message body may not scale to everything else, because files have no canonical encoding. For regular files, stream based text IO should always go through explicit text wrappers. We may provide convenience helpers for reading text directly, though

Prior arts

Rust

• when stdio channel is a Windows console, Rust use ReadConsoleW/WriteConsoleW to read unicode content and encode/decode them to UTF-8 to simulate a UTF-8 encoded binary stream
• (reqwest) HTTP message body is a binary byte stream with whole-body byte helper & whole-body string helper.
• files are binary stream, with whole-file binary/text helpers.

Note that Rust use UTF-8 for strings, so streaming text IO files with UTF-8 encoding is no different from binary IO. Other encoding requires explicit encode/decode operations.

Java

• text IO is implemented by encode to current console code page plus WriteConsole. Binary IO uses current console code page. If stdio is not console, system locale will be used for text IO
• HTTP message body is a generic type, and the desired encoding (binary or text, whole message or streaming) is decided at message initiation time. There is no support for streaming text write of message body, though
• file objects provide binary stream API. There are helpers for whole-file binary/text IO, plus helpers for creating text stream reader/writer from file path directly

.NET

• text IO on stdio is implemented by encode to current console code page plus WriteConsole. Binary IO uses current console code page. If stdio is not console, code page of the actual console will be used for text IO
• the message body objects provide binary stream operation & whole-message byte/text operations. There is an extension that provides streaming text write on the response object directly (not the response body object), but there is no equivalent for request body
• file objects provide binary stream API. There are helpers for whole-file binary/text IO, plus helpers for creating text stream reader/writer from file path directly

Python

• when stdio channel is a Windows console, reading is ReadConsoleW -> trans-code as UTF-8, while writing is trans-code from UTF-8 to UTF-16 -> WriteConsoleW. When stdio is not a console, current locale is used for text IO
• http message body provides whole-message binary/text + binary stream API
• for streaming IO on files, binary v.s. text at open time. There are separated whole-fie binary/text helpers

Node.js

• text IO on stdio channel is UTF-8 wrapper over binary IO. Binary IO on Windows console trans-code input from UTF-8 to UTF-16 and uses WriteConsoleW
• all Node.js stream support both binary and text mode. For readable stream, the encoding depend on a mutable setting on the stream object. For writable stream, text and binary chunk can be written to the same stream
• same as the HTTP case, both file and HTTP message body are Node.js stream. Whole-file helpers exist for both binary and text

OkHttp

For reading message body, OkHttp provides multiple views for the same HTTP message body object, covering all combinations of whole-message v.s. stream and binary v.s. text. For writing message body, OkHttp provides both binary/text operations over the same writer object.

Our design

The first question is what encoding should be used for text IO over files/pipes/HTTP message body. There are three options here:

1. use UTF-8
2. use system locale (or the encoding specified in the header for HTTP)
3. allow explicitly specifying encoding, may or may not use one of the above as the default

I don't think we currently have any budget to handle arbitrary code page/locale. For HTTP non UTF-8 should be extremely rare. So we'll probably go option (1) here.

For text IO on Windows console, the one and only correct behavior is to properly handle unicode using ReadConsoleW/WriteConsoleW. But binary IO behavior for windows console can be a tough choice here. There are three options:

1. use UTF-8
2. use current console code page
3. use UTF-16

Note that we cannot simply forbid binary IO on Windows console, because it is valid to use non-console stdio channels to pass binary data, so stdio channels (which may be Windows console) must support binary IO. The UTF-16 options is kinda non sense and no one is really taking that option (note that Windows do have a special UTF-16 code page, but that is different from unconditionally using UTF-16). That leave us with two options:

• use UTF-8 for binary IO on Windows console. The merit of this approach is that we can now assume all text IO are effectively UTF-8 wrapper over binary IO. It is still possible to provide a direct UTF-16 path for Windows console as an optimization (no one seems to be doing this though. Console IO is usually not performance critical, so UTF-16 -> UTF-8 -> UTF-16 is not unacceptable here)
• use current console code page for Windows console. This would force binary IO and text IO to be independent interface

Given that we have chosen UTF-8 as the canonical encoding for everything else, the UTF-8 approach here feels more attractive for simplicity.

Now in terms of surface API design, whole-message/whole-file helpers exist in most language/libraries for both HTTP message body and files, we also provide such helpers via @io.Reader::read_all/@fs.read_file/@fs.write_file &@io.Data interface currently. The real problem is streaming IO interface here. There are roughly four API styles in existing libraries:

• provide binary stream interface as primitive, text stream IO requires explicit encoder/decoder wrapper. Example:

  let response = request(url)
  // let binary = response.read_all()
  let reader = Utf8Reader(response)
  let text = reader.read_all()

• provide builtin UTF-8 text IO support for every binary stream. This come for free in UTF-8 languages such as Rust and Go. This is also the approach we are currently taking. Example:

  let response = request(url)
  // let binary = response.read_all()
  let text = response.read_all_text()

  Currently we use the a bit tricky @io.Data to simply API names. An alternative approach is to use more explicit names for binary/text operations as illustrated above.

  Since binary/text operations are bound on the same object, there is a serious API naming problem here: what should be the semantic of generic method name such as write? Possible options:

  1. do not provide generic method name, only provide explicit name such as write_bytes, write_string
  2. always bind the generic name to binary, because all text streams are UTF-8 wrapper over binary stream
  3. bind the generic name to the most frequent operation on this particular stream

  In MoonBit, this problem is particularly annoying because the template writing syntax desugar to write, so option (1) and (2) above effectively means template writing cannot be used for writers. Option (3) is also kinda controversial, because "the most frequent operation" is a subjective concept that varies from stream to stream. For raw TCP sockets binary is obviously more important, while for HTTP message body text is probably more important. But what about regular files? Even if we can come up with a reasonable default for every stream type, option (3) will create an inconsistency between different stream types.

• make the message body/file type generic, and let the user specify encoding when making request/opening the file. Example:

  let response = request(url, encoding=@encoding.text) // `encoding` and `response` has generic type
  let text = response.read_all()

  or, for files:

  let file = open(path, encoding=@encoding.text)
  let text = file.read_all()

  the downside of this approach is more verbose types (due to the generic type parameter) and more complex API for HTTP message writer (we are forced to have a separated type for outgoing message due to the generic type parameter)

• provide multiple views on the same stream object. Example:

  let response = request(url)
  // let binary = response.binary.read_all()
  let text = response.text.read_all()

[Guest0x0]

Reader buffering

Many reader operations, in particular searching API such as read_line, requires buffering. The buffer for searching read-ahead must be integrated with the reader system, otherwise remaining data in the read-ahead buffer will get lost. So the design problem becomes: which readers have an integrated buffer?

Currently, every reader in moonbitlang/async has an integrated buffer. The size of that buffer is initially zero, effectively disabling buffering. However, when search API such as read_until is invoked, the reader automatically allocate a non-empty buffer for searching.

The builtin reader buffer is also useful for stream transformers, such as TLS and gzip. If the input buffer is already buffered, there is no need to introduce one additional layer of buffer. For example, HTTP header parsing requires searching, so the HTTP message parser requires buffering on the underlying TCP connection. If the payload of the HTTP message is gzip compressed, the gzip decoder again need a buffer for decoding. In our current design, these two buffers can be shared.

The builtin reader buffer also has also ergonomic benefits. When searching through stdin etc., users no longer need to manually add a buffered reader wrapper. It is unsafe to create multiple buffered wrapper over the same stream, so the builtin buffer is especially attractive for global stream such as stdin.

Prior arts

Rust

In Rust, most readers are not buffered, hence do not support search operations by default. Buffering must be enabled explicitly via a BufReader wrapper. stdin is a notable exception though: it has its own global buffer for convenience. The asynchronous stdin object in tokio does not have builtin buffering, though.

Go

There is no builtin buffering for reader streams. Buffering must always be performed explicitly. One interesting part in Go's buffered reader design is the ReadSlice method, which return a volatile slice to the internal buffer. Using ReadSlice can avoid unnecessary copy, but is a bit dangerous because the returned slice would become invalid once the reader advance, so the user must finish using the slice as fast as possible.

Python, .NET, Node.js

Both file and stdin are buffered by default. In Node.js other streams including sockets are buffered as well.

Our design

The current "every reader has a buffer which is initially empty" design looks good to me. It provides good ergonomic for searching API without sacrificing BYOB performance (the buffer will be skipped when there is no buffered data). The detailed API and behavior certainly need some refinement here, though:

• traits cannot store their own metadata. So currently @io.Reader requires implementations to provide a _get_internal_buffer() method, the name is ugly
• currently @io.Reader requires implementation to provide a _direct_read method. The destination of this method may or may not be the internal buffer. Since _direct_read ignores currently-buffered data, it is unsafe as a end-user API. This is also the reason behind the ugly name
• currently search API such as read_until are unbounded by default, which is dangerous and prone to resource exhaustion attack on pathological input

For the method naming problem, maybe we could introduce some sort of "reader controller" context to replace the raw buffer interface. That would make the API naming make more sense here. As for _direct_read, if we want to avoid duplicated implementation, the presence of such "unsafe" API is probably unavoidable. But the naming could be improved. Maybe direct_read_skip_buffered?

[Guest0x0]

Writer buffering

Currently, there is no implicit buffering for writers. Buffered writers can only be created via the explicit @io.BufferedWriter wrapper. I think the current design is fine, because unlike buffered reading, which is required by search API, buffered writing has performance implication only. In addition, buffered writing introduce the need for flushing, introducing additional mental complexity and bug pattern. There is a dilemma here:

• if close() does not do automatic flushing, it is easy to forget flushing before closing, result in data loss
• if close() performs automatic flushing, the semantic is very tricky on error/cancelled scenario

So I don't plan to introduce implicit buffering to writers as in the reader case.

[Guest0x0]

@io.Data

One questionable part in current API is the @io.Data trait (#201). Currently @io.Data is overloaded for both input and output:

@io.Data as input

When @io.Data is used as input, the same method can be overloaded for binary/text/json input. This solves the API naming problem mentioned in the "binary v.s. text" section above. However, the API signature is less readable and obvious for user not familiar with trait object implicit conversion semantic. In addition, using trait objects as input is conflicting with check mode/literal overloading. So currently write cannot accept a JSON literal directly.

@io.Data as output

@io.Reader::read_all, @fs.read_file etc. currently returns a &@io.Data, which can be converted to Bytes/String/Json via .binary()/.text()/.json(). Similar to the input case, this allows using the same method name for multiple output formats. However:

• we are abusing the same &@io.Data type with completely different mechanism for input/output here, making the API even more confusing for most users
• data conversion is performed in .text()/.json() methods, so users may accidentally perform duplicated conversion. This API style also make incremental/streaming decoding impossible

What to do with @io.Data

This largely depend on our answer to the "binary v.s. text" section above. Some solutions do not have the API name ambiguity problem at all, making @io.Data completely useless.

Assume we still take the current "same object, multiple interface" approach, possible options are:

• keep @io.Data as-is
• remove @io.Data and use separated methods for different format
  • but in this case, we need to figure out what to do with the "default" method name such as write/read_file

The input side and the output side is kinda independent here.