gemini_text (gemini_text v3.0.2)

Parse Gemini-formatted text.

parse/1 takes a binary and returns a list of elements (see tagged_line/0) in document order, or an error, for example {error, {invalid_utf8, 3}} if line 3 is not valid UTF-8 (see parse_error/0).

A document can also be fed in chunks, of any size and split anywhere, with new_context/0, parse/2 and finish/1; the result is the same as for parse/1 on the whole text. new_context/1 takes options (see option/0).

links/1 and urls/1 are helpers which extract just the links, or just their URLs.

Summary

Types

The state of a streaming parse: see new_context/0, parse/2 and finish/1. It is opaque.

Why parsing failed: the input was not valid UTF-8 (invalid_utf8), or a line was longer than the max_line_length option (line_too_long), or had a control character and control_chars is reject (invalid_char).

An option for new_context/1.

A parse error: the reason and the 1-based number of the offending line. Every line counts, including blank ones.

A preformatted block: {preformatted, Alt, Lines}.

One element of a Gemini document. It is one of

Functions

End a streaming parse.

Extract all the links in the given Gemini page.

Start a streaming parse, with nothing seen yet and the default options.

Start a streaming parse with the given options (see option/0).

Parse Text as a Gemini document.

Parse the next chunk of a document.

Extract all the URLs in the links in the given Gemini page.

Types

context()

-opaque context()

The state of a streaming parse: see new_context/0, parse/2 and finish/1. It is opaque.

parse/2 does not change the context it is given: it returns a new one. So an earlier context can be used again, and parsing the same chunks from it gives the same results. finish/1 ends the stream and returns no context, so a stream cannot be continued after it.

error_reason()

-type error_reason() :: invalid_utf8 | line_too_long | invalid_char.

Why parsing failed: the input was not valid UTF-8 (invalid_utf8), or a line was longer than the max_line_length option (line_too_long), or had a control character and control_chars is reject (invalid_char).

option()

-type option() :: {max_line_length, pos_integer() | infinity} | {control_chars, pass | strip | reject}.

An option for new_context/1.

  • {max_line_length, Bytes | infinity}: the longest line allowed, in bytes, counted before the LF and including a CR before it. Default 1048576 (1 MiB). A longer line is the error line_too_long.

  • {control_chars, pass | strip | reject}: what to do with control characters, that is C0 controls other than tab, DEL and the C1 controls (U+0080 to U+009F), including a CR inside a line. pass (the default) leaves them in; strip removes them, everywhere including preformatted blocks; reject is the error invalid_char. This is done after the UTF-8 check and before the line is classified, so the result is the same as parsing the text with those characters already removed (for example strip turns the line <<1, "# t">> into a heading).

parse_error()

-type parse_error() :: {error_reason(), pos_integer()}.

A parse error: the reason and the 1-based number of the offending line. Every line counts, including blank ones.

pref_block()

-type pref_block() :: {preformatted, binary(), [binary()]}.

A preformatted block: {preformatted, Alt, Lines}.

Alt is the text after the opening fence, trimmed of spaces and tabs (it may be empty). Lines are the lines of the body, verbatim and without line endings.

tagged_line()

-type tagged_line() :: gemini_text_line:line_element() | pref_block().

One element of a Gemini document. It is one of:

  • {h1 | h2 | h3 | para | list_item | quote, Text}

  • {link, Url, Label}
  • pref_block/0
  • blank

Functions

finish(Context)

-spec finish(Context) -> Result
                when Context :: context(), Result :: {ok, [tagged_line()]} | {error, parse_error()}.

End a streaming parse.

Parses the last line if the document did not end in a line ending, and returns the elements still pending, including a preformatted block which was never closed. The errors are those of parse/1.

links(Text)

-spec links(Text) -> Result
               when
                   Text :: binary(),
                   Result :: {ok, [gemini_text_line:link_line()]} | {error, parse_error()}.

Extract all the links in the given Gemini page.

Each link is {link, Url, Label}, in document order. Links inside preformatted blocks are not extracted. The result is an error in the same cases as parse/1.

new_context()

-spec new_context() -> context().

Start a streaming parse, with nothing seen yet and the default options.

parse/1 uses this too, so it also has the default max_line_length.

new_context(Options)

-spec new_context(Options) -> Result
                     when
                         Options :: [option()],
                         Result :: {ok, context()} | {error, {bad_option, term()}}.

Start a streaming parse with the given options (see option/0).

The options are a proplist: the first occurrence of a key wins. An unknown or malformed option gives {error, {bad_option, Option}}.

parse(Text)

-spec parse(Text) -> Result
               when Text :: binary(), Result :: {ok, [tagged_line()]} | {error, parse_error()}.

Parse Text as a Gemini document.

Returns {ok, Elements} in document order (see tagged_line/0), or {error, {invalid_utf8, N}} if line N (counting from 1, blank lines included) is not valid UTF-8, stopping at the first such line. Lines may end in LF or CRLF, mixed freely; a bare CR is not a line break. Blank lines are kept, as blank. A preformatted block still open at the end of the input is returned as if it had been closed.

A line over 1 MiB (1048576 bytes, counted before the LF and including any CR) is the error {line_too_long, N}; this is checked before UTF-8. Use new_context/1 with parse/2 to change the limit.

The input must be UTF-8; text in any other charset must be transcoded first. Text is passed through unfiltered, including control characters, so renderers should sanitise it before display.

parse(Chunk, Context)

-spec parse(Chunk, Context) -> Result
               when
                   Chunk :: binary(),
                   Context :: context(),
                   Result :: {ok, [tagged_line()], context()} | {error, parse_error()}.

Parse the next chunk of a document.

The chunk may be of any size and may end anywhere, even in the middle of a line, a CRLF or a multibyte character. Returns the elements completed by this chunk, in document order, and the new context to use for the next chunk. A line is only parsed, and checked, once its LF has arrived. A preformatted block is returned as one element when its closing fence arrives, or by finish/1.

On an error, no context is returned, so the stream cannot be continued. The errors are those of parse/1.

urls(Text)

-spec urls(Text) -> Result when Text :: binary(), Result :: {ok, [binary()]} | {error, parse_error()}.

Extract all the URLs in the links in the given Gemini page.

URLs are returned as written, in document order, and are not resolved against any base. Links inside preformatted blocks are not extracted. The result is an error in the same cases as parse/1.