Tokenize tag

Split a character vector based on a tag. The function tokenizes MDA tagged texts by splitting on each space not followed by an <MDA> tag. It will also work on _ST tags by default.

Usage

tokenize_tag(vec, regex = "\\s(?!<\\w+>)")

Arguments

vec

A character vector.

regex

The regular expression used for splitting the vector. Use the default setting for <MDA> tags.

Value

A list of character vectors split by tag.