whisper_tokenize method

int whisper_tokenize(
  1. Pointer<whisper_context> ctx,
  2. Pointer<Char> text,
  3. Pointer<whisper_token> tokens,
  4. int n_max_tokens,
)

Convert the provided text into tokens. The tokens pointer must be large enough to hold the resulting tokens. Returns the number of tokens on success, no more than n_max_tokens Returns a negative number on failure - the number of tokens that would have been returned TODO: not sure if correct

Implementation

int whisper_tokenize(
  ffi.Pointer<whisper_context> ctx,
  ffi.Pointer<ffi.Char> text,
  ffi.Pointer<whisper_token> tokens,
  int n_max_tokens,
) {
  return _whisper_tokenize(
    ctx,
    text,
    tokens,
    n_max_tokens,
  );
}