createChat method

Future<InferenceChat> createChat({
  1. double temperature = .8,
  2. int randomSeed = 1,
  3. int topK = 1,
  4. double? topP,
  5. int tokenBuffer = 256,
  6. String? loraPath,
})

Creates a chat interface wrapping InferenceChat.

temperature, randomSeed, topK, topP — parameters for sampling. loraPath — optional path to LoRA model. tokenBuffer — token buffer size for chat for not to exceed max tokens.

Implementation

Future<InferenceChat> createChat({
  double temperature = .8,
  int randomSeed = 1,
  int topK = 1,
  double? topP, // Optional topP for chat too
  int tokenBuffer = 256,
  String? loraPath,
}) async {
  chat = InferenceChat(
    sessionCreator: () => createSession(
      temperature: temperature,
      randomSeed: randomSeed,
      topK: topK,
      topP: topP,
      loraPath: loraPath,
    ),
    maxTokens: maxTokens,
    tokenBuffer: tokenBuffer,
  );
  await chat!.initSession();
  return chat!;
}