GPT4All Python Generation API

The GPT4All python package provides bindings to our C/C++ model backend libraries. The source code and local build instructions can be found here.

Quickstart

pip install gpt4all

from gpt4all import GPT4All
model = GPT4All("orca-mini-3b-gguf2-q4_0.gguf")

This will:

• Instantiate GPT4All, which is the primary public API to your large language model (LLM).
• Automatically download the given model to ~/.cache/gpt4all/ if not already present.

Read further to see how to chat with this model.

Chatting with GPT4All

To start chatting with a local LLM, you will need to start a chat session. Within a chat session, the model will be prompted with the appropriate template, and history will be preserved between successive calls to generate().

GPT4All ExampleOutput

model = GPT4All(model_name='orca-mini-3b-gguf2-q4_0.gguf')
with model.chat_session():
    response1 = model.generate(prompt='hello', temp=0)
    response2 = model.generate(prompt='write me a short poem', temp=0)
    response3 = model.generate(prompt='thank you', temp=0)
    print(model.current_chat_session)

[
   {
      'role': 'user',
      'content': 'hello'
   },
   {
      'role': 'assistant',
      'content': 'What is your name?'
   },
   {
      'role': 'user',
      'content': 'write me a short poem'
   },
   {
      'role': 'assistant',
      'content': "I would love to help you with that! Here's a short poem I came up with:\nBeneath the autumn leaves,\nThe wind whispers through the trees.\nA gentle breeze, so at ease,\nAs if it were born to play.\nAnd as the sun sets in the sky,\nThe world around us grows still."
   },
   {
      'role': 'user',
      'content': 'thank you'
   },
   {
      'role': 'assistant',
      'content': "You're welcome! I hope this poem was helpful or inspiring for you. Let me know if there is anything else I can assist you with."
   }
]

When using GPT4All models in the chat_session() context:

• Consecutive chat exchanges are taken into account and not discarded until the session ends; as long as the model has capacity.
• A system prompt is inserted into the beginning of the model's context.
• Each prompt passed to generate() is wrapped in the appropriate prompt template. If you pass allow_download=False to GPT4All or are using a model that is not from the official models list, you must pass a prompt template using the prompt_template parameter of chat_session().

NOTE: If you do not use chat_session(), calls to generate() will not be wrapped in a prompt template. This will cause the model to continue the prompt instead of answering it. When in doubt, use a chat session, as many newer models are designed to be used exclusively with a prompt template.

Streaming Generations

To interact with GPT4All responses as the model generates, use the streaming=True flag during generation.

GPT4All Streaming ExampleOutput

from gpt4all import GPT4All
model = GPT4All("orca-mini-3b-gguf2-q4_0.gguf")
tokens = []
with model.chat_session():
    for token in model.generate("What is the capital of France?", streaming=True):
        tokens.append(token)
print(tokens)

[' The', ' capital', ' of', ' France', ' is', ' Paris', '.']

The Generate Method API

generate(prompt, *, max_tokens=200, temp=0.7, top_k=40, top_p=0.4, min_p=0.0, repeat_penalty=1.18, repeat_last_n=64, n_batch=8, n_predict=None, streaming=False, callback=empty_response_callback)

Generate outputs from any GPT4All model.

Parameters:

• prompt (str) –

  The prompt for the model the complete.

• max_tokens (int, default: 200 ) –

  The maximum number of tokens to generate.

• temp (float, default: 0.7 ) –

  The model temperature. Larger values increase creativity but decrease factuality.

• top_k (int, default: 40 ) –

  Randomly sample from the top_k most likely tokens at each generation step. Set this to 1 for greedy decoding.

• top_p (float, default: 0.4 ) –

  Randomly sample at each generation step from the top most likely tokens whose probabilities add up to top_p.

• min_p (float, default: 0.0 ) –

  Randomly sample at each generation step from the top most likely tokens whose probabilities are at least min_p.

• repeat_penalty (float, default: 1.18 ) –

  Penalize the model for repetition. Higher values result in less repetition.

• repeat_last_n (int, default: 64 ) –

  How far in the models generation history to apply the repeat penalty.

• n_batch (int, default: 8 ) –

  Number of prompt tokens processed in parallel. Larger values decrease latency but increase resource requirements.

• n_predict (int | None, default: None ) –

  Equivalent to max_tokens, exists for backwards compatibility.

• streaming (bool, default: False ) –

  If True, this method will instead return a generator that yields tokens as the model generates them.

• callback (ResponseCallbackType, default: empty_response_callback ) –

  A function with arguments token_id:int and response:str, which receives the tokens from the model as they are generated and stops the generation by returning False.

Returns:

• Any –

  Either the entire completion or a generator that yields the completion token by token.

Source code in gpt4all/gpt4all.py

def generate(
    self,
    prompt: str,
    *,
    max_tokens: int = 200,
    temp: float = 0.7,
    top_k: int = 40,
    top_p: float = 0.4,
    min_p: float = 0.0,
    repeat_penalty: float = 1.18,
    repeat_last_n: int = 64,
    n_batch: int = 8,
    n_predict: int | None = None,
    streaming: bool = False,
    callback: ResponseCallbackType = empty_response_callback,
) -> Any:
    """
    Generate outputs from any GPT4All model.

    Args:
        prompt: The prompt for the model the complete.
        max_tokens: The maximum number of tokens to generate.
        temp: The model temperature. Larger values increase creativity but decrease factuality.
        top_k: Randomly sample from the top_k most likely tokens at each generation step. Set this to 1 for greedy decoding.
        top_p: Randomly sample at each generation step from the top most likely tokens whose probabilities add up to top_p.
        min_p: Randomly sample at each generation step from the top most likely tokens whose probabilities are at least min_p.
        repeat_penalty: Penalize the model for repetition. Higher values result in less repetition.
        repeat_last_n: How far in the models generation history to apply the repeat penalty.
        n_batch: Number of prompt tokens processed in parallel. Larger values decrease latency but increase resource requirements.
        n_predict: Equivalent to max_tokens, exists for backwards compatibility.
        streaming: If True, this method will instead return a generator that yields tokens as the model generates them.
        callback: A function with arguments token_id:int and response:str, which receives the tokens from the model as they are generated and stops the generation by returning False.

    Returns:
        Either the entire completion or a generator that yields the completion token by token.
    """

    # Preparing the model request
    generate_kwargs: dict[str, Any] = dict(
        temp=temp,
        top_k=top_k,
        top_p=top_p,
        min_p=min_p,
        repeat_penalty=repeat_penalty,
        repeat_last_n=repeat_last_n,
        n_batch=n_batch,
        n_predict=n_predict if n_predict is not None else max_tokens,
    )

    if self._history is not None:
        # check if there is only one message, i.e. system prompt:
        reset = len(self._history) == 1
        self._history.append({"role": "user", "content": prompt})

        fct_func = self._format_chat_prompt_template.__func__  # type: ignore[attr-defined]
        if fct_func is GPT4All._format_chat_prompt_template:
            if reset:
                # ingest system prompt
                # use "%1%2" and not "%1" to avoid implicit whitespace
                self.model.prompt_model(self._history[0]["content"], "%1%2",
                                        empty_response_callback,
                                        n_batch=n_batch, n_predict=0, reset_context=True, special=True)
            prompt_template = self._current_prompt_template.format("%1", "%2")
        else:
            warnings.warn(
                "_format_chat_prompt_template is deprecated. Please use a chat session with a prompt template.",
                DeprecationWarning,
            )
            # special tokens won't be processed
            prompt = self._format_chat_prompt_template(
                self._history[-1:],
                self._history[0]["content"] if reset else "",
            )
            prompt_template = "%1"
            generate_kwargs["reset_context"] = reset
    else:
        prompt_template = "%1"
        generate_kwargs["reset_context"] = True

    # Prepare the callback, process the model response
    output_collector: list[MessageType]
    output_collector = [
        {"content": ""}
    ]  # placeholder for the self._history if chat session is not activated

    if self._history is not None:
        self._history.append({"role": "assistant", "content": ""})
        output_collector = self._history

    def _callback_wrapper(
        callback: ResponseCallbackType,
        output_collector: list[MessageType],
    ) -> ResponseCallbackType:
        def _callback(token_id: int, response: str) -> bool:
            nonlocal callback, output_collector

            output_collector[-1]["content"] += response

            return callback(token_id, response)

        return _callback

    # Send the request to the model
    if streaming:
        return self.model.prompt_model_streaming(
            prompt,
            prompt_template,
            _callback_wrapper(callback, output_collector),
            **generate_kwargs,
        )

    self.model.prompt_model(
        prompt,
        prompt_template,
        _callback_wrapper(callback, output_collector),
        **generate_kwargs,
    )

    return output_collector[-1]["content"]

Examples & Explanations

Influencing Generation

The three most influential parameters in generation are Temperature (temp), Top-p (top_p) and Top-K (top_k). In a nutshell, during the process of selecting the next token, not just one or a few are considered, but every single token in the vocabulary is given a probability. The parameters can change the field of candidate tokens.

• Temperature makes the process either more or less random. A Temperature above 1 increasingly "levels the playing field", while at a Temperature between 0 and 1 the likelihood of the best token candidates grows even more. A Temperature of 0 results in selecting the best token, making the output deterministic. A Temperature of 1 represents a neutral setting with regard to randomness in the process.

• Top-p and Top-K both narrow the field:

  • Top-K limits candidate tokens to a fixed number after sorting by probability. Setting it higher than the vocabulary size deactivates this limit.
  • Top-p selects tokens based on their total probabilities. For example, a value of 0.8 means "include the best tokens, whose accumulated probabilities reach or just surpass 80%". Setting Top-p to 1, which is 100%, effectively disables it.

The recommendation is to keep at least one of Top-K and Top-p active. Other parameters can also influence generation; be sure to review all their descriptions.

Specifying the Model Folder

The model folder can be set with the model_path parameter when creating a GPT4All instance. The example below is is the same as if it weren't provided; that is, ~/.cache/gpt4all/ is the default folder.

from pathlib import Path
from gpt4all import GPT4All
model = GPT4All(model_name='orca-mini-3b-gguf2-q4_0.gguf', model_path=Path.home() / '.cache' / 'gpt4all')

If you want to point it at the chat GUI's default folder, it should be:

macOSWindowsLinux

from pathlib import Path
from gpt4all import GPT4All

model_name = 'orca-mini-3b-gguf2-q4_0.gguf'
model_path = Path.home() / 'Library' / 'Application Support' / 'nomic.ai' / 'GPT4All'
model = GPT4All(model_name, model_path)

from pathlib import Path
from gpt4all import GPT4All
import os
model_name = 'orca-mini-3b-gguf2-q4_0.gguf'
model_path = Path(os.environ['LOCALAPPDATA']) / 'nomic.ai' / 'GPT4All'
model = GPT4All(model_name, model_path)

from pathlib import Path
from gpt4all import GPT4All

model_name = 'orca-mini-3b-gguf2-q4_0.gguf'
model_path = Path.home() / '.local' / 'share' / 'nomic.ai' / 'GPT4All'
model = GPT4All(model_name, model_path)

Alternatively, you could also change the module's default model directory:

from pathlib import Path
from gpt4all import GPT4All, gpt4all
gpt4all.DEFAULT_MODEL_DIRECTORY = Path.home() / 'my' / 'models-directory'
model = GPT4All('orca-mini-3b-gguf2-q4_0.gguf')

Managing Templates

When using a chat_session(), you may customize the system prompt, and set the prompt template if necessary:

GPT4All Custom Session Templates ExamplePossible Output

from gpt4all import GPT4All
model = GPT4All('wizardlm-13b-v1.2.Q4_0.gguf')
system_template = 'A chat between a curious user and an artificial intelligence assistant.\n'
# many models use triple hash '###' for keywords, Vicunas are simpler:
prompt_template = 'USER: {0}\nASSISTANT: '
with model.chat_session(system_template, prompt_template):
    response1 = model.generate('why is the grass green?')
    print(response1)
    print()
    response2 = model.generate('why is the sky blue?')
    print(response2)

The color of grass can be attributed to its chlorophyll content, which allows it
to absorb light energy from sunlight through photosynthesis. Chlorophyll absorbs
blue and red wavelengths of light while reflecting other colors such as yellow
and green. This is why the leaves appear green to our eyes.

The color of the sky appears blue due to a phenomenon called Rayleigh scattering,
which occurs when sunlight enters Earth's atmosphere and interacts with air
molecules such as nitrogen and oxygen. Blue light has shorter wavelength than
other colors in the visible spectrum, so it is scattered more easily by these
particles, making the sky appear blue to our eyes.

Without Online Connectivity

To prevent GPT4All from accessing online resources, instantiate it with allow_download=False. When using this flag, there will be no default system prompt by default, and you must specify the prompt template yourself.

You can retrieve a model's default system prompt and prompt template with an online instance of GPT4All:

Prompt Template RetrievalOutput

from gpt4all import GPT4All
model = GPT4All('orca-mini-3b-gguf2-q4_0.gguf')
print(repr(model.config['systemPrompt']))
print(repr(model.config['promptTemplate']))

'### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n\n'
'### User:\n{0}\n### Response:\n'

Then you can pass them explicitly when creating an offline instance:

from gpt4all import GPT4All
model = GPT4All('orca-mini-3b-gguf2-q4_0.gguf', allow_download=False)

system_prompt = '### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n\n'
prompt_template = '### User:\n{0}\n\n### Response:\n'

with model.chat_session(system_prompt=system_prompt, prompt_template=prompt_template):
    ...

Interrupting Generation

The simplest way to stop generation is to set a fixed upper limit with the max_tokens parameter.

If you know exactly when a model should stop responding, you can add a custom callback, like so:

GPT4All Custom Stop CallbackOutput

from gpt4all import GPT4All
model = GPT4All('orca-mini-3b-gguf2-q4_0.gguf')

def stop_on_token_callback(token_id, token_string):
    # one sentence is enough:
    if '.' in token_string:
        return False
    else:
        return True

response = model.generate('Blue Whales are the biggest animal to ever inhabit the Earth.',
                          temp=0, callback=stop_on_token_callback)
print(response)

 They can grow up to 100 feet (30 meters) long and weigh as much as 20 tons (18 metric tons).

API Documentation

GPT4All

Python class that handles instantiation, downloading, generation and chat with GPT4All models.

Source code in gpt4all/gpt4all.py

class GPT4All:
    """
    Python class that handles instantiation, downloading, generation and chat with GPT4All models.
    """

    def __init__(
        self,
        model_name: str,
        *,
        model_path: str | os.PathLike[str] | None = None,
        model_type: str | None = None,
        allow_download: bool = True,
        n_threads: int | None = None,
        device: str | None = "cpu",
        n_ctx: int = 2048,
        ngl: int = 100,
        verbose: bool = False,
    ):
        """
        Constructor

        Args:
            model_name: Name of GPT4All or custom model. Including ".gguf" file extension is optional but encouraged.
            model_path: Path to directory containing model file or, if file does not exist, where to download model.
                Default is None, in which case models will be stored in `~/.cache/gpt4all/`.
            model_type: Model architecture. This argument currently does not have any functionality and is just used as
                descriptive identifier for user. Default is None.
            allow_download: Allow API to download models from gpt4all.io. Default is True.
            n_threads: number of CPU threads used by GPT4All. Default is None, then the number of threads are determined automatically.
            device: The processing unit on which the GPT4All model will run. It can be set to:
                - "cpu": Model will run on the central processing unit.
                - "gpu": Model will run on the best available graphics processing unit, irrespective of its vendor.
                - "amd", "nvidia", "intel": Model will run on the best available GPU from the specified vendor.
                - A specific device name from the list returned by `GPT4All.list_gpus()`.
                Default is "cpu".

                Note: If a selected GPU device does not have sufficient RAM to accommodate the model, an error will be thrown, and the GPT4All instance will be rendered invalid. It's advised to ensure the device has enough memory before initiating the model.
            n_ctx: Maximum size of context window
            ngl: Number of GPU layers to use (Vulkan)
            verbose: If True, print debug messages.
        """
        self.model_type = model_type
        # Retrieve model and download if allowed
        self.config: ConfigType = self.retrieve_model(model_name, model_path=model_path, allow_download=allow_download, verbose=verbose)
        self.model = LLModel(self.config["path"], n_ctx, ngl)
        if device is not None and device != "cpu":
            self.model.init_gpu(device)
        self.model.load_model()
        # Set n_threads
        if n_threads is not None:
            self.model.set_thread_count(n_threads)

        self._history: list[MessageType] | None = None
        self._current_prompt_template: str = "{0}"

    def __enter__(self) -> Self:
        return self

    def __exit__(
        self, typ: type[BaseException] | None, value: BaseException | None, tb: TracebackType | None,
    ) -> None:
        self.close()

    def close(self) -> None:
        """Delete the model instance and free associated system resources."""
        self.model.close()

    @property
    def backend(self) -> Literal["cpu", "kompute", "metal"]:
        """The name of the llama.cpp backend currently in use. One of "cpu", "kompute", or "metal"."""
        return self.model.backend

    @property
    def device(self) -> str | None:
        """The name of the GPU device currently in use, or None for backends other than Kompute."""
        return self.model.device

    @property
    def current_chat_session(self) -> list[MessageType] | None:
        return None if self._history is None else list(self._history)

    @staticmethod
    def list_models() -> list[ConfigType]:
        """
        Fetch model list from https://gpt4all.io/models/models3.json.

        Returns:
            Model list in JSON format.
        """
        resp = requests.get("https://gpt4all.io/models/models3.json")
        if resp.status_code != 200:
            raise ValueError(f'Request failed: HTTP {resp.status_code} {resp.reason}')
        return resp.json()

    @classmethod
    def retrieve_model(
        cls,
        model_name: str,
        model_path: str | os.PathLike[str] | None = None,
        allow_download: bool = True,
        verbose: bool = False,
    ) -> ConfigType:
        """
        Find model file, and if it doesn't exist, download the model.

        Args:
            model_name: Name of model.
            model_path: Path to find model. Default is None in which case path is set to
                ~/.cache/gpt4all/.
            allow_download: Allow API to download model from gpt4all.io. Default is True.
            verbose: If True (default), print debug messages.

        Returns:
            Model config.
        """

        model_filename = append_extension_if_missing(model_name)

        # get the config for the model
        config: ConfigType = {}
        if allow_download:
            available_models = cls.list_models()

            for m in available_models:
                if model_filename == m["filename"]:
                    tmpl = m.get("promptTemplate", DEFAULT_PROMPT_TEMPLATE)
                    # change to Python-style formatting
                    m["promptTemplate"] = tmpl.replace("%1", "{0}", 1).replace("%2", "{1}", 1)
                    config.update(m)
                    break

        # Validate download directory
        if model_path is None:
            try:
                os.makedirs(DEFAULT_MODEL_DIRECTORY, exist_ok=True)
            except OSError as e:
                raise RuntimeError("Failed to create model download directory") from e
            model_path = DEFAULT_MODEL_DIRECTORY
        else:
            model_path = Path(model_path)

        if not model_path.exists():
            raise FileNotFoundError(f"Model directory does not exist: {model_path!r}")

        model_dest = model_path / model_filename
        if model_dest.exists():
            config["path"] = str(model_dest)
            if verbose:
                print(f"Found model file at {str(model_dest)!r}", file=sys.stderr)
        elif allow_download:
            # If model file does not exist, download
            filesize = config.get("filesize")
            config["path"] = str(cls.download_model(
                model_filename, model_path, verbose=verbose, url=config.get("url"),
                expected_size=None if filesize is None else int(filesize), expected_md5=config.get("md5sum"),
            ))
        else:
            raise FileNotFoundError(f"Model file does not exist: {model_dest!r}")

        return config

    @staticmethod
    def download_model(
        model_filename: str,
        model_path: str | os.PathLike[str],
        verbose: bool = True,
        url: str | None = None,
        expected_size: int | None = None,
        expected_md5: str | None = None,
    ) -> str | os.PathLike[str]:
        """
        Download model from https://gpt4all.io.

        Args:
            model_filename: Filename of model (with .gguf extension).
            model_path: Path to download model to.
            verbose: If True (default), print debug messages.
            url: the models remote url (e.g. may be hosted on HF)
            expected_size: The expected size of the download.
            expected_md5: The expected MD5 hash of the download.

        Returns:
            Model file destination.
        """

        # Download model
        if url is None:
            url = f"https://gpt4all.io/models/gguf/{model_filename}"

        def make_request(offset=None):
            headers = {}
            if offset:
                print(f"\nDownload interrupted, resuming from byte position {offset}", file=sys.stderr)
                headers['Range'] = f'bytes={offset}-'  # resume incomplete response
                headers["Accept-Encoding"] = "identity"  # Content-Encoding changes meaning of ranges
            response = requests.get(url, stream=True, headers=headers)
            if response.status_code not in (200, 206):
                raise ValueError(f'Request failed: HTTP {response.status_code} {response.reason}')
            if offset and (response.status_code != 206 or str(offset) not in response.headers.get('Content-Range', '')):
                raise ValueError('Connection was interrupted and server does not support range requests')
            if (enc := response.headers.get("Content-Encoding")) is not None:
                raise ValueError(f"Expected identity Content-Encoding, got {enc}")
            return response

        response = make_request()

        total_size_in_bytes = int(response.headers.get("content-length", 0))
        block_size = 2**20  # 1 MB

        partial_path = Path(model_path) / (model_filename + ".part")

        with open(partial_path, "w+b") as partf:
            try:
                with tqdm(desc="Downloading", total=total_size_in_bytes, unit="iB", unit_scale=True) as progress_bar:
                    while True:
                        last_progress = progress_bar.n
                        try:
                            for data in response.iter_content(block_size):
                                partf.write(data)
                                progress_bar.update(len(data))
                        except ChunkedEncodingError as cee:
                            if cee.args and isinstance(pe := cee.args[0], ProtocolError):
                                if len(pe.args) >= 2 and isinstance(ir := pe.args[1], IncompleteRead):
                                    assert progress_bar.n <= ir.partial  # urllib3 may be ahead of us but never behind
                                    # the socket was closed during a read - retry
                                    response = make_request(progress_bar.n)
                                    continue
                            raise
                        if total_size_in_bytes != 0 and progress_bar.n < total_size_in_bytes:
                            if progress_bar.n == last_progress:
                                raise RuntimeError("Download not making progress, aborting.")
                            # server closed connection prematurely - retry
                            response = make_request(progress_bar.n)
                            continue
                        break

                # verify file integrity
                file_size = partf.tell()
                if expected_size is not None and file_size != expected_size:
                    raise ValueError(f"Expected file size of {expected_size} bytes, got {file_size}")
                if expected_md5 is not None:
                    partf.seek(0)
                    hsh = hashlib.md5()
                    with tqdm(desc="Verifying", total=file_size, unit="iB", unit_scale=True) as bar:
                        while chunk := partf.read(block_size):
                            hsh.update(chunk)
                            bar.update(len(chunk))
                    if hsh.hexdigest() != expected_md5.lower():
                        raise ValueError(f"Expected MD5 hash of {expected_md5!r}, got {hsh.hexdigest()!r}")
            except:
                if verbose:
                    print("Cleaning up the interrupted download...", file=sys.stderr)
                try:
                    os.remove(partial_path)
                except OSError:
                    pass
                raise

            # flush buffers and sync the inode
            partf.flush()
            _fsync(partf)

        # move to final destination
        download_path = Path(model_path) / model_filename
        try:
            os.rename(partial_path, download_path)
        except FileExistsError:
            try:
                os.remove(partial_path)
            except OSError:
                pass
            raise

        if verbose:
            print(f"Model downloaded to {str(download_path)!r}", file=sys.stderr)
        return download_path

    @overload
    def generate(
        self, prompt: str, *, max_tokens: int = ..., temp: float = ..., top_k: int = ..., top_p: float = ...,
        min_p: float = ..., repeat_penalty: float = ..., repeat_last_n: int = ..., n_batch: int = ...,
        n_predict: int | None = ..., streaming: Literal[False] = ..., callback: ResponseCallbackType = ...,
    ) -> str: ...
    @overload
    def generate(
        self, prompt: str, *, max_tokens: int = ..., temp: float = ..., top_k: int = ..., top_p: float = ...,
        min_p: float = ..., repeat_penalty: float = ..., repeat_last_n: int = ..., n_batch: int = ...,
        n_predict: int | None = ..., streaming: Literal[True], callback: ResponseCallbackType = ...,
    ) -> Iterable[str]: ...
    @overload
    def generate(
        self, prompt: str, *, max_tokens: int = ..., temp: float = ..., top_k: int = ..., top_p: float = ...,
        min_p: float = ..., repeat_penalty: float = ..., repeat_last_n: int = ..., n_batch: int = ...,
        n_predict: int | None = ..., streaming: bool, callback: ResponseCallbackType = ...,
    ) -> Any: ...

    def generate(
        self,
        prompt: str,
        *,
        max_tokens: int = 200,
        temp: float = 0.7,
        top_k: int = 40,
        top_p: float = 0.4,
        min_p: float = 0.0,
        repeat_penalty: float = 1.18,
        repeat_last_n: int = 64,
        n_batch: int = 8,
        n_predict: int | None = None,
        streaming: bool = False,
        callback: ResponseCallbackType = empty_response_callback,
    ) -> Any:
        """
        Generate outputs from any GPT4All model.

        Args:
            prompt: The prompt for the model the complete.
            max_tokens: The maximum number of tokens to generate.
            temp: The model temperature. Larger values increase creativity but decrease factuality.
            top_k: Randomly sample from the top_k most likely tokens at each generation step. Set this to 1 for greedy decoding.
            top_p: Randomly sample at each generation step from the top most likely tokens whose probabilities add up to top_p.
            min_p: Randomly sample at each generation step from the top most likely tokens whose probabilities are at least min_p.
            repeat_penalty: Penalize the model for repetition. Higher values result in less repetition.
            repeat_last_n: How far in the models generation history to apply the repeat penalty.
            n_batch: Number of prompt tokens processed in parallel. Larger values decrease latency but increase resource requirements.
            n_predict: Equivalent to max_tokens, exists for backwards compatibility.
            streaming: If True, this method will instead return a generator that yields tokens as the model generates them.
            callback: A function with arguments token_id:int and response:str, which receives the tokens from the model as they are generated and stops the generation by returning False.

        Returns:
            Either the entire completion or a generator that yields the completion token by token.
        """

        # Preparing the model request
        generate_kwargs: dict[str, Any] = dict(
            temp=temp,
            top_k=top_k,
            top_p=top_p,
            min_p=min_p,
            repeat_penalty=repeat_penalty,
            repeat_last_n=repeat_last_n,
            n_batch=n_batch,
            n_predict=n_predict if n_predict is not None else max_tokens,
        )

        if self._history is not None:
            # check if there is only one message, i.e. system prompt:
            reset = len(self._history) == 1
            self._history.append({"role": "user", "content": prompt})

            fct_func = self._format_chat_prompt_template.__func__  # type: ignore[attr-defined]
            if fct_func is GPT4All._format_chat_prompt_template:
                if reset:
                    # ingest system prompt
                    # use "%1%2" and not "%1" to avoid implicit whitespace
                    self.model.prompt_model(self._history[0]["content"], "%1%2",
                                            empty_response_callback,
                                            n_batch=n_batch, n_predict=0, reset_context=True, special=True)
                prompt_template = self._current_prompt_template.format("%1", "%2")
            else:
                warnings.warn(
                    "_format_chat_prompt_template is deprecated. Please use a chat session with a prompt template.",
                    DeprecationWarning,
                )
                # special tokens won't be processed
                prompt = self._format_chat_prompt_template(
                    self._history[-1:],
                    self._history[0]["content"] if reset else "",
                )
                prompt_template = "%1"
                generate_kwargs["reset_context"] = reset
        else:
            prompt_template = "%1"
            generate_kwargs["reset_context"] = True

        # Prepare the callback, process the model response
        output_collector: list[MessageType]
        output_collector = [
            {"content": ""}
        ]  # placeholder for the self._history if chat session is not activated

        if self._history is not None:
            self._history.append({"role": "assistant", "content": ""})
            output_collector = self._history

        def _callback_wrapper(
            callback: ResponseCallbackType,
            output_collector: list[MessageType],
        ) -> ResponseCallbackType:
            def _callback(token_id: int, response: str) -> bool:
                nonlocal callback, output_collector

                output_collector[-1]["content"] += response

                return callback(token_id, response)

            return _callback

        # Send the request to the model
        if streaming:
            return self.model.prompt_model_streaming(
                prompt,
                prompt_template,
                _callback_wrapper(callback, output_collector),
                **generate_kwargs,
            )

        self.model.prompt_model(
            prompt,
            prompt_template,
            _callback_wrapper(callback, output_collector),
            **generate_kwargs,
        )

        return output_collector[-1]["content"]

    @contextmanager
    def chat_session(
        self,
        system_prompt: str | None = None,
        prompt_template: str | None = None,
    ):
        """
        Context manager to hold an inference optimized chat session with a GPT4All model.

        Args:
            system_prompt: An initial instruction for the model.
            prompt_template: Template for the prompts with {0} being replaced by the user message.
        """

        if system_prompt is None:
            system_prompt = self.config.get("systemPrompt", "")

        if prompt_template is None:
            if (tmpl := self.config.get("promptTemplate")) is None:
                warnings.warn("Use of a sideloaded model or allow_download=False without specifying a prompt template "
                              "is deprecated. Defaulting to Alpaca.", DeprecationWarning)
                tmpl = DEFAULT_PROMPT_TEMPLATE
            prompt_template = tmpl

        if re.search(r"%1(?![0-9])", prompt_template):
            raise ValueError("Prompt template containing a literal '%1' is not supported. For a prompt "
                             "placeholder, please use '{0}' instead.")

        self._history = [{"role": "system", "content": system_prompt}]
        self._current_prompt_template = prompt_template
        try:
            yield self
        finally:
            self._history = None
            self._current_prompt_template = "{0}"

    @staticmethod
    def list_gpus() -> list[str]:
        """
        List the names of the available GPU devices.

        Returns:
            A list of strings representing the names of the available GPU devices.
        """
        return LLModel.list_gpus()

    def _format_chat_prompt_template(
        self,
        messages: list[MessageType],
        default_prompt_header: str = "",
        default_prompt_footer: str = "",
    ) -> str:
        """
        Helper method for building a prompt from list of messages using the self._current_prompt_template as a template for each message.

        Warning:
            This function was deprecated in version 2.3.0, and will be removed in a future release.

        Args:
            messages:  List of dictionaries. Each dictionary should have a "role" key
                with value of "system", "assistant", or "user" and a "content" key with a
                string value. Messages are organized such that "system" messages are at top of prompt,
                and "user" and "assistant" messages are displayed in order. Assistant messages get formatted as
                "Response: {content}".

        Returns:
            Formatted prompt.
        """

        full_prompt = default_prompt_header + "\n\n" if default_prompt_header != "" else ""

        for message in messages:
            if message["role"] == "user":
                user_message = self._current_prompt_template.format(message["content"])
                full_prompt += user_message
            if message["role"] == "assistant":
                assistant_message = message["content"] + "\n"
                full_prompt += assistant_message

        full_prompt += "\n\n" + default_prompt_footer if default_prompt_footer != "" else ""

        return full_prompt

backend: Literal['cpu', 'kompute', 'metal'] property

The name of the llama.cpp backend currently in use. One of "cpu", "kompute", or "metal".

device: str | None property

The name of the GPU device currently in use, or None for backends other than Kompute.

__init__(model_name, *, model_path=None, model_type=None, allow_download=True, n_threads=None, device='cpu', n_ctx=2048, ngl=100, verbose=False)

Constructor

Parameters:

• model_name (str) –

  Name of GPT4All or custom model. Including ".gguf" file extension is optional but encouraged.

• model_path (str | PathLike[str] | None, default: None ) –

  Path to directory containing model file or, if file does not exist, where to download model. Default is None, in which case models will be stored in ~/.cache/gpt4all/.

• model_type (str | None, default: None ) –

  Model architecture. This argument currently does not have any functionality and is just used as descriptive identifier for user. Default is None.

• allow_download (bool, default: True ) –

  Allow API to download models from gpt4all.io. Default is True.

• n_threads (int | None, default: None ) –

  number of CPU threads used by GPT4All. Default is None, then the number of threads are determined automatically.

• device (str | None, default: 'cpu' ) –

  The processing unit on which the GPT4All model will run. It can be set to: - "cpu": Model will run on the central processing unit. - "gpu": Model will run on the best available graphics processing unit, irrespective of its vendor. - "amd", "nvidia", "intel": Model will run on the best available GPU from the specified vendor. - A specific device name from the list returned by GPT4All.list_gpus(). Default is "cpu".

  Note: If a selected GPU device does not have sufficient RAM to accommodate the model, an error will be thrown, and the GPT4All instance will be rendered invalid. It's advised to ensure the device has enough memory before initiating the model.

• n_ctx (int, default: 2048 ) –

  Maximum size of context window

• ngl (int, default: 100 ) –

  Number of GPU layers to use (Vulkan)

• verbose (bool, default: False ) –

  If True, print debug messages.

Source code in gpt4all/gpt4all.py

def __init__(
    self,
    model_name: str,
    *,
    model_path: str | os.PathLike[str] | None = None,
    model_type: str | None = None,
    allow_download: bool = True,
    n_threads: int | None = None,
    device: str | None = "cpu",
    n_ctx: int = 2048,
    ngl: int = 100,
    verbose: bool = False,
):
    """
    Constructor

    Args:
        model_name: Name of GPT4All or custom model. Including ".gguf" file extension is optional but encouraged.
        model_path: Path to directory containing model file or, if file does not exist, where to download model.
            Default is None, in which case models will be stored in `~/.cache/gpt4all/`.
        model_type: Model architecture. This argument currently does not have any functionality and is just used as
            descriptive identifier for user. Default is None.
        allow_download: Allow API to download models from gpt4all.io. Default is True.
        n_threads: number of CPU threads used by GPT4All. Default is None, then the number of threads are determined automatically.
        device: The processing unit on which the GPT4All model will run. It can be set to:
            - "cpu": Model will run on the central processing unit.
            - "gpu": Model will run on the best available graphics processing unit, irrespective of its vendor.
            - "amd", "nvidia", "intel": Model will run on the best available GPU from the specified vendor.
            - A specific device name from the list returned by `GPT4All.list_gpus()`.
            Default is "cpu".

            Note: If a selected GPU device does not have sufficient RAM to accommodate the model, an error will be thrown, and the GPT4All instance will be rendered invalid. It's advised to ensure the device has enough memory before initiating the model.
        n_ctx: Maximum size of context window
        ngl: Number of GPU layers to use (Vulkan)
        verbose: If True, print debug messages.
    """
    self.model_type = model_type
    # Retrieve model and download if allowed
    self.config: ConfigType = self.retrieve_model(model_name, model_path=model_path, allow_download=allow_download, verbose=verbose)
    self.model = LLModel(self.config["path"], n_ctx, ngl)
    if device is not None and device != "cpu":
        self.model.init_gpu(device)
    self.model.load_model()
    # Set n_threads
    if n_threads is not None:
        self.model.set_thread_count(n_threads)

    self._history: list[MessageType] | None = None
    self._current_prompt_template: str = "{0}"

chat_session(system_prompt=None, prompt_template=None)

Context manager to hold an inference optimized chat session with a GPT4All model.

Parameters:

• system_prompt (str | None, default: None ) –

  An initial instruction for the model.

• prompt_template (str | None, default: None ) –

  Template for the prompts with {0} being replaced by the user message.

Source code in gpt4all/gpt4all.py

@contextmanager
def chat_session(
    self,
    system_prompt: str | None = None,
    prompt_template: str | None = None,
):
    """
    Context manager to hold an inference optimized chat session with a GPT4All model.

    Args:
        system_prompt: An initial instruction for the model.
        prompt_template: Template for the prompts with {0} being replaced by the user message.
    """

    if system_prompt is None:
        system_prompt = self.config.get("systemPrompt", "")

    if prompt_template is None:
        if (tmpl := self.config.get("promptTemplate")) is None:
            warnings.warn("Use of a sideloaded model or allow_download=False without specifying a prompt template "
                          "is deprecated. Defaulting to Alpaca.", DeprecationWarning)
            tmpl = DEFAULT_PROMPT_TEMPLATE
        prompt_template = tmpl

    if re.search(r"%1(?![0-9])", prompt_template):
        raise ValueError("Prompt template containing a literal '%1' is not supported. For a prompt "
                         "placeholder, please use '{0}' instead.")

    self._history = [{"role": "system", "content": system_prompt}]
    self._current_prompt_template = prompt_template
    try:
        yield self
    finally:
        self._history = None
        self._current_prompt_template = "{0}"

close()

Delete the model instance and free associated system resources.

Source code in gpt4all/gpt4all.py

def close(self) -> None:
    """Delete the model instance and free associated system resources."""
    self.model.close()

download_model(model_filename, model_path, verbose=True, url=None, expected_size=None, expected_md5=None) staticmethod

Download model from https://gpt4all.io.

Parameters:

• model_filename (str) –

  Filename of model (with .gguf extension).

• model_path (str | PathLike[str]) –

  Path to download model to.

• verbose (bool, default: True ) –

  If True (default), print debug messages.

• url (str | None, default: None ) –

  the models remote url (e.g. may be hosted on HF)

• expected_size (int | None, default: None ) –

  The expected size of the download.

• expected_md5 (str | None, default: None ) –

  The expected MD5 hash of the download.

Returns:

• str | PathLike[str] –

  Model file destination.

Source code in gpt4all/gpt4all.py

@staticmethod
def download_model(
    model_filename: str,
    model_path: str | os.PathLike[str],
    verbose: bool = True,
    url: str | None = None,
    expected_size: int | None = None,
    expected_md5: str | None = None,
) -> str | os.PathLike[str]:
    """
    Download model from https://gpt4all.io.

    Args:
        model_filename: Filename of model (with .gguf extension).
        model_path: Path to download model to.
        verbose: If True (default), print debug messages.
        url: the models remote url (e.g. may be hosted on HF)
        expected_size: The expected size of the download.
        expected_md5: The expected MD5 hash of the download.

    Returns:
        Model file destination.
    """

    # Download model
    if url is None:
        url = f"https://gpt4all.io/models/gguf/{model_filename}"

    def make_request(offset=None):
        headers = {}
        if offset:
            print(f"\nDownload interrupted, resuming from byte position {offset}", file=sys.stderr)
            headers['Range'] = f'bytes={offset}-'  # resume incomplete response
            headers["Accept-Encoding"] = "identity"  # Content-Encoding changes meaning of ranges
        response = requests.get(url, stream=True, headers=headers)
        if response.status_code not in (200, 206):
            raise ValueError(f'Request failed: HTTP {response.status_code} {response.reason}')
        if offset and (response.status_code != 206 or str(offset) not in response.headers.get('Content-Range', '')):
            raise ValueError('Connection was interrupted and server does not support range requests')
        if (enc := response.headers.get("Content-Encoding")) is not None:
            raise ValueError(f"Expected identity Content-Encoding, got {enc}")
        return response

    response = make_request()

    total_size_in_bytes = int(response.headers.get("content-length", 0))
    block_size = 2**20  # 1 MB

    partial_path = Path(model_path) / (model_filename + ".part")

    with open(partial_path, "w+b") as partf:
        try:
            with tqdm(desc="Downloading", total=total_size_in_bytes, unit="iB", unit_scale=True) as progress_bar:
                while True:
                    last_progress = progress_bar.n
                    try:
                        for data in response.iter_content(block_size):
                            partf.write(data)
                            progress_bar.update(len(data))
                    except ChunkedEncodingError as cee:
                        if cee.args and isinstance(pe := cee.args[0], ProtocolError):
                            if len(pe.args) >= 2 and isinstance(ir := pe.args[1], IncompleteRead):
                                assert progress_bar.n <= ir.partial  # urllib3 may be ahead of us but never behind
                                # the socket was closed during a read - retry
                                response = make_request(progress_bar.n)
                                continue
                        raise
                    if total_size_in_bytes != 0 and progress_bar.n < total_size_in_bytes:
                        if progress_bar.n == last_progress:
                            raise RuntimeError("Download not making progress, aborting.")
                        # server closed connection prematurely - retry
                        response = make_request(progress_bar.n)
                        continue
                    break

            # verify file integrity
            file_size = partf.tell()
            if expected_size is not None and file_size != expected_size:
                raise ValueError(f"Expected file size of {expected_size} bytes, got {file_size}")
            if expected_md5 is not None:
                partf.seek(0)
                hsh = hashlib.md5()
                with tqdm(desc="Verifying", total=file_size, unit="iB", unit_scale=True) as bar:
                    while chunk := partf.read(block_size):
                        hsh.update(chunk)
                        bar.update(len(chunk))
                if hsh.hexdigest() != expected_md5.lower():
                    raise ValueError(f"Expected MD5 hash of {expected_md5!r}, got {hsh.hexdigest()!r}")
        except:
            if verbose:
                print("Cleaning up the interrupted download...", file=sys.stderr)
            try:
                os.remove(partial_path)
            except OSError:
                pass
            raise

        # flush buffers and sync the inode
        partf.flush()
        _fsync(partf)

    # move to final destination
    download_path = Path(model_path) / model_filename
    try:
        os.rename(partial_path, download_path)
    except FileExistsError:
        try:
            os.remove(partial_path)
        except OSError:
            pass
        raise

    if verbose:
        print(f"Model downloaded to {str(download_path)!r}", file=sys.stderr)
    return download_path

generate(prompt, *, max_tokens=200, temp=0.7, top_k=40, top_p=0.4, min_p=0.0, repeat_penalty=1.18, repeat_last_n=64, n_batch=8, n_predict=None, streaming=False, callback=empty_response_callback)

Generate outputs from any GPT4All model.

Parameters:

• prompt (str) –

  The prompt for the model the complete.

• max_tokens (int, default: 200 ) –

  The maximum number of tokens to generate.

• temp (float, default: 0.7 ) –

  The model temperature. Larger values increase creativity but decrease factuality.

• top_k (int, default: 40 ) –

  Randomly sample from the top_k most likely tokens at each generation step. Set this to 1 for greedy decoding.

• top_p (float, default: 0.4 ) –

  Randomly sample at each generation step from the top most likely tokens whose probabilities add up to top_p.

• min_p (float, default: 0.0 ) –

  Randomly sample at each generation step from the top most likely tokens whose probabilities are at least min_p.

• repeat_penalty (float, default: 1.18 ) –

  Penalize the model for repetition. Higher values result in less repetition.

• repeat_last_n (int, default: 64 ) –

  How far in the models generation history to apply the repeat penalty.

• n_batch (int, default: 8 ) –

  Number of prompt tokens processed in parallel. Larger values decrease latency but increase resource requirements.

• n_predict (int | None, default: None ) –

  Equivalent to max_tokens, exists for backwards compatibility.

• streaming (bool, default: False ) –

  If True, this method will instead return a generator that yields tokens as the model generates them.

• callback (ResponseCallbackType, default: empty_response_callback ) –

  A function with arguments token_id:int and response:str, which receives the tokens from the model as they are generated and stops the generation by returning False.

Returns:

• Any –

  Either the entire completion or a generator that yields the completion token by token.

Source code in gpt4all/gpt4all.py

def generate(
    self,
    prompt: str,
    *,
    max_tokens: int = 200,
    temp: float = 0.7,
    top_k: int = 40,
    top_p: float = 0.4,
    min_p: float = 0.0,
    repeat_penalty: float = 1.18,
    repeat_last_n: int = 64,
    n_batch: int = 8,
    n_predict: int | None = None,
    streaming: bool = False,
    callback: ResponseCallbackType = empty_response_callback,
) -> Any:
    """
    Generate outputs from any GPT4All model.

    Args:
        prompt: The prompt for the model the complete.
        max_tokens: The maximum number of tokens to generate.
        temp: The model temperature. Larger values increase creativity but decrease factuality.
        top_k: Randomly sample from the top_k most likely tokens at each generation step. Set this to 1 for greedy decoding.
        top_p: Randomly sample at each generation step from the top most likely tokens whose probabilities add up to top_p.
        min_p: Randomly sample at each generation step from the top most likely tokens whose probabilities are at least min_p.
        repeat_penalty: Penalize the model for repetition. Higher values result in less repetition.
        repeat_last_n: How far in the models generation history to apply the repeat penalty.
        n_batch: Number of prompt tokens processed in parallel. Larger values decrease latency but increase resource requirements.
        n_predict: Equivalent to max_tokens, exists for backwards compatibility.
        streaming: If True, this method will instead return a generator that yields tokens as the model generates them.
        callback: A function with arguments token_id:int and response:str, which receives the tokens from the model as they are generated and stops the generation by returning False.

    Returns:
        Either the entire completion or a generator that yields the completion token by token.
    """

    # Preparing the model request
    generate_kwargs: dict[str, Any] = dict(
        temp=temp,
        top_k=top_k,
        top_p=top_p,
        min_p=min_p,
        repeat_penalty=repeat_penalty,
        repeat_last_n=repeat_last_n,
        n_batch=n_batch,
        n_predict=n_predict if n_predict is not None else max_tokens,
    )

    if self._history is not None:
        # check if there is only one message, i.e. system prompt:
        reset = len(self._history) == 1
        self._history.append({"role": "user", "content": prompt})

        fct_func = self._format_chat_prompt_template.__func__  # type: ignore[attr-defined]
        if fct_func is GPT4All._format_chat_prompt_template:
            if reset:
                # ingest system prompt
                # use "%1%2" and not "%1" to avoid implicit whitespace
                self.model.prompt_model(self._history[0]["content"], "%1%2",
                                        empty_response_callback,
                                        n_batch=n_batch, n_predict=0, reset_context=True, special=True)
            prompt_template = self._current_prompt_template.format("%1", "%2")
        else:
            warnings.warn(
                "_format_chat_prompt_template is deprecated. Please use a chat session with a prompt template.",
                DeprecationWarning,
            )
            # special tokens won't be processed
            prompt = self._format_chat_prompt_template(
                self._history[-1:],
                self._history[0]["content"] if reset else "",
            )
            prompt_template = "%1"
            generate_kwargs["reset_context"] = reset
    else:
        prompt_template = "%1"
        generate_kwargs["reset_context"] = True

    # Prepare the callback, process the model response
    output_collector: list[MessageType]
    output_collector = [
        {"content": ""}
    ]  # placeholder for the self._history if chat session is not activated

    if self._history is not None:
        self._history.append({"role": "assistant", "content": ""})
        output_collector = self._history

    def _callback_wrapper(
        callback: ResponseCallbackType,
        output_collector: list[MessageType],
    ) -> ResponseCallbackType:
        def _callback(token_id: int, response: str) -> bool:
            nonlocal callback, output_collector

            output_collector[-1]["content"] += response

            return callback(token_id, response)

        return _callback

    # Send the request to the model
    if streaming:
        return self.model.prompt_model_streaming(
            prompt,
            prompt_template,
            _callback_wrapper(callback, output_collector),
            **generate_kwargs,
        )

    self.model.prompt_model(
        prompt,
        prompt_template,
        _callback_wrapper(callback, output_collector),
        **generate_kwargs,
    )

    return output_collector[-1]["content"]

list_gpus() staticmethod

List the names of the available GPU devices.

Returns:

• list[str] –

  A list of strings representing the names of the available GPU devices.

Source code in gpt4all/gpt4all.py

@staticmethod
def list_gpus() -> list[str]:
    """
    List the names of the available GPU devices.

    Returns:
        A list of strings representing the names of the available GPU devices.
    """
    return LLModel.list_gpus()

list_models() staticmethod

Fetch model list from https://gpt4all.io/models/models3.json.

Returns:

• list[ConfigType] –

  Model list in JSON format.

Source code in gpt4all/gpt4all.py

@staticmethod
def list_models() -> list[ConfigType]:
    """
    Fetch model list from https://gpt4all.io/models/models3.json.

    Returns:
        Model list in JSON format.
    """
    resp = requests.get("https://gpt4all.io/models/models3.json")
    if resp.status_code != 200:
        raise ValueError(f'Request failed: HTTP {resp.status_code} {resp.reason}')
    return resp.json()

retrieve_model(model_name, model_path=None, allow_download=True, verbose=False) classmethod

Find model file, and if it doesn't exist, download the model.

Parameters:

• model_name (str) –

  Name of model.

• model_path (str | PathLike[str] | None, default: None ) –

  Path to find model. Default is None in which case path is set to ~/.cache/gpt4all/.

• allow_download (bool, default: True ) –

  Allow API to download model from gpt4all.io. Default is True.

• verbose (bool, default: False ) –

  If True (default), print debug messages.

Returns:

• ConfigType –

  Model config.

Source code in gpt4all/gpt4all.py

@classmethod
def retrieve_model(
    cls,
    model_name: str,
    model_path: str | os.PathLike[str] | None = None,
    allow_download: bool = True,
    verbose: bool = False,
) -> ConfigType:
    """
    Find model file, and if it doesn't exist, download the model.

    Args:
        model_name: Name of model.
        model_path: Path to find model. Default is None in which case path is set to
            ~/.cache/gpt4all/.
        allow_download: Allow API to download model from gpt4all.io. Default is True.
        verbose: If True (default), print debug messages.

    Returns:
        Model config.
    """

    model_filename = append_extension_if_missing(model_name)

    # get the config for the model
    config: ConfigType = {}
    if allow_download:
        available_models = cls.list_models()

        for m in available_models:
            if model_filename == m["filename"]:
                tmpl = m.get("promptTemplate", DEFAULT_PROMPT_TEMPLATE)
                # change to Python-style formatting
                m["promptTemplate"] = tmpl.replace("%1", "{0}", 1).replace("%2", "{1}", 1)
                config.update(m)
                break

    # Validate download directory
    if model_path is None:
        try:
            os.makedirs(DEFAULT_MODEL_DIRECTORY, exist_ok=True)
        except OSError as e:
            raise RuntimeError("Failed to create model download directory") from e
        model_path = DEFAULT_MODEL_DIRECTORY
    else:
        model_path = Path(model_path)

    if not model_path.exists():
        raise FileNotFoundError(f"Model directory does not exist: {model_path!r}")

    model_dest = model_path / model_filename
    if model_dest.exists():
        config["path"] = str(model_dest)
        if verbose:
            print(f"Found model file at {str(model_dest)!r}", file=sys.stderr)
    elif allow_download:
        # If model file does not exist, download
        filesize = config.get("filesize")
        config["path"] = str(cls.download_model(
            model_filename, model_path, verbose=verbose, url=config.get("url"),
            expected_size=None if filesize is None else int(filesize), expected_md5=config.get("md5sum"),
        ))
    else:
        raise FileNotFoundError(f"Model file does not exist: {model_dest!r}")

    return config