[CONFIG] Add Support for Local Speculative Decoding Draft Models #251
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Configure the harness to use Gemma 2B as a draft model for speculative decoding when running larger models like Hermes 14B.