You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Refactor LLM runner to bypass the upstream ET runner #1458
Currently the LLMRunner is a thin wrapper over ET's native runner which is experimental and doesn't support many features that it should (e.g. batched prefill - required for models exported with seqlen < ctxlen, custom sampling strategies, constrained decoding, etc.). The scope of this issue is to reimplement it inside TS using the Model primitive and drive the generation directly from TS.
Activity