Tokenizers for Large Code Language Models
AssigneeAurora Labs Ltd.
InventorsCarmit SAHAR, Daniel YELLIN, Stojancho GANCHEV, Zohar FOX
Life of this patent
Estimating the term…
Abstract
Disclosed herein are techniques for creating and using tokens representing portions of programming code. Techniques include identifying a body of programming code; associating a plurality of tokens with respective portions of the body of programming code; configuring model input data for a code language processing model, the model input data comprising the plurality of tokens; and analyzing at least a part of the body of programming code using the code language processing model influenced by the model input data.
Claims29
A non-transitory computer-readable medium including instructions that, when executed by at least one processor, cause the at least one processor to perform operations for creating and using tokens representing portions of programming code, the operations comprising:
- identifying a body of programming code;
- associating a plurality of tokens with respective portions of the body of programming code;
- configuring model input data for a code language processing model, the model input data comprising the plurality of tokens;
- and
- analyzing at least a part of the body of programming code using the code language processing model influenced by the model input data.
A computer-implemented method for creating and using tokens representing portions of programming code, the method comprising:
- identifying a body of programming code;
- associating a plurality of tokens with respective portions of the body of programming code;
- configuring model input data for a code language processing model, the model input data comprising the plurality of tokens;
- and
- analyzing at least a part of the body of programming code using the code language processing model influenced by the model input data.
Dependencies are inferred from the claim text.