Mojo function
fused_token_sampling_cpu
fused_token_sampling_cpu[dtype: DType, out_idx_type: DType](max_k: Int, input: LayoutTensor[dtype, layout, origin, address_space=address_space, element_layout=element_layout, layout_int_type=layout_int_type, linear_idx_type=linear_idx_type, masked=masked, alignment=alignment], out_idxs: LayoutTensor[out_idx_type, layout, origin, address_space=address_space, element_layout=element_layout, layout_int_type=layout_int_type, linear_idx_type=linear_idx_type, masked=masked, alignment=alignment], k: OptionalReg[LayoutTensor[DType.int64, Layout.row_major(-1), MutableAnyOrigin]] = OptionalReg[LayoutTensor[DType.int64, Layout.row_major(-1), MutableAnyOrigin]]({:i1 0, 1}), temperature: OptionalReg[LayoutTensor[DType.float32, Layout.row_major(-1), MutableAnyOrigin]] = OptionalReg[LayoutTensor[DType.float32, Layout.row_major(-1), MutableAnyOrigin]]({:i1 0, 1}), top_p: OptionalReg[LayoutTensor[DType.float32, Layout.row_major(-1), MutableAnyOrigin]] = OptionalReg[LayoutTensor[DType.float32, Layout.row_major(-1), MutableAnyOrigin]]({:i1 0, 1}), seed: OptionalReg[LayoutTensor[DType.uint64, Layout.row_major(-1), MutableAnyOrigin]] = OptionalReg[LayoutTensor[DType.uint64, Layout.row_major(-1), MutableAnyOrigin]]({:i1 0, 1}))
Generalized implementation of the Top K algorithm with sampling. Returns the sampled index from the innermost dimension of the input tensor for each row/subvolume.
Parameters:
- dtype (
DType
): Data type of the input buffer. - out_idx_type (
DType
): Data type of the output indices.
Args:
- max_k (
Int
): Largest number of top elements. - input (
LayoutTensor
): NDBuffer[dtype, rank] (Any shape)- The input tensor. - out_idxs (
LayoutTensor
): NDBuffer[out_idx_type, rank] (shape of [input_shape[:-1]] + [1]) - The output indices. - k (
OptionalReg
): Optional device buffer of top elements to keep for each batch element. - temperature (
OptionalReg
): The temperature based scaling. - top_p (
OptionalReg
): Only use the tokens whose cumulative probability exceeds this threshold. - seed (
OptionalReg
): The seed to use for the random number generator.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!