Is attention mask wrong for batch generation?
#33
by
qingsonglv
- opened
For batch generation, the attention_mask is set to a single 1 referring to this line: https://huggingface.co/THUDM/chatglm-6b/blob/main/modeling_chatglm.py#L948
However, for a batch with various lengths, the left padded tokens are not masked in this case.
position id has the same problem I guess.
I tried to fix it with this PR: https://huggingface.co/THUDM/chatglm-6b/discussions/35
seems like my fault... there's no bug
qingsonglv
changed discussion status to
closed