Needle-in-a-Haystack sentence retrieval test, model performance and loss changes with data scaling

100M tokens, acc 37.8

300M tokens, acc 59.0

500M tokens, acc 81.1

0%

1.

50%

ft

i

1

fo

100%

|

[I

[|

[|

1K

16K

32K

64K

128K

1K

16K 32K

64K

128K 1K

16K 32K

64K

128K

Loss@nk

1.87 1.65

1.50

1.45

1.61

1.82 1.57

1.45

1.35

1.44

1.81 1.59

1.43

1.34

1.34

1B tokens, acc 85.3

5B tokens, acc 88.0

10B tokens, acc 84.0

0% |

|

Ti

a

AY

a

nt

50% |

|

i

i

7

Ht

100% }

LT

1K

16K

32K

64K

128K

1K

16K 32K

64K

128K 1K

16K 32K

64K

128K

Loss@nk

1.81 1.57

1.42

1.34

1.38

1.80 1.56

1.41

1.32

1.36

1.79 1.55

1.40

1.31

1.35
