{"title":"Q: ChatGPT(gpt-3.5-turbo)\u306e\u6700\u5927\u30c8\u30fc\u30af\u30f3\u6570\u306f4096\u3067\u3059\u3002\u306a\u306e\u3067\u3001\u30d7\u30ed\u30f3\u30d7\u30c8\u306e\u9577\u3055\u304c30000\u6587\u5b57\u3092\u8d85\u3048\u308b\u3068\u304d\u306b\u56de\u7b54\u3092\u5f97\u308b\u3053\u3068\u306f\u3067\u304d\u306a\u3044\u3002\u25ef\u304b\u2613\u304b","url":"https://nikkie-ftnext.hatenablog.com/entry/chatgpt-delimiter-length-power-of-2-is-1-token","width":"100%","blog_url":"https://nikkie-ftnext.hatenablog.com/","author_url":"https://blog.hatena.ne.jp/nikkie-ftnext/","published":"2023-05-23 23:23:28","description":"\u7b54\u3048\u306f\u2613(\u30d0\u30c4) \u6700\u5927\u30c8\u30fc\u30af\u30f3\u6570\uff08max tokens\uff09\u306f40961\u3067\u3059\u304c\u3001\u30d7\u30ed\u30f3\u30d7\u30c8\u3092\u5de5\u592b\u3059\u308c\u307030000\u5b57\u3092\u8d85\u3048\u308b\u9577\u3055\u3067\u3042\u3063\u3066\u3082\u56de\u7b54\u3092\u5f97\u3089\u308c\u307e\u3059\uff01 \u306a\u304a\u3001\u3053\u306e\u77e5\u898b\u306f\u300c\u3078\u3047\u300d\u30ec\u30d9\u30eb\u3067\u3001LLM\u3092\u4f7f\u3063\u3066\u3044\u304f\u4e0a\u3067\u306f\u5f79\u306b\u7acb\u305f\u306a\u3044\u3068\u601d\u3044\u307e\u3059\uff08\u3053\u3053\u3067\u5f15\u304d\u8fd4\u3057\u3066\u3082\u3048\u3048\u3093\u3084\u3067\uff09\u3002 \u76ee\u6b21 \u7b54\u3048\u306f\u2613(\u30d0\u30c4) \u76ee\u6b21 \u5148\u884c\u7814\u7a76\uff1a\u533a\u5207\u308a\u7dda\u306f2\u306e\u51aa\u4e57\u30671\u30c8\u30fc\u30af\u30f3 2\u306e12\u4e57\uff084096\u6587\u5b57\uff09\u306e\u533a\u5207\u308a\u7dda\u3067\u3082\u3044\u3051\u308b\u3063\u3066\u3053\u3068\uff1f \u52d5\u4f5c\u74b0\u5883 4096\u6587\u5b57\u306e\u533a\u5207\u308a\u7dda\u3092\u542b\u3093\u3060\u30d7\u30ed\u30f3\u30d7\u30c8\u3092\u9001\u308b 2\u306e12\u4e57\uff084096\uff09\u6587\u5b57\u306e\u533a\u5207\u308a\u7dda\u3092tiktoken\u3067\u898b\u3066\u307f\u308b 2\u306e\u51aa\u4e57\u306e\u6587\u5b57\u6570\u306e\u533a\u5207\u308a\u7dda\u306f64\u6587\u5b57\u30671\u30c8\u30fc\u30af\u30f3\u3068\u52b9\u7387\u3088\u304f\u30c8\u30fc\u30af\u30f3\u5316\u3055\u308c\u308b\u306e\u3067\u2026","author_name":"nikkie-ftnext","height":"190","categories":["\u9b54\u6cd5\u5c11\u5973\u30de\u30eb\u30ab\u30d0\u30c4\u30ab"],"provider_name":"Hatena Blog","type":"rich","provider_url":"https://hatena.blog","image_url":null,"version":"1.0","blog_title":"nikkie-ftnext\u306e\u65e5\u8a18","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Fnikkie-ftnext.hatenablog.com%2Fentry%2Fchatgpt-delimiter-length-power-of-2-is-1-token\" title=\"Q: ChatGPT(gpt-3.5-turbo)\u306e\u6700\u5927\u30c8\u30fc\u30af\u30f3\u6570\u306f4096\u3067\u3059\u3002\u306a\u306e\u3067\u3001\u30d7\u30ed\u30f3\u30d7\u30c8\u306e\u9577\u3055\u304c30000\u6587\u5b57\u3092\u8d85\u3048\u308b\u3068\u304d\u306b\u56de\u7b54\u3092\u5f97\u308b\u3053\u3068\u306f\u3067\u304d\u306a\u3044\u3002\u25ef\u304b\u2613\u304b - nikkie-ftnext\u306e\u65e5\u8a18\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>"}