{"url":"https://chakoku.hatenablog.com/entry/2025/08/12/163616","author_url":"https://blog.hatena.ne.jp/chakoku/","image_url":null,"description":"\u80cc\u666f\uff1aLLM\u3092\u7406\u89e3\u3059\u3079\u304fLLM from scratch\u672c\u3067\u30d7\u30ed\u30b0\u30e9\u30e0\u3092\u66f8\u304f \u9032\u6357\uff1a\u300c2.2 \u30c6\u30ad\u30b9\u30c8\u3092\u30c8\u30fc\u30af\u30f3\u5316\u3059\u308b\u300d(P21\uff09 \u8ab2\u984c\uff1a\u6587\u66f8\u3092LLM\u3067\u6271\u3046\u306b\u306f\u3001\u5358\u8a9e\u3092\u30c8\u30fc\u30af\u30f3\uff08ID\uff09\u306b\u7f6e\u304d\u63db\u3048\u308b\u5fc5\u8981\u304c\u3042\u308b\u3002\u3053\u306e\u305f\u3081\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u304c\u5fc5\u8981 \u53d6\u308a\u7d44\u307f\uff1a\u81ea\u4f5c\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u3092\u4f5c\u6210 \u7d50\u8ad6\uff1a\u81ea\u4f5c\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u306f\u3067\u304d\u305f\u8a73\u7d30\uff1a \u672c\u3067\u306f\u6b63\u898f\u8868\u73fe\u3067split\u3055\u305b\u308b\u65b9\u6cd5\u3067\u6587\u66f8\u304b\u3089\u5358\u8a9e\u3092\u5207\u308a\u51fa\u3057\u3066\u3001\u8f9e\u66f8\u306b\u767b\u9332\u3057\u3066\u3044\u304f\u65b9\u6cd5\u304c\u63a1\u7528\u3055\u308c\u3066\u3044\u308b\u3002\u591a\u5206\u4e00\u756a\u77ed\u3044\u30d7\u30ed\u30b0\u30e9\u30e0\u3060\u3068\u601d\u3046\u3002\u4ed6\u4eba\u306e\u771f\u4f3c\u3092\u3059\u308b\u306e\u304c\u5acc\u306a\u3078\u305d\u66f2\u304c\u308a\u306a\u306e\u3067\u3001\u4e00\u6587\u5b57\u305a\u3064\u9806\u756a\u306b\u6587\u5b57\u3092\u8aad\u307f\u8fbc\u307f\u306a\u304c\u3089\u5358\u8a9e\u306b\u5206\u5272\u3059\u308b\u30a2\u30eb\u30b4\u30ea\u30ba\u30e0\u3067\u7d44\u3093\u3067\u307f\u308b\u3002\u306f\u3063\u304d\u308a\u3063\u3066\u3001\u3053\u3063\u3061\u306f\u52b9\u7387\u304c\u60aa\u3044\u3002\u3057\u304b\u2026","type":"rich","provider_name":"Hatena Blog","version":"1.0","published":"2025-08-12 16:36:16","height":"190","title":"LLM from scratch; tokenizer\u3092\u4f5c\u6210","blog_title":"chakoku\u306e\u30d6\u30ed\u30b0(rev4)","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Fchakoku.hatenablog.com%2Fentry%2F2025%2F08%2F12%2F163616\" title=\"LLM from scratch; tokenizer\u3092\u4f5c\u6210 - chakoku\u306e\u30d6\u30ed\u30b0(rev4)\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","width":"100%","provider_url":"https://hatena.blog","categories":[],"blog_url":"https://chakoku.hatenablog.com/","author_name":"chakoku"}