{"description":"\u4eca\u65e5\u306f\uff0c\u30af\u30e9\u30b9\u8a2d\u8a08\u306b\u95a2\u3059\u308b\u304a\u8a71\uff0e \u6587\u5b57\u5217\u51e6\u7406\u3092\u884c\u3046\u30d7\u30ed\u30b0\u30e9\u30e0\u3092\u66f8\u3044\u3066\u3044\u308b\u3068\uff0ctokenizer\uff08\u3072\u3068\u3064\u306e\u6587\u5b57\u5217\u3092\u7279\u5b9a\u306e\u6587\u5b57\u3092\u533a\u5207\u308a\u6587\u5b57\u3068\u3057\u3066\u8907\u6570\u306e\u6587\u5b57\u5217\u306b\u5207\u308a\u5206\u3051\u308b\u3053\u3068\uff09\u304c\u5fc5\u8981\u3068\u306a\u308b\u5834\u9762\u306b\u591a\u3005\u51fa\u304f\u308f\u3057\u307e\u3059\uff0etokenizer\u306a\u3089boost\u3067\u3082\u4f7f\u3063\u3066\u304a\u3051\u3070\u3044\u3044\u306e\u3067\u3057\u3087\u3046\u304c\uff0cboost\u306etokenizer\u306f\u30e9\u30f3\u30c0\u30e0\u30a2\u30af\u30bb\u30b9\uff08tok[i]\u307f\u305f\u3044\u306a\uff09\u304c\u3067\u304d\u306a\u3044\u306e\u3067\u500b\u4eba\u7684\u306b\u306f\u3061\u3087\u3063\u3068\u4f7f\u3044\u3065\u3089\u3044\u3067\u3059\uff0e\u305d\u3046\u3044\u3046\u8a33\u3067\uff0c\u305b\u3063\u304b\u304f\u306a\u306e\u3067tokenizer\u30af\u30e9\u30b9\u3092\u4f5c\u6210\u3057\u3088\u3046\u3068\u601d\u3044\u7acb\u3061\u307e\u3057\u305f\uff0e \u30af\u30e9\u30b9\u306f\uff0c\u3053\u3093\u306a\u611f\u3058\u306b\u306a\u308a\u307e\u3059\uff0e template< class String, class TokenFunction, \u2026","url":"https://clown.cube-soft.jp/entry/20051013/p1","blog_title":"Cube Lilac","width":"100%","version":"1.0","author_name":"tt_clown","author_url":"https://blog.hatena.ne.jp/tt_clown/","categories":["Development"],"blog_url":"https://clown.cube-soft.jp/","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Fclown.cube-soft.jp%2Fentry%2F20051013%2Fp1\" title=\" \u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u5f15\u6570\u306b\u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u3092\u7528\u3044\u308b - Cube Lilac\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","provider_url":"https://hatena.blog","height":"190","provider_name":"Hatena Blog","published":"2005-10-13 00:00:00","image_url":null,"title":" \u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u5f15\u6570\u306b\u30c6\u30f3\u30d7\u30ec\u30fc\u30c8\u3092\u7528\u3044\u308b","type":"rich"}