{"provider_name":"Hatena Blog","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Fnikkie-ftnext.hatenablog.com%2Fentry%2Fobserve-huggingface-tokenizers-normalizers\" title=\"huggingface/tokenizers\u306eNormalizer\u89b3\u5bdf\u8a18 \u301c\u51e6\u7406\u306e\u90e8\u54c1\u5316\u3068\u7d71\u4e00\u3055\u308c\u305f\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30fc\u30b9\u301c - nikkie-ftnext\u306e\u65e5\u8a18\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","description":"\u306f\u3058\u3081\u306b \u3046\u301c\u3093\u3001\u304a\u3044\u3057\u301c\ud83d\ude0b1 \u3001nikkie\u3067\u3059\ud83d\udc1f Python\u88fdOSS\u306e\u30bd\u30fc\u30b9\u30b3\u30fc\u30c9\u3092\u8aad\u3080\u306e\u304c\u597d\u304d\u3067\u3001\u305d\u3053\u3067\u5f97\u3089\u308c\u305f\u77e5\u898b\uff08\u65b0\u3057\u304f\u77e5\u3063\u305f\u66f8\u304d\u65b9\u3084\u8a2d\u8a08\u4f8b\uff09\u3092\u5b9f\u88c5\u306e\u53c2\u8003\u306b\u3057\u307e\u3059\u3002 \u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u306e\u524d\u51e6\u7406\u306b\u3064\u3044\u3066\u53c2\u8003\u306b\u3057\u305f\u304f\u3001huggingface/tokenizers\u306e\u30bd\u30fc\u30b9\u30b3\u30fc\u30c9\uff08\u53b3\u5bc6\u306b\u306f\u578b\u5b9a\u7fa9\u306e\u30b9\u30bf\u30d6\uff09\u3092\u8aad\u307f\u307e\u3057\u305f\u3002 \u4eca\u56de\u306f\u8aad\u3093\u3067\u8003\u3048\u305f\u3053\u3068\u3092\u30e1\u30e2\u30ec\u30d9\u30eb\u3067\u30a2\u30a6\u30c8\u30d7\u30c3\u30c8\u3057\u307e\u3059\u3002 \u76ee\u6b21 \u306f\u3058\u3081\u306b \u76ee\u6b21 huggingface/tokenizers Normalizers\u306e\u30bd\u30fc\u30b9\u30b3\u30fc\u30c9\u30ea\u30fc\u30c7\u30a3\u30f3\u30b0 \u3059\u3079\u3066\u306enormalizer\u306e\u30d9\u30fc\u30b9\u30af\u30e9\u30b9Normalizer Normalizer\u30af\u30e9\u30b9\u3092\u7d99\u627f\u3057\u305f\u5177\u4f53\u2026","height":"190","url":"https://nikkie-ftnext.hatenablog.com/entry/observe-huggingface-tokenizers-normalizers","blog_title":"nikkie-ftnext\u306e\u65e5\u8a18","published":"2022-12-06 23:50:43","author_url":"https://blog.hatena.ne.jp/nikkie-ftnext/","image_url":null,"blog_url":"https://nikkie-ftnext.hatenablog.com/","title":"huggingface/tokenizers\u306eNormalizer\u89b3\u5bdf\u8a18 \u301c\u51e6\u7406\u306e\u90e8\u54c1\u5316\u3068\u7d71\u4e00\u3055\u308c\u305f\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30fc\u30b9\u301c","categories":["NLP(\u81ea\u7136\u8a00\u8a9e\u51e6\u7406)","\u8a2d\u8a08"],"version":"1.0","type":"rich","width":"100%","provider_url":"https://hatena.blog","author_name":"nikkie-ftnext"}