{"author_name":"radiology-nlp","blog_url":"https://radiology-nlp.hatenablog.com/","title":"SentencePiece\u3067\u306e\u65e5\u672c\u8a9e\u5206\u304b\u3061\u66f8\u304d\u3092Transformers\u306e\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u306b\u7d44\u307f\u8fbc\u3080","provider_url":"https://hatena.blog","categories":[],"published":"2021-06-08 01:35:49","version":"1.0","height":"190","blog_title":"radiology-nlp\u2019s blog","width":"100%","provider_name":"Hatena Blog","type":"rich","author_url":"https://blog.hatena.ne.jp/radiology-nlp/","description":"\u80cc\u666f PyTorch\u3067Hugging Face Transformers\u3092\u4f7f\u3063\u3066\u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u3092\u884c\u3046\u3068\u304d\uff0c\u6587\u7ae0\u3092\u30e2\u30c7\u30eb\u306b\u5165\u529b\u3059\u308b\u305f\u3081\u306b\u306f\u307e\u305a\u5358\u8a9e\u5358\u4f4d\u306b\u5206\u304b\u3061\u66f8\u304d (tokenize) \u3057\u306a\u3051\u308c\u3070\u306a\u308a\u307e\u305b\u3093\uff0e \u3053\u306e\u524d\u51e6\u7406\u304c\u601d\u3063\u305f\u3088\u308a\u3082\u3084\u3063\u304b\u3044\u306a\u306e\u3067\u3059\uff0e \u4e8b\u524d\u5b66\u7fd2\u6e08\u307f\u306e\u30e2\u30c7\u30eb\u3092Transformers\u516c\u5f0f\u304b\u3089\u63d0\u4f9b\u3055\u308c\u3066\u3044\u308b\u3082\u306e\u304b\u3089\u9078\u3093\u3067\u3057\u307e\u3048\u3070\uff0c\u3053\u3053\u306f\u3042\u307e\u308a\u554f\u984c\u306b\u306a\u308a\u307e\u305b\u3093\uff0eTransformers\u4ed8\u5c5e\u306etokenizer\u3092\u4f7f\u3063\u3066\u4e00\u767a\u3067\u5206\u304b\u3061\u66f8\u304d\u3067\u304d\u308b\u304b\u3089\u3067\u3059\uff0e \u5b9f\u969b\uff0c\u6771\u5317\u5927\u304b\u3089Transformers\u3092\u901a\u3058\u3066\u65e5\u672c\u8a9eBERT\u4e8b\u524d\u5b66\u7fd2\u6e08\u307f\u30e2\u30c7\u30eb\u304c\u516c\u958b\u3055\u308c\u3066\u4e45\u3057\u3044\u306e\u3067\uff0c\u65e5\u672c\u8a9eBERT\u30e2\u30c7\u30eb\u3092\u4f7f\u3046\u306e\u306f\u3060\u3044\u2026","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Fradiology-nlp.hatenablog.com%2Fentry%2F2021%2F06%2F08%2F013549\" title=\"SentencePiece\u3067\u306e\u65e5\u672c\u8a9e\u5206\u304b\u3061\u66f8\u304d\u3092Transformers\u306e\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u306b\u7d44\u307f\u8fbc\u3080 - radiology-nlp\u2019s blog\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","image_url":null,"url":"https://radiology-nlp.hatenablog.com/entry/2021/06/08/013549"}