{"image_url":"https://cdn-ak.f.st-hatena.com/images/fotolife/N/Ninhydrin/20220921/20220921101350.png","blog_url":"https://ninhydrin.hatenablog.com/","provider_url":"https://hatena.blog","height":"190","width":"100%","published":"2022-09-21 19:26:21","blog_title":"\u306b\u3093\u3072\u3069\u308a\u3093\u306f\u3093\u306e\u3046","author_url":"https://blog.hatena.ne.jp/Ninhydrin/","url":"https://ninhydrin.hatenablog.com/entry/2022/09/21/192621","description":"arxiv.orgCADL2022\u52b9\u7387\u7684\u306amulti-head attention\u306e\u63d0\u6848 transformer\u306eattention\u306f\u30c8\u30fc\u30af\u30f3\u306e\u6570\u306e2\u4e57\u30aa\u30fc\u30c0\u30fc\u306e\u8a08\u7b97\u30b3\u30b9\u30c8\u3092\u5fc5\u8981\u3068\u3059\u308b\u3002 \u305d\u306e\u305f\u3081Vision Transformer(ViT)\u306a\u3069\u3067\u9ad8\u89e3\u50cf\u5ea6\u306e\u753b\u50cf\u3092\u6271\u3046\u3068\u30c8\u30fc\u30af\u30f3\u6570\u304c\u81a8\u5927\u306b\u306a\u308a\u3001\u8a08\u7b97\u306e\u307b\u3068\u3093\u3069\u3092attention matrix\u306e\u751f\u6210\u3068\u9069\u7528\u306b\u8cbb\u3084\u3059\u3053\u3068\u306b\u306a\u308b\u3002 \u3053\u308c\u3092\u306a\u3093\u3068\u304b\u3057\u305f\u3044\u3068\u3044\u3046\u304a\u6c17\u6301\u3061\u3002 \u63d0\u6848\u624b\u6cd5 \u4e00\u822c\u7684\u306asoftmax self-attention\u306f\u4ee5\u4e0b\u306e\u5f0f(1) \u30c8\u30fc\u30af\u30f3\u6570\u3092\u3001\u7279\u5fb4\u91cf\u306e\u6b21\u5143\u6570\u3092\u3068\u3059\u308b\u3068\u8a08\u7b97\u91cf\u306f\u3002 softmax\u3092\u3068\u306e\u985e\u4f3c\u5ea6\u3092\u56f3\u308b\u95a2\u6570\u3068\u3057\u3066\u4e00\u822c\u5316\u3057\u305f\u306e\u304c\u2026","version":"1.0","author_name":"Ninhydrin","type":"rich","categories":["Attention","ViT"],"title":" [\u8ad6\u6587\u30e1\u30e2] Hydra Attention: Efficient Attention with Many Heads","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Fninhydrin.hatenablog.com%2Fentry%2F2022%2F09%2F21%2F192621\" title=\" [\u8ad6\u6587\u30e1\u30e2] Hydra Attention: Efficient Attention with Many Heads - \u306b\u3093\u3072\u3069\u308a\u3093\u306f\u3093\u306e\u3046\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","provider_name":"Hatena Blog"}