{"author_url":"https://blog.hatena.ne.jp/TadaoYamaoka/","published":"2026-05-20 21:30:40","provider_name":"Hatena Blog","width":"100%","blog_title":"TadaoYamaoka\u306e\u958b\u767a\u65e5\u8a18","blog_url":"https://tadaoyamaoka.hatenablog.com/","image_url":"https://cdn-ak.f.st-hatena.com/images/fotolife/T/TadaoYamaoka/20260520/20260520211336.png","version":"1.0","type":"rich","author_name":"TadaoYamaoka","description":"\u6700\u8fd1\u306eLLM\u3067\u306f\u3001Attention\u306bQKNorm\u304c\u4f7f\u308f\u308c\u3066\u3044\u308b\u3002 \u7279\u306b\u3001RMSNorm\u3092\u4f7f\u3046\u5b9f\u88c5\u304c\u4e3b\u6d41\u306b\u306a\u3063\u3066\u3044\u308b\u3002\u4e16\u754c\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u5c06\u68cb\u306e\u4f1a\u5834\u3067nshogi\u306e\u958b\u767a\u8005\u3068\u8a71\u3057\u305f\u969b\u306b\u3001SwiGLU\u3068QKNorm\u304c\u52b9\u679c\u304c\u3042\u3063\u305f\u3068\u3044\u3046\u3053\u3068\u3060\u3063\u305f\u3002 SwiGLU\u306f\u3001dlshogi\u3067\u3082\u63a1\u7528\u3057\u3066\u52b9\u679c\u304c\u9ad8\u3044\u3053\u3068\u3092\u78ba\u8a8d\u3057\u3066\u3044\u305f\u304c\u3001QKNorm\u306f\u8a66\u3057\u3066\u3044\u306a\u304b\u3063\u305f\u306e\u3067\u8a66\u3057\u3066\u307f\u305f\u3002 QKNorm Attention\u306e Query \u3068 Key \u3092\u5185\u7a4d\u524d\u306b\u6b63\u898f\u5316\u3057\u3066\u3001attention logits \u306e\u30b9\u30b1\u30fc\u30eb\u66b4\u8d70\u3084 softmax saturation \u3092\u9632\u304e\u3001\u5b66\u7fd2\u3092\u5b89\u5b9a\u5316\u3059\u308b\u624b\u6cd5\u3067\u3042\u308b\u3002 \u5143\u306e\u8ad6\u6587\u3067\u306f\u3001\u6b63\u898f\u5316\u306bL2 \u2026","title":"\u3010dlshogi\u3011QKNorm\u3092\u8a66\u3059","categories":["dlshogi","QKNorm","RMSNorm"],"provider_url":"https://hatena.blog","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Ftadaoyamaoka.hatenablog.com%2Fentry%2F2026%2F05%2F20%2F213040\" title=\"\u3010dlshogi\u3011QKNorm\u3092\u8a66\u3059 - TadaoYamaoka\u306e\u958b\u767a\u65e5\u8a18\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","url":"https://tadaoyamaoka.hatenablog.com/entry/2026/05/20/213040","height":"190"}