{"width":"100%","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Ftadaoyamaoka.hatenablog.com%2Fentry%2F2026%2F05%2F30%2F111704\" title=\"dropless MoE(Mixture of Experts)\u3092\u8a66\u3059 \u305d\u306e6(\u63a8\u8ad6\u901f\u5ea6\u6bd4\u8f03) - TadaoYamaoka\u306e\u958b\u767a\u65e5\u8a18\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","version":"1.0","provider_url":"https://hatena.blog","title":"dropless MoE(Mixture of Experts)\u3092\u8a66\u3059 \u305d\u306e6(\u63a8\u8ad6\u901f\u5ea6\u6bd4\u8f03)","description":"\u524d\u56de\u3001C++\u3067\u5b9f\u88c5\u3057\u305fTensorRT\u30d7\u30e9\u30b0\u30a4\u30f3\u3092\u4f7f\u3063\u305fMoE\u306e\u63a8\u8ad6\u51e6\u7406\u306e\u63a8\u8ad6\u901f\u5ea6\u3092\u6bd4\u8f03\u3059\u308b\u3002 \u6bd4\u8f03\u5bfe\u8c61\u306f\u3001 Dense MoE Sparse MoE (\u81ea\u524d\u5b9f\u88c5CUDA\u30ab\u30fc\u30cd\u30eb) Sparse MoE (CUTLASS Grouped GEMM) \u306e3\u30d1\u30bf\u30fc\u30f3\u3068\u3059\u308b\u3002 \u6bd4\u8f03\u6761\u4ef6 SwinTransformer\u306eStage 0/1\u3092MoE\u5316 Stage 0\u306e\u89e3\u50cf\u5ea6 8x8\u3001State 1\u306e\u89e3\u50cf\u5ea6 4x4 Stage 0\u306ehidden_features 256\u3001State 1\u306ehidden_features 512 Expert\u65704 top_k=2 \u30d0\u30c3\u30c1\u30b5\u30a4\u30ba128 FP16 RTX 4090\u2026","published":"2026-05-30 11:17:04","url":"https://tadaoyamaoka.hatenablog.com/entry/2026/05/30/111704","blog_url":"https://tadaoyamaoka.hatenablog.com/","author_name":"TadaoYamaoka","categories":["MoE"],"author_url":"https://blog.hatena.ne.jp/TadaoYamaoka/","provider_name":"Hatena Blog","blog_title":"TadaoYamaoka\u306e\u958b\u767a\u65e5\u8a18","type":"rich","image_url":null,"height":"190"}