{"type":"rich","provider_url":"https://hatena.blog","author_url":"https://blog.hatena.ne.jp/iwiwi/","description":"arxiv.org host memory\u3084NVMe\u3092\u4f7f\u3063\u305fout-of-core\u306e\u5b9f\u88c5\u3092\u771f\u9762\u76ee\u306b\u30af\u30bd\u9811\u5f35\u308b\u3053\u3068\u3067\u3001\u3088\u308a\u5927\u304d\u306atransformer\u3092\u9ad8\u3044\u5b9f\u884c\u52b9\u7387\u3092\u7dad\u6301\u3057\u306a\u304c\u3089\u5b66\u7fd2\u3067\u304d\u308b\u3088\u3046\u306b\u3059\u308b\u8a71\u3002 \u624b\u6cd5 \u57fa\u672c\u7684\u306b\u306f\u5404\u7a2e\u30c7\u30fc\u30bf\uff08param, optimizer stats, activation\u306a\u3069\uff09\u306e\u30a2\u30af\u30bb\u30b9\u30d1\u30bf\u30fc\u30f3\u3092\u8003\u3048\u308b\u3002\u6b21\u306b\u3001\u305d\u308c\u306b\u5fdc\u3058\u3066\u8981\u6c42\u3055\u308c\u308b\u8aad\u307f\u8fbc\u307fbandwidth\u3092\u8a08\u7b97\u3059\u308b\u3002\u5b9f\u884c\u52b9\u7387\u3092\u7dad\u6301\u3057\u305f\u3044\u306e\u3067\u3001\u300c\u8a08\u7b97\u3068overlap\u3055\u305b\u3089\u308c\u308b\u304b\u3069\u3046\u304b\u300d\u3067\u8981\u6c42bandwidth\u304c\u6c7a\u307e\u308b\u3002\u3042\u3068\u306f\u3001\u305d\u308c\u306b\u5fdc\u3058\u3066\u30c7\u30fc\u30bf\u306e\u914d\u7f6e\u3092\u3059\u308b\u3002 \u3053\u308c\u3092\u6c17\u5408\u3092\u5165\u308c\u3066\u5b9f\u88c5\u3059\u308b\u3002 \u611f\u60f3 \u4f55\u304b\u3073\u3063\u304f\u308a\u3059\u308b\u3088\u3046\u306a\u2026","blog_url":"https://iwiwi.hatenadiary.jp/","html":"<iframe src=\"https://hatenablog-parts.com/embed?url=https%3A%2F%2Fiwiwi.hatenadiary.jp%2Fentry%2F2023%2F03%2F25%2F000000\" title=\"ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning - iwiwi \u5099\u5fd8\u9332\" class=\"embed-card embed-blogcard\" scrolling=\"no\" frameborder=\"0\" style=\"display: block; width: 100%; height: 190px; max-width: 500px; margin: 10px 0px;\"></iframe>","title":"ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning","published":"2023-03-25 00:00:00","width":"100%","provider_name":"Hatena Blog","blog_title":"iwiwi \u5099\u5fd8\u9332","image_url":null,"categories":["LLM"],"version":"1.0","height":"190","url":"https://iwiwi.hatenadiary.jp/entry/2023/03/25/000000","author_name":"iwiwi"}