We host the intermediate checkpoints of DeepSeek LLM 7B/67B on AWS S3 (Simple Storage Service). Similar to prefilling, we periodically decide the set of redundant consultants in a sure interval, primarily based on the statistical skilled load from our online service. During decoding, we deal with the shared expert as a routed one. From this perspective, every token will choose 9 experts during routing, where the shared expert is regarded as a heavy-load one that may all the time be selected. D ...