Kind of interesting, really what experts do is sort/organise weights into categories that are optimal to work together. Seems like a lot of research could be done to extend this concept to group weights together for common inputs ahead of time to achieve the same purpose.
Yeah, I tried both rearranging experts on disk and predicting the next expert using statistical approach. Reordering helped on the test prompt, but failed on another prompt. Markov and cross layer prediction didn't work either
