Maybe some or many of us are similar in the way our immune systems processed and remembered the infectious antigens?
A past infection would have made the immune cells (for example the B cell antibodies and T cell receptors) remember the surface proteins of the antigen.
Once the immune system has done it's job of memorising the antigen's protein marker, later down the line if any of the proteins in the semen resembles the surface proteins of the antigen, the immune system would attack it instantly because it instantly recognizes that protein.
Regarding fernab's observation that people who had similar infections but do not develop POIS - maybe their immune systems processed the antigen and remembered the antigen's surface protein differently. Or maybe those kinds of proteins just don't exist in their semen.
There are studies which show that many people infected with EBV do not develop multiple sclerosis, but 100% of the patients suffering from multiple sclerosis have EBV antibodies. It's still ongoing research. You can google this.