(a) Both particles are moving in the x-direction with kinetic energies K=5m.
(b) One particle moves in the y-direction with K=5m and the other moves in the +x-direction with K=5m.
(c) Given that both systems contain two mass-m particles with K=5m, explain any difference between your answers to parts (a) and (b).
In some older textbooks "relativistic mass" is defined as total energy divided by c^2. But this is NOT the modern definition of mass. After the invention of relativity, it became clear that mass and energy are the same things; the equation E = m c^2 merely expresses this fact. The factor c^2 is only there because we still use our old units in which time and space are considered to be different quantities.
Today, the word "mass" in modern physics is reserved for the total energy in the rest frame of the system divided by c^2. Obviously this is an invariant quantity. If you and I are moving with respect to each other and there is some particle moving with some velocity w.r.t. me, then the mass is the energy/c^2 in the rest frame of that particle. I can calculate that if I know the energy and momentum of the particle. You can do that to with your figures, but we must both find the same value. Because of this invariance, the word "invariant mass" is sometimes used for the mass. The word "rest mass" is more often encountered in the older literature to distinguish it from the now obsolete term "relativistic mass". By putting c = 1, you can measure mass and energy in the same units.
So, how do we ...
A detailed solution is given.