Mack ,Oren's answer is maybe too decisive, but the bottom line is that though rats may not be aggressive, by their very nature are opportunistic feeders and will view the smaller hamsters as nothing more than prey.
The only mix I know of that works is gerbils and hamsters, and even then you have to start when they are barely walking.