I would not worry about silver fillings in the slightest imo. I've had a mouthful of silver fillings put in through the years -after- developing health problems and it didn't effect my health. I'm a pretty good candidate for this observation because I am effected by any little fraction of a pill I put in my body, yet I've felt no effect from amalgram fillings, and am better now after following diet.
That being said, microscopically, it could potentially leak toxic material into your body, but the fraction would be so small you'd probably get more toxins from the food that you eat (totally speaking out of my ass here/didn't research that). But it wouldn't, imo, cause symptoms you can feel.
Also, fillings fall out all the time and I'm sure that people swallow them from time to time. The filling would just come out the other end and I'm sure taking that trip through your body would be same as if it never came out in the first place?