Yes, you should connect the scope probe ground to the circuit ground near where you are probing.
The unit under test (UUT) and the scope may have their grounds connected, but that is via a round-about path. This causes problems:
- The ground path can have significant impedance at high frequencies. Put another way, no, the grounds aren't really connected at high frequencies.
- The loop can pick up noise via inductive coupling. Any voltage induced in the loop will show up on the trace directly as a signal. This can be from the 50 or 60 Hz power line frequency, but also from spikes on the power line, radio transmissions, etc.
- There can be ground offset between two different branches of the power distribution network. This offset again shows up on the trace like a signal.
An easy way see this for yourself is to connect just the probe tip to the ground of UUT, with the scope ground clip unconnected. In theory you should see a flat line at 0 V, but you won't. Everything you see there is noise that will be added to any other UUT signals you look at without grounding the scope probe. Usually this noise is enough to be visible even at something like 1 V/div. If you happen to be in an unusually quiet environment, crank up the gain and you'll see it. Note that anything you see here is noise due to bad scope probe grounding. If your theory were correct, you'd always see flat 0 V, but you won't.
Now hold the grounding clip against the UUT ground nearby, and watch most of the noise disappear. The same experiment works with any DC voltage, like a power supply output. In that case you may have to switch to AC coupling so that the DC level doesn't force the trace off the screen before the gain is enough to seen the noise. You'll see a solid difference between probe ground connected to the UUT and not.