Visibility is one of the most subjective aspects of diving. Ask 10 people after a dive what they think the visibility was, and you'll get 9 different answers. The last person will spend a half hour trying to guess.
People generally suck at estimating visibility ("viz"). Why is that? I think it comes down to a lack of common definition. I will attempt to define visibility in concrete terms.
The best place to start for this sort of thing is the dictionary. Webster's defines visibility as
The distance at which something can be seen.
We're getting closer, but what exactly is the "something" to be seen? I've heard a few options: the distance at which hand signals are no longer recognizable, the distance you can't see a diver's bubbles, and the distance at which you can't distinguish an object. The first might be too strict for general use, but the last two start to get the point. Let's define visibility as the distance at which an object cannot be discerned. The point at which you can't tell for sure if that is a diver in the distance is the extent of the visibility.
That was the easy part, now comes the hard part. How do we measure viz? Believe it or not, there is an established manner. It requires something called a Secchi disk. The disc is attached to a line and gradually moved from the measurer. The distance where the disk cannot be seen is recorded, with the length of the line being the visibility. Even more advanced methods with higher precision exist using photometers.
That's nice for Jacques Cousteau, but not for Joe Diver who doesn't have a Secchi disk on hand.
First of all, don't feel bad. Viz is hard to guesstimate, but gets easier with experience. Even then, in general a range is a better indicator for estimating viz rather than a hard, fixed number. Try to get the lower end where things get blurry and the upper end where things can no longer be seen. As you get better with distances, this range will get smaller and the "true" viz will probably lie somewhere in between.
One rule with measuring viz for scuba: it measures horizontal distance. Vertical distance can be deceiving, as light from the surface increases visibility. However, it can be useful both for an upper bound on your range (vertical viz will always be higher than horizontal) and for estimating distances, since you always know your depth. Some, as a rule of thumb, define the viz as the vertical visibility (the depth where you can no longer see the surface) minus 3 m / 10 ft. Probably a good guideline, but useless in most of the tropics, where the viz always exceeds the dive depth.
Try to use anything you have to your advantage. Length of a wreck, for instance. You can find this out easily and use it to gauge distances more accurately.
I'll let you in on a secret: it doesn't really matter. Viz is cool for bragging ("We could easily see over 100 meters!"), but there aren't too many practical reasons where an accurate number is necessary. There are exceptions, however. For instance, a photographer probably cares more so than others, since the equipment they take can depend on these numbers.
In general, ranges are fine, and even broad descriptions sufficing as well. A scale that goes something like amazing -> great -> good -> ok -> where's my buddy? -> I can't see my hand, is descriptive and covers most situations. Make an educated guess, compare to the guesses of others to refine your estimation abilities, but don't get too hung up on the ever-elusive viz.