An electron beam system inspection or measurement apparatus comprising an electron optical system including an electron beam source, a deflection element for deflecting electron beams emitted from the electron beam source, and an objective lens for converging and irradiating electron beams deflected by the deflection element on an inspected object, an electron beam image detection optical system for detecting a secondary electron beam image generated from the inspected object by the electron beams deflected by the electron optical system and converged and irradiated, an optical height detection apparatus for optically detecting a height of a surface in an area on the inspected object irradiated by electron beams deflected and converged by the electron optical system, a focus controller for focusing electron beams on inspected object in a properly-focused state by controlling a current flowed to or a voltage applied to said objective lens of said electron optical system on the basis of the height of a surface on the inspected object detected by the optical height detection apparatus, a deflection controller for correcting an image distortion containing a magnification error of electron beams generated on the basis of the focus control by correcting a deflection amount of the electron optical system to the deflection element on the basis of the height of the surface on said inspected object detected by the optical height detection apparatus, and an image processor for inspecting or measuring a pattern formed on the inspected object on the basis of a secondary electron beam image detected by the electron beam detection optical system, wherein the detection of the secondary electron beam image, optical height detection by the optical height detection apparatus, and a real time feed back auto-focus control of the converging electron beam by the focus controller are simultaneously effected.