A method of forming a non-magnetic portion of a composite magnetic member, comprising the steps of hot working an Fe—Cr—C-base alloy steel containing 0.1 to 5.0 weight % Al at a temperature not higher than 1100° C., annealing said alloy steel at least once at a temperature not higher than A3 transformation point so that said ferromagnetic portion is obtained in which a number of carbides with a grain size of not less than 0.1 μm is regulated to not more than 50 in an area of 100 μm2 and in which a proportion of another number of carbides with a grain size of not less than 1.0 μm to the number of carbides of not less than 0.1 μm in grain size in an area of 100 μm2 is regulated to not less than 15%, heating a part of said ferromagnetic portion in a temperature range of 1050° C. to the melting point, and cooling said heated part so that said non-magnetic portion is obtained.