Prove that the determinant of $\left( {\begin{array}{*{20}{c}}
a&h&g\\
h&b&f\\
g&f&c
\end{array}} \right)$is $abc + 2fgh - a{f^2} - {h^2}c - g{b^2}$.

Hint: To find the determinant of the $3 \times 3$matrix first break down the matrix into $2 \times 2$determinant problems which will be easy to do. Notice that the top row elements are $a$,$h$and $g$are the scalar multipliers to a corresponding $2 \times 2$matrix. That is scalar
$a$is being multiplied to the $2 \times 2$matrix of left over elements created when vertical and horizontal line segments are drawn passing through $a$. Do the same for $h$and $g$also.


Complete step by step solution
Given:
The given matrix is $\left( {\begin{array}{*{20}{c}}
a&h&g\\
h&b&f\\
g&f&c
\end{array}} \right)$.

The formula to find the determinant of $3 \times 3$matrix is a break down to $2 \times 2$determinant problems which are very easy to handle.

Given a matrix $\left( {\begin{array}{*{20}{c}}
a&h&g\\
h&b&f\\
g&f&c
\end{array}} \right)$

The determinant of a matrix is calculated by using the key points.

First take one element in the row that is $a$.

Now draw a line horizontally and vertically from $a$. We get

$\left( {\begin{array}{*{20}{c}}
a&{\rlap{-} h}&{\rlap{-} g}\\
{\rlap{-} h}&b&f\\
{\rlap{-} g}&f&c
\end{array}} \right)$.

The remaining terms are $\left( {\begin{array}{*{20}{c}}
b&f\\
f&c
\end{array}} \right)$.
Multiply $a$with the determinant of the $2 \times 2$matrix $\left| {\begin{array}{*{20}{c}}
b&f\\
f&c
\end{array}} \right|$.


Second take one element in the row that is $h$.

Now draw a line horizontally and vertically from $h$. We get

$\left( {\begin{array}{*{20}{c}}
{\rlap{-} a}&h&{\rlap{-} g}\\
h&{\rlap{-} b}&f\\
g&{\rlap{-} f}&c
\end{array}} \right)$.

The remaining terms are $\left( {\begin{array}{*{20}{c}}
h&f\\
g&c
\end{array}} \right)$.


Multiply $h$with the determinant of the $2 \times 2$matrix $\left| {\begin{array}{*{20}{c}}
h&f\\
g&c
\end{array}} \right|$.

Thirdly take one element in the row that is $g$.

Now draw a line horizontally and vertically from
$g$. We get

$\left( {\begin{array}{*{20}{c}}
{\rlap{-} a}&{\rlap{-} h}&g\\
h&b&{\rlap{-} f}\\
g&f&{\rlap{-} c}
\end{array}} \right)$.

The remaining terms are $\left( {\begin{array}{*{20}{c}}
h&b\\
g&f
\end{array}} \right)$.

Multiply $g$with the determinant of the $2 \times 2$matrix
$\left| {\begin{array}{*{20}{c}}
h&b\\
g&f
\end{array}} \right|$.

Then the determinant will be ,


$a\left| {\begin{array}{*{20}{c}}
b&f\\
f&c
\end{array}} \right| - h\left| {\begin{array}{*{20}{c}}
h&f\\
g&c
\end{array}} \right| + g\left| {\begin{array}{*{20}{c}}
h&b\\
g&f
\end{array}} \right|$..........(1)

To find the determinant of
$\left| {\begin{array}{*{20}{c}}
b&f\\
f&c
\end{array}} \right|$is
$bc - {f^2}$

The determinant of $\left| {\begin{array}{*{20}{c}}
h&f\\
g&c
\end{array}} \right|$is $hc - gf$

The determinant of $\left| {\begin{array}{*{20}{c}}
h&b\\
g&f
\end{array}} \right|$is $hf - gb$.


Substitute the determinants in (1) we get


$\begin{array}{l}
a\left| {\begin{array}{*{20}{c}}
b&f\\
f&c
\end{array}} \right| - h\left| {\begin{array}{*{20}{c}}
h&f\\
g&c
\end{array}} \right| + g\left| {\begin{array}{*{20}{c}}
h&b\\
g&f
\end{array}} \right|\\
a\left( {bc - {f^2}} \right) - h\left( {hc - gf} \right) + g\left( {hf - {b^2}} \right)\\
abc - a{f^2} - {h^2}c + hgf + hgf - g{b^2}
\end{array}$

Hence the determinant of $\left( {\begin{array}{*{20}{c}}
a&h&g\\
h&b&f\\
g&f&c
\end{array}} \right)$ is $abc - a{f^2} - {h^2}c + 2hgf - g{b^2}$ .

Hence proved


Note: This process of finding determinant can be done for higher matrix but finally need to reduce the higher order also to $2$by $2$matrix determinant form and the minus sign will be alternate form.