Find a unit vector perpendicular to both $\vec a$ and $\vec b$, where $\vec a = \hat i - 2\hat j + 3\hat k$ and $\vec b = \hat i + 2\hat j - \hat k$.

Hint: In cross product (or vector product) of two nonzero vectors $\vec a$ and $\vec b$, the resultant vector is perpendicular to both vectors $\vec a$ and $\vec b$.
So you got the hint, to find a vector perpendicular to two nonzero vectors $\vec a$ and $\vec b$, we have to find the cross product of those two vectors.
Remember that the resultant vector may or may not be a unit vector.
Unit vectors are those vectors whose magnitude is 1.
Therefore, find the unit vector by dividing by vector by its magnitude.

Complete step-by-step answer:
Step 1: Find the cross product of $\vec a$ and $\vec b$.
$\hat a \times \hat b$ is the determinant of the matrix $\left[ {\begin{array}{*{20}{c}}
  {\hat i}&{\hat j}&{\hat k} \\
  1&{ - 2}&3 \\
  1&2&{ - 1}
\end{array}} \right]$
$\hat a \times \hat b$ $ = \left| {\begin{array}{*{20}{c}}
  {\hat i}&{\hat j}&{\hat k} \\
  1&{ - 2}&3 \\
  1&2&{ - 1}
\end{array}} \right|$
\[   \Rightarrow \hat i\left( {2 - 6} \right) - \hat j\left( { - 1 - 3} \right) + \hat k\left( {2 + 2} \right) \]
\[   \Rightarrow \hat i\left( { - 4} \right) - \hat j\left( { - 4} \right) + \hat k\left( 4 \right) \]
\[   \Rightarrow - 4\hat i + 4\hat j + 4\hat k \]
Let $\vec c = \hat a \times \hat b$
Step 2: Find the unit vector $\hat c$:
\[\vec c = - 4\hat i + 4\hat j + 4\hat k\]
Magnitude of \[\vec c\]:
$
  \left| {\vec c} \right| = \sqrt {{{\left( { - 4} \right)}^2} + {{\left( 4 \right)}^2} + {{\left( 4 \right)}^2}} \\
   \Rightarrow {\text{ }} = \sqrt {16 + 16 + 16} \\
   \Rightarrow {\text{ }} = 4\sqrt 3 \\
 $
Unit vector $\hat c = \dfrac{{\vec c}}{{\left| {\vec c} \right|}}$
Therefore, $\hat c$ \[ = \dfrac{{ - 4\hat i + 4\hat j + 4\hat k}}{{4\sqrt 3 }}\]
So, $\hat c$\[ = - \dfrac{1}{{\sqrt 3 }}\hat i + \dfrac{1}{{\sqrt 3 }}\hat j + \dfrac{1}{{\sqrt 3 }}\hat k\]
Unit vector perpendicular to both $\vec a$ and $\vec b$, where $\vec a = \hat i - 2\hat j + 3\hat k$ and $\vec b = \hat i + 2\hat j - \hat k$ is \[ - \dfrac{1}{{\sqrt 3 }}\hat i + \dfrac{1}{{\sqrt 3 }}\hat j + \dfrac{1}{{\sqrt 3 }}\hat k\].

Additional Information:
The unit vector $\hat i$ is along the direction of the x-axis, the unit vector $\hat j$ is along the direction of the y-axis, and the unit vector $\hat k$ is along the direction of the z-axis. Thus, $\hat i,{\text{ }}\hat j{\text{ }}\& {\text{ }}\hat k$ are unit vectors mutually perpendicular to each other.

Note: You can check the final answer by finding the magnitude of the vector $\hat c$.
$\left| {\hat c} \right| = $\[\sqrt {{{\left( { - \dfrac{1}{{\sqrt 3 }}} \right)}^2} + {{\left( {\dfrac{1}{{\sqrt 3 }}} \right)}^2} + {{\left( {\dfrac{1}{{\sqrt 3 }}} \right)}^2}} \]
\[= \sqrt {\dfrac{1}{3} + \dfrac{1}{3} + \dfrac{1}{3}} \]
\[ = \sqrt {\dfrac{3}{3}} = \sqrt 1 \]