Let $\alpha \in R$ and the three vectors $\overrightarrow{a}=\alpha \hat{i}+\hat{j}+3\hat{k}$, $\overrightarrow{b}=2\hat{i}+\hat{j}-\alpha \hat{k}$ and $\overrightarrow{c}=\alpha \hat{i}-2\hat{j}+3\hat{k}$. Then the set $S=\left\{ \alpha :\overrightarrow{a},\overrightarrow{b}\text{ and }\overrightarrow{c}\text{ are coplanar} \right\}$.
(A) is singleton
(B) contains only two numbers only one of which is positive
(C) contains exactly two positive numbers
(D) is empty

Hint: We start solving this problem by going through the concept of scalar triple product and vectors being coplanar. Then we find the scalar triple product of given vectors using the formula $\left[ \begin{matrix}
   \overrightarrow{a} & \overrightarrow{b} & \overrightarrow{c} \\
\end{matrix} \right]=\left| \begin{matrix}
   {{a}_{1}} & {{a}_{2}} & {{a}_{3}} \\
   {{b}_{1}} & {{b}_{2}} & {{b}_{3}} \\
   {{c}_{1}} & {{c}_{2}} & {{c}_{3}} \\
\end{matrix} \right|$. Then we equate it to zero to find the condition for which they are coplanar. So, we find the set S as it is a set of values of $\alpha $ for which the given vectors are coplanar.


Complete step-by-step answer:
First, let us go through the concept of scalar triple product.
For any three vectors $\overrightarrow{a}={{a}_{1}}\hat{i}+{{a}_{2}}\hat{j}+{{a}_{3}}\hat{k}$, $\overrightarrow{b}={{b}_{1}}\hat{i}+{{b}_{2}}\hat{j}+{{b}_{3}}\hat{k}$ and $\overrightarrow{c}={{c}_{1}}\hat{i}+{{c}_{2}}\hat{j}+{{c}_{3}}\hat{k}$ scalar triple product is defined as
$\left[ \begin{matrix}
   \overrightarrow{a} & \overrightarrow{b} & \overrightarrow{c} \\
\end{matrix} \right]=\left| \begin{matrix}
   {{a}_{1}} & {{a}_{2}} & {{a}_{3}} \\
   {{b}_{1}} & {{b}_{2}} & {{b}_{3}} \\
   {{c}_{1}} & {{c}_{2}} & {{c}_{3}} \\
\end{matrix} \right|$
Now, let us go through the concept of coplanar. Any three vectors are said to be coplanar if they lie on the same plane. The three vectors $\overrightarrow{a}={{a}_{1}}\hat{i}+{{a}_{2}}\hat{j}+{{a}_{3}}\hat{k}$, $\overrightarrow{b}={{b}_{1}}\hat{i}+{{b}_{2}}\hat{j}+{{b}_{3}}\hat{k}$ and $\overrightarrow{c}={{c}_{1}}\hat{i}+{{c}_{2}}\hat{j}+{{c}_{3}}\hat{k}$ are said to be coplanar if their scalar triple product is equal to zero, that is $\left[ \begin{matrix}
   \overrightarrow{a} & \overrightarrow{b} & \overrightarrow{c} \\
\end{matrix} \right]=0$.

So, now let us consider the scale triple product of given three vectors $\overrightarrow{a}=\alpha \hat{i}+\hat{j}+3\hat{k}$, $\overrightarrow{b}=2\hat{i}+\hat{j}-\alpha \hat{k}$ and $\overrightarrow{c}=\alpha \hat{i}-2\hat{j}+3\hat{k}$.
\[\begin{align}
  & \Rightarrow \left[ \begin{matrix}
   \overrightarrow{a} & \overrightarrow{b} & \overrightarrow{c} \\
\end{matrix} \right]=\left| \begin{matrix}
   \alpha & 1 & 3 \\
   2 & 1 & -\alpha \\
   \alpha & -2 & 3 \\
\end{matrix} \right| \\
 & \Rightarrow \left[ \begin{matrix}
   \overrightarrow{a} & \overrightarrow{b} & \overrightarrow{c} \\
\end{matrix} \right]=\alpha \left| \begin{matrix}
   1 & -\alpha \\
   -2 & 3 \\
\end{matrix} \right|-\left| \begin{matrix}
   2 & -\alpha \\
   \alpha & 3 \\
\end{matrix} \right|+3\left| \begin{matrix}
   2 & 1 \\
   \alpha & -2 \\
\end{matrix} \right| \\
 & \Rightarrow \left[ \begin{matrix}
   \overrightarrow{a} & \overrightarrow{b} & \overrightarrow{c} \\
\end{matrix} \right]=\alpha \left( 3-2\alpha \right)-\left( 6+{{\alpha }^{2}} \right)+3\left( -4-\alpha \right) \\
 & \Rightarrow \left[ \begin{matrix}
   \overrightarrow{a} & \overrightarrow{b} & \overrightarrow{c} \\
\end{matrix} \right]=3\alpha -2{{\alpha }^{2}}-6-{{\alpha }^{2}}-12-3\alpha \\
 & \Rightarrow \left[ \begin{matrix}
   \overrightarrow{a} & \overrightarrow{b} & \overrightarrow{c} \\
\end{matrix} \right]=-3{{\alpha }^{2}}-18 \\
\end{align}\]
Now, let us equate the obtained scalar triple product to zero as they are coplanar.
\[\begin{align}
  & \Rightarrow \left[ \begin{matrix}
   \overrightarrow{a} & \overrightarrow{b} & \overrightarrow{c} \\
\end{matrix} \right]=-3{{\alpha }^{2}}-18=0 \\
 & \Rightarrow -3{{\alpha }^{2}}-18=0 \\
 & \Rightarrow 3{{\alpha }^{2}}=-18 \\
 & \Rightarrow {{\alpha }^{2}}=-6 \\
\end{align}\]
But square of any number is always positive. So, such an \[\alpha \] does not exist. So, given vectors are not coplanar in any case.
As S is a set such that $S=\left\{ \alpha :\overrightarrow{a},\overrightarrow{b}\text{ and }\overrightarrow{c}\text{ are coplanar} \right\}$, we get S is an empty set.

Note: The major mistake one does while solving this question is after getting the value \[{{\alpha }^{2}}=-6\], most people forget that square of a real number is positive and note the answer as 2 possibilities and mark the answer as Option B. But we need to remember that the square of a real number is always positive.