A Bag contains $ 5 $ blue balls and an unknown number of red balls, two balls are drawn at random. The probability of both of them are blue is $ \dfrac{5}{14} $ , then the number of red balls is
A. $ 3 $
B. $ 2 $
C. $ 4 $
D. $ 5 $

Hint: In this problem, we have the number of blue balls in a bag and assume the number of red balls in the bag as $ x $. Now we will calculate the total number of balls in the bag adding the number of blue balls to the number of red balls. If we draw $ 2 $ balls from the bag then the total number of events in the sample space is equal to the number of ways of picking two balls from the bag. If the drawn two balls are to be blue, we need to pick the two balls from the blue balls, and it will become our number of possible ways for the event. In the problem we have the probability of the event, so we will calculate the probability of the event from calculated data and equate it to given value to find the number of red balls in the bag.

Complete step by step answer:
Given that. A bag contains $ 5 $ blue balls and an unknown number of red balls.
Let the number of red balls in the bag is to be $ x $.
Now the total number of balls in the bag is $ 5+x $.
Let $ A $ be the event of picking two blue balls from the bag.
We can pick $ 2 $ balls from $ 5+x $ balls in $ {}^{5+x}{{C}_{2}} $ ways. So, the total number of events in the same space is $ n\left( S \right)={}^{5+x}{{C}_{2}} $ .
Now we can pick $ 2 $ balls from $ 5 $ blue balls in $ {}^{5}{{C}_{2}} $ ways. So, the number of possible events for the event $ A $ is $ n\left( A \right)={}^{5}{{C}_{2}} $ .
We will get the probability of the event $ A $ by calculating the ratio of the number of possible ways to the event $ A $ to the total number of events in the sample space.
 $ \begin{align}
  & \therefore P\left( A \right)=\dfrac{n\left( A \right)}{n\left( S \right)} \\
 & \Rightarrow P\left( A \right)=\dfrac{{}^{5}{{C}_{2}}}{{}^{5+x}{{C}_{2}}} \\
\end{align} $
In the problem we have given that the probability of the event $ A $ is $ \dfrac{5}{14} $ , so
 $ \begin{align}
  & \therefore P\left( A \right)=\dfrac{5}{14} \\
 & \Rightarrow \dfrac{{}^{5}{{C}_{2}}}{{}^{5+x}{{C}_{2}}}=\dfrac{5}{14} \\
 & \Rightarrow \dfrac{\dfrac{5!}{2!\left( 5-2 \right)!}}{\dfrac{\left( 5+x \right)!}{2!\left( 5+x-2 \right)!}}=\dfrac{5}{14} \\
 & \Rightarrow \dfrac{20\left( 3+x \right)!}{\left( 5+x \right)!}=\dfrac{5}{14} \\
 & \Rightarrow \dfrac{4\left( 3+x \right)!}{\left( 5+x \right)\left( 5+x-1 \right)\left( 5+x-2 \right)!}=\dfrac{1}{14} \\
 & \Rightarrow \dfrac{4\left( 3+x \right)!}{\left( 5+x \right)\left( 4+x \right)\left( 3+x \right)!}=\dfrac{1}{14} \\
 & \Rightarrow {{x}^{2}}+20+9x=56 \\
 & \Rightarrow {{x}^{2}}+9x-36=0 \\
\end{align} $
To get the number of red balls we need to solve the above equation. To solve the quadratic equation, we will split the middle term as $ ax+bx $ where $ ab=-36 $ .
 $ \begin{align}
  & \therefore {{x}^{2}}+12x-3x-36=0 \\
 & \Rightarrow x\left( x+12 \right)-3\left( x+12 \right)=0 \\
 & \Rightarrow \left( x+12 \right)\left( x-3 \right)=0 \\
\end{align} $
 $ x $ can’t be zero so the value of $ x $ is $ 3 $ .
Hence the number of red balls in the bag is equal to $ 3 $ .

Note:
 We can solve the quadratic equation by using the formula $ \dfrac{-b\pm \sqrt{{{b}^{2}}-4ac}}{2a} $ , where $ a=1 $ , $ b=9 $ , $ c=-36 $ .
 $ \begin{align}
  & \therefore x=\dfrac{-9\pm \sqrt{{{9}^{2}}-4\left( 1 \right)\left( -36 \right)}}{2\left( 1 \right)} \\
 & \Rightarrow x=\dfrac{-9\pm \sqrt{81+144}}{2} \\
 & \Rightarrow x=\dfrac{-9\pm 15}{2} \\
 & \Rightarrow x=\dfrac{-9+15}{2}\text{ or }\dfrac{-9-15}{2} \\
 & \Rightarrow x=3\text{ or }-12 \\
\end{align} $
 $ x $ values can’t be negative, so the value of $ x $ is $ 3 $ .
Hence the number of red balls in the bag is equal to $ 3 $.
From both the methods we got the same result.