JEE Mathematics 2026 Question with Solution

Given: The direction cosines of two lines satisfy $4l + m - n = 0$ and $2mn + 5nl + 3lm = 0$.

Find: The cosine of the acute angle between these two lines.

From the first equation,

$$n = 4l + m$$

Substitute this into the second equation:

$$2m(4l + m) + 5l(4l + m) + 3lm = 0$$ $$8lm + 2m^2 + 20l^2 + 5lm + 3lm = 0$$ $$20l^2 + 16lm + 2m^2 = 0$$ $$10l^2 + 8lm + m^2 = 0$$

Divide by $m^2$:

$$10\left(\frac{l}{m}\right)^2 + 8\left(\frac{l}{m}\right) + 1 = 0$$

Let the roots be $\frac{l_1}{m_1}$ and $\frac{l_2}{m_2}$. Using the quadratic roots together with the identity $l^2 + m^2 + n^2 = 1$, we obtain the cosine of the acute angle between the two lines as

$$\cos \theta = \frac{10}{3\sqrt{38}}$$

Therefore, the correct option is A.

Given: $4l + m - n = 0$ and $2mn + 5nl + 3lm = 0$.

Find: The cosine of the acute angle between the corresponding lines.

Treat $l:m:n$ as direction ratios of the lines. From

$$n = 4l + m$$

put this into

$$2mn + 5nl + 3lm = 0$$

to get

$$2m(4l + m) + 5l(4l + m) + 3lm = 0$$ $$8lm + 2m^2 + 20l^2 + 5lm + 3lm = 0$$ $$20l^2 + 16lm + 2m^2 = 0$$ $$10l^2 + 8lm + m^2 = 0$$

Now let $x = \frac{l}{m}$. Then

$$10x^2 + 8x + 1 = 0$$

So the two possible values of $x$ correspond to the two lines. For each value of $x$,

$$l : m : n = x : 1 : (4x + 1)$$

These give the two directions. Converting these direction ratios into direction cosines by normalization and then using

$$\cos \theta = \left| l_1l_2 + m_1m_2 + n_1n_2 \right|$$

we get

$$\cos \theta = \frac{10}{3\sqrt{38}}$$

Hence, the cosine of the acute angle between the lines is $\frac{10}{3\sqrt{38}}$.