You may have correctly used the definition of cosine, but you did not use the Pythagorean Theorem. You can draw a triangle with $\displaystyle{o}={5}\usepackage{cmbright}\Huge\$ and $\displaystyle{a}={3}\usepackage{cmbright}\Huge\$ . Then use the Pythagorean Theorem to find the length h, of the hypotenuse:

$\displaystyle{{5}}^{{2}}+{{3}}^{{2}}={{h}}^{{2}}\usepackage{cmbright}\Huge\$

Solve this equation for h. Then use the definition of cosine. The correct answer is $\displaystyle\frac{{3}}{\sqrt{{34}}}\usepackage{cmbright}\Huge\$ .