It appears that you tried to subtract first term minus first term, second term minus second term, and third term minus third term. This is not possible because they are not like terms. Instead, combine the $\displaystyle{{a}}^{{2}}\usepackage{cmbright}\Huge\$ terms, combine the $\displaystyle{a}{b}\usepackage{cmbright}\Huge\$ terms, and combine the $\displaystyle{{b}}^{{2}}\usepackage{cmbright}\Huge\$ terms.