Any way to get rid of just 2 feathers?

When you have OCD 👀

...Lol no. You'll have to live with your mistake of not thinking and carefully calculating the feathers you'd get. >:)

Wait until tomorrow, then with the 15 you should collect from your castle the 2 will be gone (of course you will then have a 7)!

Some rewards from events gave out uneven feathers. Greeting your friends list can also uneven your feathers.

My feather count ends in 1 or 6 depending on the day. I've had *a single feather* to my name once.

I'm just glad that I don't get stuck at 19,999.

Delete all your friends until 3 left.
You'll get 3 from greetings.
Presto.