First things first, you must have a heavy weapon.

Then when you find a guard with the captains sword, kill all other guards around him to make it easy to retrieve the sword.

Then you either throw the heavy weapon at him or just throw it away by holding X.

If you decide to throw the weapon away you must kill the guard with any weapon.

When he's dead you can pick up the sword and keep it, but it will not be on the weapon rack at Ezio's hidout.