"The edges look bad" covers several distinct problems with different causes. Diagnose first.
1. Stair-Stepping (Aliasing)
The edge shows visible square steps, like a staircase. This happens when the alpha channel is binary — every pixel is either fully opaque or fully transparent, with nothing in between. Diagonal and curved edges can't be represented that way.
Fix: use a tool that produces anti-aliased (partial alpha) edges, or apply a small feather (0.5–1 px). Note that GIF and 8-bit PNG palettes support only binary transparency; if you need smooth edges, use 24/32-bit PNG or WebP.
2. White or Colored Fringe (Halo)
A thin outline of the *old* background survives around the subject. This is the partial-pixel problem: edge pixels were a blend of subject and background, and the tool kept them.
Fix: contract the selection by a pixel, desaturate the edge, or use a tool with spill suppression. Placing the cutout on a background of similar brightness to the original also hides it.
3. Dark Halo After Compositing
Sometimes the opposite: a dark line appears when you place a cutout on a light background. This usually comes from incorrect alpha handling — specifically premultiplied vs straight alpha being mixed up by a tool along the chain. Re-exporting from the master, or exporting as straight-alpha PNG, generally resolves it.
4. Crunchy Edges After Scaling or Compression
The cutout looked fine, then you resized it or saved it and the edges became rough.
- Scaling a binary-alpha image magnifies the stair-steps. Always scale from a master with soft alpha, and scale down rather than up.
- Saving transparency as a palette format (8-bit PNG, GIF) quantizes the alpha to on/off.
- Lossy WebP with alpha can introduce artifacts along edges at low quality — keep quality above ~80 for cutouts.
- Sharpening after cutting out amplifies edge artifacts. Sharpen before the cutout, or not at all.
The Order That Avoids All Four
- Cut out at full resolution, with anti-aliased/partial alpha.
- Do color and spill corrections on the edge immediately, while the master is still large.
- Then resize down to the target size — downscaling smooths edges rather than roughening them.
- Export as 24/32-bit PNG or high-quality WebP.
- Inspect at 100% on both light and dark backgrounds.
If a cutout must be delivered at several sizes, always generate each size from the full-resolution master rather than from an already-shrunk copy — every re-scale of an alpha edge compounds the roughness.