Two long beaches, a network of concrete paths, and not a single car.
Find a room before you go.
Koh Phayam lies off Ranong near the Burmese border, and it has stayed small on purpose. There are no cars — the island runs on motorbike taxis and narrow concrete paths through the cashew plantations — and the whole population is a couple of thousand people spread between two bays.

Ao Yai on the west is the long sunset beach with the bungalows and the surf in season. Ao Khao Kwai in the north — Buffalo Bay — is quieter and calmer. Between them are cashew farms, a couple of bars that stay open late, and a rhythm that is genuinely slow rather than marketed as such.
Affiliate link. If you book through it we earn a commission, at no extra cost to you.

Ranong is the gateway. Buses run from Bangkok overnight, or fly to Ranong from Bangkok in a little over an hour. From Ranong's Saphan Pla pier, the slow boat takes two hours and the speedboat about forty-five minutes.
From the Gulf coast this is one of the easier crossings: Chumphon to Ranong is a two-hour drive straight across the isthmus, which makes Phayam a natural pairing with Koh Tao on the other side.
Boats are less frequent in low season and can stop altogether in bad weather — leave a day of slack if you have a flight to catch.

Ao Yai for the long beach, the sunsets and most of the bars. Ao Khao Kwai for calm water and early nights. Both have simple bungalows and a few comfortable places, and nothing on the island is more than fifteen minutes away by scooter.