TigranMets, so I think it was your brother you declined my offer
Yes, you're in a nice position to attack Peloponnesus. If you develop strongly and even set up an order you'll be a menace to the Byzantines. I hope your lord is Sicilian?
I'd suggest you'd establish a port on your island - it is bad that you haven't got one and that the only port on the island is captured by a Byzantine guy. Once you have a port you'd build koggs and go and capture ports in South Peloponnesus and Western Crete.
Given that Andrew901 settled in the northern part of your island - maybe you should build strong contours to claim 90% of the island as yours or he may expand.
On Outremer, without a port you're toast