How to avoid the polygon to be complex when optimizing its vertices?

Suppose there's a set of 2D points to represent an initial simple polygon.开发者_JAVA百科 Now I want to optimize the positions of each point according to some cost function. But this could make the polygon complex, i.e. the polygon intersects with itself. How can I avoid this? Thanks!

---

If the polygon could be presumed to be convex, then it is simple. Simply compute the angles between each side and the next side. Each angle must be between 0 and 180 degrees for a convex polygon. The sum of those angles is well known for a closed polygon with N sides. This will result in a simple constrained optimization. (Actually, you can write those constraints in a "simpler" form than computing the angles with a trigonometric function. Cross products will suffice.)

If the polygon need not be convex, then you need to worry about edges crossing, or other degeneracies.