You would need to find the place at which a location on the hexagon (blue), or an extension of one of its edges, is the same distance from a corner of the hexagon and from one side or the other of the inner Circle.  The series of green and red Points represent the path of the intersection of the green and red arcs as they change size with their distances from those references equal.  For both the green and red, one of the arc radii is measured from the corner of the hexagon that the star-arm Arcs must meet.  In the case of the green arcs, the other radius is measured through the center of the inner Circle to the far side of it; in the case of the red, it's from the near side of it.

 

Where those paths of Points intersect the hexagon or its extension determines the center points of the star-arm Arcs.  Those series of Points are not linear (I filled in the whole path with the green ones to make that obvious), but I'm not sure whether that would be a hyperbolic curve, or perhaps parabolic or elliptical, or none of those.  I drew Splines along a series of them in the area of intersection, and used the intersections of the Splines with the hexagon and its extension as the centers for the Arcs, with their radii defined at the corner of the hexagon.  The green layout elements determine the cyan Arc, and the red elements determine the yellow Arc.  Because the Splines don't precisely represent the correct path of the Points, the results are not completely precise -- the yellow Arc intersects the inner Circle twice very close together, and the cyan Arc doesn't quite intersect it.

 

I doubt that AutoLISP's mathematical functions would be capable of calculating the precise positions of those center points.  You would need to come up with equations defining the paths of Points (the hard part), and for an edge of the hexagon (easy enough), and solve for their intersection.  Maybe that's possible, but it's beyond my capabilities.  It could be done with a routine by trial and error, refining the determination until some defined level of precision is achieved, but I doubt it can be done with absolute accuracy.