In geometry, squaring the circle was a long-standing puzzle that was proved impossible in the late 19th century. The term also has metaphorical meanings, and it has been used as a symbol in alchemy, particularly in the 17th century.

Mathematics and Geometry

According to Wikipedia (offsite link), squaring the circle:

"is the challenge of constructing a square with the same area as a given circle by using only a finite number of steps with compass and straightedge. More abstractly and more precisely, it may be taken to ask whether specified axioms of Euclidean geometry concerning the existence of lines and circles entail the existence of such a square."

In 1882 the puzzle was proven impossible.

Metaphorical Meaning

To say that one is attempting to square the circle means that they are attempting an impossible task.
That is different from trying to fit a square peg in a round hole, which implies two things are inherently incompatible.

Alchemy

A symbol of a circle within a square within a triangle within a circle started being used in the 17th century to represent alchemy and the philosopher's stone, which is the ultimate goal of alchemy.
There are also illustrations that include a squaring the circle design, such as on in Michael Maier’s 1618 book Atalanta Fugiens. Here a man is using a compass to draw a circle around a circle within a square within a triangle. Within the smaller circle are a man and woman, the two halves of our nature that are brought together through alchemy.

Read more:Alchemical Pictures: the Red King, White Queen and the Rebis

Read more:Gender in Western Occultism (and general Western Culture)

Circles often represent the spiritual because they are infinite.

Squares are often symbols of the material because of the number of physical things that come in 4s (four seasons, four directions, four physical elements, etc.) not to mention its solid appearance. The union of man and woman in alchemy is a merging of a person's spiritual and physical natures. The triangle is then a symbol of the resulting union of body, mind and soul.

In the 17th century, squaring the circle had not yet been proved impossible. However, it was a puzzle no one had been known to solve. Alchemy was viewed very similarly: it was something few if any had ever fully completed. The study of alchemy was as much about the journey as the goal, as no one might ever actually forge a philosopher’s stone.