See this fantastic 3Blue1Brown video for an in depth explanation.

To summarize: The original idea is popularized by M. C. Escher in their artworks.

The droste effect is the idea of embedding a smaller version of an image into itself leading to an (theoretically) infinitely repeating recursive depth.

Escher took this idea even further by breaking the boundary between two recursive images and morphing them together. This leads to the visual impression that there are no separate instances of a repeated image but just a single image spiraling inwards.

This effect can also be recognized as strange loop as done by Douglas Hofstadter in his book Gödel, Escher, Bach.

Computationally, to turn any classic image into a droste image a smaller version of the classic image has just to be renderered somewhere ontop of the original image. Depending on the scaling factor between nested images this has to be repeated until the threshold of the required display resolution is met, so that nobody can recognize that there are in fact not infinite smaller and smaller images.

Generating an Escher spiral to connect all the recursive images smoothly is slightly more sophisticated. We need to transform the (x,y)-coordinates of the image into polar coordinates of radius and angle measured from the center point in the image at which the recursion takes place. This makes both coordinates periodic: The angle is obviously periodic and the radius is periodic in droste images because when moving inwards or outwards the image will repeat after some period

So just changing the images coordinate system from cartesian to polar coordinates enables us to see and to express the periodicity of the image much more clearly.

The next trick is to transform (rotate and scale) the image in its polar coordinate space. The key insight required for this is that all concetric circles from the original image are straight horzontall lines inside the polar space (because they have all the same radius but different angles in the original image).

So in the polar space all the smaller and smaller instances of the droste image are unwrapped into equally sized rows below to each other. To break their separation we just have to break the horzontal row structure seperating them. This can easily be done by just rotating everything in polar space so that a single image instance runs diagonally across multiple (actually all infinite) rows.

After the rotation both a single horzontal line and a single vertical line in polar space run through all image instances. When now transforming back into cartesian coordinates this leads to all the instaces of the images having both all radii and and angles in common, resulting in a spiralling effect. The only tricky part is to do the polar space rotation in such a way that the period of the spiral turning lines up with the image being scaled.

Overall this trick of first transforming into polar coordinates, then doing a simple rotation and finally transforming back can be compared to the application of the fourier transform in signal processing. The fourier transform can be used to apply linear filters to a signal. Instead of computing the convolution of the filter kernel with the input signal in the time domain it can be done as a simple element-wise multiplication in the frequency. Instead of doing a complicated calculation directly we first transform into another coordinate system, do a simple calculation there and them transform back. For the Escher effect for calculating a crazy spiral in cartesian space, we just move to polar coordinates, to a simple rotation and them move back.

Below you can read the WebGL vertex and and fragment shaders for doing the the transformation.

Some other people have already built some really cool demos of the same effect. For example this recreation Eschers Print Gallery and the same effect to applied to playing video.

Vertex Shader

Framgnet Shader