Projections using Neural Networks

This is a proof of concept article that shows the potential of Neural Networks to create “projections” that preserve distances.

Define a function:

Let’s start with 1D functions, in this case sin(x):

Target function.

Calculate distances:

The next step is creating a method to calculate distances on the surface of this function. I’m going to use the simplest method possible — sampling the function using an evenly spaced range and calculating the Euclidean distance between intervals.

Method to calculate surface distances.

So if you were driving yur car on the surface of sin(x) it would take approximately 24.5 units to go from x=-10 to x=10.

Sampling:

Now let’s sample random pairs of points and distances on the surface of the function, maintaining the same bounds (-10, 10). To make the task easier an evenly spaced range is going to be generated, and all the pairwise combinations between elements is going to be calculated. The range will have 1001 points this means the output will have 1001*1000*0.5 (500500) pairs of points.

Sampling strategy.

Model architecture and training:

Finally, the architecture of the network is going to be defined in Keras, and the resulting model trained using the previously generated dataset.

Neural Network architecture and Training

So to sum it up, it’s necessary to create a “custom” L2 Norm layer because of numerical instability issues of TF calculating the gradient of the square root. In some use cases, like geographic coordinate systems, it might be required to increase Keras/TF float precision to 64 bits. The architecture of the Network is very similar to a Skip-Gram model, but the input is a dense vector instead of embeddings and the output metric is L2 distance instead of Cosine distance. The same layers are applied to each point (weight sharing), the L2 Norm is then calculated between the resulting vectors.

Result:

It’s pretty much accurate.

Results and examples.

On the left the points are in their original space, on the right the points are projected on a space that preserves distances.

Left: points on original space. Right: points projected on a space that preserves distances.

Conclusion:

In conclusion, we want to measure the distance of a very curvy road with a straight tape measure. We have several options, we can try to bend the tape measure according to the road curvature, we can try to approximate the distance by taking several consecutive straight measurements with the tape, or we can straighten out the road and use the measuring tape as is (distorting space).

Future Work:

1. Test if it’s its possible to apply concept to a 2d circle.
2. Test if it’s possible to project the surface of a n-dimensional manifold into a higher dimensional Cartesian plane that preserves distances. Or in other words does it make sense to use 6 dimensions to project the surface of an object originally in 3d space?
3. If possible this concept should be applied to the surface of the earth, using the WGS-84 ellipsoid model to create a very accurate “higher dimensional projection” of earth, on which Euclidean distances of points in projected space would be an approximation of the corresponding Geodesic distance of original coordinates. This should be cheap, work for the entire globe, allow fast Nearest Neighbors, and eventually allow to hash geometries (LineStrings, Polygons) and encode other properties of the surface.
4. Create a new geospatial toolkit based mostly on Neural Networks.