Hidden Dimensions in Data
How unsupervised learning reveals latent structure — clusters, low-dimensional manifolds, and anomalies — hidden inside high-dimensional data.
A complete interactive classroom, not just a preview.
Start when you are ready to enter this Stage's 13 scenes and explore, respond, and learn as you go.
Can a machine discover hidden dimensions and unusual patterns in data that look completely random to us?
- high-dim-data
- Why real datasets live in hundreds or thousands of dimensions and why humans cannot visualize them.
- curse-of-dimensionality
- Counter-intuitive properties of high-dimensional space: distance concentration and empty neighborhoods.
- pca
- Principal Component Analysis finds orthogonal directions of maximum variance.
- nonlinear-manifold
- Many datasets lie on a curved low-dimensional manifold inside the high-dimensional space.
- autoencoders
- Neural networks that compress data to a latent code and reconstruct it, learning the manifold.
- clustering
- Unsupervised grouping: k-means, hierarchical, DBSCAN — partitioning, agglomerating, and density-based.
- anomaly-detection
- Identifying points that do not fit: statistical, density-based, and isolation methods.
More features always means more information for the model.
Show that many features are redundant or noise; informative signal often lives in far fewer dimensions.
PCA and t-SNE are interchangeable ways to visualize data.
Show that PCA preserves global linear variance while t-SNE preserves local neighborhood structure, and they answer different questions.
If clustering finds groups, those groups must be real categories in the world.
Show that clusters are patterns in the chosen features, not ground-truth labels — changing features changes clusters.
Anomalies are simply points far from the mean.
Show that distance from the mean fails in high dimensions and on irregular shapes; density-based and isolation methods are needed.
- Comfort with vectors, matrices, and basic statistics (mean, variance)
- Familiarity with plotting data on axes
- deep reinforcement learning
- generative adversarial networks
- full mathematical proofs
- production-scale MLOps
- Learner picks the right dimensionality reduction tool for a stated goal (global variance vs local structure).
- Learner explains why a clustering result may not reflect a real-world category.
- Learner identifies an appropriate anomaly detection method for a given data shape.
- Given a new high-dimensional dataset, propose which unsupervised techniques to try first and what their results would and would not tell you.
Curious beginners comfortable with basic algebra and graphs, but new to machine learning.
- 01A Question Hidden in Plain SightslideOrientation
Frame the opening question: can a machine see patterns we cannot? Set up why high-dimensional data is opaque to humans and why machines have an edge.
- Real data often has hundreds of features
- Humans see in 2D/3D — machines do not
- Unsupervised learning explores without labels
- 02The Curse of DimensionalityslideModel building
Explain why high-dimensional space behaves counter-intuitively: distances concentrate, neighborhoods become empty, and intuition breaks.
- Volume explodes as dimensions grow
- All points become roughly equidistant
- Nearest-neighbor loses meaning
- 03Predict the Hidden ShapeinteractivePredictionPredict
Learners look at a scatter of points that look random and predict how many underlying variables generated them before revealing the answer.
- Observe the point cloud
- Predict the number of latent variables
- Compare guess with the true dimensionality
- 04PCA: Finding the Lying AxesslideModel building
Introduce Principal Component Analysis as the linear way to find the axes along which data varies most, projecting into a viewable space.
- Eigenvectors of the covariance matrix
- Linear projection preserving global variance
- Loses local and nonlinear structure
- 05PCA vs t-SNE: Same Data, Different StoryinteractiveMisconception repairObserve
Learners toggle between PCA and t-SNE projections of the same dataset and notice how each surfaces different structure.
- Switch projection method
- Observe global variance view vs local cluster view
- Note that neither is 'wrong'
- 06Autoencoders: Learning the CurveslideModel building
Show how a neural network compresses data to a small latent code and reconstructs it, learning the curved manifold automatically.
- Encoder → bottleneck → decoder
- Bottleneck forces the manifold
- Nonlinear version of PCA
- 07Which Tool Wins?quizAssessment
Three short questions: pick the right reduction method for a stated goal, predict when an autoencoder will beat PCA, and explain why t-SNE distorts global distances.
- Match tool to goal
- Reason about nonlinear structure
- Interpret t-SNE's local-only guarantee
- 08Clustering: Naming the IslandsslideModel building
Survey three clustering families: k-means (centroids), hierarchical (nested merges), DBSCAN (dense regions).
- k-means: assume round blobs
- Hierarchical: produce a dendrogram
- DBSCAN: find dense islands
- 09Cluster ThisinteractivePracticeChoose
Learners choose an algorithm and parameters on a 2D toy dataset and watch the resulting clusters form.
- Switch algorithm
- Adjust parameters
- See how shape assumptions affect results
- 10Are Clusters Real?slideMisconception repair
Address the trap of treating clusters as truth: they reflect patterns in the chosen features, not objective categories.
- Clusters are patterns, not facts
- Different features give different clusters
- Always sanity-check with domain knowledge
- 11Anomalies: The Points That Don't FitslideModel building
Introduce anomaly detection: statistical (z-score), density-based (LOF), and isolation (Isolation Forest). Explain why simple distance from the mean fails in high dimensions.
- Z-score breaks in high dimensions
- LOF compares local density
- Isolation Forest uses random cuts
- 12Spot the Odd One OutinteractiveApplicationApply
Learners try to flag anomalies on a shaped dataset, then compare their picks with an Isolation Forest result and see where simple distance fails.
- Click suspicious points
- Reveal the model's picks
- Reflect on where distance-based intuition breaks
- 13What Machines Cannot SeeslideSynthesis
Pull together the lesson: unsupervised learning reveals structure but cannot tell us whether that structure is meaningful, fair, or safe.
- Hidden structure ≠ ground truth
- Choice of method shapes the answer
- Ethics of automated pattern discovery
Discussion threads for a Stage aren't available yet.
