Linear Algebra

Linear algebra studies vector spaces and linear maps between them. It is one of the most reusable pieces of mathematics because vectors can represent points, signals, documents, model weights, colors, forces, probabilities, graph nodes, embeddings, constraints, and many other structured quantities.

The central idea is that linear structure is expressive enough to model real systems and disciplined enough to compute with. A matrix can rotate a shape, solve a system, project high-dimensional data, describe a graph, encode a Markov chain, fit a least-squares model, compress an image, or move information through a neural network layer.

This page connects graph theory, graphs, topology, training neural networks, data visualization, Python, Rust, number theory, category theory, type theory, theorem proving, GPU programming, data sources, and data storage.

The basic vocabulary is compact:

• scalars are the numbers used to scale vectors;
• vectors are elements of a vector space;
• linear combinations build new vectors from old ones;
• span describes what a collection of vectors can generate;
• linear independence says no vector in a set is redundant;
• bases provide coordinate systems;
• dimension counts the size of a basis;
• linear maps preserve addition and scalar multiplication.

These definitions turn geometry into algebra. A point can be represented as coordinates, a transformation can be represented as a matrix, and a question about space can become a calculation.

A vector space is a domain where vectors can be added and scaled while obeying familiar laws. Real coordinate space is the first example, but the idea is broader. Polynomials, functions, signals, matrices, solution sets, embeddings, and finite-dimensional feature tables can all behave like vector spaces when the operations are defined correctly.

The modeling question is not only "is this an array?" It is "what does addition mean, what does scaling mean, and what equivalences are being assumed?" Adding two force vectors has a physical interpretation. Averaging two text embeddings has a model-dependent interpretation. Adding two one-hot identifiers usually has no useful semantic meaning. The same numerical operation can be valid, weak, or meaningless depending on the representation contract.

A matrix is best understood as a linear transformation written in coordinates. Matrix multiplication is function composition. The order matters because doing one transformation and then another usually differs from reversing them.

Important matrix ideas include:

• rank, the dimension of the image;
• nullspace, the vectors sent to zero;
• column space, the reachable outputs;
• row space, the constraints visible through rows;
• determinant, signed volume scaling for square matrices;
• inverse, when a transformation can be undone;
• transpose, which swaps rows and columns and exposes dual relationships;
• trace, the sum of diagonal entries and eigenvalues under suitable conditions.

The rank-nullity theorem is the basic conservation law: for a linear map from a finite-dimensional vector space, domain dimension equals rank plus nullity.

Linear algebra becomes practical only after a representation is chosen. A vector may represent a position, feature list, probability distribution, embedding, force, color, coefficient set, or latent state. A matrix may represent a transformation, adjacency relation, covariance structure, constraint system, similarity kernel, differential operator, or learned layer.

When using linear algebra in a knowledge system, record:

• what the vector space represents;
• which basis or coordinate system is being used;
• what the units are;
• what distance, angle, norm, or similarity means;
• whether values are exact, measured, normalized, learned, inferred, or simulated;
• whether matrices are dense, sparse, structured, symmetric, positive definite, stochastic, or low rank;
• which algorithm produced the numerical artifact;
• which data source and model version support the representation.

That contract keeps data visualization, graph theory, and training neural networks from treating every array as interchangeable.

In machine-learning and scientific-computing code, vectors and matrices are usually part of a larger tensor vocabulary. A tensor can represent batches, channels, time steps, tokens, heads, image patches, graph neighborhoods, or physical dimensions. The word is useful only when the axes have names and the shape contract is explicit.

Shape discipline records dimension order, broadcasting rules, batch semantics, dtype, units, padding, masks, and normalization. This is the difference between a model that merely runs and a model whose outputs can be interpreted. Transformers, multimodal AI, GPU programming, and training neural networks all depend on this discipline.

Systems of linear equations are everywhere. Gaussian elimination is the foundational algorithm; LU, QR, Cholesky, and iterative methods are the production versions used when matrices are large, sparse, symmetric, positive definite, rectangular, or numerically delicate.

The important distinction is between existence, uniqueness, computation, and interpretation. A system may have no solution, one solution, or many solutions. It may have an exact mathematical solution that is numerically unstable. It may have a least-squares solution that is useful for prediction but weak as an explanation.

Numerical linear algebra adds an engineering question: not only "is there a solution?" but "can we compute a stable answer with finite precision, bounded memory, and a reliable error estimate?"

Orthogonality gives linear algebra its geometry. Dot products measure alignment. Norms measure size. Orthogonal bases make coordinates easier to interpret. Projections find the closest point in a subspace under a chosen inner product.

Least squares is the practical workhorse here. When a system has no exact solution, a projection can find the best approximation under a defined norm. That shows up in regression, signal processing, data fitting, image compression, calibration, and many forms of model evaluation.

This intuition connects linear algebra to data visualization. A projection can make high-dimensional structure visible, but it can also hide variance, distort distance, or invent visual clusters. A 2D plot of embeddings is a view through a chosen projection, not the original space.

Eigenvectors are directions preserved by a linear map. Eigenvalues say how those directions are scaled. Spectral information reveals invariant directions, stability, oscillation, diffusion, ranking, clustering, and long-term behavior.

Examples:

• a covariance matrix exposes principal directions of variation;
• a graph Laplacian exposes connectivity and clustering structure;
• a Markov transition matrix exposes stationary behavior;
• a dynamical system's eigenvalues help diagnose stability;
• a neural-network layer can be studied through singular values and conditioning.

Spectral language is powerful because it compresses structure, but it can mislead when the underlying matrix does not match the real system. The matrix is a model, not the world.

Matrix decompositions turn one hard object into more useful pieces:

• LU decomposes a matrix for solving square systems;
• QR supports least squares and orthogonalization;
• Cholesky handles symmetric positive definite systems efficiently;
• eigendecomposition exposes invariant directions when available;
• singular value decomposition works broadly and reveals rank, compression, and conditioning;
• Schur decomposition supports stable eigenvalue algorithms;
• nonnegative matrix factorization can expose additive parts when nonnegativity is meaningful.

The decomposition is not only an algorithm. It is a claim about what structure matters: triangular solve, orthogonal basis, principal components, low-rank approximation, or interpretable parts.

Many useful matrices are mostly empty or highly structured. Graph adjacency matrices, incidence matrices, finite-difference operators, term-document matrices, recommender matrices, and feature tables often need sparse storage. Toeplitz, diagonal, banded, symmetric, stochastic, block, and low-rank matrices carry structure that a dense array would hide.

Preserving that structure changes the algorithmic problem. Sparse formats can make an impossible graph computation tractable, but the wrong operation can densify the matrix and destroy the advantage. A good record should say whether sparsity is mathematical, empirical, or merely an implementation detail.

Graphs become linear algebra when relationships are written as matrices. Adjacency matrices represent edges. Incidence matrices represent vertex-edge structure. Laplacian matrices represent flow, cuts, diffusion, smoothness, and connectivity.

This is why graph theory sits close to linear algebra. Spectral graph theory studies networks through eigenvalues, eigenvectors, singular values, random walks, expansion, clustering, and cuts. Ranking algorithms and embedding methods often turn graph traversal into matrix operations.

For knowledge graphs, this translation has to be handled carefully. A typed, sourced claim graph can be flattened into a matrix, but flattening may erase relation type, provenance, direction, confidence, temporal context, and semantic difference. Linear algebra can make a graph computable while also making parts of it invisible.

Modern machine learning is saturated with linear algebra. Model inputs become tensors. Layers apply matrix multiplications. Gradients move through vector spaces. Attention mechanisms compute similarities. Embedding models place text, image, audio, code, or graph nodes into learned vector spaces.

The useful question is not "does it have embeddings?" It is:

• which model produced the embedding;
• what training data shaped the space;
• what dimension and normalization were used;
• what similarity measure is meaningful;
• whether distances are calibrated for the task;
• how the embedding changes across model versions.

Training neural networks depends on this discipline. A model checkpoint is not just weights. It is a numerical object tied to dataset versions, initialization, optimizer state, precision, hardware, logging, and evaluation.

Semantic search turns documents, images, audio, graph nodes, or products into vectors and ranks them by a similarity rule. The ranking is only meaningful inside the embedding model's contract: training data, normalization, distance metric, dimensionality, pooling method, and index configuration all matter.

For the compendium, this links linear algebra to SEO, semantic web, data sources, and graphs. A search result is not just text relevance. It is a projection through a model, an index, and a ranking rule. The page should make those layers inspectable when a claim depends on them.

Linear algebra supplies local geometric language: vectors, tangent spaces, coordinates, bases, Jacobians, transformations, projections, and approximations. Topology asks what survives continuous deformation. Geometry and analysis add measurement, curvature, and change.

The boundary matters. A linear approximation can be useful near a point and wrong globally. A projection can preserve local neighborhoods while distorting global shape. A low-dimensional embedding can show clusters while hiding holes, boundaries, or disconnected pieces. Good mathematical writing says which structure is being preserved.

Linear algebra workloads often reduce to dense matrix multiplication, sparse matrix operations, factorizations, vector reductions, and memory movement. That makes implementation details central. Cache layout, BLAS kernels, GPU tensor cores, data transfer, precision, sparsity format, batching, and parallelism can determine whether an algorithm is practical.

GPU programming matters because many machine-learning and simulation workloads are limited by matrix and tensor throughput. Python matters because NumPy, SciPy, JAX, PyTorch, and notebooks are the everyday interface. Rust matters when systems code needs explicit memory, performance, and safety tradeoffs.

Numerical linear algebra is full of quiet traps. Floating-point arithmetic is finite. Some matrices are ill-conditioned. Sparse matrices can become dense by accident. Iterative solvers can converge slowly or to the wrong tolerance. A fast matrix multiplication can still produce an answer whose interpretation is weak.

A useful computational note should record:

• shape and dimension;
• dtype and precision;
• basis, coordinates, and units;
• sparsity and storage format;
• matrix conditioning;
• solver and decomposition method;
• tolerance and stopping rule;
• random seed when randomized algorithms are used;
• hardware and library versions when they affect the result.

For Python notebooks, this discipline makes charts and model outputs reproducible. For Rust or systems work, it clarifies where performance, memory, and correctness tradeoffs live. For theorem proving, it separates exact algebraic statements from floating-point computation.

A compendium record that invokes linear algebra should preserve:

• the object represented as a vector, matrix, tensor, subspace, operator, or decomposition;
• the semantic meaning of coordinates;
• dimensions and shape constraints;
• basis or coordinate system;
• norm, inner product, metric, or similarity rule;
• exact versus numerical status;
• algorithm and library used;
• source data and transformation path;
• interpretation limits.

Some of these invariants can be made explicit through type theory: vector dimensions, matrix shapes, units, coordinate systems, and solver assumptions can become checked boundaries instead of comments beside an array.

Linear algebra often changes the object being studied. Text becomes an embedding, a graph becomes a matrix, an image becomes a tensor, and a route becomes coordinates. A useful record should say what was preserved, what was discarded, and which basis, normalization, or model made the representation possible.

This keeps numerical artifacts connected to data sources, training neural networks, graphs, and data visualization without treating every vector as self-explanatory.

Linear algebra provides the bridge between symbolic graph structure and numerical representation. A graph can become an adjacency matrix. A corpus can become an embedding space. A map can become coordinates. A neural network can become layers of matrix operations. Those translations are powerful, but each one changes what questions can be asked.

Useful graph metadata includes vector dimension, basis, norm, model version, matrix type, decomposition method, solver tolerance, source data, library version, and hardware context. That makes numerical artifacts traceable instead of treating embeddings, projections, rankings, and scores as opaque magic.

Useful graph predicates include represents_as, projects_to, embeds_with, decomposes_into, solves_system, approximates, preserves_norm, changes_basis, factors_as, ranks_by, regularizes, and conditions_on.

Linear algebra claims need more than a formula. A useful matrix record says what the rows and columns mean, which field or numeric type is used, whether the matrix is dense or sparse, how missing values were handled, what normalization or weighting was applied, and which operation produced the result. Without those fields, an embedding, projection, similarity score, or eigenvector becomes difficult to audit.

For data sources, the row and column labels are provenance. For data storage, sparsity, shape, dtype, and compression decide whether the artifact can be preserved and recomputed. For data visualization, the reader needs to know whether a 2D plot is a projection, embedding, factorization, layout, or hand-authored diagram. For training neural networks, tensor shape, precision, initialization, gradient scale, and batch layout become part of the evidence.

A good record also preserves solver settings. Tolerance, iteration limit, seed, preconditioner, library version, hardware, and stopping reason can change the answer. "SVD," "least squares," "nearest neighbors," and "eigenvectors" are not full claims until the computation boundary is named. The compendium graph should keep that boundary visible so numerical claims do not look more exact than their inputs and algorithms allow.

Embedding spaces are one of the most common places linear algebra touches this site. Articles, images, sources, graph nodes, and queries can all become vectors. The useful question is not whether the vectors exist, but what distance means inside that model. Cosine similarity, dot product, Euclidean distance, learned reranking, and graph-neighborhood similarity express different assumptions.

For search and recommendation, preserve the embedding model, version, input text, chunking policy, normalization, index settings, and reranking method. That makes semantic web identifiers and graphs complementary to dense vectors rather than competitors. Symbolic links explain identity and provenance; vectors help find near neighbors. A high-quality compendium needs both.

• Treating matrix formulas as exact when floating-point conditioning dominates the result.
• Ignoring sparsity and turning a tractable problem into a memory problem.
• Confusing correlation, projection, and causation when interpreting dimensions.
• Using high-dimensional embeddings without checking what distance means.
• Forgetting that a visually clean 2D projection may hide important structure.
• Comparing vectors from different models, bases, units, or normalization schemes.
• Treating a matrix as a knowledge graph after provenance and relation types have been erased.
• Reporting more precision than the input data or algorithm can justify.

Learn vectors, matrices, row reduction, bases, determinants, linear maps, eigenvalues, orthogonality, projections, least squares, SVD, and numerical stability. Then branch toward optimization, statistics, graphics, topology, graph theory, machine learning, or scientific computing.

For implementation, build small examples in NumPy first, then compare with SciPy routines and GPU-backed tensor libraries. For mathematical depth, keep exact examples beside numerical ones so the conceptual structure does not vanish into library calls.

• MIT OpenCourseWare: Linear Algebra (opens in new tab)
• Immersive Linear Algebra (opens in new tab)
• Fundamentals of Numerical Computation (opens in new tab)
• LAPACK: Linear Algebra PACKage (opens in new tab)
• LAPACK Users' Guide (opens in new tab)
• BLAS at Netlib (opens in new tab)
• NumPy linear algebra documentation (opens in new tab)
• SciPy linear algebra documentation (opens in new tab)
• The Matrix Cookbook (opens in new tab)
• Eigen documentation (opens in new tab)
• JAX documentation (opens in new tab)
• PyTorch documentation (opens in new tab)

• Mathematics for the larger abstraction map.
• Graph Theory for spectral and matrix views of networks.
• Graphs for knowledge graphs, adjacency, traversal, and graph embeddings.
• Training Neural Networks for tensor computation in practice.
• Transformers and Multimodal AI for attention, embeddings, and tensor shape contracts.
• GPU Programming for parallel matrix and tensor kernels.
• Python for the main everyday scientific-computing ecosystem.
• Rust for systems-oriented numerical software.
• Data Visualization for projection, dimensionality reduction, and visual evidence.
• Category Theory for structure-preserving maps at a higher level of abstraction.
• Type Theory for typed dimensions, shapes, and checked invariants.
• Data Sources and Data Storage for provenance, derived artifacts, and reproducible numerical records.