Linear Regression, Explained With a Straight Line and No Calculus
Before a computer fits a line through a scatter of points, the idea is exactly the one you'd use with a ruler: find the straight line that comes closest to all of them at once.
Handed a scatter of dots on graph paper -- house sizes plotted against their sale prices, say -- and asked to draw the one straight line that best represents the overall trend, most people can do it reasonably well by eye: not a line that hits every point exactly, since that's usually impossible, but one that runs as close as possible to all of them at once. Linear regression is the precise, mathematical version of exactly that instinct.
A straight line is fully described by two numbers: its slope (how steeply it rises or falls) and its intercept (where it crosses the vertical axis) -- the familiar y = mx + b. Linear regression's actual job is finding the specific slope and intercept that make that line the "best" fit for a given set of points, where "best" has a precise mathematical definition: the line that minimizes the total squared distance between itself and every point, measured straight up-and-down rather than at an angle.
Squaring each point's distance from the line before adding them up does two things a plain distance sum wouldn't: it makes every term positive, so points above and below the line don't cancel each other out and hide how far off the fit really is, and it penalizes a large miss far more heavily than a small one, pushing the best-fit line to avoid being badly wrong about any single point even at the cost of being slightly less perfect everywhere else.
Finding that exact best line can be solved directly with a closed-form formula when the dataset is small enough, or approximated iteratively using gradient descent, covered elsewhere in this category -- nudging a candidate slope and intercept a small amount at a time in whichever direction reduces the total squared error, the same downhill-in-the-fog process that trains far larger models, just applied here to only two adjustable numbers instead of millions.
Once a straight line fits reasonably well, the same underlying idea -- pick numbers that minimize total squared error against real data -- extends directly into far more complex models with many more adjustable numbers than just a slope and an intercept, which is exactly why linear regression is usually the first model taught in any statistics or machine learning course: it's the whole idea, in its smallest and most visual possible form.