I’m sure there’s some short elegant solution here that uses beautiful vector math. Instead I went for the butt ugly coordinate geometry solution.

Let u = <a, b> and v = <c, d>

See diagram. WLOG, assume u has a smaller angle than v. We can find cos(θ) by constructing a right triangle as shown, and by finding a new vector w, in the same direction as u, which has the correct length to complete our right triangle. Once done, we will have cos(θ) = |w| / |v|.

Let’s consider each vector to be anchored at the origin. So we can say u lies on the line y = (b/a)x, and v lies on the line y = (d/c)x. To find w, let us first find z - the third side of the triangle, which we know must be perpendicular to u, and pass through (c, d).

The line perpendicular to y = (b/a)x and passing through (c, d) is y = (-a/b)(x-c) + d. So this is the line containing the third side of our constructed triangle, z. To find w, then, let’s find its point of intersection with y = (b/a)x, the line containing w.

(b/a)x = (-a/b)(x-c) + d → Setup

(b/a)x = (-a/b)x + (ca/b) + d → Distribute on right

x(b/a + a/b) = (ca/b) + d → Add (a/b)x to both sides, factor out x on left

x(a² + b²) = ca² + dab → Multiply both sides by ab

x = (ca² + dab) / (a² + b²) → Divide both sides by (a² + b²)

x = (ca² + dab) / |u|² → a² + b² is |u|²

y = (b/a)x = (b/a)(ca² + dab) / |u|² → Plug solution for x into (b/a)x, don’t bother simplifying as this form is useful in a moment

So w = <(ca² + dab) / |u|², (b/a)(ca² + dab) / |u|²>. Now we need |w|.

|w| = sqrt((ca² + dab)² / |u|⁴ + (b²/a²)(ca² + dab)² / |u|⁴) → Plugging into normal |w| formula

|w| = sqrt(((ca² + dab)² / |u|⁴) * (1 + b² / a²)) → Factor out (ca² + dab)² / |u|⁴

|w| = (ca² + dab) / |u|² * sqrt(1 + b² / a²) → Pull (ca² + dab)² / |u|⁴ out of the root

|w| = (ca² + dab) / |u|² * sqrt((a² + b²) / a²) → Combine terms in root to one fraction

|w| = (ca² + dab) / |u|² * |u| / a → Evaluate sqrt

|w| = (ca + db) / |u| → Cancel out |u| and a from numerator and denominator

So this is the length of our adjacent side in the constructed right triangle. Finally, to find cos(θ), divide it by the length of the hypotenuse - which is |v|.

cos(θ) = |w| / |v| = (ac + bd) / (|u||v|)

And note that ac + bd is u•v. So we’re done.

cos(θ) = u•v / (|u||v|), or |u||v|cos(θ) = u•v.

Note that since this formula is symmetric with respect to u and v, our assumption that u’s angle was smaller than v’s angle did not matter - so this should hold regardless of which is larger.