Parameter Games: Guessing For Profit

Suppose you’ve got a really nifty way to measure a certain physical property.  It could be height, weight, temperature, pressure, etc…  This measurement wows everyone that takes a few moments to get the gist of it.

Due to TANSTAAFL a measurement so nifty will of course have drawbacks.  In this case the drawback is that this measurement takes a pretty long time relative to other measurements of interest.
So what do you do when you want the benefit of the nifty measurement without the cost of waiting for it to be done everywhere you’d like to do it?

Enter interpolation.  In mathematics, interpolation “fills in” a function given N or more known values.  There are lots of ways to interpolate a function some of which perform better (that is, they more closely approximate the underlying function over the interval of interest) in some circumstances than others.  There are, among others, linear interpolation, quadratic interpolation (or more generally, polynomial interpolation), piecewise interpolation and splines.  Interpolation itself is a specific case of the more general concept of approximation.

In a sense neural networks perform approximation in that they are trained on existing data sets (these are the “known values”) and are later used to produce values on inputs where the outputs are not known.

A common theme with all of these approaches to “filling in” functions is the parameter game.  They’ve all got parameters, some more than others, and tuning those parameters for a specific application (the tuning game) is usually more art than science.

Convolution confusion

So suppose you have a function that you’d like to modify.  Maybe you don’t like the way the function treats certain inputs.  Maybe you wish the function emphasized some aspect of its input over other aspects.  Maybe you just don’t think the function is aesthetically pleasing.

So you modify the function.  One way to modify the function is to add a constant value at every point.  This is a linear modification.  Since the fourier transform of a function is based on integrating sinusoidal basis functions, and linearity is preserved in integration, you can make this modification in either the time domain OR the frequency domain.

Suppose adding a constant to the function at every point doesn’t modify the function to your liking.  It’s a pretty broad brush change.  Instead of enhancing a particular aspect of the function it, at least in the time domain, merely shifts the function.

Another way to modify the function would be to multiply it by a constant value.  This scaling will also survive the translation to the frequency domain because the Integral(a * f(x) dx) is equal to a * Integral(f(x) dx).  This modification is similar to the first modifcation (adding a constant) in that it’s not selective at all.  It’s very broad brush.

What you really want to be able to do is to modify the input function selectively.  That is, you’d like to use another function to modify the original function.

Enter convolution.  Through some rather heroic mathematics this wonderful operation allows you to use another function, call it h(x), to modify your original function, call it f(x), to produce your output function, call it g(x).

Convolution is a binary operation that is implemented by multiplying the Fourier transforms of the 2 operands.

If f(x) and h(x) are the original function and the modifying function in the time domain then F(u) and H(u) are their corresponding Fourier transforms (we call this the frequency domain).

f(x) convolve h(x) can be performed by multiplying F(u) * H(u) then taking the inverse Fourier transform to produce g(x) – the modified version of the original input.  H(u) has many names, one of which is the transfer function.

Once we’re in the frequency domain, we can make H(u) pretty much whatever we want.  The simplest non-trivial function would be one that multiples F(u) by 1 if it’s below a certain value and 0 if it’s above.  This is an ideal low pass filter.  By ideal it is meant that there is a sharp cutoff – the signal is passed entirely when inside the cutoff and not passed at all when outside.

A key point to keep in mind is that H(u) operates in the frequency domain.  That is, we’re modifying frequencies, which will ultimately produce a modification in the time domain.  I believe the art part of using the Fourier Transform, is figuring out how to modify the function in the frequency domain in a way that produces the desired result in the time domain.

So, for instance, we know that sharp edges in the time/spatial domain require high frequency components in the frequency domain.  To enhance sharp edges via convolution requires that we filter out low frequency components.  To smooth/blur an image (that is, reduce sharp edges) requires the exact opposite (filtering out high frequency components instead).

Notes on Fourier Series

Stanford’s course on the Fourier Transform has, for the first 6 lectures, been almost entirely about Fourier Series.

Fourier Series can be used to represent any periodic phenomena.  Phenomena = functions in math, signals = engineering.

Periodic phenomena are broadly classified as either periodic in space (e.g., a ring) or periodic in time (e.g., a wave).  Periodicity arises from symmetry inherent in some property of the periodic phenomenon.

Prof Osgood is a genius.  His enthusiasm is surpassed only by his insight into mathematics.

The Fourier series is based on linear combinations of sine and cosine.  These are in turn based on the unit circle.  Mentioning this relationship to the unit circle because it isn’t the only basis for trigonometric functions.  For example, the Hyperbolic sine is based on a hyperbola.

Any periodic function can be expressed as a linear combination (a sum) of these trigonometric basis functions.  The hard part is figuring out the coefficients to apply to each linear component of the sum.

For mathematical convenience, the exponential form of the sine and cosine are often used when deriving the coefficients of the Fourier series for a particular function.  There’s a lot of calculus involved here but, at least up to lecture 7, it’s all pretty basic plug and chug integration and differentiation.  Despite that, it makes me yearn for a refresher in things like Integration by Parts.  Thank God for Wikipedia and Schaum’s Outlines!

Apparently the key breakthrough in the theoretical justification for Fourier Series occured when mathematicians gave up on trying to prove that the Fourier series converges exactly to the function being represented.  Instead, they were able to prove that the mean squared error (difference between the value of the function and its Fourier series) of the Fourier series for a given function, in the limit, approaches zero.

Sine and Cosine are both continuous and smooth (infinitely differentiable).  Because of this edges, which are either jump discontinuities or sharp changes in the derivative, require higher and higher frequency components to express.  I visualize this as higher and higher frequency sinusoids “bunching up” together to produce a sharp change in the value of the function at any given point.  The more such points in a function (e.g., the more edges), the more these high frequency sinusoids are needed to represent the function.

I believe it takes an infinite number of such high frequency components to perfectly reproduce any discontinuous function since the sum of a finite number of continuous functions is a continuous function.

In lecture 6 Prof Osgood formally bridges Fourier Transforms to Fourier Series.  The Fourier Transform is a limiting case of Fourier Series.  Limited in the sense that the phenomena need not be periodic (which means, mathematically, that the period tends to infinity).

A Quick Note on Correlation

The correlation coefficient of 2 variables measures the strength and direction of the linear relationship between the 2 variables.

Strength: Expressed visually by how “line-like” a plot of both variables will appear.  Lines are thin (infinitely thin in the most abstract sense).  Strength is indicated by how close the absolute value of the correlation coefficient is to 1.

Direction: Do the variables rise and fall together?  Or does one variable fall as the other rises?  This is indicated by the sign of the correlation coefficient (positive or negative respectively).

Converse considerations.  While a non-zero correlation coefficient implies some degree of linear dependence between the 2 variables, a correlation coefficient of 0 does not imply independence.

• The relationship might be non-linear.  The correlation coefficient identifies linear relationships.

Why did I bother to write this note?

Given a linear relationship between 2 variables (or, put another way, a linear dependence of one variable on another) one variable might be used to predict the other.  If the variables in question represent measurements than this can be incredibly valuable because some measurements are much harder to perform than others.  Substituting a prediction based on a cheap measurement for an expensive actual measurement can yield cost savings.  Cost might be “measured” in dollars, time or some other way so this statistic can turn out to be valuable in many industries/disciplines.