The "understanding-first" approach is so right, especially with a topic like Bayesian methods where the logic makes _so much sense_, and yet is so easy to miss by brute-force memorization of equations. IPython notebook is a great choice. I've only skimmed the first couple of chapters but cloned and will definitely spend some time with this.
This seems inspired by the awesome Yudkowsky article that comes from a similar (introductory, not programming-specific) place: http://yudkowsky.net/rational/bayes/
For anyone that hasn't used IPython notebook and is interested in scientific computing in python, you need to check it out. The ability to mix prose and live python, with effortless plotting, storable in git, sharable with links or nbviewer, is just magic. That probably seems mundane, but for things like exploratory data analysis across a team, it's a game-changer. Another staple in this stack is Pandas ( http://pandas.pydata.org/ )
The most powerful stuff revolves around the Dynamic symbol[1]. If you use this, for example:
Dynamic[Graphics[Disk[p]]]
And then somewhere else, either manually or programmatically, you change the value of p, you will see the disk move around. (In link [1], open up the "Neat Examples" dropdown).
Basically, with Dynamic you are saying "I want this updated as soon as any of its components are updated." And the system takes care of the rest. Furthermore, the dynamic content can be completely arbitrary. Mathematica's Manipulate symbol[2] is essentially a wrapper around Dynamic. See [3] for countless examples built using Manipulate.
To be clear, these aren't "generated programs." All this dynamism resides comfortably and natively inside notebooks. (Mathematica also allows data that persists across sessions, through DynamicModule).
See the last example in [4], combine that with the fact that you can have arbitrary expressions anywhere, and you can start to get a sense of the power the system gives you. For example, you can create an ad-hoc tool for making diagrams or whatever, and then plop that tool right in the middle of a piece of source code, inline.
As another example of the power of Mathematica's notebooks, links [1] [2] and [4] are HTML exports of Mathematica notebooks. Note all of the fancy formatting.
But wait, there's more, as Mathematica notebooks are themselves fundamentally Mathematica expressions (essentially M-expressions. Mathematica is in part a Lisp on top of symbolic semantics). Thus you can construct/alter them programmatically, etc. In other words, Mathematica is a ridiculously homoiconic system, not only in the language sense but also in the broader systemic sense. As an example, [5] is the Mathematica expression behind the notebook of [4].
This seems inspired by the awesome Yudkowsky article that comes from a similar (introductory, not programming-specific) place: http://yudkowsky.net/rational/bayes/
For anyone that hasn't used IPython notebook and is interested in scientific computing in python, you need to check it out. The ability to mix prose and live python, with effortless plotting, storable in git, sharable with links or nbviewer, is just magic. That probably seems mundane, but for things like exploratory data analysis across a team, it's a game-changer. Another staple in this stack is Pandas ( http://pandas.pydata.org/ )