In the previous posts, we looked a lot into fluid simulation. One major aspect of fluid simulation is the neighborhood search. Previously we just glanced over the fact and solved the issue using a library. Today we want to explore one method in more detail to achieve a faster neighborhood search.

2. Neighborhood Search

For simplicity, let’s say we want to create a simple particle simulation, where we spawn a bunch of particles in the beginning and just run collision detection between the particles and a bounding box.

First we need to detect collisions, which will be the main focus. Then later on we will talk about the collision resolution. From our collision we need two major pieces of information. How deep is the collision, so how big is the overlap and in which direction. So for particles, the direction is the difference of center coordinates of both particles.

Checking if a particle collides with an AABB (axis aligned bounding box) is easy, by comparing the x,y,z coordinates to the x,y,z coordinates of the two points defining the AABB. Therefore we just iterate over all particles, compare if the particle center is closer than the particle radius to the boundary and apply forces accordingly to resolve the collision.

The more complicated part is, finding inter particle collisions. First lets define the distance between two spheres a and b: \(d = ||p_1 - p_2|| - (r_1 + r_2).\) If the distance is smaller than 0, we have a collision, and already the value we need for resolution.

So one way to check, would be to simply compare all sphere positions to each other:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
FOR (size_t i = 0; i < particles.size(); ++i) {
    FOR (size_t j = i + 1; j < particles.size(); ++j) {
        glm::vec3 direction = particles[j].position - particles[i].position;
        float distance = glm::length(direction);
        float minDistance = particles[i].radius + particles[j].radius;

        if (distance < minDistance) {
            float overlap = minDistance - distance;
            CollisionPair collision{
                i, j, overlap, glm::normalize(direction)
            };
            collisions.push_back(collision);
        }
    }

Since we have two nested for loops both with length particles.size() we have a runtime of \(O(n^2)\) with n being the number of particles. For 100 particles this might be still ok, but for 1.000 particles we already need to run 1.000.000 comparisons and for 10.000 particles 100.000.000 comparisons. Considering that our basic fluid simulations ran with 20.000 particles already, we need to find a faster solution.

3. Spatial Hashing

A major component of accelerating the search is the use of spatial acceleration data structures. For one we could add a voxel grid to our domain with a voxel size of the particle radius. Now we denote which particle lives in which voxel cell. If we now want to know which particles might have a collision with particle a, we simply in 2D look at the cell that particle a is in and the 8 surrounding cells. For the 3D case we then have to search 27 cells. Therefore for each particle we have to only look at a fixed amount of cells. This approach is considered a dense grid representation. Let’s take a look at an example:

Logically this means that first we iterate over all particles, determine each particle’s cell id and increment the count of particles for each cell. The cell id is the index in the cell array. For later access we then create an entry points array. At the index i, the entry points array tells us at which index in the particle array we have to look for the particles of the cells. And at the index i + 1 of the entry points array we get the index of the next starting particle for the next cell (and thereby the index of the last element as the predecessor).

Performing the neighborhood search now is easy, by determining the indices of the neighboring 26 cells, looking up the particles inside the cells and iterating over them.

This approach has a few issues though. First it forces us to determine the domain size in the beginning, which limits the dynamic evolution of the scene. Second, most simulations have a high air to particle ratio, rendering many cells empty. And lastly, by design this approach works best if there is only a small amount of particles in a cell. Therefore the cells should be small. All factors combined can lead to very big grids with very few entries, being slow and memory hungry.

To circumvent this, instead of calculating our index like in the marching cubes example, we introduce a hash function:

First of all, now we can also see negative indices, since an integer coordinate is simply the world coordinate divided by the cell size. Therefore we need to force the result into a positive range. Next we can preselect how many cells we want to have. Choosing too many cells leads to many sparse entries. Choosing too few cells can lead to overlap. Two different integer coordinates can lead to the same index. To minimize this, literature suggests using large prime numbers. If the overlap is not too big, we will simply iterate over a few more entries.

Therefore the general process is very similar. One thing we didn’t look at yet is how to determine the particle array and how to actually build the entry points array. The second image shows a simple way of creating the particle array, by first building up a partial sums array that simply adds up as the entries continue over the index. Next we fill in each particle into the particle array, with the index being determined by the subsequent partial sums entry and decrement the entry in the partial sums array. This leaves us with a particle array and a partial sums array.

Dependent on how often the neighborhood information needs to be accessed, the neighborhood information can also be cached in a neighbors array, as done in my code sample https://github.com/hanusspi/BasicParticleSimulation.

Considering that this implementation is very poorly optimized and majorly created as a test bed for a bigger framework, we can still see drastic performance improvements growing with the number of particles. And most importantly, doubling the number of particles only halves instead of quarters the framerate now.

4. Summary

Today we analyzed one major bottleneck in particle based simulations and derived two new methods of improving the performance. Firstly we built up an easier system using dense grid spatial data structure and then improved performance even further by applying sparse hash grids. While this helps performance a lot, especially the fluid simulation still runs very slow, as it is highly serialized. One means to improve performance would be to increase clock rates of the CPU. But here we are very limited. And if we cannot scale in depth, we have to scale in width. While CPUs are still very limited in this department, GPUs are not. Therefore next time we will take a deeper look at leveraging GPUs to gain massive performance improvements.

In the last post we created our first pressure solver and got some pretty good results. Real fluids though do not really look like a bunch of spheres, but have a closed fluid surface. To resolve this issue, we define a function that can tell us how far a point in space is from the fluid surface. To do this we will define a signed distance function, that works with the lagrangian fluid simulation. Furthermore we need a strategy to create points, for which we can sample the distance to the surface and turn those points into a grid. And this is what we will start with.

2. Marching Cubes

To start off with marching cubes algorighm [LC87] we overlay the simulation domain with a voxel grid. The length of a voxel we will denote with \(l\). Next we sample some function, in most cases a signed distance field (sdf) \(\Phi\) at each corner point \(x_i\) of each voxel.

The first major issue of this algorithm is indexing. We have a 3d voxel grid, but need to store everything in a 1d array. Therefore we determine unique integer coordinates (i,j,k) (which are easy to translate into world space) with (0,0,0) at the origin of the grid. Given the size of our simulation domain, which we can represent as an Axis Aligned Bounding Box (AABB) we can determine how many voxel cells the grid has in each dimension, denoted by \(n_x, n_y, n_z\). Next we need to map each integer coordinate to a vertex (corner of a voxel) with

\(V = in_y n_z + j n_z + k\).

In the same fashion a cell can be identified with

\(C = in_y n_z + j n_z + k\).

Even though we have one less cell in each direction than vertices. But for simplicity we keep it like that.

And lastly we need a function to define edges

where dir is 0,1,2 for x,y,z.

For the full implementation we will have to handle some literal edge cases, because vertices on the outer side of the grid do not have edges in all directions.

The nice thing about those functions is, that they are bijective, therefore from a vertex id we can backtrack to its integer and finally world coordinates. Given the world coordinate x we can sample \(\phi(x)\) and get for the vertex i \(\Phi_i = \Phi(x_i)\).

With this information we know if a vertex is inside or outside of the surface. This already gives us some basic information, but not a mesh yet.

Therefore next we will iterate over all edges and check if they intersect the surface, i.e. if the sign between the vertices making up the edge changes. If we have a sign change, we calculate an edge weight by linear interpolation giving us the intersection point. This gives us a solid guess of where the actual surface could be. It is important that we store these points in relation to the edge they belong to. Since our surface is going to have a lot fewer points than we have voxels, it is sensible to store them in a map.

In the last step we need to append the vertex data with geometric connectivity. For this a marching cubes table exists, that we can use. As visible in the image above, the marching cube table tells us, based on which edges experiencing a sign change, which vertices to connect. And for this to work properly, all the previous hassle with indexing was required.

Lastly, to get a smooth mesh, we do not just need to have vertices and connectivity, but also vertex normals. These we can either calculate in the beginning for the voxel corners and interpolate later on or calculate in a second sweep for the exact vertex positions.

Applying this for an object with a known signed distance function, like a torus yields this result:

3. Fluid Surface Reconstruction

To make marching cubes work we therefore need a \(\Phi\) that we can sample. While for primitive shapes, like a torus, sphere or square these are analytically defined, for our fluid they are not. Therefore we need a function that is close to 1, if it is inside the fluid and 0 or even negative if it is outside of the fluid. Applying some fundamentals of the SPH idea and using the method how we determine the value of quantites in SPH (as discusses in the last posts) gives us [BJ25]

where \(\rho_j\) is now the normalized density estimation

We are required to add a slightly negative c to force a sign change, since the second part of the function will never get smaller than 0. By choosing the c we can determine how tight or loose we want the surface to be. A good starting value is 0.55.

Since this formulation seemingly requires a neighborhood search, it seems to be pretty expensive. Since unlike the particles, our vertices are grid aligned and indexable though, we can turn things around and iterate over the particles, determine an AABB around each particle, check which vertices are in the interior of the AABB and then write the values to the vertices. The only disadvantage of this approach is that it is very hard to parallelize.

In the very beginning we defined the cell size as a parameter. We want to set the cell size so that a sole particle can create a single mesh, to prevent particles suddenly popping up and vanishing later on. Therefore a good cell size is 1.0 times the particle radius.

4. Results

Applying marching cubes and the newly defined sdf gives us this lovely result for our previously generated simulation

Depending on the rendering settings in Blender the look of the fluid can be varied even further.

5. Summary

In this post we started out by defining a voxel grid with many indices turning it into the marching cube algorithm [LC87]. Finally we managed to apply it to a particle based fluid simulation turning it into a real fluid. The code can be found under https://github.com/hanusspi/BasicPressureSolverLab, where the mesh gets generated as a vtk and can be imported into blender using the same workflow as for the particles in the previous blog post.

References

[LC87] W.E. Lorensen and H.E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM Computer Graphics, 21(4):163–169, 1987. [BJ25] Prof. Dr. Jan Bender, M.Sc. Timna B¨ottcher, M.Sc. Jos´e Fern´andez, M.Sc. Lukas Westhofen: Fluid Simulation in Computer Graphics

In the previous chapters I talked alot about physics and maths building up some fundamental ideas of why and how the lagrangian viewpoint can be applied to fluid simulation. Now it is time, to turn the formulas into an algorithm and code to start some fluid simulation. The algorightms we use to simulate fluid, are generally called pressure solvers, since that is exactly what they do.

2. WCSPH

There is many ways to implement an SPH Solver, but one of the oldest and simplest (and easiest to understand) is Weakly Compressible SPH.

Let me start by showing you the general algorithm and then jump into the details and some practically relevant side notes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// SPH (Smoothed Particle Hydrodynamics) Algorithm

FOR each particle i:
    find_neighbors(i) → neighbors_j

FOR each particle i:
    density_i = SUM(mass_j * kernel_weight(i,j)) for all neighbors j
    pressure_i = state_equation(density_i)

FOR each particle i:
    pressure_acceleration_i = -(1/density_i) * gradient(pressure_i)
    viscosity_acceleration_i = viscosity * laplacian(velocity_i)
    other_acceleration_i = gravity
    
    total_acceleration_i = pressure_force_i + viscosity_force_i + other_force_i

FOR each particle i:
    velocity_i(t + dt) = velocity_i(t) + dt * total_force_i 
    position_i(t + dt) = position_i(t) + dt * velocity_i(t + dt)

The first step is the neighborhood search. For starters we will be using CompactNSearch to do this for us. This offers a relatively well optimized, fast and accesible way to do the neighborhood search.

In the next step we need to see, how the current situation looks. Therefore we compute the density for all fluid particles including boundaries as we have discussed in the last post. Finally we can apply the EOS (equation of state) to determine the pressure for each particle.

Knowing the pressure, we can now determine a resulting force. The force is dependent the particle mass, which is static, the density, which we just calculated and the gradient of the pressure.

Here we can see the use of the altered derivative to ensure symmetric forces between particles. Dividing the pressure force by the particles mass then gives us an acceleration:

Combine this with the acceleration due to viscosity:

And applying gravity gives us all the resulting accelerations for our particle.

In the final step we apply the accleration using the Semi Implicit Euler method to update the velocity and finally the position.

3. Discusstion and Specialties

Having looked at the general proccess it is easier to explain the name. What we are doing is looking at the current situation and based on that we derive an update. But we do not know, if the update will improve the situation. After all we could push many particles into one area creating a new pressure hot spot. This use of an explicit solving method makes the weackly compressible in the name.

To minimze the error we make by doing this, we have to select a very small timestep size. Which size exactly always depends on the dynamics of the simulation and all other parameters. A second factor that limits our timestep size is the boundary handling method. Currently we have a single layer of boundary particles that the fluid particle should never cross or it will be gone for ever. To express and calculate this limitation, we use the CFL-Condition:

Which we can either use to determine a dynamic timestep size by looking at the fastest particle or which we can use to limit the particle velocities for a fixed timestep size.

In the previous post we also discussed (at least a bit) an extra model for cohesiona and adhesion. This model gives us forces, that we can simply add to the other Forces in the algorithm and include it that way.

Lastly we need to discuss an issue with the equation of state. Currently we simply derive a pressure from the density deviation. Furthermore we simulate a single phase liquid, therefore we are not having any surrounding air particles or something like this. This means, that especially on the surface and in splashes we will see negative pressures. These pressures would suck up other fluid particles and build clumps. While this might sound like surface tension, it is a very much unwanted and bad looking effect. Therefore we simply modify our EOS: \(p_i = \max(0, \kappa (\rho_i - \rho_0))\) to ignore negative pressures and only handle postive pressures.

4. Turning it into code

Having an algorithm and having a running simulator are two very different pairs of shoes. Therefore i want to take a few lines to talk about the actual implementation and some potential error sources. The full code for this project is available here: https://github.com/hanusspi/BasicPressureSolverLab

The procect is coded in C++ as most libraries in this domain are available in C++. Furthermore, it offers the best compromise between beeing fast and still readable enough, to be well understood and fast programmed.

First of all, we need to find a way to define boundaries. For this I used the tinyobject loader and object files, which can be created in blender. Next we need a method to sample the particles. Here I got some good code from a class at my Uni. As did I get a few more helping functions. Afther we have loaded our boundary, and populated it with boundary particles, we apply a first neighborhood search and determine the boundary weights and volumes.

Next we need to define our fluid particles. For simplicity we will start out with sampling a cube with a regular grid and make every grid cell corner a particle. While from a readability stand point it might me nicer to create now a struct, that holds all the particles properties, like its mass, density, position and acceleration. But having a vector of structs is fairly slow compared to having a struct of vectors (or just many vectors). Vectors being like lists, not the mathematical thing.

In the next step we can implement a simulation loop that and update our particle positions. Sadly we do not really have any way of seeing the results right now. Of course there are many ways, for visualizaation, like adding a graphics api and showing the results directly.

One thing you will note though, the simulation runs very slow. Therefore for now we will simply export the particles into .vtk files. That is file format originally created for paraview, but also useable in blender using: .

Now we can run our simulation, drink a cup of coffee, maybe have full meal (or two) and then we will look at the results.

5. Results

After 5 minutes we can look at the first results of the breaking dam scene. For this we spawn a grid of particles and advect them according to the pressure solver. For starters we set the viscosity very low. As particle diameter we choose 0.005m and we run the simulation with 3000Hz. This means for one second of footage we need to run the neighborhood search 3000 times. In future we will explore different solvers that can hopefully run at a lower frequency. This is what makes the computation very slow. As we can see, the particles look like fluid, but non the less, we still have the look of a ball pit and not really the look of water. This issue will be fixed in a future post.

6. Summary

Today we took a big step and implemented the first pressure solver. It still is relativley basic, but it works. Right now we still face to big problems. First, a simulation takes very long, second it does not look like water. Those will be things, that we hopefully can change in the next posts.

In the last post, we began exploring the fundamentals of SPH Fluid Simulation. First, we introduced the general physical framework of fluid simulation and then began turning it into a more computer-friendly system. But all that we have so far is a fluid-fluid interaction. Therefore, we are now able to spawn some fluid in vacuum and have it behave (not even really realistically). So the next step will be to explore how we can compute some coupling between the fluid and solids. Furthermore, we explored the first part of the Navier-Stokes equation, but have not yet explored viscosity. Last but not least, we will take a quick peek at some more effects that the Navier-Stokes equation does not account for.

2. Fluid-Solid Coupling

Since we are representing our fluid as discrete particles that are interacting with each other, the intuitive approach is to simply represent the boundary as particles as well. And exactly that is what we are going to do.

First, we sample the boundary with boundary particles. Those particles need to be treated differently in our simulation, since they are never going to move. Next, we need to determine the weight of the boundary particles. Unlike fluid particles, who get all the same weight, we would have to sample the boundary particles all with perfectly equal distance to be able to do this. Since this is not very practical, we simply apply our kernel function and first determine the implied volume of each boundary particle depending on its neighborhood.

You might see a pattern arise here. Most of fluid simulation works by summing up some kernel function over its neighborhood. The advantage of the volume approach is that it is independent of the mass or density. Now the question is, which density our boundary should have. Maybe one could guess that it depends on the boundary material, but since we only care about the effect on the fluid, we assume they have the rest density of the fluid.

With that in mind, the density of a fluid particle turns into:

Knowing this, we can apply our previous knowledge for building a fluid simulator.

3. Viscosity

Viscosity is defined by a fluid’s rate/dependent resistance to a change in shape or to movement of its neighboring positions relative to one another. At least by Wikipedia…

What this means is that viscosity describes the internal friction between adjacent layers of fluid. Since we are having particles and the particle’s velocity, we can therefore derive something to simulate viscosity:

While this looks huge, mean, and complicated, we just need to know a few things. First, we get an acceleration of our fluid particle. The magnitude of the acceleration depends on the velocity difference of the particle relative to its neighbors or boundary (since the boundary doesn’t move, we do not have to subtract anything). Furthermore, we get two parameters. One for the fluid viscosity \(\nu_f\) and one for the boundary viscosity \(\nu_b\) with which we can set up the behavior of our fluid.

While one might argue that especially water is not viscous, we will still apply a small bit of viscosity to ensure numerical stability. This means, by discretization we make a small error every timestep. And this might lead to an increasingly growing error and run away: an explosion (at least visually). And to prevent this behavior, viscosity is a good dampening mechanic.

4. Surface Tension (Adhesion and Cohesion)

Lastly, I want to introduce a simple version of surface tension. If we were to spawn a cube of water in space, in nature the cube would turn into a droplet, since water always strives to minimize its surface area. This is an effect that SPH by default does not recreate. The same effect is also responsible for droplet building on surfaces.

Therefore, we introduce a cohesion force which, by applying a different kernel, strives to minimize the surface area of the fluid. For simplicity, I’ll just link to the authors of the paper and not introduce any new formulas.

Next, we add an adhesion force, which simulates the wetting effect of fluids and the tendency to stick to the boundary.

Taking a look at cohesion, we can see with an increasing cohesion factor we get droplet building.

5. Summary

In this a bit shorter chapter, we collected the final puzzle pieces to start building a first fluid simulator. For that, we first thought of a way to handle boundaries to create interesting simulations. Then we introduced viscosity as a natural dampening factor and added adhesion and cohesion forces to generate a more natural fluid behavior.

1. Introduction to Fluid Simulation

One of the more interesting topics in physics-based animation is the simulation of water and fluids in general. The relevance of this topic reaches from computer games, like Sea of Thieves, where a real-time simulation of the ocean is relevant, over VFX art for movies, like Avatar: The Way of Water, to many industrial applications like casting steel.

While there is a lot of research being done on this topic, I first came into contact with it during a physics-based animations lecture and have since worked hard on getting a deeper insight into the topic. Here I want to begin by introducing the fundamentals and then take a dive into some deeper topics that I am currently interested in.

There exist two (and a half) approaches for fluid simulation: the Lagrangian particle-based one and the Eulerian grid-based one. Furthermore, there is a lot of research going into hybrid models that try to combine the best out of both worlds. For now, I will focus on the Lagrangian approach.

2. Navier-Stokes Equation

First, let’s think about what characteristics make a fluid a fluid and especially let’s think about water. First, of course, it can flow freely and will always try to find a position of lowest energy. Furthermore, it is not compressible. So, (to an extent) no matter how deep you dive into the ocean, 1 cubic meter of water will always have the same amount of water molecules, no matter how high the pressure is. (Very much unlike air, which is very compressible) Finally, we have some more specific properties like surface tension.

The fundamental aspect of fluid simulation though is maintaining the incompressibility through time. And while this post will have a few formulas, I strive to make it as easy and interesting to understand as possible. And I promise in future posts, things will get more programming-centric. The incompressibility can simply be described by \(\frac{D \rho}{D t} = 0 \Leftrightarrow \nabla \cdot v = 0\). $\rho$ denotes the density. The second part implies the same, but we will not use it any further.

From this we can derive the Navier-Stokes equation: \(\frac{D\mathbf{v}}{Dt}=-\frac{1}{\rho}\nabla p+\nu\nabla^2\mathbf{v}+\frac{\mathbf{f}}{\rho}\) First of all, we are describing an acceleration. This acceleration has three components:

1. \(-\frac{1}{\rho}\nabla p\): Acceleration due to pressure differences in the fluid.

2. \(\nu\nabla^2\mathbf{v}\): Acceleration due to friction forces between particles in the fluid. This is generally known as viscosity.

3. \(\frac{\mathbf{f}}{\rho}\): Accelerations due to external forces. The forces are normalized to a unit volume. The major force being used is gravity, but it can also be used to induce waves.

3. Smoothed Particle Hydrodynamics (SPH)

Currently, we managed to describe the properties of what we want to simulate with a formula. This is a good beginning, but the formula describes something temporally and spatially continuous, which we cannot compute. Therefore, we introduce a spatial discretization into finitely many particles. These particles are not really particles (but we call them that for simplicity) but more like probes giving us information about important field values like density and velocity at a position. This form of discretization makes it a Lagrangian viewpoint.

Having done the first step, we will use a trick to use this for discretization using the Dirac-Delta Distribution. This distribution has the simple characteristic that it is always zero, except at the point x where it jumps to infinity. Formally this looks like:

Given a function \(f:\Omega\to\mathbb{R}^n\) that maps a position vector \(x\) in the domain \(\Omega \subset \mathbb{R}^3\) to a scalar or vector value, it can be rewritten using the Dirac delta identity:

While this does not really help us yet to determine somehow what values/properties our particles have, we can now simply rewrite the Dirac Delta identity using a kernel function.

Our kernel function will have to fulfill a few requirements. As we saw, the Dirac delta identity sums up to 1. This must be the case for the kernel function as well. Furthermore, it needs to be symmetric, positive, and have a compact condition. This simply means that it only returns a positive value as long as we are in a set radius, and otherwise we are just 0. For now, we will use the cubic spline kernel, which looks like this:

where \(q = \frac{\|\mathbf{x}_i - \mathbf{x}_j\|}{h}\) and $\alpha$ is the normalization constant depending on in how many dimensions we run the simulation.

Taking a closer look at the relationship between the kernel function and the Dirac Delta identity:

It becomes more obvious to see how an increasingly smaller h pulls the kernel function to a Dirac delta identity. Changing the h will later also influence how our simulation behaves. If the h is too small, it will explode; if it is too big, it will be very lazy, since there will be too much averaging between particles.

Finally, we can put it together and rewrite the continuous function into a discrete sum:

Now the quantity of f can be simply determined by a sum over its neighborhood in a fixed radius. Doing the neighborhood search in a naive brute force way is very expensive and scales horribly (\(O(n^3)\)). But luckily, there are some smarter ways to handle this. We will take a look into this at a later point, but for the beginning, we will use a library that does this for us. Using some smart math, we can now express the density of a particle using:

What this means is that the closer the neighbors (the particle itself being the closest neighbor) are, the higher the influence of the density will be, which is exactly what we want. Many particles clustered together should result in a high density and sparsely seeded ones into a low one. This is visualized here:

With this, we did the first big logical step into understanding fluid simulation.

4. Equation of State

Knowing the density of all our particles in the system now enables us to enforce the continuity equation. If the particle density is too high, this can be directly translated into a high pressure. Pressure forces are the forces that counteract high densities. Furthermore, taking the gradient of the density into account, we can determine in which direction the pressure is forcing our particle to retain constant density.

A mathematical formulation of this is presented in the equation of state (EOS):

where the pressure \(p\) of the particle $i$ is dependent on the pressure difference between the rest density \(\rho_0\), which is the natural density of water and therefore for us \(1000kg/m^3\), and the actual density of the particle. Furthermore, \(\gamma, \kappa\) are stiffness parameters. For now, we will simply set \(\kappa\) to 1, giving us \(p_i = \kappa (\rho_i - \rho_0)\). Formulating it like this shows great similarity to a spring force, just defined as a density deviation times a stiffness constant.

While this gives us the pressure, later on we will also need the direction of the pressure. Therefore, it is noteworthy to quickly talk about the derivative of our kernel and density calculation. Mathematically, the derivative keeps the factors the same and just derives the kernel function itself. This derivative does not preserve linear and angular momentum though, since it is not symmetric. To maintain symmetry and satisfy Newton’s laws, we therefore rephrase it as:

5. Summary

With this, we already did the first steps into understanding the fundamentals of fluid simulation. First, we took a look into the general physical principles behind fluids and then derived a way to make it more computable. For the next part, we will take a look at a few more building blocks before we can start building a first fluid simulator.

So far we have discussed bayesian analysis of data. So we took data, a hypothesis and prior knowledge to gain an insight on the posterior. For this we considered rejection sampling and importance sampling, which were both absed on independent samples from q(z). However, for many problems of practical interests, is is difficult or even impossible to find a fitting q(z) with the required accuracy.

For that reason it would be better to simply abondon independent sampling and simply rely on a stochastic process to generate dependent samples from the target distribution. For this we use a Markov Chain and a Monte Carlo estimate. This might sound a bit abstrct for now, but in a bit we will figure out how this works in detail.

The core topic is now called MCMC (Markov Chain Monte Carlo) and allows us to sample from a large class of distributions. For this a new sample is generated within a probability distribution of the previous sample in the chain. MCMC is popular, as it scales better with higher dimensions.

For a Markov Chain we have a set of states and a state transition relation. Futher more the state transitions can have different probabilites. Now we keep a record tof the current state \(z^{(\tau)}\) and the proposal distribution depends on the current state: \(q(z|z^{(\tau)})\). The sequence of samples from the chain we will denote with \(z^{(1)}, z^{(2)},...\) and again we assume, that we can evaluate \(p(z) = \frac{1}{Z_p}\tilde{p}(z)\). Now at each time step, we generate a candidate sample from the proposal distribution and accept the sample according to a criterion.

2. Metropolis-Hastings Algorighm

The idea of MCMC is that the chainis a sequence of states created through iterative estimations of new states. A state is a set of parameters of an analytic model over which MCMC is iterating in order to maximize the posterior. For a markov chain, the markov property has to hold: \(P[S|S'] = P[S|S',S'']\) . So the probability of state \(S\) only depends on the previous state \(S'\) and not on any other earlier visited states. In the Bayesian context, the accpetance of a new state often involves comparing the likelihood of the current state, with respect to a uniformly sampled acceptance threshold. This might sound a bit confusing in the beginning but will become clearer in the code.

For now lets assume, that the proposal distribution is symmetric, therefore \(q(z_A|z_B) = q(z_B|z_A)\) holds. Now we need to determine the probability with which we accept a new candidate sample \(z^*\) :

Now we can again sample \(u\) from a uniform distribution and accept the sample, if \(A(z^*, z^{(\tau)}) > u\) . What we can already note is, that the new candiate sample will always be accepted if \(\tilde{p}(z^*) \geq \tilde{p}^{(\tau)}\) . So if the new sample has a higher probability than the previous one, we will accpet it. But every now and then the algorithm also accpets samples with lower probability. For the next step this means, that if the new sample is accepted, that \(z^{(\tau+1)} = z^*\) and if the new sample is not accepted, \(z^{(\tau+1)} = z^{(\tau)}\) . If we compare this to rejection sampling, we discarded rejected samples. Now we still discard samples, but copy accepted samples in return. So there can be multiple copies of a sample. This simpler case is the original Metropolis Algorithm and has the property, that if \(q(z_A|z_B)>0\) for all \(z\) , the distribution of \(z^{\tau}\) tends to \(p(z)\) as \(\tau \rightarrow \inf\) .

To compensate for non symmetric proposal distributions, we can append to our acceptance probability and define it with:

where k labels the members of the set of possible transitions considered. The evaluation of the acceptance criterion does not require the normalizing constant \(Z_p\) . And now we can also see, that if \(q_k\) is symmetrical, Metropolis Hastings becomes the Metropolis Algorithm.

3. Implementation in Python

For our implementation example lets stay with a 1D mixture of gausisians as given for the previous sampling methods. Furthermore we will use a simple gaussian as proposal distribution. Lets start by defining the distributions:

1
2
3
4
5
6
7
8
9
10
11
12
def gaussian(x, mu, sigma):
    factor = 1 / (sigma * np.sqrt(2 * np.pi))
    return factor * np.exp(-0.5 * ((x - mu) / sigma) ** 2)

def target(x):
    return 0.3 * norm.pdf(x, 0, 2) + 0.7 * norm.pdf(x, 10, 2)

def proposal(mu, sigma):
    return mu + sigma * np.random.randn()

def proposal_conditional(x, y, sigma_prop):
    return gaussian(x=x, mu=y, sigma=sigma_prop)

This style of formulating the distributions seems a bit weird, but makes sense, since i wanted to play around a bit with the functools.partial() function, which turns a parametrized function into a predefined function for fixed inputs. Next take a look at the Metropolis Hastings Algorithm:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
def MH(target, proposal, proposal_conditional, n_iterations):
    samples = []
    n_accepted = 0

    # initial state
    x = np.random.rand()
    proposal_old = target(x)

    for i in range(n_iterations):
        xprime = proposal(x)    
        proposal_new = target(xprime) # P(x')
        alpha = proposal_new / (proposal_old + np.finfo(float).eps) # P(x') / P(x)
        alpha *= proposal_conditional(x, xprime) / proposal_conditional(xprime, x) # q(x|x’), q(x'|x)
        alpha = min(1, alpha)
        # Acceptance probability A “accepts” the proposed value x'
        # otherwise rejects it by keeping the current value x
        
        if np.random.rand() <= alpha:
            x = xprime
            proposal_old = proposal_new
            n_accepted += 1

        if i > 2000: # Burn-in period
            samples.append(x)

    return samples, n_accepted

Here we take the three distribution functions and the number of iterations as input. Then we just run the conditions over n_iterations and only accept the samples which have an alpha greater than our random linear variable. Since we are using a Markov Model it is a helpful trick, to give the model a burn in period, which is implemented with the last condition. Since our samples will only converge, the more transitions we take, early samples tend to be off and can skew the result. The trouble is, that if we don’t know the the target distribution, we don’t have any indication, when we ran for long enough. So it could happen, that we stop to early and get bad results.

Now lets put it all together:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
n_iterations = 50000
np.random.seed(42)
sigma_prop = 10
mu_prop = 3

target_fn = partial(target)
proposal_fn = partial(proposal, sigma=sigma_prop)
proposal_conditional_fn = partial(proposal_conditional, sigma_prop = sigma_prop)

# Generate samples using Metropolis-Hastings
x, naccept = MH(
    target=target_fn,
    proposal=proposal_fn,
    proposal_conditional=proposal_conditional_fn,
    n_iterations=n_iterations,
)

Ns = [3000, 5000, 8000, n_iterations]
print(f"number of accepted samples = {naccept}")

x_t = np.linspace(-10, 20, 1000)
gauss = target_fn(x_t)

fig, axes = plt.subplots(2, 2, figsize=(10, 7), sharey=True)
for i, ax in enumerate(fig.axes):
    ax.set_title(f"N={Ns[i]}")
    sns.histplot(ax=ax, data=x[: Ns[i]], element="step", kde=True, stat="density")
    ax.plot(x_t, gauss, color="r")
    ax.axis([-10, 20, 0, 0.2])

plt.tight_layout()
plt.show()

Which results in the following plot:

What we can notice is, that the more we sample, the better we match the target distribution. But even for a little amount of samples we get a relativly good match. Further more, we have a pretty high rejection rate. For the last test with 50000 samples only 18171 samples are being accepted. This is pretty bad for us, since it means, we just wasted over 60% of our computing power to produce samples, that we discard.

4 Summary

In this article we took the jump from general independent sampling to MCMC sampling. For this we began introducing some general ideas about Markov Chains and then the Metropolis Hastings algorith,. In the implementation we got a better feeling for how it works in practice and identified the biggest issue, that we will never be sure, if we ran for long enough. With this, the approach has shown to be simple and effective, even though it is not very efficient. In the next article we will talk about Gibbs Sampling, which will be completly parameter free.

In the previous post, we explored Rejection Sampling as a method for sampling from complex distributions. While effective, rejection sampling can be inefficient, especially when a large number of samples are rejected. Importance Sampling offers an alternative approach that improves efficiency by assigning weights to samples instead of rejecting them.

Importance sampling is widely used in Bayesian inference, Monte Carlo simulations, and statistical estimation. In this post, we’ll dive into the concept of importance sampling, how it works, and play around with it in python.

2. The Idea behind Importance Sampling

For most use-cases the goal is not sampling from \(p(z)\) by itself, but to evaluate expectations of the form

The exact solution of this integral is most of the time intractable, so we need to find a discrete solution. A simple strategy would be the use of grid sampling. For this we discretize \(z\)-space into a uniform grid. Now we evaluate the integrand as a sum of the form

For a low dimensional problem this is a good a solution, but the number of terms grows exponentially with number of dimensions.

But we can apply the same trick of rejection sampling and try to make the problem easier using a proposal distribution \(q(z)\) from which it is easy to draw samples. Now we can use some trickes to reformulate the problem:

where \(\frac{p(z^{(l)})}{q(z^{(l)})}\) is considered the importance weight.

Now we need to consider the typical setting, in which we can only evaluate \(p(z)\) up to an unkown normalization constant \(p(z) = \tilde{p}(z)/Z_p\). With \(q(z)\) we can do the same: \(q(z) =\tilde{q}(z)/Z_q\). From this we can construct:

\(\mathbb{E}[f] = \int f(z)p(z) dz = \frac{Z_q}{Z_p} \int f(z)\frac{p(z)}{q(z)}q(z) dz\) \(\simeq \frac{Z_q}{Z_p} \frac{1}{L} \sum_{l=1}^L \tilde{r_l}f(z^{(l)})\)

with \(\tilde{r_l} = \frac{\tilde{p}(z^{(l)})}{\tilde{q}(z^{(l)})}\)

As you might have noticed, we introduced two constants expressing our uncertaintity in the quality of our sample. These constants are not known, but we only need to know their ratio anyways. This can be done

and with this we can express the expectancy as

$$ \mathbb{E}[f] \simeq \sum_{l=1}^L w_l f(z^{(l)})

with

For the weights we can see, that the sum acts as a normalization factor. With this the first difference to rejection sampling should become obvious. In rejection sampling we were worried about a bad fit and throwing away to many samples. Here we use every sample and give it a weight. In all fairness though, we might keep the sample, but it still could end up getting a weight very close to zero.

Now lets take a look about the resulting implications.

• The success of importance sampling depends still crucially on how well the sampling distribution \(q(z)\) matches the desired target distribution \(p(z)\)
• Often, \(p(z)f(z)\) is strongly varying and has a significant proportion of its mass concentrated over small regions of the z-space
• Expectancy may be dominated by a few heavy weights
• The results can be wrong and produce arbitraty errors with no diagnostic indications. This will happen, if the proposal distribution is to far off or the samples are by random chance in none of the relevant areas of \(p(z)f(z)\)

3. Example of Importance Sampling

Lets now take a look at how this can look in code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
import numpy as np
from scipy.integrate import quad
import matplotlib.pyplot as plt
from scipy.stats import norm

np.random.seed(42)

# Define the lower and upper bounds of the integral
lower_bound = -2
upper_bound = 8
# Define the target distribution (Mixture of Gaussians)
def target_distribution(x):
    return 0.3 * norm.pdf(x, 1, 0.9) + 0.7 * norm.pdf(x, 5, .6)

# Define the proposal distribution (Gaussian distribution)
def proposal_distribution(x):
    return norm.pdf(x, 3, 2)

# Define the importance sampling function
def importance_sampling(num_samples):
    samples = np.random.uniform(lower_bound, upper_bound, num_samples)  # Generate samples from the proposal distribution
    weights = target_distribution(samples) / proposal_distribution(samples)  # Compute the importance weights
    weights /= np.sum(weights)  # Normalize the weights
    
    # Estimate the expectation of a function using importance sampling
    def estimate_expectation(func):
        return np.sum(func(samples) * weights)

    return estimate_expectation

def combined(x):
    return target_distribution(x)*sigmoid(x)

# Define the exact solution
def exact_solution():
    return quad(combined, lower_bound, upper_bound)[0]

# Define the sigmoid function
def sigmoid(x):
    return 1 / (1 + np.exp(-x+3))-.2

# Perform importance sampling with 10000 samples
estimate = importance_sampling(100)(sigmoid)
exact = exact_solution()

print('Estimate:', estimate)
print('Exact:', exact)

# Print the estimates
x = np.linspace(lower_bound, upper_bound, 1000)  # Define "x" as an array of values
plt.plot(x, sigmoid(x), label='Sigmoid')
plt.plot(x, proposal_distribution(x), label='Proposal Distribution')
plt.plot(x, target_distribution(x), label='Target Distribution')

plt.axhline(y=0, color='black', linestyle='--')
plt.legend()
plt.show()

So, what is happening here. First we define our target function, which is a Mixture of Gaussians, then our proposal distribution, which is a simple normal distribution and \(f(x)\) is a sigmoid function. Then we generate the samples and calculate the corresponding weights and their normalization factor and get the following expectancy for the importance sampled expectancy: 0.413 and using numerical integration we get an “exact” value of: 0.451, which seems to be a pretty good guess. But moving the mean of our propsal distribution by 2 to the left turns our estimate already to 0.705 and keeping the mean at 3 and changing the variance to 1 turns the expactancy to -0.05. If we visulize the functions, we can also see why: The proposal distribution only covers the low part of the target distribution leading to the bad result.

4. Conclusion and What’s Next

In this post, we’ve introduced a more versatile sampling method than rejection sampling. We implemented rejection sampling in Python and used the example to better understand the implications of importance sampling.

In the next post, we’ll explore a more universal approach using the Metropolis Hastings Algorithm with Markov Chains

When dealing with complex probability distributions \(p(z)\), direct sampling can be difficult or even impossible. This is where we can use Rejection Sampling, which allows us to sample from a complex distribution with the aid of a proposal distribution, that we can evaluate.

Rejection sampling is especially useful in Bayesian inference when dealing with non-standard distributions that don’t have a straightforward method for drawing samples. In this post, we’ll explore the concept of rejection sampling.

2. Why Rejection Sampling?

Rejection sampling is used when:

• Direct Sampling is Difficult: When we can’t easily sample from the target distribution, rejection sampling provides an alternative method.

• Target Distribution is Known Up to a Constant: To perform rejection sampling, we need to be able to evaluate the target distribution to some normalization factor \(Z_p\) \(p(z) = \frac{1}{Z_p}\tilde{p}(z)\) Preferably with very little effort.

3. How Rejection Sampling Works

The idea of rejection sampling is to sample from a proposal distribution and then “reject” or “accept” samples based on how likely they are under the target distribution.

Let’s break it down into simple steps:

1. Choose a Proposal Distribution: Select a proposal distribution \(q(z)\) that is easy to sample from and that roughly covers the shape of the target distribution \(\tilde{p}(z)\). The proposal distribution should be scaled by a constant \(k\), such that \(\forall z: kq(x) \geq \tilde{p}(z)\).

2. Generate a Sample: Draw a sample \(z\) from the proposal distribution \(q(z)\).

3. Generate a Uniform Random Variable: Draw a uniform random variable \(u\) from \(\operatorname{Uniform}(0,1)\).

4. Acceptance Criterion: Accept the sample \(z\) if \(u\leq\frac{\tilde{p}(z)}{kq(z)}\)​. If not, reject the sample and go back to step 2.

5. Repeat: Continue the process until you have enough accepted samples.

The accepted samples will be approximately distributed according to the target distribution \(p(x)\).

4. Example: Rejection Sampling from a Gaussian Mixture Model

Let’s now consider an example where we want to sample from a Gaussian Mixture Model (GMM), which is a combination of multiple Gaussian distributions as a demonstration.

We’ll use a simple GMM composed of two Gaussian distributions.

The target distribution \(p(x)\) is a mixture of two Gaussian distributions: \(p(x)=0.3\cdot \mathcal{N}(x;−2,0.5^2)+0.7\cdot \mathcal{N}(x;2,1^2)\)

The normalization factor \(Z_p\) in this case is 1, since we can evaluate this function precisly.

We will use a simple normal distribution as the proposal distribution \(q(x)\).

1. Python Code Example: Rejection Sampling

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

# Define the target distribution (Gaussian Mixture Model)
def target_distribution(x):
    return 0.3 * norm.pdf(x, -2, 0.5) + 0.7 * norm.pdf(x, 2, 1.0)

# Define the proposal distribution (Normal distribution)
proposal_distribution = norm(0, 3)  # Mean 0, std 3

# Define the constant M such that M * proposal_distribution >= target_distribution
M = 1.5  # Chosen to ensure the proposal upper-bounds the target

# Number of samples
n_samples = 10000

# Rejection Sampling
samples = []
while len(samples) < n_samples:
    x = proposal_distribution.rvs()  # Sample from proposal distribution
    u = np.random.uniform(0, 1)  # Uniform random variable

    # Accept or reject the sample
    if u <= target_distribution(x) / (M * proposal_distribution.pdf(x)):
        samples.append(x)

# Convert to numpy array for easier handling
samples = np.array(samples)

# Plot the histogram of the samples
plt.figure(figsize=(10, 6))
plt.hist(samples, bins=50, density=True, alpha=0.6, color='b', label="Sampled Data")

# Overlay the target distribution
x = np.linspace(-6, 6, 1000)
plt.plot(x, target_distribution(x), 'r', linewidth=2, label="Target Distribution")
plt.title("Rejection Sampling from a Gaussian Mixture Model")
plt.xlabel("Value")
plt.ylabel("Density")
plt.legend()
plt.show()

5. Limitations of Rejection Sampling

Now that we gained a rough understanding of how rejection sampling works, we can see that it can be very usefull, but there are also some obvious limitations:

• Efficiency: Rejection sampling can be inefficient if the proposal distribution doesn’t closely match the target distribution. The more samples we reject, the longer it takes to generate enough accepted samples.
• Parameters: Rejection Sampling cannot be simply used. For every usecase we need to find a good matching proposal distribution, which in itself can be very challenging or risk loosing alot of efficiency.
• Dimensionality: In high-dimensional spaces, finding a suitable proposal distribution that covers the target distribution well can be even more challenging, leading to high rejection rates.

Conclusion and What’s Next

In this post, we’ve introduced the concept of rejection sampling, a versatile technique for sampling from complex distributions. We implemented rejection sampling in Python and visualized how the method works in practice. Rejection sampling is a powerful tool but comes with trade-offs in terms of efficiency and complexity.

In the next post, we’ll explore Importance Sampling, a related method that improves efficiency by weighting samples from the proposal distribution instead of rejecting them.

Sampling is a foundational concept in statistics and machine learning, and it plays a crucial role in Bayesian inference, particularly in approximate methods. Simply put, sampling involves drawing random values from a probability distribution. These samples can be used to estimate properties of the distribution, such as its mean, variance, or other moments.

In this post, we’ll explore the basics of sampling, its significance in Bayesian inference, and how to perform sampling from a specific distribution using the Inverse Transform Sampling technique.

2. Why Sampling is Important in Inference

Sampling is essential in approximate inference methods, especially when dealing with complex models where exact analytical solutions are not feasible. It allows us to estimate expectations, probabilities, and other quantities by drawing samples from the desired distribution.

This is crucial in many applications, including:

• Monte Carlo Methods: Approximating expectations using random samples.
• Bayesian Inference: Sampling from the posterior distribution when it cannot be computed analytically.
• Optimization: Exploring complex probability landscapes using sampling techniques.

In this post, we’ll focus on a practical example of sampling from an exponential distribution using Python.

3. The Exponential Distribution

The exponential distribution is a common distribution used to model the time between independent events that happen at a constant average rate. It is characterized by the rate parameter \(λ\).

The probability density function (PDF) of the exponential distribution is given by: \(f(x;λ)=λe−λxforx≥0\)

where \(λ\) is the rate parameter.

The cumulative distribution function (CDF) is the integral of the PDF: \(F(x;λ)=1−e−λx\)

Using the inverse of this CDF, we can sample from the exponential distribution using a uniform random variable.

4. Inverse Transform Sampling

Inverse Transform Sampling is a method to generate random samples from a given distribution by inverting its CDF. The steps are:

Generate a uniform random variable \(u∼Uniform(0,1)\). Compute the inverse of the CDF at uu to obtain the corresponding sample \(x\).

For the exponential distribution, the CDF is: \(F(x;λ)=1−e−λx\)

To invert this, solve for xx: \(u=1−e−λx\) \(e−λx=1−u\) \(x=−1λln⁡(1−u)\)

Thus, to generate a sample from the exponential distribution, we can use the formula: \(x=−1λln⁡(1−u)\)

5. Python Code Example: Sampling from an Exponential Distribution

Here’s how you can implement inverse transform sampling for the exponential distribution in Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import numpy as np
import matplotlib.pyplot as plt

# Define the rate parameter lambda
lambda_param = 1.0

# Number of samples
n_samples = 1000

# Generate uniform random variables
u = np.random.uniform(0, 1, n_samples)

# Inverse transform sampling to generate samples from the exponential distribution
samples = -np.log(1 - u) / lambda_param

# Plot the histogram of the samples
plt.figure(figsize=(10, 6))
plt.hist(samples, bins=30, density=True, alpha=0.6, color='b')
plt.title("Sampling from an Exponential Distribution using Inverse Transform Sampling")
plt.xlabel("Value")
plt.ylabel("Density")

# Overlay the true exponential distribution
x = np.linspace(0, 8, 1000)
plt.plot(x, lambda_param * np.exp(-lambda_param * x), 'r', linewidth=2, label='True PDF')
plt.legend()
plt.show()

6. Applications of Sampling

Sampling isn’t just a theoretical exercise; it has practical applications in various domains:

• Monte Carlo Methods: Sampling forms the basis of Monte Carlo methods, which are used to estimate integrals, simulate systems, and perform Bayesian inference.
• Bootstrap Methods: In statistics, sampling is used in bootstrap methods to estimate the sampling distribution of a statistic by resampling from the data.
• MCMC Algorithms: Markov Chain Monte Carlo methods rely heavily on sampling to explore complex posterior distributions.

These applications show how sampling is a versatile tool in both statistics and machine learning.

Conclusion and What’s Next

In this post, we’ve introduced sampling from the exponential distribution using the Inverse Transform Sampling technique. Sampling is a crucial technique in Bayesian inference, especially when exact solutions are not feasible.

In the next post, we’ll dive into Rejection Sampling, a powerful method for sampling from complex distributions when direct sampling is difficult.

Bayesian inference is a powerful statistical technique that allows us to update our beliefs about the world as we gather more evidence. Unlike classical (frequentist) approaches, which often focus on fixed parameters and p-values, Bayesian inference provides a probabilistic framework where uncertainty is quantified and incorporated into the analysis.

At its core, Bayesian inference is built upon Bayes’ Theorem:

• \(P(H \vert D)\): The posterior probability, which represents our updated belief in the hypothesis H after observing data \(D\).
• \(P(D \vert H)\): The likelihood, the probability of the data given that the hypothesis \(H\) is true.
• \(P(H)\): The prior probability, which represents our initial belief in the hypothesis before seeing the data.
• \(P(D)\): The evidence or marginal likelihood, which is the total probability of observing the data under all possible hypotheses.

Bayesian inference is particularly useful when dealing with uncertainty, incorporating prior knowledge, and updating beliefs as new information becomes available.

2. Exact vs. Approximate Inference

While Bayesian inference provides a robust framework, it can be computationally challenging. In simple cases, we can calculate the posterior distribution exactly, which is known as exact Bayesian inference. However, in more complex models, calculating the exact posterior distribution becomes intractable due to high-dimensional integrals or complex likelihood functions.

This is where approximate inference comes into play. Approximate methods aim to estimate the posterior distribution without solving it exactly. These methods allow us to perform Bayesian inference in situations where exact methods would be computationally impossible.

Here’s a brief comparison:

• Exact Inference:
  • Pros: Provides precise posterior distributions.
  • Cons: Computationally expensive or impossible for complex models.
• Approximate Inference:
  • Pros: Scalable to complex models and large datasets.
  • Cons: Introduces approximations, which may lead to less precise results.

This blog series will dive into several approximate methods, but before we do, let’s walk through a simple example of exact Bayesian inference.

3. Example of Exact Bayesian Inference

Let’s consider a classic problem in Bayesian statistics: estimating the probability of a coin being biased.

Imagine you have a coin, and you’re not sure whether it’s fair (i.e., whether it lands heads or tails with equal probability). You decide to model this uncertainty using Bayesian inference.

• Prior Belief: You believe that the coin is probably fair, but you’re open to the possibility that it might be biased. This belief can be represented by a Beta distribution, a common choice for modeling probabilities.

• Data: You flip the coin 10 times and observe 7 heads.

• Likelihood: The likelihood function here is binomial since we’re counting the number of successes (heads) in a fixed number of trials (10 flips).

• Posterior Distribution: Using Bayes’ Theorem, we update our belief about the fairness of the coin.

Here’s the Python code that performs exact Bayesian inference for this problem:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta

# Define prior: Beta(2, 2) (initial belief that the coin is fair)
a_prior = 2
b_prior = 2

# Data: 7 heads out of 10 flips
heads = 7
tails = 3

# Posterior distribution parameters: Beta(a + heads, b + tails)
a_post = a_prior + heads
b_post = b_prior + tails

# Generate x values for plotting
x = np.linspace(0, 1, 100)

4. Motivation for Approximate Inference

While the above example was simple, imagine trying to apply exact Bayesian inference to a model with thousands of parameters or complex data structures. In these cases, computing the exact posterior distribution might require solving high-dimensional integrals, which is often infeasible.

This is where approximate inference methods come into play. These methods provide ways to estimate the posterior distribution without needing to compute it exactly. In the upcoming posts, we’ll explore several key techniques for approximate inference, including:

• Sampling Methods: Techniques like rejection sampling, importance sampling, and Markov Chain Monte Carlo (MCMC).
• Variational Methods: Approximating the posterior with simpler distributions.
• Latent Variable Models: Inference in models with hidden variables.

These methods are not just theoretical tools; they are essential for modern machine learning and data science applications, from probabilistic models to deep learning.

Conclusion:

Bayesian inference is a foundational concept in statistics and machine learning, allowing us to update our beliefs as new data arrives. However, the complexity of real-world problems often necessitates approximate methods. In this blog series, we’ll embark on a journey to understand these methods, starting with basic sampling techniques and progressing to more advanced topics like MCMC and variational inference.

Stay tuned for the next post, where we’ll dive into the basics of sampling from a distribution!