Obtaining Diagnostic Information from Linking¶

Details¶

1. Diagnostics typically slow down linking by a few percent and consume significant additional memory, so they are turned off by default.
2. If you already ran linking with diagnostics turned on, but no longer want the extra columns cluttering up your results, you can use the strip_diagnostics() function to automatically remove all columns whose names start with “diag_”.
3. The set of diagnostic of columns, and the order in which they appear, is not fixed. Each column is added only if that diagnostic datum was recorded. For example, below we will encounter the diag_remembered column that gives information about the memory feature. If you did not specify the memory option for linking, this column will never be present in your results.
4. For performance reasons, link_df() ordinarily does not make a copy of its input DataFrame, instead modifying it in-place. For practical reasons, however, with diagnostics enabled link_df() must work on a copy. Depending on how it was written, your program may therefore behave differently when diagnostics are turned on! If you suspect a problem, you can instruct link_df() to behave more consistently by supplying copy_features=True.

diag_search_range¶

trackpy didn’t record every kind of diagnostic data for every feature. We do notice, however, that there is a diag_search_range for everything past the first frame. This column is the value of search_range that was used to find a particle and link it. Since the particles in the first frame did not need to be found, diag_search_range is null there. But for the subsequent frames, it is the same as the search_range we told link_df() to use, because those particles were successfully linked.

diag_search_range would be a nearly pointless column except that it reveals the workings of adaptive search when that feature is turned on. See the tutorial on adaptive search for a more meaningful look at diag_search_range.

diag_remembered¶

When a particle seems to disappear, memory allows its “ghost” to stick around for a few more frames in case that particle reappears. In that event, the diag_remembered column records the number of frames for which the particle was missing. Let’s look in the 4th frame to see whether this actually happened. Again, we show just the first 5 entires and transpose to fit on the page:

remembered_frame3 = t[(t.frame == 3) & (t.diag_remembered > 0)]

print(len(remembered_frame3), 'remembered particles in frame number 3.')
remembered_frame3.head().T

13 remembered particles in frame number 3.

One of these particles has a diag_remembered value of 2, which means that it was present in the first frame, went missing for 2 frames, and then seemingly reappeared in the fourth frame.

If you are not sure whether memory (or a specific value of the memory option) is appropriate for your movie, diag_remembered shows you which trajectories and frames to spot-check.

diag_subnet¶

As you may have also learned in other trackpy tutorials (including the one devoted to the topic), how easily subnetworks arise, and how they are resolved, can be crucial to the feasibility and correctness of linking. trackpy records diagnostic data whenever it encounters a subnetwork. (If there were no subnetworks at all, the columns discussed here will not be present.) As with memory, these data can help you focus on potential trouble spots.

In our example data, we can see what subnetworks were encountered when linking frames 0 and 1:

subnet_particles = t[(t.frame == 1) & ~(t.diag_subnet.isnull())]

print(len(subnet_particles), 'particles involved in subnets between frames 0 and 1.')
subnet_particles.head().T

87 particles involved in subnets between frames 0 and 1.

The subnet columns are as follows:

``diag_subnet`` sequentially numbers each subnet as it is encountered. This allows you to count subnets, and identify particles that are in the same subnet. Note that the numbering is sequential over the entire movie. Therefore, to count the subnets in a particular frame, use

len(subnet_particles.diag_subnet.unique())

45

We can also take this opportunity to show these subnet particles. This would be the starting point for more in-depth spot checks of how subnets were solved.

tp.annotate(subnet_particles, frames[1], plot_style={'markersize': 10});

<IPython.core.display.Javascript object>

Like diag_subnet, the columns diag_subnet_size and diag_subnet_iterations also pertain to the entire subnet; all members of a subnet have the same values. It would be nice to consolidate these data into a shorter table of the subnets and their properties. To do this, we can use the powerful grouping and aggregation features of Pandas:

subnets = subnet_particles.groupby('diag_subnet')[['diag_subnet_size', 'diag_subnet_iterations']].first()

# Count how many particles in the *present* frame are in each subnet,
# and include it in the results.
subnet_current_sizes = subnet_particles.diag_subnet.value_counts()
subnet_current_sizes.name = 'current_size'
subnets = subnets.join(subnet_current_sizes)

subnets.head()

Each subnet involves a set of trajectories in the past (generally the previous frame), and a set of candidate particles in the present (the frame in which the diagnostic info was recorded). ``diag_subnet_size`` gives the number of particles in the past, and in this example, we computed the current_size column to see the number in the present. You can see that some subnets necessitated the ending of a trajectory (e.g. 3 particles to 2), while others required creation (e.g. 1 to 2).

Finally, the ``diag_subnet_iterations`` column counts the number of possible solutions that were explored by the subnet-solving algorithm before it decided which was best. It is therefore roughly proportional to the time spent solving each subnet. These data are recorded by the numba linker only (link_strategy='numba'), which is used by default if numba is installed. Almost all of the subnets we’re examining are small and were solved quickly:

plt.figure()
plt.hist(subnets.diag_subnet_iterations)
plt.xlabel('Time to solve subnet (arbitrary units)')
plt.ylabel('Number of occurrences');

<IPython.core.display.Javascript object>

However, a subnet of \(\sim\) 25 particles can easily take \(10^9\) iterations or more to solve.