BAMI: Sparse Distributed Representations

Created: 2021-01-12T01:23:47-06:00

• Hot bit: a bit which is 1.
• Bit array: an array of bits, zeroes or ones.
• Sparsity: if only some small section of this array may be set at once this array can be called sparse.
• SDR: Sparse Distributed Representation
• Individual neurons are not important to the cortex but the behavior of populations are (sidenote: could be related to neurons falling apart and dying, being replaced, constantly.)
• Neurons predicting multiple outcomes for a given scenario; when an input does not match any of the predictions one might interpret it to mean something is amiss.
• Brain memory is associative; it has to connect one thing to the next.

Sparsity

• No matter how large an SDR is only the amount of bits which can be set at once matters for storage.
• An SDR can be stored as a list of indices for the hot bits.
• A size of 2048 with only 40 hot bits is "typical."

Union

• Of two bit arrays: a new bit array. for each bit: where the bit is set in both arrays, the bit is hot. otherwise zero.
• Of two SDRs: a new SDR which contains only the indices of bits which are bot in both SDRs.
• Union Set: when multiple SDRs are OR'd together to merge them to a single matcher. Allows testing against an entire battery of SDRs at once but only tells you if something matches the whole group. Has no risk of false negatives since bits cannot be turned on unless they belong to the set.

Matching

• Take the union of two SDRs.
• Count the number of hot bits in the union (hamming distance.)
• If overlap > theta then its a match.
• You have to figure out what theta is on your own :blobcatshrug:

Noise

• SDRs with a higher amount of hot bits are more resistant to noise (tested up to 50% noise.)

Subsampling

• A smaller, incomplete chunk of an SDR can be used instead of the full SDR. This exploits the noise tolerance of SDRs to use less storage space.

Math

• Vector cardinality: the number of hot bits.
• N Choose W: choose(n,w) is calculated by factorial(n) / (factorial(w) * (n-w))
• The number of possible reprsentations of a dense bit array is 2^b, where b is the number of bits in the array.
• The number of possible reprsentations of an SDR are choose(n,w), where n is the number of bits in the array and w is the maximum number which can be hot at one time.
• The probability two SDRs are completely identical is 1/choose(n,w)

TODO care about the math for false positives of mathing two SDRs partially

TODO care about the math for false positives on subsampling