HDBSCAN and BERTopic producing different clusters

[JWOsmond]

Hi Maarten, I hope you're well! This may be a silly question but I've been having a little bit of trouble understanding exactly what's going on under the hood with BERTopic regarding choosing cluster labels, and wondered if someone can clarify. Please let me know if this has been asked before.

I've found a counter-example to an assumption I was making (namely, if I embed my documents, reduce them, and cluster them with HDBSCAN, then the cluster labels I get from HDBSCAN will correspond to the topic labels I later get from running the full BERTopic algorithm with the same embedding, dimensionality reduction, and clustering models), by choosing a particular set of parameters on the documents I'm analysing:

import numpy as np
import pandas as pd
from bertopic import BERTopic
from hdbscan import HDBSCAN
from sentence_transformers import SentenceTransformer
from sklearn.decomposition import PCA

docs = pd.read_csv("docs.csv")["0"].tolist()

# Embed
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embedding_model.encode(docs)

# Reduce
reducer = PCA(n_components=13, random_state=42)
reduced_embeddings = reducer.fit_transform(embeddings)

# Cluster
clusterer = HDBSCAN(
    min_cluster_size=13,
    min_samples=8,
    metric="euclidean",
    cluster_selection_method="eom",
    prediction_data=True,
)
clustered = clusterer.fit(reduced_embeddings)
# Show unique cluster labels
np.unique(clustered.labels_)

array([-1])

As you can see, all my documents have been assigned the label -1 by HDBSCAN.

Now we plug the same combination of models into BERTopic:

topic_model = BERTopic(
    embedding_model=embedding_model,
    umap_model=reducer,
    hdbscan_model=clusterer,
    # nr_topics="auto",
)
topics, probs = topic_model.fit_transform(docs, embeddings)
topic_info = topic_model.get_topic_info()
topic_info[["Topic", "Count"]]

index	Topic	Count
0	-1	351
1	0	54
2	1	13

Now, despite choosing the same embedding, dimensionality reduction, and clustering models to pass to BERTopic(), I am getting a different set of topics from if I just ran those steps sequentially as above. Should I not be expecting HDBSCAN to produce the same labels as BERTopic in this way?

I hope this makes sense.

Thank you,
Jack

[MaartenGr]

iirc, it should be indeed the same output as long as both the reducer and clusterer will generate the same output every time. Do you get the same output each time when you run them without BERTopic (so your first example)??

[JWOsmond]

Thank you for looking at this. Yes this should all be totally reproducible - the reducer produces exactly the same reduced embeddings every time I run that cell, and every time the clusterer assigns every document to the -1 group.

BERTopic() and .get_topic_info() also produce exactly the same table every time - it's just not the same as when the reducer and clusterer are run in isolation.

Is there a way to see the reduced embeddings that are produced when BERTopic() is run?

[MaartenGr]

There's currently not an easy way to inspect reduced embedding. You can also pass an empty dimensionality reduction class to BERTopic and pass the reduced_embeddings instead of the embeddings.

[JWOsmond]

Thanks, that does appear to make the BERTopic() results the same as the first code chunk.

I'm attaching another similar example to this discussion that uses a public dataset - where running PCA and HDBSCAN separately produces different results to running BERTopic().

[JWOsmond]

Hopefully this is a slightly more reproducible example (if a bit contrived) using a public dataset:

import numpy as np
import pandas as pd
from bertopic import BERTopic
from hdbscan import HDBSCAN
from sentence_transformers import SentenceTransformer
from sklearn.datasets import fetch_20newsgroups
from sklearn.decomposition import PCA

docs = fetch_20newsgroups(
    subset='all', remove=('headers', 'footers', 'quotes')
)['data'][:2000]

# Embed
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embedding_model.encode(docs)

# Reduce
reducer = PCA(n_components=13, random_state=42)
reduced_embeddings = reducer.fit_transform(embeddings)

# Cluster
clusterer = HDBSCAN(
    min_cluster_size=60,
    min_samples=30,
    metric="euclidean",
    cluster_selection_method="eom",
    prediction_data=True,
)
clustered = clusterer.fit(reduced_embeddings)
# Show unique cluster labels
pd.DataFrame(
    np.unique(clustered.labels_, return_counts=True)
).T.rename(columns={0: "Topic", 1: "Count"})

index	Topic	Count
0	-1	1604
1	0	69
2	1	327

topic_model = BERTopic(
    embedding_model=embedding_model,
    umap_model=reducer,
    hdbscan_model=clusterer,
    calculate_probabilities=True,
    # nr_topics="auto",
)
topics, probs = topic_model.fit_transform(docs, embeddings)
topic_info = topic_model.get_topic_info()
topic_info[["Topic", "Count"]]

index	Topic	Count
0	-1	1622
1	0	309
2	1	69

class EmptyReducer:
    def __init__(self):
        pass
    def fit(self, embeddings):
        return embeddings
    def fit_transform(self, embeddings):
        return embeddings
    def transform(self, embeddings):
        return embeddings

topic_model = BERTopic(
    embedding_model=embedding_model,
    umap_model=EmptyReducer(),
    hdbscan_model=clusterer,
    calculate_probabilities=True,
    # nr_topics="auto",
)
topics, probs = topic_model.fit_transform(docs, reduced_embeddings)
topic_info = topic_model.get_topic_info()
topic_info[["Topic", "Count"]]

index	Topic	Count
0	-1	1604
1	0	327
2	1	69

All three of these outputs are slightly different (the only difference between the first and third is that clusters 0 and 1 get swapped around I suppose?). But the second one has different numbers of documents assigned to each cluster.

This is all using bertopic v0.17.3.

[JWOsmond]

Hi @MaartenGr , I found the difference between what BERTopic() does and what I was doing.

If instead of:

reducer = PCA(n_components=13, random_state=42)
reduced_embeddings = reducer.fit_transform(embeddings)

I do:

reducer = PCA(n_components=13, random_state=42)
reducer.fit(embeddings)
reduced_embeddings = reducer.transform(embeddings)

Then the outputs of both approaches will align, as this latter one matches the behaviour of BERTopic._reduce_dimensionality(). Was expecting self.umap_model.fit(embeddings) followed by umap_embeddings = self.umap_model.transform(embeddings) to be equivalent to self.umap_model.fit_transform(embeddings), but apparently this is not the case!