How to improve K-means clustering with TF-IDF?

Question

Geovane Gomes 2024-10-7

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2158035-how-to-improve-k-means-clustering-with-tf-idf

评论： Christopher Creutzig 2024-10-22，5:57

采纳的回答： Sandeep Mishra

segmentos95Translated.xlsx

在 MATLAB Online 中打开

Hi all,

I’m currently working on a project where I need to classify company segments based on their activity descriptions.

I’ve implemented K-means clustering using TF-IDF for feature extraction from text data. However, the current clustering results aren’t entirely accurate, especially when it comes to grouping semantically similar segments (e.g., "cars" and "vehicles" are placed into separate clusters). Is this possible to optmise it, or use another approche rather than TF-IDF.

See cluster 13. More than 50% of the items were assigned to this cluster. I also tried using other distance parameters, but the results didn't improve.

Here is my code:

clear
close
% load and preprocess
d = readtable("segmentos95Translated.xlsx");
t = d.TRANSLATED;
for i = 1:height(t)
str = t{i};
splitStr = strsplit(str, 'EXCEPT');
t{i} = strtrim(splitStr{1});
end
for i = 1:height(t)
str = t{i};
splitStr = strsplit(str, 'WITHOUT PREDOMINANCE');
t{i} = strtrim(splitStr{1});
end
% tokenization
t = lower(t);
t = tokenizedDocument(t);
t = removeStopWords(t);
t = normalizeWords(t);
customStopWords = ["manufactur","activ",",","rental","(",")","*","exempt"...
"commerci","repres","agent","trade","product","retail","sale","waiv","special","wholesal"];
t = removeWords(t,customStopWords);
% bag of words and TF-IDF
bag = bagOfWords(t);
tfidfMatrix = tfidf(bag);
X = full(tfidfMatrix);
% kmeans
rng(1)
numClusters = 25; % about 10%
[idx, C, sumd, D] = kmeans(X, numClusters);
d.clusters = idx;
% display results
for i = 1:numClusters
fprintf('Cluster %d:\n', i);
disp(d.TRANSLATED(idx == i));
end
Cluster 1:
    {'MANUFACTURE OF PRESERVED FISH, CRUSTACES AND MOLLUSCS'}
    {'PRESERVATION OF FISH, CRUSTACES AND MOLLUSCS'         }
Cluster 2:
    {'WEAVING ARTIFICIAL AND SYNTHETIC FIBER YARN'}
    {'SPINNING OF ARTIFICIAL AND SYNTHETIC FIBERS'}
    {'PREPARATION AND SPINNING OF COTTON FIBERS'  }
Cluster 3:
    {'WHOLESALE TRADE SPECIALIZED IN OTHER INTERMEDIATE PRODUCTS NOT PREVIOUSLY SPECIFIED'                              }
    {'INTERMEDIATION AND AGENCY ACTIVITIES FOR SERVICES AND BUSINESS IN GENERAL, EXCEPT REAL ESTATE'                    }
    {'MANUFACTURE OF OTHER TEXTILE PRODUCTS NOT PREVIOUSLY SPECIFIED'                                                   }
    {'MANUFACTURE OF OTHER HOUSEHOLD APPLIANCES NOT PREVIOUSLY SPECIFIED, PARTS AND ACCESSORIES'                        }
    {'WHOLESALE TRADE SPECIALIZED IN OTHER FOOD PRODUCTS NOT PREVIOUSLY SPECIFIED'                                      }
    {'RETAIL SALE OF OTHER PRODUCTS NOT PREVIOUSLY SPECIFIED'                                                           }
    {'MANUFACTURE OF RUBBER ARTIFACTS NOT PREVIOUSLY SPECIFIED'                                                         }
    {'MANUFACTURE OF OTHER METAL PRODUCTS NOT PREVIOUSLY SPECIFIED'                                                     }
    {'METALLURGY OF OTHER NON-FERROUS METALS AND THEIR ALLOYS NOT PREVIOUSLY SPECIFIED'                                 }
    {'MANUFACTURE OF OTHER FOOD PRODUCTS NOT PREVIOUSLY SPECIFIED'                                                      }
    {'SPECIALIZED WHOLESALE OF CONSTRUCTION MATERIALS NOT PREVIOUSLY SPECIFIED'                                         }
    {'MANUFACTURE OF OTHER NON-METALLIC MINERAL PRODUCTS NOT PREVIOUSLY SPECIFIED'                                      }
    {'MANUFACTURE OF OTHER MACHINES AND GENERAL USE EQUIPMENT NOT PREVIOUSLY SPECIFIED, PARTS AND ACCESSORIES'          }
    {'MANUFACTURE OF MACHINES AND EQUIPMENT FOR SPECIFIC INDUSTRIAL USE NOT PREVIOUSLY SPECIFIED, PARTS AND ACCESSORIES'}
    {'WHOLESALE OF BEVERAGES NOT PREVIOUSLY SPECIFIED'                                                                  }
    {'WHOLESALE TRADE OF AGRICULTURAL RAW MATERIALS NOT PREVIOUSLY SPECIFIED'                                           }
    {'MANUFACTURE OF OTHER ELECTRICAL EQUIPMENT AND APPARATUS NOT PREVIOUSLY SPECIFIED'                                 }
Cluster 4:
    {'MANUFACTURE OF CARS, TRUCKS AND UTILITIES'                 }
    {'MANUFACTURE OF ENGINES FOR CARS, TRUCKS AND UTILITIES'     }
    {'WHOLESALE TRADE OF NEW AND USED CARS, TRUCKS AND UTILITIES'}
    {'RETAIL SALE OF NEW CARS, TRUCKS AND UTILITIES'             }
Cluster 5:
    {'WHOLESALE OF NEW PARTS AND ACCESSORIES FOR MOTOR VEHICLES'                                            }
    {'MANUFACTURE OF DIRECT AND ALTERNATING CURRENT GENERATORS, PARTS AND ACCESSORIES'                      }
    {'MANUFACTURE OF OTHER PARTS AND ACCESSORIES FOR MOTOR VEHICLES NOT PREVIOUSLY SPECIFIED'               }
    {'MANUFACTURE OF TRACTORS, PARTS AND ACCESSORIES, EXCEPT AGRICULTURAL'                                  }
    {'MANUFACTURE OF AGRICULTURAL TRACTORS, PARTS AND ACCESSORIES'                                          }
    {'MANUFACTURE OF TRANSFORMERS, INDUCTORS, CONVERTERS, SYNCHRONIZERS AND THE LIKE, PARTS AND ACCESSORIES'}
    {'MANUFACTURE OF PARTS AND ACCESSORIES FOR THE MOTOR VEHICLE MOTOR SYSTEM'                              }
    {'MANUFACTURE OF MACHINE TOOLS, PARTS AND ACCESSORIES'                                                  }
    {'MANUFACTURE OF PARTS AND ACCESSORIES FOR THE STEERING AND SUSPENSION SYSTEM OF MOTOR VEHICLES'        }
    {'MANUFACTURE OF PARTS AND ACCESSORIES FOR THE MOTOR VEHICLE BRAKE SYSTEM'                              }
    {'MANUFACTURE OF SEATS AND UPHOLSTERY FOR MOTOR VEHICLES'                                               }
    {'RETAIL OF NEW PARTS AND ACCESSORIES FOR MOTOR VEHICLES'                                               }
    {'WHOLESALE OF PARTS AND ACCESSORIES FOR MOTORCYCLES AND SCOOTERS'                                      }
    {'MANUFACTURE OF COMPRESSORS FOR INDUSTRIAL USE, PARTS AND ACCESSORIES'                                 }
    {'MANUFACTURE OF PARTS AND ACCESSORIES FOR MOTOR VEHICLE DRIVE AND TRANSMISSION SYSTEMS'                }
    {'MANUFACTURE OF ELECTRIC MOTORS, PARTS AND ACCESSORIES'                                                }
    {'MANUFACTURE OF NON-MOTORIZED BICYCLES AND TRICYCLES, PARTS AND ACCESSORIES'                           }
Cluster 6:
    {'WHOLESALE TRADE OF GENERAL GOODS, WITHOUT PREDOMINANCE OF FOOD OR AGRICULTURAL INPUTS'}
    {'WHOLESALE TRADE OF GENERAL GOODS, WITH A PREDOMINANCE OF FOOD PRODUCTS'               }
    {'RETAIL TRADE OF GENERAL GOODS, WITH A PREDOMINANCE OF FOOD PRODUCTS - HYPERMARKETS'   }
    {'WHOLESALE TRADE OF GENERAL GOODS, WITH A PREDOMINANCE OF AGRICULTURAL INPUTS'         }
    {'RETAIL TRADE OF GENERAL GOODS, WITH A PREDOMINANCE OF FOOD PRODUCTS - SUPERMARKETS'   }
Cluster 7:
    {'WHOLESALE OF MACHINES AND EQUIPMENT FOR INDUSTRIAL USE PARTS AND PARTS'                                                      }
    {'WHOLESALE OF OTHER MACHINES AND EQUIPMENT NOT PREVIOUSLY SPECIFIED PARTS AND PARTS'                                          }
    {'WHOLESALE OF MACHINERY AND EQUIPMENT FOR COMMERCIAL USE PARTS AND PARTS'                                                     }
    {'WHOLESALE OF MACHINES, APPARATUS AND EQUIPMENT FOR AGRICULTURAL USE; PARTS AND PARTS'                                        }
    {'MANUFACTURE OF MACHINES AND EQUIPMENT FOR OIL PROSPECTION AND EXTRACTION, PARTS AND ACCESSORIES'                             }
    {'MANUFACTURE OF MACHINERY AND EQUIPMENT FOR AGRICULTURE AND LIVESTOCK, PARTS AND ACCESSORIES, EXCEPT FOR IRRIGATION'          }
    {'MANUFACTURE OF MACHINES, EQUIPMENT AND APPARATUS FOR TRANSPORTING AND LIFTING LOADS, PARTS AND ACCESSORIES'                  }
    {'WHOLESALE OF MACHINES, APPARATUS AND EQUIPMENT FOR DENTAL-MEDICAL-HOSPITAL USE; PARTS AND PARTS'                             }
    {'MANUFACTURE OF MACHINES, EQUIPMENT AND APPARATUS FOR TRANSPORTING AND LIFTING PEOPLE, PARTS AND ACCESSORIES'                 }
    {'MANUFACTURE OF REFRIGERATION AND VENTILATION MACHINES AND APPARATUS FOR INDUSTRIAL AND COMMERCIAL USE, PARTS AND ACCESSORIES'}
Cluster 8:
    {'MANUFACTURE OF ALLOPATHIC MEDICINES FOR HUMAN USE'   }
    {'WHOLESALE TRADE OF MEDICINES AND DRUGS FOR HUMAN USE'}
Cluster 9:
    {'LOGISTICS ORGANIZATION OF CARGO TRANSPORTATION'                                                                  }
    {'ROAD CARGO TRANSPORTATION, EXCEPT DANGEROUS PRODUCTS AND MOVEMENTS, INTERMUNICIPAL, INTERSTATE AND INTERNATIONAL'}
Cluster 10:
    {'MANUFACTURE OF PLASTIC MATERIAL FLAT AND TUBULAR LAMINATES'}
    {'PRODUCTION OF ALUMINUM LAMINATES'                          }
    {'PRODUCTION OF ALUMINUM AND ITS ALLOYS IN PRIMARY FORMS'    }
    {'RECOVERY OF ALUMINUM SCRAP'                                }
Cluster 11:
    {'MANUFACTURE OF MACHINES AND EQUIPMENT FOR EARTHMOVING, PAVING AND CONSTRUCTION, PARTS AND ACCESSORIES, EXCEPT TRACTORS'}
    {'WHOLESALE OF MACHINES, EQUIPMENT FOR EARTHMOVING, MINING AND CONSTRUCTION; PARTS AND PARTS'                            }
Cluster 12:
    {'MANUFACTURING OF OTHER ARTIFACTS AND PRODUCTS FROM CONCRETE, CEMENT, FIBER CEMENT, PLASTER AND SIMILAR MATERIALS'}
Cluster 13:
    {'TRUCK AND BUS MANUFACTURING'                                                                  }
    {'WHOLESALE TRADE OF RESINS AND ELASTOMERS'                                                     }
    {'WHOLESALE TRADE OF TIRES AND TUMBLES'                                                         }
    {'WHOLESALE TRADE OF PROCESSED YARN AND FIBERS'                                                 }
    {'WHOLESALE TRADE OF FOOD PRODUCTS IN GENERAL'                                                  }
    {'WHOLESALE TRADE OF AGRICULTURAL PESTICIDES, FERTILIZERS, FERTILIZERS AND SOIL IMPROVEMENTS'   }
    {'CONSULTING ACTIVITIES IN BUSINESS MANAGEMENT, EXCEPT SPECIFIC TECHNICAL CONSULTANCY'          }
    {'WHOLESALE TRADE OF HARDWARE AND TOOLS'                                                        }
    {'FABRIC WHOLESALE TRADE'                                                                       }
    {'MANUFACTURE OF TIRES AND TUMBLES'                                                             }
    {'WHOLESALE OF CLOTHING ARTICLES AND ACCESSORIES, EXCEPT PROFESSIONAL AND SECURITY'             }
    {'WHOLESALE OF ELECTRONIC COMPONENTS AND TELEPHONE AND COMMUNICATION EQUIPMENT'                 }
    {'MANUFACTURE OF AGRICULTURAL PROTECTIVES'                                                      }
    {'MANUFACTURE OF ADDITIVES FOR INDUSTRIAL USE'                                                  }
    {'WHOLESALE TRADE OF BEEF AND PORK MEAT AND DERIVATIVES'                                        }
    {'MANUFACTURE OF THERMOPLASTIC RESINS'                                                          }
    {'MARINE AGENCY ACTIVITIES'                                                                     }
    {'MANUFACTURE OF FERTILIZERS AND FERTILIZERS, EXCEPT ORGANO-MINERALS'                           }
    {'WHOLESALE TRADE OF STEEL AND METALLURGICAL PRODUCTS, EXCEPT FOR CONSTRUCTION'                 }
    {'MANUFACTURE OF COSMETICS, PERFUMERY AND PERSONAL HYGIENE PRODUCTS'                            }
    {'ELECTRIC POWER GENERATION'                                                                    }
    {'WHOLESALE TRADE OF COMPUTER EQUIPMENT'                                                        }
    {'MANUFACTURE OF ANIMAL FOOD'                                                                   }
    {'WHOLESALE PACKAGING TRADE'                                                                    }
    {'CARGO AGENCY, EXCEPT FOR SEA TRANSPORTATION'                                                  }
    {'WHOLESALE OF OFFICE AND STATIONERY SUPPLIES'                                                  }
    {'IRON ORE EXTRACTION'                                                                          }
    {'MANUFACTURE OF SPECIAL FABRICS, INCLUDING ARTIFACTS'                                          }
    {'MANUFACTURE OF PAINTS, VARNISHES, ENAMELS AND LACQUERS'                                       }
    {'MANUFACTURE OF PAPER PACKAGING'                                                               }
    {'SPECIALIZED RETAIL SALE OF COMPUTER EQUIPMENT AND SUPPLIES'                                   }
    {'ACTIVITIES TO SUPPORT OIL AND NATURAL GAS EXTRACTION'                                         }
    {'WHOLESALE TRADE OF MINERAL EXTRACTION PRODUCTS, EXCEPT FUELS'                                 }
    {'MANUFACTURE OF KNITTED FABRICS'                                                               }
    {'MANUFACTURE OF INSULATED ELECTRICAL WIRES, CABLES AND CONDUCTORS'                             }
    {'RETAIL SALE OF TIRES AND TUMBERS'                                                             }
    {'MANUFACTURE OF THERMOSET RESINS'                                                              }
    {'MANUFACTURE OF PRODUCTS DERIVED FROM COCOA AND CHOCOLATES'                                    }
    {'WHOLESALE OF COSMETICS AND PERFUMERY PRODUCTS'                                                }
    {'MANUFACTURING OF GARMENTS, EXCEPT UNDERWEAR AND CUSTOM MADE CLOTHING'                         }
    {'MANUFACTURE OF PERIPHERALS FOR COMPUTER EQUIPMENT'                                            }
    {'WHOLESALE TRADE OF PAPER AND RAW CARDBOARD'                                                   }
    {'WHOLESALE OF PET FOOD'                                                                        }
    {'WHOLESALE TRADE OF MOTORCYCLES AND SCOOTERS'                                                  }
    {'MANUFACTURE OF TELEPHONE DEVICES AND OTHER COMMUNICATION EQUIPMENT, PARTS AND ACCESSORIES'    }
    {'PORT OPERATOR’S ACTIVITIES'                                                                   }
    {'PRODUCTION OF FLAT ROLLED CARBON STEEL PLATES, COATED OR NOT'                                 }
    {'WHOLESALE OF CHANDELIERS, LIGHTS AND LAMPSHADES'                                              }
    {'MANUFACTURE OF DISPOSABLE DIAPERS'                                                            }
    {'CONSTRUCTION OF LARGE VESSELS'                                                                }
    {'MANUFACTURE OF ADHESIVES AND SEALANTS'                                                        }
    {'MANUFACTURE OF CABINS, BODYWORKS AND TRAILERS FOR TRUCKS'                                     }
    {'DISPATCH OFFICE'                                                                              }
    {'WHOLESALE OF GLASS, MIRRORS AND STAINED GLASS WINDOWS'                                        }
    {'DEPARTMENT STORES OR MAGAZINES, EXCEPT DUTY FREE STORES'                                      }
    {'MANUFACTURE OF AUDIO AND VIDEO RECEIVING, REPRODUCTION, RECORDING AND AMPLIFICATION APPARATUS'}
    {'MANUFACTURE OF MARGARINE AND OTHER VEGETABLE FATS AND NON-EDIBLE ANIMAL OILS'                 }
    {'RETAIL SALE OF HARDWARE AND TOOLS'                                                            }
    {'PRODUCTION OF LONG ROLLED STEEL, EXCEPT TUBES'                                                }
    {'WHOLESALE OF BICYCLES, TRICYCLES AND OTHER RECREATIONAL VEHICLES'                             }
    {'CASTING OF NON-FERROUS METALS AND THEIR ALLOYS'                                               }
    {'MANUFACTURE OF COMPUTER EQUIPMENT'                                                            }
    {'MANUFACTURE OF COMMUNICATION TRANSMITTING EQUIPMENT, PARTS AND ACCESSORIES'                   }
    {'ENGINEERING SERVICES'                                                                         }
    {'IRON AND STEEL CASTING'                                                                       }
    {'MANUFACTURE OF CLEANING AND POLISHING PRODUCTS'                                               }
    {'MANUFACTURE OF METALLIC PACKAGING'                                                            }
    {'SALES REPRESENTATIVES AND TRADE AGENTS OF FOOD, BEVERAGE AND TOCCO PRODUCTS'                  }
    {'MANUFACTURE OF FERTILIZER INTERMEDIATES'                                                      }
    {'PRODUCTION OF SEAMLESS STEEL PIPE'                                                            }
    {'DAIRY PRODUCTION'                                                                             }
    {'MANUFACTURE OF OTHER PETROLEUM PRODUCTS, EXCEPT REFINING PRODUCTS'                            }
    {'ZINC PRODUCTION IN PRIMARY FORMS'                                                             }
    {'WHOLESALE OF HYGIENE, CLEANING AND HOME MAINTENANCE PRODUCTS'                                 }
    {'PAPER MANUFACTURING'                                                                          }
    {'MANUFACTURE OF AIR CONDITIONING APPLIANCES AND EQUIPMENT FOR NON-INDUSTRIAL USE'              }
    {'MANUFACTURE OF ELECTRONIC COMPONENTS'                                                         }
    {'WHOLESALE FOOTWEAR'                                                                           }
    {'GENERAL WAREHOUSES - WARRANT ISSUANCE'                                                        }
    {'TOOL MANUFACTURING'                                                                           }
    {'MANUFACTURE OF CUTLERY ARTICLES'                                                              }
    {'OIL AND NATURAL GAS EXTRACTION'                                                               }
    {'MANUFACTURE OF LUMINAIRES AND OTHER LIGHTING EQUIPMENT'                                       }
    {'MANUFACTURE OF HYDRAULIC AND PNEUMATIC EQUIPMENT, PARTS AND ACCESSORIES, EXCEPT VALVES'       }
    {'MANUFACTURE OF FLAT AND SAFETY GLASS'                                                         }
    {'MANUFACTURING APPARATUS AND EQUIPMENT FOR DISTRIBUTION AND CONTROL OF ELECTRICAL ENERGY'      }
    {'PRODUCTION OF STEEL SEMI-FINISHED FINISHES'                                                   }
    {'PRODUCTION OF FLAT ROLLED SPECIAL STEEL'                                                      }
    {'MANUFACTURE OF TRUCK AND BUS ENGINES'                                                         }
    {'MANUFACTURE OF WATERPROOFING AGENTS, SOLVENTS AND RELATED PRODUCTS'                           }
    {'MANUFACTURE OF BASIC PETROCHEMICAL PRODUCTS'                                                  }
    {'CIGARETTE MANUFACTURING'                                                                      }
    {'MANUFACTURE OF GLASS PACKAGING'                                                               }
    {'FERROLALLOYS PRODUCTION'                                                                      }
    {'MANUFACTURE OF PULP AND OTHER PULP FOR PAPER MANUFACTURING'                                   }
    {'MANUFACTURE OF TILES AND FLOORS'                                                              }
    {'MANUFACTURE OF ORGANO-MINERAL FERTILIZERS AND FERTILIZERS'                                    }
    {'WHOLESALE OF COMPUTER SUPPLIES'                                                               }
    {'MANUFACTURE OF REFRACTORY CERAMIC PRODUCTS'                                                   }
    {'MANUFACTURE OF LEATHER FOOTWEAR'                                                              }
    {'MANUFACTURE OF BRUSHES, BRUSHES AND BROOMS'                                                   }
    {'PRODUCTION OF STEEL PIPE WITH SEAM'                                                           }
    {'DEPOSITS OF GOODS FOR THIRD PARTIES, EXCEPT GENERAL WAREHOUSES AND FURNITURE STORES'          }
    {'MANUFACTURE OF PRESERVED VEGETABLES AND OTHER VEGETABLES, EXCEPT PALMIT'                      }
    {'FURNITURE RETAIL TRADE'                                                                       }
    {'INDUSTRIAL SMOKE PROCESSING'                                                                  }
    {'BOOK EDITING'                                                                                 }
    {'RETAIL OF SPORTING GOODS'                                                                     }
    {'WHOLESALE TRADE OF LUBRICANTS'                                                                }
    {'WHOLESALE TRADE OF PAINTS, VARNISHES AND SIMILAR'                                             }
    {'MANUFACTURE OF SPICES, SAUCES, SEASONINGS AND CONDIMENTS'                                     }
    {'WHOLESALE OF CLOTHING AND ACCESSORIES FOR PROFESSIONAL AND WORK SAFETY USE'                   }
    {'MULTIPLE BANKS, WITH COMMERCIAL CARD'                                                         }
    {'MANUFACTURE OF FOOD POWDER'                                                                   }
    {'TRAVEL AGENCIES'                                                                              }
    {'WHOLESALE TRADE OF PERSONAL HYGIENE PRODUCTS'                                                 }
    {'WHOLESALE OF PROSTHETICS AND ORTHOPEDIC ARTICLES'                                             }
    {'CUSTOMS CLEARANCE ACTIVITIES'                                                                 }
    {'AIRCRAFT MANUFACTURING'                                                                       }
    {'MANUFACTURING OF CLOTHING ARTICLES, PRODUCED IN KNITTING AND KNITTING, EXCEPT SOCKS'          }
    {'WHOLESALE TRADE OF PROCESSED CEREALS AND LEGUMES'                                             }
    {'MANUFACTURE OF MEDICINES FOR VETERINARY USE'                                                  }
Cluster 14:
    {'WHOLESALE OF OTHER EQUIPMENT AND ITEMS FOR PERSONAL AND HOUSEHOLD USE NOT PREVIOUSLY SPECIFIED'               }
    {'WHOLESALE OF ELECTRONIC DEVICES FOR PERSONAL AND HOUSEHOLD USE'                                               }
    {'WHOLESALE OF ELECTRICAL EQUIPMENT FOR PERSONAL AND HOUSEHOLD USE'                                             }
    {'WHOLESALE TRADE OF HARDWARE ITEMS'                                                                            }
    {'MANUFACTURE OF STOVES, REFRIGERATORS AND WASHING AND DRYING MACHINES FOR HOUSEHOLD USE, PARTS AND ACCESSORIES'}
    {'MANUFACTURE OF TEXTILE ARTIFACTS FOR HOUSEHOLD USE'                                                           }
    {'WHOLESALE OF BAGS, SUITCASES AND TRAVEL ITEMS'                                                                }
    {'RETAIL OF CLOTHING ITEMS AND ACCESSORIES'                                                                     }
    {'WHOLESALE OF BEDDING, TABLE AND BATH ITEMS'                                                                   }
    {'WHOLESALE OF FURNITURE AND QUILT ITEMS'                                                                       }
    {'WHOLESALE OF TAPESTRY ITEMS BLINDS AND CURTAINS'                                                              }
    {'SPECIALIZED RETAIL SALE OF HOUSEHOLD APPLIANCES AND AUDIO AND VIDEO EQUIPMENT'                                }
    {'MANUFACTURE OF METAL ITEMS FOR HOUSEHOLD AND PERSONAL USE'                                                    }
    {'MANUFACTURE OF GLASS ITEMS'                                                                                   }
    {'RETAIL OF OTHER HOUSEHOLD ITEMS NOT PREVIOUSLY SPECIFIED'                                                     }
    {'MANUFACTURE OF TRAVEL ITEMS, BAGS AND SIMILAR ITEMS FROM ANY MATERIAL'                                        }
Cluster 15:
    {'COMMERCIAL REPRESENTATIVES AND AGENTS IN THE TRADE OF GENERAL, NON-SPECIALIZED GOODS'}
Cluster 16:
    {'MANUFACTURE OF CORRUGATED CARDBOARD SHEET AND PACKAGING'}
    {'MANUFACTURE OF CARDBOARD AND PAPER CARDBOARD PACKAGING' }
Cluster 17:
    {'MANUFACTURING OF FURNITURE WITH A PREDOMINANCE OF METAL'                       }
    {'MANUFACTURE OF FURNITURE WITH PREDOMINANCE OF WOOD'                            }
    {'MANUFACTURE OF LAMINATED WOOD AND PLYWOOD SHEET, PRESSED AND AGGLOMERATED WOOD'}
    {'WHOLESALE TRADE OF WOOD AND DERIVATIVE PRODUCTS'                               }
Cluster 18:
    {'MANUFACTURE OF PAPER, CARDBOARD, CARDBOARD AND CORRUGATED CARDBOARD PRODUCTS FOR COMMERCIAL AND OFFICE USE'           }
    {'MANUFACTURE OF CELLULOUS PULP, PAPER, CARDBOARD, CARDBOARD AND CORRUGATED CARDBOARD PRODUCTS NOT PREVIOUSLY SPECIFIED'}
Cluster 19:
    {'WHOLESALE OF FRUITS, VEGETABLES, ROOTS, TUBERS, VEGETABLES AND FRESH VEGETABLES'}
Cluster 20:
    {'TANNING AND OTHER LEATHER PREPARATIONS'}
Cluster 21:
    {'WHOLESALE TRADE OF MARBLE AND GRANITE'}
Cluster 22:
    {'MANUFACTURE OF PLASTIC MATERIAL PACKAGING'                                                      }
    {'WHOLESALE OF INSTRUMENTS AND MATERIALS FOR MEDICAL, SURGICAL, HOSPITAL AND LABORATORY USE'      }
    {'WHOLESALE TRADE OF ELECTRICAL MATERIALS'                                                        }
    {'MANUFACTURE OF PLASTIC MATERIAL ARTIFACTS FOR PERSONAL AND HOUSEHOLD USE'                       }
    {'MANUFACTURE OF PLASTIC MATERIAL ARTIFACTS FOR INDUSTRIAL USES'                                  }
    {'MANUFACTURE OF PLASTIC MATERIAL ARTIFACTS FOR OTHER USES NOT PREVIOUSLY SPECIFIED'              }
    {'WHOLESALE TRADE OF CONSTRUCTION MATERIALS IN GENERAL'                                           }
    {'MANUFACTURE OF ELECTRICAL AND ELECTRONIC MATERIALS FOR MOTOR VEHICLES, EXCEPT BATTERIES'        }
    {'MANUFACTURE OF PLASTIC MATERIAL ARTIFACTS FOR USE IN CONSTRUCTION, EXCEPT TUBES AND ACCESSORIES'}
    {'RECOVERY OF PLASTIC MATERIALS'                                                                  }
    {'PRINTING AND TEXTURIZING IN YARN, FABRIC, TEXTILE ARTIFACTS AND GARMENT PARTS'                  }
    {'RETAIL TRADE OF CONSTRUCTION MATERIALS IN GENERAL'                                              }
    {'MANUFACTURE OF MATERIALS FOR MEDICINE AND DENTISTRY'                                            }
    {'MANUFACTURE OF PIPE AND ACCESSORIES FROM PLASTIC MATERIAL FOR USE IN CONSTRUCTION'              }
    {'MANUFACTURE OF FOOTWEAR FROM SYNTHETIC MATERIAL'                                                }
    {'PRINTING MATERIAL FOR OTHER USES'                                                               }
Cluster 23:
    {'WHOLESALE OF OTHER CHEMICAL AND PETROCHEMICAL PRODUCTS NOT PREVIOUSLY SPECIFIED'}
    {'MANUFACTURE OF OTHER CHEMICAL PRODUCTS NOT PREVIOUSLY SPECIFIED'                }
    {'MANUFACTURE OF ORGANIC CHEMICALS NOT PREVIOUSLY SPECIFIED'                      }
    {'MANUFACTURE OF OTHER INORGANIC CHEMICAL PRODUCTS NOT PREVIOUSLY SPECIFIED'      }
Cluster 24:
    {'WHOLESALE TRADE OF BEER, BEER AND SOFT DRINKS'}
    {'MANUFACTURING OF BEERS AND BEERS'             }
Cluster 25:
    {'WHOLESALE TRADE OF ROUGH MINERAL FUEL'}
sortrows(groupcounts(d,"clusters"),"Percent","descend")
ans = 25x3 table
    clusters    GroupCount    Percent
    ________    __________    _______

       13          122         50.413
        3           17         7.0248
        5           17         7.0248
       14           16         6.6116
       22           16         6.6116
        7           10         4.1322
        6            5         2.0661
        4            4         1.6529
       10            4         1.6529
       17            4         1.6529
       23            4         1.6529
        2            3         1.2397
        1            2        0.82645
        8            2        0.82645
        9            2        0.82645
       11            2        0.82645

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Sandeep Mishra 2024-10-8

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2158035-how-to-improve-k-means-clustering-with-tf-idf#answer_1528615

在 MATLAB Online 中打开

Hi Geovane,

I can observe that you are trying to enhance the accuracy of your K-means clustering implementation.

The current implementation using 'TF-IDF' fails to capture the semantic meanings between words, which can lead to unrelated synonyms or related terms being treated as distinct.

To resolve this, you can use word embeddings such as 'fastText' which represent words in a continuous vector space, capturing semantic meanings.

You can leverage the 'Text Analytics Toolbox Model for fastText English 16 Billion Token Word Embedding' add-on in MATLAB to implement 'fastText' word embedding.

Consider the following implementation:

% Converting tokenized documents to cell array 
textData = arrayfun(@(doc) joinWords(doc), t, 'UniformOutput', false);
% Loading fastText word embedding
emb = fastTextWordEmbedding; 
% Converting text to embedding
X = zeros(numel(textData), emb.Dimension);
for i = 1:numel(textData)
    words = split(textData{i});
    validWords = words(isVocabularyWord(emb, words));
    if ~isempty(validWords)
        vecs = word2vec(emb, validWords);
        X(i, :) = mean(vecs, 1);
    end
end
[idx, C] = kmeans(X, numClusters);

Refer to the following MathWorks Documentation to learn more about ‘Text Analytics Toolbox Model for fastText English 16 Billion Token Word Embedding’ function in MATLAB: https://www.mathworks.com/matlabcentral/fileexchange/66229-text-analytics-toolbox-model-for-fasttext-english-16-billion-token-word-embedding

I hope this helps.

4 个评论
显示 2更早的评论隐藏 2更早的评论

Geovane Gomes 2024-10-16

Thanks, @Sandeep Mishra.

Christopher Creutzig 2024-10-22，5:57

Also worth checking out are documentEmbedding and, for a different workflow with “soft clustering,” fitlda.

请先登录，再进行评论。

How to improve K-means clustering with TF-IDF?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

4 个评论
显示 2更早的评论隐藏 2更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

How to improve K-means clustering with TF-IDF?

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

4 个评论 显示 2更早的评论隐藏 2更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

4 个评论
显示 2更早的评论隐藏 2更早的评论