tfidf, am I understanding it right?

Posted by alskndalsnd on Stack Overflow See other posts from Stack Overflow or by alskndalsnd
Published on 2010-03-29T07:08:25Z Indexed on 2010/03/29 7:13 UTC
Read the original article Hit count: 368

Filed under:

information-retrieval

|

text-processing

|

language-agnostic

|

algorithm

Hey everyone, I am interested in doing some document clustering, and right now I am considering using TF-IDF for this.

If I am not wrong, TFIDF is particularly used for evaluating the relevance of a document given a query. If I do not have a particular query, how can I apply tfidf to clustering?

© Stack Overflow or respective owner

Related posts about information-retrieval

Suggestion needed to learn Machine Learning and Information Retrieval

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi! I want lo learn about Information Retrieval and Machine Learning. Which books do you recommend and in what order do you think is better to read them? The idea is to reach a good understanding of recommendation systems. Thanks! Jonathan >>> More
Fetching templates via API. Who provides this service?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I'm mainly a server side developer. I'm not a designer, even if I understand web layouts, grids, CSS, typography, valid markup, etc. and I'm able to do some graphic work too (almost). It just takes a lot of time and the result is not always beautiful. I know there are tons of website templates sites… >>> More
How do search engines handle hyphenated words?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I am not sure my title fully explains what I mean. I thought this might be an interesting question. If I had a set of keywords, broken with a dash or 2, will search engines consider the dashed split keyword as maybe a full keyword? Say I have a site that sort of breaks words down, like the dictionary… >>> More
entity set expansion python

as seen on Stack Overflow - Search for 'Stack Overflow'
Do you know of any existing implementation in any language (preferably python) of any entity set expansion algorithms, such that the one from Google sets ? ( http://labs.google.com/sets ) I couldn't find any library implementing such algorithms and I'd like to play with some of those to see how they… >>> More
Assistance with building an inverted-index

as seen on Stack Overflow - Search for 'Stack Overflow'
It's part of an information retrieval thing I'm doing for school. The plan is to create a hashmap of words using the the first two letters of the word as a key and any words with the two letters saved as a string value. So, hashmap["ba"] = "bad barley base" Once I'm done tokenizing a line I take… >>> More

Related posts about text-processing

Performing text processing on flatpage content to include handling of custom tag

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi. I'm using flatpages app in my project to manage some html content. That content will include images, so I've made a ContentImage model allowing user to upload images using admin panel. The user should then be able to include those images in content of the flatpages. He can of course do that by… >>> More
Fuzzy Regex, Text Processing, Lexical Analysis?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm not quite sure what terminology to search for, so my title is funky... Here is the workflow I've got: Semi-structured documents are scanned to file. The files are OCR'd to text. The text is parsed into Python objects The objects are serialized (to SQL, JSON, whatever) for use. The documents… >>> More
Getting started with character and text processing (encoding, regular expressions)

as seen on Stack Overflow - Search for 'Stack Overflow'
I'd like to learn foundations of encodings, characters and text. Understanding these is important for dealing with a large set of text whether that are log files or text source for building algorithms for collective intelligence. My current knowledge is pretty basic: something like "As long as I use… >>> More
Improve Efficiency for This Text Processing Code

as seen on Stack Overflow - Search for 'Stack Overflow'
I am writing a program that counts the number of words in a text file which is already in lowercase and separated by spaces. I want to use a dictionary and only count the word IF it's within the dictionary. The problem is the dictionary is quite large (~100,000 words) and each text document has also… >>> More
Tools for modelling data and workflows using structured text files

as seen on Programmers - Search for 'Programmers'
Consider a case when I want to try some idea of an application. But I want to avoid investing a lot of effort in coding UI/work flows/database schema etc before I see that it's going to be useful to me (as example of potential user). My idea is stay lightweight and put all the data in text files.… >>> More