n-grams from text in PostgreSQL
- by harshsinghal
I am looking to create n-grams from text column in PostgreSQL. I currently split(on white-space) data(sentences) in a text column to an array.
select regexp_split_to_array(sentenceData,E'\s+') from tableName
Once I have this array, how do I go about:
Creating a loop to find n-grams, and write each to a row in another table
Using unnest I can obtain all the elements of all the arrays on separate rows, and maybe I can then think of a way to get n-grams from a single column, but I'd loose the sentence boundaries which I wise to preserve.
Sample SQL code for PostgreSQL to emulate the above scenario
create table tableName(sentenceData text);
INSERT INTO tableName(sentenceData) VALUES('This is a long sentence');
INSERT INTO tableName(sentenceData) VALUES('I am currently doing grammar, hitting this monster book btw!');
INSERT INTO tableName(sentenceData) VALUES('Just tonnes of grammar, problem is I bought it in TAIWAN, and so there aint any englihs, just chinese and japanese');
select regexp_split_to_array(sentenceData,E'\s+') from tableName;
select unnest(regexp_split_to_array(sentenceData,E'\s+')) from tableName;
