Explain techniques for extracting, choosing and engineering features from text data

Text is turned into features in stages. Tokenisation splits it into words; normalisation lowercases, removes stop words, and applies stemming or lemmatisation. A bag of words records which tokens occur, ignoring order, and a document term matrix has one row per document and one column per token. N-grams such as bigrams keep short sequences like "not profitable". Feature selection uses frequency measures: term frequency is a token's count in a sentence divided by the sentence's tokens, document frequency is the share of sentences containing it, and TF–IDF is TF × ln(1 ÷ DF). Feature engineering adds numbers, named entities and parts of speech. The usual slip is swapping rows and columns.