Explain how text-based data is prepared, wrangled and explored for use in financial forecasting

Text preparation starts with cleansing: removing HTML tags, most punctuation, and extra white space, and replacing numbers with a placeholder token such as /number/. Preprocessing then tokenises and normalises: lowercasing, removing stop words, and stemming (cutting to a root such as "downgrad") or lemmatisation (mapping to a dictionary word). Exploration studies token frequencies and may use a word cloud. Tokens that appear in nearly every document carry little distinguishing information, and very rare tokens add sparse noise, so both are often removed. Chi-square tests and mutual information rank the remaining tokens. Watch that normalisation does not destroy meaning, for example by removing "not" before sentiment analysis.