Generative AI is AI that creates new content — text, images, code, audio, video — by learning the patterns of its training data, rather than only classifying or predicting a label. Examples: ChatGPT and Claude (text), DALL·E and Midjourney (images).
How LLMs work
- Tokenization: text is split into tokens (word pieces) and each becomes a numeric vector (embedding).
- Transformer architecture: layers of self-attention let every token weigh its relevance to every other token, capturing context and long-range meaning; positional encoding keeps word order.
- Pre-training: on huge text corpora the model learns to predict the next token, which teaches grammar, facts and reasoning patterns. Billions of parameters are adjusted by gradient descent.
- Fine-tuning and alignment: instruction tuning and RLHF (reinforcement learning from human feedback) make it helpful and safe.
- Generation (inference): given a prompt, it predicts a probability distribution for the next token, samples one, appends it and repeats.
Applications: chat assistants, content and report writing, code generation, translation, summarization, customer service, document analysis, education and tutoring.
Challenges
- Hallucination: confident but false answers.
- Bias inherited from training data.
- Privacy and security: leakage of sensitive data, prompt injection.
- Cost: heavy compute and energy for training and serving.
- Copyright, misuse (deepfakes, misinformation), and limited explainability.
— rhs