Jump to Content
chapter 1, Probability Theory
Table of Contents
Chapter Contents
- 1.1 Probability Space
- 1.1.1. Events
- 1.1.2. Conditional Probability
- 1.2. Random Variables
- 1.2.1. Probability Distributions
- 1.2.2. Probability Densities
- 1.2.3. Functions of a Random Variable
- 1.3. Moments
- 1.3.1. Expectation and Variance
- 1.3.2. Moment-Generating Function
- 1.4. Important Probability Distributions
- 1.4.1. Binomial Distribution
- 1.4.2. Poisson Distribution
- 1.4.3. Normal Distribution
- 1.4.4. Gamma Distribution
- 1.4.5. Beta Distribution
- 1.4.6. Computer Simulation
- 1.5. Multivariate Distributions
- 1.5.1. Jointly Distributed Random Variables
- 1.5.2. Conditioning
- 1.5.3. Independence
- 1.6. Functions of Several Random Variables
- 1.6.1. Basic Arithmetic Functions of Two Random Variables
- 1.6.2. Distributions of Sums of Independent Random Variables
- 1.6.3. Joint Distributions of Output Random Variables
- 1.6.4. Expectation of a Function of Several Random Variables
- 1.6.5. Covariance
- 1.6.6. Multivariate Normal Distribution
- 1.7. Laws of Large Numbers
- 1.7.1. Weak Law of Large Numbers
- 1.7.2. Strong Law of Large Numbers
- 1.7.3. Central Limit Theorem
- 1.8. Parametric Estimation via Random Samples
- 1.8.1. Random-Sample Estimators
- 1.8.2. Sample Mean and Sample Variance
- 1.8.3. Minimum-Variance Unbiased Estimators
- 1.8.4. Method of Moments
- 1.8.5. Order Statistics
- 1.9. Maximum-Likelihood Estimation
- 1.9.1. Maximum-Likelihood Estimators
- 1.9.2. Additive Noise
- 1.9.3. Minimum Noise
- 1.10 Entropy
- 1.10.1. Uncertainty
- 1.10.2. Information
- 1.10.3. Entropy of a Random Vector
- 1.11 Source Coding
- 1.11.1. Prefix Codes
- 1.11.2. Optimal Coding
- Exercises for Chapter 1
Excerpt
1.1. Probability Space
Probability theory is concerned with measurements of random phenomena and the properties of such measurements. This opening section discusses the formulation of event structures, the axioms that need to be satisfied by a measurement to be a valid probability measure, and the basic set-theoretic properties of events and probability measures.
1.1.1. Events
At the outset we posit a set S, called the sample space, containing the possible experimental outcomes of interest. Mathematically, we simply postulate the existence of a set S to serve as a universe of discourse. Practically, all statements concerning the experiment must be framed in terms of elements in S and therefore S must be constrained relative to the experiment. Every physical outcome of the experiment should refer to a unique element of S. In effect, this practical constraint embodies two requirements: every physical outcome of the experiment must refer to some element in S and each physical outcome must refer to only one element in S. Elements of S are called outcomes.
Probability theory pertains to measures applied to subsets of a sample space. For mathematical reasons, subsets of interest must satisfy certain conditions. A collection E of subsets of a sample space S is called a σ-algebra if three conditions are satisfied:
(S1) S∈E.
(S2) If E∈E, then Ec∈E, where Ec is the complement of E.
(S3) If the countable (possibly finite) collection E1,E2,…∈E, then the union E1∪E2∪…∈E.
Subsets of S that are elements of E are called events. If S is finite, then we usually take the set of all subsets of S to be the σ-algebra of events. However, when S is infinite, the problem is more delicate. An in-depth discussion of σ-algebras properly belongs to a course on measure theory and here we will restrict ourselves to a few basic points.
Since S is an event and the complement of an event is an event, the null set Ø is an event.
©1999 Society of Photo-Optical Instrumentation Engineers
