(archived) Probability and statistics notes

Also using this as $LATEX$ practice.

Grade Distribution

Exam Score: 50%

Usual Performance: 50%

• Homework
• Midterm + Quizzes
• Non-standard assessments
• In-class performance

Features

• Emphasis on comprehension and memorization

• Requires strong calculus foundation

• Fixed question types

Probability Section

Fundamentals of Probability Theory

Use $A,B,C$, etc., to denote events.

$\bar{A}$ denotes the complementary event of $A$ (complementarity is a sufficient condition for mutual exclusivity).

$AB=A\cap B$

$A-B=A\bar{B}=A-AB$

$\overline{A\cap B}=\bar{A}\cup \bar{B}$

$\overline{A\cup B}=\bar{A}\cap \bar{B}$

The probability of event $A$ occurring after event $B$ has occurred is $P(A|B)=\frac{P(AB)}{P(B)}$, where $P(B)>0$.

Law of Total Probability: Let $B_1,\dots ,B_m$ be a complete set of events, then $P(A)=\sum _{i=1}^{m}P(A|B_i)P(B_i)$.

Bayes' Theorem: Let $B_1,B_2,\dots ,B_m$ be a complete set of events, then
$P(B_i|A)=\frac{P(B_{i}A)}{P(A)}=\frac{P(B_i)P(A|B_i)}{\sum _{i=1}^{m}P(A|B_i)P(B_i)}$,
i.e., Bayes' Theorem is a combination of the multiplication rule and the law of total probability.

If $P(AB)=P(A)P(B)$, then $A$ and $B$ are independent. Similarly, $A$ and $\bar{B}$, $\bar{A}$ and $\bar{B}$, and $\bar{A}$ and $B$ are also independent.

Binomial distribution probability: $P(k)={\mathrm{C}}_n^k{p}^k(1-p{)}^{n-k}$

Multinomial distribution: Suppose a random experiment has $k$ possible outcomes $C_1\sim C_k$, with probabilities $p_1\sim p_k$. In $N$ independent trials, let the random variables $x_1\sim x_k$ denote the number of occurrences of $C_1\sim C_k$, respectively. The probability that $C_1$ occurs $x_1$ times, $C_2$ occurs $x_2$ times, ..., $C_k$ occurs $x_k$ times is

$\frac{N!}{x_1!x_2!\dots x_k!}{p}_1^{x_1}{p}_2^{x_2}\dots p_k^{x_k}$, where $\sum _{i=1}^k{x}_i=N$ and $\sum _{i=1}^k{p}_i=1$.

Random Variables and Their Distributions (Memorize)

A random variable $X(\omega )$ is a function, where $\omega$ is the sample space, and $X$ is a mapping from the sample space to the real numbers.

The distribution function $F(x)=P(X\le x)$ has a domain of $\mathbb{R}$, is non-decreasing (left limit is 0, right limit is 1), and is right-continuous.

The density function $f(x)$ satisfies $F(x)={\int }_{-\mathrm{\infty }}^{x}f(x)\mathrm{d}x$.

When $f(x)$ is continuous, it simplifies to $F^{\prime }(x)=f(x)$. Also, $f(x)\ge 0$ and ${\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)\mathrm{d}x=1$.

Probability problems can be transformed into integrals of the probability density function: $P(X\in G)={\int }_{G}f(x)\mathrm{d}x$.

Non-continuous Distributions

Geometric and hypergeometric distributions are omitted.

Binomial distribution: $X\sim \mathrm{B}(n,p)$, $P(X=k)={\mathrm{C}}_n^k{p}^k(1-p{)}^{n-k}$

The binomial distribution is unimodal. Since

$\frac{P\{X=k\}}{P\{X=k-1\}}=1+\frac{(n+1)p-k}{kq}$, the maximum value is attained when $k$ is closest to $(n+1)p$.

Poisson distribution: $X\sim \mathrm{P}(\lambda )$, $P(X=k)=\frac{{\lambda }^k}{k!}{e}^{-\lambda }$

The Poisson distribution corresponds to the normalization of the Taylor expansion terms of $e^x$.

Poisson's theorem: The binomial distribution converges to the Poisson distribution. Specifically, for $X\sim \mathrm{B}(n,p)$, when $n$ is sufficiently large (≥100), it can be approximated as $X\sim \mathrm{P}(\lambda )$, where $\lambda =np$.

Continuous Distributions

Uniform Distribution

$$f(x)=\{\begin{array}{}\frac{1}{b-a}& & x\in [a,b]\\ 0& & others\end{array},x\sim U(a,b)$$

Exponential Distribution

$$f(x)=\{\begin{array}{}\lambda e^{-\lambda x}& & x>0\\ 0& & others\end{array},x\sim e(\lambda )$$

Gamma Distribution

$$f(x)=\{\begin{array}{}\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{x}^{\alpha -1}{\mathrm{e}}^{-\beta x}& & x>0\\ 0& & others\end{array},x\sim \mathrm{\Gamma }(\alpha ,\beta )$$

$\mathrm{\Gamma }(x)={\int }_0^{\mathrm{\infty }}{t}^{x-1}{\mathrm{e}}^{-t}\mathrm{d}t$

Methods for Finding the Distribution of a Function of a Random Variable Y = f(X)

Taking the probability density function $f(x)$ as an example:

• Determine the range of values corresponding to $Y$.
• Express $X$ in terms of $Y$, and replace $F(X)$ with $F(Y)$.
• Differentiate $F(Y)$ to obtain $f(y)$.

For problems requiring case-by-case discussion:

• Determine the range of values corresponding to $Y$ and identify the breakpoints.
• Within each interval, solve the inequality to obtain $X\in g(Y)$, and compute $F(Y)={\int }_{x\in g(Y)}f(x)\mathrm{d}x$.
• Differentiate $F(Y)$ to obtain $f(y)$, and summarize the results.

Note: When integrating $+C$, pay attention to the continuity of the distribution function.

Multidimensional (Two-Dimensional) Random Variables and Their Distributions

Two-Dimensional Discrete Random Variable Distribution

It is essentially a table where the sum of all entries is 1. To find probabilities, simply add the probabilities at the corresponding positions.

Marginal Distributions:

• $p_{i\bullet }=\sum _j{p}_{ij}$

• $p_{\bullet j}=\sum _i{p}_{ij}$

Conditional Distribution Laws:

• $P(Y=y_j|X=x_i)=\frac{p_{ij}}{p_{i\bullet }}$

• $P(X=x_i|Y=y_j)=\frac{p_{ij}}{p_{\bullet j}}$

Two-Dimensional Continuous Random Variable Distribution

Equivalent to evaluating double integrals, with attention to the domain of integration.

Often, the property that the integral over the domain sums to 1 is used to find parameters, and then the double integral over a specified region is evaluated to compute probabilities.

Two-dimensional distribution function and density function:
$F(x,y)=P(X\le x,Y\le y)={\int }_{-\mathrm{\infty }}^x{\int }_{-\mathrm{\infty }}^{y}f(u,v)\mathrm{d}u\mathrm{d}v$

$$P(x_1<X\le x_2,y_1<Y\le y_2)=F(x_2,y_2)-F(x_1,y_2)-F(x_2,y_1)+F(x_1,y_1)$$

Marginal densities:

• $f_X(x)=f(x,+\mathrm{\infty })={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x,y)\mathrm{d}y$
• $f_Y(y)=f(+\mathrm{\infty },y)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x,y)\mathrm{d}x$

Conditional distribution functions:

• $F_{X|Y}(x|y)={\int }_{-\mathrm{\infty }}^x\frac{f(u,y)}{f_Y(y)}\mathrm{d}u$ (the other half is omitted for brevity)

• $f_{X|Y}(x|y)=\frac{f(x,y)}{f_Y(y)},f_{Y|X}(y|x)=\frac{f(x,y)}{f_X(x)}$

Common question types:

One density, two conditionals, two marginals

• Given one density, find the remaining four.
• Given one conditional and its corresponding marginal, find the remaining three.

Given the two-dimensional density function $f(x,y)$ and $Z=g(X,Y)$, find the density function $f_Z(z)$ of $Z$:

• Determine the effective region $R$ of $f(x,y)$.
• Compute $F_Z(z)={\iint }_{(x,y)\in R}f(x,y)\mathrm{d}x\mathrm{d}y$, paying attention to case analysis.
• Differentiate with respect to $z$ to obtain the density function $f_Z(z)$.

Farewell to convolution

If $Z=max(X,Y)$ and $X,Y$ are independent, then $F_Z(z)=F_X(z)F_Y(z)$.

If $Z=min(X,Y)$ and $X,Y$ are independent, then $F_Z(z)=1-(1-F_X(z))(1-F_Y(z))$.

Mathematical Expectation

Definition of Mathematical Expectation

Expectation for discrete variables: $E(X)={\mathrm{\Sigma }}_{k=1}^{\mathrm{\infty }}{x}_k{P}_k$

Expectation for continuous variables: $E(X)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}xf(x)\mathrm{d}x$

The mathematical expectation exists when the sum of the series (or the integral) is absolutely convergent.

For the two-dimensional case, it can also be calculated as:

$E(X)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}x{f}_X(x)\mathrm{d}x$

$E(Y)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}y{f}_Y(y)\mathrm{d}y$

Expectation of a Function of a Random Variable

$E(g(x))={\mathrm{\Sigma }}_{k=1}^{\mathrm{\infty }}g(x_k)P_k={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x)f(x)\mathrm{d}x$

$E(g(x,y))=\mathrm{\Sigma }\mathrm{\Sigma }g(x_i,y_j)P_{ij}={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x,y)f(x,y)\mathrm{d}x\mathrm{d}y$

Properties of Mathematical Expectation

$E(C)=C$

$E(C_{1}X+C_{2}Y)=C_{1}E(X)+C_{2}E(Y)$

If $X$ and $Y$ are independent, then $E(XY)=E(X)E(Y)$

Variance

Definition of Variance

$D(X)=E(X-E(X){)}^2=E(X^2)-E(X{)}^2$

Let $E(X)=c$,

$=E(X-c{)}^2=E(X^2-2cX+c^2)$

$=E(X^2)-2cE(x)+c^2$

$=E(X^2)-c^2=E(X^2)-E(X{)}^2$

The standard deviation is $\sqrt{D(X)}$

Properties of Variance

$D(C)=0$

$D(aX)=a^{2}D(X)$

When $X$ and $Y$ are independent, $D(X\pm Y)=D(X)+D(Y)$

When $X$ and $Y$ are independent, $D({\mathrm{\Sigma }}_{i=1}^n{c}_i{X}_i)={\mathrm{\Sigma }}_{i=1}^n{c}_i^{2}D(X_i)$

$D(X)=0⟺\mathrm{\exists }c,P(X=c)=1$, but this does not imply $X=c$ (Similarly, an event with probability 1 is not necessarily a certain event)

Coefficient of variation: $C_v=\frac{\sqrt{D(X)}}{|E(X)|}$

Common Distributions: Expectations and Variances (Memorize)

$E(\mathrm{\Gamma }(\alpha ,\beta ))=\frac{\alpha }{\beta },D(\mathrm{\Gamma }(\alpha ,\beta ))=\frac{\alpha }{{\beta }^2}$

Raw Moments and Central Moments

$m_k=E(X^k)$

${\mu }_k=E(X-E(X){)}^k$

Therefore, variance is the second-order central moment.

Covariance and Correlation Coefficient

Definition of Covariance

$\mathrm{Cov}(X,Y)=E((X-EX)(Y-EY))=E(XY)-E(X)E(Y)$

The proof is similar to that of variance and is omitted here.

Properties of Covariance

$\mathrm{Cov}(X,X)=D(X)$

$\mathrm{Cov}(X,Y)=\mathrm{Cov}(Y,X)$

$\mathrm{Cov}(X,a)=0$

$\mathrm{Cov}(aX,bY)=ab\mathrm{Cov}(X,Y)$

$\mathrm{Cov}(X+Y,Z)=\mathrm{Cov}(X,Z)+\mathrm{Cov}(Y,Z)$

$D(X\pm Y)=D(X)+D(Y)\pm 2\mathrm{Cov}(X,Y)$

If $X$ and $Y$ are independent, then $\mathrm{Cov}(X,Y)=0$

Proof: This follows directly from the properties of expectation.

Consequently, if $X$ and $Y$ are independent, $D(X\pm Y)=D(X)+D(Y)$

Standardization of Random Variables

$X^{\ast }=\frac{X-E(X)}{\sqrt{D(X)}}$

It has an expectation of 0, a variance of 1, and is dimensionless.

Definition of Correlation Coefficient (Memorize)

The correlation coefficient $(X,Y)=\mathrm{Cov}(X^{\ast },Y^{\ast })=\frac{1}{\sqrt{D(X)}}\frac{1}{\sqrt{D(Y)}}\mathrm{Cov}(X,Y)$

Obviously, $X$ and $Y$ cannot be constants.

To compute the correlation coefficient, five expectations are required: $E(X),E(Y),E(X^2),E(Y^2),E(XY)$

That is, $R(X,Y)=\frac{E(XY)-E(X)E(Y)}{\sqrt{E(X^2)-E(X{)}^2}\sqrt{E(Y^2)-E(Y{)}^2}}$

Properties of the Correlation Coefficient

$0\le |R(X,Y)|\le 1$

When $R(X,Y)=1$, $\mathrm{\exists }{t}_0>0,P(Y=t_{0}X)=1$, meaning $X$ and $Y$ are positively correlated.

When $R(X,Y)=-1$, $\mathrm{\exists }{t}_0<0,P(Y=t_{0}X)=1$, meaning $X$ and $Y$ are negatively correlated.

$R(X,Y)=0$ indicates that $X$ and $Y$ are uncorrelated, which is a necessary condition for the independence of $X$ and $Y$.

To prove that $X$ and $Y$ are not independent, one should select an appropriate interval such that $P(X,Y)\ne P(X)P(Y)$.

Normal Distribution

Standard Normal Distribution

$\mathrm{N}(0,1)=\phi (x)=\frac{1}{\sqrt{2\pi }}{e}^{-\frac{x^2}{2}},x\in \mathbb{R}$

Even function, bell-shaped curve.

$\mathrm{\Phi }(x)={\int }_{-\mathrm{\infty }}^x\phi (x)\mathrm{d}x$

$\mathrm{\Phi }(0)=\frac{1}{2}$
$\mathrm{\Phi }(x)+\mathrm{\Phi }(-x)=1$
These two properties are often used in exams when calculating probabilities, and $\mathrm{\Phi }(x)$ should be retained instead of $\mathrm{\Phi }(-x)$ in the answer (or when looking up values in a table).

Normal Distribution

Since $\frac{X-\mu }{\sigma }\sim \mathrm{N}(0,1)$, it follows that $X\sim \mathrm{N}(\mu ,{\sigma }^2)$, and thus $F(x)=\mathrm{\Phi }(\frac{x-\mu }{\sigma })$.

Differentiating yields $\mathrm{N}(\mu ,{\sigma }^2)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}},x\in \mathbb{R}$.

$\mu$ determines the horizontal shift of the graph (expectation), and $\sigma$ determines the "peakedness" of the graph (standard deviation).

A linear combination of multiple independent normal distributions is still a normal distribution.

In particular, if they are all $\mathrm{N}(\mu ,{\sigma }^2)$, their average is $\mathrm{N}(\mu ,\frac{{\sigma }^2}{n})$.

Bivariate Normal Distribution

Special Case:

$(X,Y)\sim ({\mu }_1,{\mu }_2;{\sigma }_1^2,{\sigma }_2^2)=\mathrm{N}({\mu }_1,{\sigma }_1^2)\mathrm{N}({\mu }_2,{\sigma }_2^2)$

General Case (Including Correlation Coefficient):

$(X,Y)\sim ({\mu }_1,{\mu }_2;{\sigma }_1^2,{\sigma }_2^2;r)$

$=\frac{1}{{\sigma }_1{\sigma }_{2}2\pi \sqrt{1-r^2}}{e}^{-\frac{1}{2(1-r^2)}((\frac{x-{\mu }_1}{{\sigma }_1}{)}^2+(\frac{y-{\mu }_2}{{\sigma }_2}{)}^2-2(\frac{x-{\mu }_1}{{\sigma }_1})(\frac{y-{\mu }_2}{{\sigma }_2})r)}$

For the bivariate normal distribution, its marginal and conditional distributions are normal. Additionally, being uncorrelated is equivalent to being independent.

Natural Exponential Family

$f(x,\theta )=e^{\theta x-\phi (\theta )}h(x)$

Among common distributions, all except the uniform distribution can be expressed in this form.

Its mean parameter (expectation $m$) is ${\phi }^{\prime }(\theta )$, and the variance function is ${\phi }^{(2)}(\theta )$.

Limit Theorems

Chebyshev's Inequality

$P(|X-E(X)|\ge \epsilon )\le \frac{D(x)}{{\epsilon }^2}$

$P(|X-E(X)|<\epsilon )\ge 1-\frac{D(x)}{{\epsilon }^2}$

Law of Large Numbers

Consider a sequence of random variables $\{X_n\}$ and their mean $\bar{X}$.

If $\bar{X}-E(\bar{X})\stackrel{P}{⟶}0$, then $\{X_n\}$ satisfies the Law of Large Numbers.

• Chebyshev's Law of Large Numbers:

  A necessary and sufficient condition is that the variances of the random variables are uniformly bounded, i.e., there exists a constant $C$ such that $D(X_k)\le C$, which ensures compliance with the Law of Large Numbers.

• Law of Large Numbers for Independent and Identically Distributed (i.i.d.) Variables:

  As long as the $X_k$ are independent and identically distributed with $E(X_k)=\mu$ and $D(X_k)={\sigma }^2$, the Law of Large Numbers is guaranteed to hold.

• Bernoulli's Law of Large Numbers:

  If $X_k\sim \mathrm{B}(n,p)$, then $\frac{n_A}{n}\stackrel{P}{⟶}p$.

Central Limit Theorem

Let $X_i$ be independent and identically distributed. Then $Z_n=\sum _{i=1}^n{X}_i$ approximately follows a normal distribution.

$E(X_k)=\mu$, $D(X_k)={\sigma }^2\to P(\frac{n\bar{X}-n\mu }{\sigma \sqrt{n}}\le x)=\mathrm{\Phi }(x)$

It follows that $Z$ approximately follows $\mathrm{N}(n\mu ,n{\sigma }^2)$, and $\bar{X}$ approximately follows $N(\mu ,\frac{{\sigma }^2}{n})$.

Suppose $X\sim \mathrm{B}(n,p)$, and let $Z_n=\sum _{i=1}^n{X}_i$

$\underset{n\to \mathrm{\infty }}{lim}P(\frac{X_n-np}{\sqrt{npq}}\le x)=\mathrm{\Phi }(x)$

That is, $Z$ approximately follows $\mathrm{N}(np,npq)$

Statistics Section

Common Distributions

Chi-Square Distribution — Sum of Squares of Normals

The sum of squares of normally distributed variables $X_i\sim \mathrm{N}(0,1)$.

${\chi }^2={\mathrm{\Sigma }}_{i=1}^n{X}_i^2$ follows a chi-square distribution with $n$ degrees of freedom, denoted as ${\chi }^2(n)$.

${\chi }^2(n)=\mathrm{\Gamma }(\frac{n}{2},\frac{1}{2})$, hence $E({\chi }^2)=n$, $D({\chi }^2)=2n$.

The chi-square distribution satisfies additivity.

t-Distribution — Normal Divided by a Number

$X\sim \mathrm{N}(0,1),Y\sim {\chi }^2(n)$

$t(n)=\frac{X}{\sqrt{\frac{Y}{n}}}$

The distribution is similar to the normal distribution but has thicker tails.

F-Distribution — Ratio of Normal Sums of Squares

$X\sim {\chi }^2(n),Y\sim {\chi }^2(m)$

$F(n,m)=\frac{X/n}{Y/m}$

$\frac{1}{F}=F(m,n)$

Common Statistical Measures

Sample Mean

$\bar{X}=\frac{1}{n}{\mathrm{\Sigma }}_{i=1}^n{X}_i$

Sample Variance

$S^2=\frac{1}{n-1}{\mathrm{\Sigma }}_{i=1}^n(X_i-\bar{X}{)}^2$ (Note: it is n-1, not n)

Sampling Distribution Theorem

Case 1 (Known Variance)

Let the sample $X_1,X_2,\dots ,X_n$ be drawn from $\mathrm{N}(\mu ,{\sigma }^2)$. Then,

${\chi }^2=\frac{(n-1)S^2}{{\sigma }^2}=\frac{1}{{\sigma }^2}\sum (X_i-\bar{X}{)}^2\sim {\chi }^2(n-1)$,

and $\bar{X}$ and ${\chi }^2$ are independent.

Part 2 (Known Mean)

The sample $X_1,X_2,\dots ,X_n$ is drawn from $\mathrm{N}(\mu ,{\sigma }^2)$. Then,

$t=\frac{\bar{X}-\mu }{S\sqrt{\frac{1}{n}}}\sim t(n-1)$

Third (Multiple Populations)

Probability and Statistics has successfully turned into a liberal art

If $X\sim \mathrm{N}({\mu }_1,{\sigma }^2)$ and $Y\sim \mathrm{N}({\mu }_2,{\sigma }^2)$, then:

$\bar{X}-\bar{Y}\sim \mathrm{N}({\mu }_1-{\mu }_2,\frac{{\sigma }_1^2}{n_1}+\frac{{\sigma }_2^2}{n_2})$

$\frac{(n_1-1)S_1^2+(n_2-1)S_2^2}{{\sigma }^2}\sim {\chi }^2(n_1+n_2-2)$

$\frac{(\bar{X}-\bar{Y})-({\mu }_1-{\mu }_2)}{S_w\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}\sim t(n_1+n_2-2)$

where $S_w^2=\frac{(n_1-1)S_1^2+(n_2-1)S_2^2}{n_1+n_2-2}$

If $X\sim \mathrm{N}({\mu }_1,{\sigma }_1^2)$ and $Y\sim \mathrm{N}({\mu }_2,{\sigma }_2^2)$, then:

$\frac{S_1^2/{\sigma }_1^2}{S_2^2/{\sigma }_2^2}\sim F(n_1-1,n_2-1)$

Point Estimation

Method of Moments Estimation

Set the sample mean equal to the expectation function (which can also be the squared expectation), express the parameter $\theta$ of the expectation function in terms of the expected value $m$ to obtain $\theta (m)$, and then substitute $m$ with $\bar{x}$ to obtain $\hat{\theta }=\theta (\bar{x})$.

Maximum Likelihood Estimation

Discrete Case: Write the probability of the observed event as a function of the parameter $\theta$ (usually a product of some discrete distribution), and assume this function reaches its maximum. Solve the logarithmic maximum likelihood equation $\frac{\mathrm{d}L(x)}{\mathrm{d}\theta }=0$ to obtain the corresponding parameter estimate $\hat{\theta }$.

Continuous Case: The function to be considered is $\prod _{i=1}^{n}f(x_i,\theta )$, where $x_i$ is treated as a constant. The subsequent steps are the same.

The maximum likelihood estimate is the maximum value of $x_i$, and the maximum likelihood estimator is the maximum value of $X_i$.

Criteria for Estimator Evaluation

Unbiased Estimator: An estimator is unbiased if the expected value of the estimate equals the true parameter, i.e., $E(\hat{\theta })=\theta$; otherwise, it is biased.

Asymptotically Unbiased Estimator: An estimator is asymptotically unbiased if $\underset{n\to \mathrm{\infty }}{lim}E(\hat{\theta })-\theta =0$.

Efficiency Criterion: If $D({\hat{\theta }}_1)\le D({\hat{\theta }}_2)$, then ${\hat{\theta }}_1$ is more efficient than ${\hat{\theta }}_2$.

Consistency Criterion: If ${\hat{\theta }}_n\stackrel{P}{⟶}\theta$, then ${\hat{\theta }}_n$ is a consistent estimator of $\theta$.

Mean Squared Error (MSE) Criterion: If $E({\hat{\theta }}_1-\theta {)}^2\le E({\hat{\theta }}_2-\theta {)}^2$, then ${\hat{\theta }}_1$ is more efficient than ${\hat{\theta }}_2$ in terms of MSE.

Specifically, $B_2$ is more efficient than $S^2$ in terms of MSE.

Interval Estimation (Memorize)

Only for normal distribution

Two-Sided Confidence Interval

Estimate the $(1-\alpha )$ confidence interval for $\mu$.

If ${\sigma }^2$ is known, it is $(\bar{X}-\frac{\sigma }{\sqrt{n}}{u}_{1-\frac{\alpha }{2}},\bar{X}+\frac{\sigma }{\sqrt{n}}{u}_{1-\frac{\alpha }{2}})$.

If ${\sigma }^2$ is unknown, it is $(\bar{X}-\frac{S}{\sqrt{n}}{t}_{1-\frac{\alpha }{2}}(n-1),\bar{X}+\frac{S}{\sqrt{n}}{t}_{1-\frac{\alpha }{2}}(n-1))$.

Estimate the $(1-\alpha )$ confidence interval for ${\sigma }^2$.

$(\frac{(n-1)s^2}{{\chi }_{1-\frac{\alpha }{2}}^2(n-1)},\frac{(n-1)s^2}{{\chi }_{\frac{\alpha }{2}}^2(n-1)})$

One-Sided Confidence

Estimate the $(1-\alpha )$ confidence interval for $\mu$.

If ${\sigma }^2$ is known, the one-sided lower confidence limit is $\bar{X}-\frac{\sigma }{\sqrt{n}}{u}_{1-\alpha }$, and the upper limit is $\bar{X}+\frac{\sigma }{\sqrt{n}}{u}_{1-\alpha }$.

If ${\sigma }^2$ is unknown, the one-sided lower confidence limit is $\bar{X}-\frac{S}{\sqrt{n}}{t}_{1-\alpha }(n-1)$, and the upper limit is $\bar{X}+\frac{S}{\sqrt{n}}{t}_{1-\alpha }(n-1)$.

Estimate the $(1-\alpha )$ confidence interval for ${\sigma }^2$.

The one-sided lower confidence limit is $\frac{(n-1)s^2}{{\chi }_{1-\alpha }^2(n-1)}$, and the upper limit is $\frac{(n-1)s^2}{{\chi }_{\alpha }^2(n-1)}$.

Hypothesis Testing

Process

Proof by contradiction with a probabilistic nature

1. First, state the null hypothesis $H_0$ and the alternative hypothesis $H_1$.
2. Under the assumption that $H_0$ holds, construct the distribution satisfied by the sample.
3. Determine the rejection region $W$ based on the value of $\alpha$.
4. Substitute the observed value $u$. If $u\in W$, reject the null hypothesis.

Type I and Type II Errors and Their Probabilities

• Type I Error (False Positive): When $H_0$ is true, the sample value $u\in W$.
• Type II Error (False Negative): When $H_1$ is true, the sample value $u\notin W$.

The probability of a Type I error, $P_1=P(W)$, is less than or equal to $\alpha$.

The probability of a Type II error is $P_2=P(\bar{W})$.

$P_1$ and $P_2$ cannot be reduced simultaneously. However, by fixing one of them and increasing the sample size $n$, the other can be reduced. The choice of which one to reduce depends on the severity of their respective consequences.