EPSY 514 - Nonparametric Statistics and Regression

Semester: Fall 2007
Professor: George Karabatsos
Time: Monday 5:00-8:00pm Phone: 312-413-1816
Room: EPASW 2217 (second floor);
Also, we will often meet
at EPASW 2027 (computer lab).		 E-mail: georgek@uic.edu
Office Hours:
Mon 2-4 (EPASW 1034)		 CRN: 25570

Course Description:
Nonparametric models provide ways to perform statistical inference, while allowing the statistician to make minimal assumptions about the true data-generating process. This course covers nonparametric models for two related tasks that are central to statistical inference: (1) Density estimation, and (2) Regression. Nonparametric density estimation provide a means to infer the (unknown) true distribution that generated a given set of data, while nonparametric regression provides a means to estimate the (unknown) true regression curve representing the actual relationship between a covariate and the outcome variable. In the task of density (distribution) estimation, no assumption is made about the shape of the true distribution (e.g., no assumption that that true distribution is a normal distribution), and in nonparametric regression, no assumption is made about the shape of the true regression function (e.g., no assumption that that true regression curve is linear). Finally, the course also covers
Semiparametric Regression, which provides a flexible way to model many predictor variables, while relaxing the assumptions of linear models and generalized linear-mixed models. These assumptions include normally-distributed errors, normally-distributed random effects, and the assumption that the link function is defined by a logistic (or probit) cumulative distribution function.

A nonparametric (or semiparametric) model, in making minimal assumptions about the data generating process, enables more accurate conclusions in data analysis. In contrast, parametric models of statistical inference usually make strong assumptions about the true data-generating process. For example, the ANOVA model assumes that the data arise from normal distributions, and the classical regression model assumes that each covariate has a linear relationship with the outcome variable, the random effects are normally distributed, and that the distribution of the errors is normal. While such simplifying assumptions help maintain the mathematical elegance of such models, these assumptions are incorrect in real practice (e.g., usually, the true distribution of the data is not normal, and the true relationship between two variables is not linear).

Students who successfully complete the course will have learned to use various flexible statistical models that handle virtually all problems in data analysis.

In particular, this course will cover classical and modern Bayesian approaches to density estimation, nonparametric regression, and semiparametric regression. The classical approaches are based on kernel estimation, and the Bayesian approaches or based on hierarchical models where a nonparametric prior is specified on the entire space of distributions. Among nonparametric priors, we consider Dirichlet Process priors, Pólya Tree priors, Bernstein polynomial priors, Bivariate Dirichlet-Bernstein priors, as well as nonparametric priors corresponding to the classical bootstrap, and the Bayesian bootstrap.
Specifically, this course shows how to:
(1) Compare distributions (densities) and hazard rates between two or more groups of observations, where these groups may either be independent or dependent;
(2) Perform the estimation of quantiles (i.e., the estimated value of a random variable, for a given percentile in the estimated distribution);
(3) Perform nonlinear regression using Kernel smoothing and Bayesian nonparametric priors;
(4) Perform linear regression and ANOVA modeling with a nonparametric prior for the error distribution (i.e., no normality assumption for the error distribution);.
(5) Perform data analysis with semi-parametric versions of generalized linear mixed models, binary regression models, ordinal regression models, and Rasch models, which treat the true distribution of the random effects as unknown (instead of assuming that the true distribution is normal).

This course will illustrate such nonparametric and semiparametric models on data sets arising from education, psychology, and medicine. Furthermore, students will learn how to perform nonparametric data analysis through applications on real data, using either the R software or the Winbugs software. Such applications will count as credit for the take-home midterm exam and the take-home final exam, and students will be given course time to complete these two exams.
The R software and the readings for the course (listed below) are all available at no cost.
Various (free) packages in the R software are needed for the course, including the DP package which can be downloaded from:
http://cran.r-project.org/src/contrib/Descriptions/DPpackage.html
You may install the software on your personal computer, and I will provide all the readings to you as pdf files.

This course places strong emphasis on the practical applications of nonparametric models. However, this emphasis is not made at the expense of rigor. In particular, this course intends to provide the theoretical background that is needed to describe the properties of the various nonparametric models, and to provide justifications for the use of these models in practical data analysis.

Prerequisites: At least two graduate courses in statistics.

Background Readings: (I will provide these articles to you)
Escobar, M.D. (2007). Applied Bayesian Methods. Course Notes On Powerpoint (Many thanks to Michael Escobar for providing this valuable source of information for students).
Escobar, M.D., & West, M. (1995) Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90, 577-588.
Ferguson, T.S. (1973). A Bayesian analysis of some nonparametric problems. The Annals of Statistics, 1, 209-230.
Ferguson, T.S. (1974). Prior distributions on space of probability measures. The Annals of Statistics, 2, 615-629.
Gutiérrez-Peña, E., & Walker, S.G. (2005). Statistical Decision Problems and Bayesian Nonparametric Methods. International Statistical Review, 3, 309-330.
Hanson, T., and Johnson, W. (2002). Modeling regression error with a mixture of Polya trees. Journal of the American Statistical Association, 97, 1020-1033.
Jara, A. (2007). Documentation for the DPpackage Package software.
Jara, A. (2007). Applied Bayesian Non- and Semi-parametric Inference using DPpackage. To appear, Journal of Statistical Software.
Kleinman K.P., & Ibrahim, J.G. (1998a). A semi-parametric Bayesian approach to the random effects model. Biometrics, 54, 921-938.
Kleinman K.P., & Ibrahim, J.G. (1998b). A semi-parametric Bayesian approach to generalized linear mixed models. Statistics In Medicine, 17, 2579-2596.
Lavine, M. (1992). Some aspects of Polya tree distributions for statistical modeling, Annals of Statistics, 20, 1222-1235.
Lavine, M. (1994). More aspects of Polya tree distributions for statistical modeling, Annals of Statistics, 22, 1161-1176.
Muliere, P., & Secchi, P. (1996). Bayesian nonparametric predictive inference and bootstrap techniques. Annals of the Institute of Statistical Mathematics, 48, 663-673.
Müller, P., & Quintana, F.A. (2004). Nonparametric Bayesian Data Analysis. Statistical Science, 19 (1), 95-110.
Newton, M.A., Czado, C., and Chapell, R. (1996). Bayesian inference for semiparametric regression. Journal of the American Statistical Association, 91, 142-153.
Petrone, S. (1999). Random Bernstein polynomials. Scandinavian Journal of Statistics, 26, 373-393.
Rubin, D.B. (1981). The Bayesian boostrap. The Annals of Statistics, 9, 130-134.
Schucany, W.R. (2004). Kernel smoothers: An overview of curve estimators for the first graduate course in nonparametric statistics. Statistical Science, 19(4), 663-675.
Sheather, S.J. (2004). Density estimation. Statistical Science, 19(4), 588-597.
Walker, S.G., & Muliere, P. (2003). A bivariate Dirichlet process. Statistics and Probability Letters, 64, 1-7.
Walker, S.G., Damien, P., Laud, P.W., & Smith, A.F.M. (1999). Bayesian nonparametric Inference for random distributions and related functions.
Journal of the Royal Statistical Society, Series B, 61, 485-527.


COURSE SCHEDULE

Date Topic
Suggested Readings

Aug27

 

 

 Assignments/tasks. Introduction and motivation for Nonparametric Statistical Inference.
Review of Theory of Probability and Statistical Inference
-- Review set theory, sigma-algebra, countable additivity, Lebesgue integration, Lebesgue-Stieltjes measure, Lebesgue-Stieltjes integration.
-- Axioms of Probability, statistical independence, conditional probability, conditional independence, Bayes theorem.
-- Cumulative distribution functions, probability density functions, probability mass functions.

Course
Notes

Sep3

Labor Day, no class.

Sep10 Review of Theory of Probability and Statistical Inference (continued)
-- Comaparing distributions: Mean order, variance order, stochstic-precedence order-stochastic order, hazard order, likelihood ratio order, Csiszar divergence (including chi-square divergence, Kullback-Leibler divergence, Hellinger distance).
-- A probability model as a prior distribution defined on the space of distribution functions.
-- Frequentist point estimation, and Bayesian inference.
Course
Notes
Sep17

Assignments/tasks. Introduction and motivation for Nonparametric Statistical Inference.
Review of Theory of Probability and Statistical Inference
-- DeFinetti's Theorem for fully exchangeable, and partially exchangeable, sequences of observations
-- Bayesian (nonparametric) statistical decision theory, with emphasis on model selection, and point estimation (including maximum-likelihood estimation).
-- The bootstrap corresponds to inference under a noninformative Dirichlet Process prior.
-- Some Large Sample Theory: Asymptotic consistency, efficiency, bias, rate of convergence.

Gutiérrez-Peña;
Rubin

Sep24

 Density Estimation, frequentist approaches.
-- The histogram estimator.
-- The kernel approach.
-- Bootstrap approaches to learn the uncertainty of kernel density estimates (Classical bootstrap and Bayesian bootstrap).
-- Comparing densities (distributions), cumulative distribution functions, and hazard rates between two or more independent samples.
-- Quantile estimation.
-- Using the (classical and Bayesian) bootstrap to learn the distribution of a correlation coefficient
(Pearson correlation, Kendall's tau correlation, Spearman's correlation, Cohen's kappa coefficient of judge agreement, reliability coefficient).
Sheather

Oct1

 Density Estimation, hierarchical Bayesian nonparametric approaches.
-- The Dirichlet Process Prior.
-- The Dirichlet-Multinomial Process prior.
-- Mixture of Dirichlet Processes.
-- Quantile estimation.
-- Comparing distributions and cumulative distribution functions between two or more independent samples.
Ferguson73
Escobar & West
Müller&Quintana
Walker et.al
Oct8

Density Estimation, Bayesian nonparametric approaches (continued).
-- Bernstein polynomial prior (i.e., Random Bernstein Polynomials).
-- Pólya Tree prior.
-- The Dirichlet Process as a special case of the Pólya Tree prior.
-- Quantile Estimation.
-- Comparing distributions and cumulative distribution functions between two or more independent samples.
Ferguson74
Lavine
Petrone
Oct15

Multivariate Density Estimation, frequentist and Bayesian nonparametric approaches.
-- The kernel approach.
-- The curse of dimensionality.
-- The Mixture of Dirichlet Process model for multivariate density estimation..
-- The bivariate Dirichlet-Bernstein prior, and modeling dependence between two or more samples of observations.
-- Comparing distributions, cumulative distribution functions, and hazard rates between two or more independent samples, using the bivariate Dirichlet Process.

Escobar & West
Walker & Muliere
Oct22 Nonparametric Regression, frequentist approaches.
-- The Kernel approach.
-- Locally-weighted Scatter-Plot Smoothing (LOESS)
-- Bootstrap approaches to learn the uncertainty of regression estimates (Classical bootstrap and Bayesian bootstrap).
-- Bootstrap approaches to comparing two (or more) regression functions.
-- The curse of dimensionality.
-- Isotonic regression using the pooled-adjacent-violators algorithm.
-- Isotonic regression through a nonparametric prior.
-- Bootstrap approaches to learn the uncertainty of isotonic regression estimates (Classical bootstrap and Bayesian bootstrap).
TAKE-HOME MIDTERM EXAM DUE
Schucany
Oct29

Semiparametric Regression, the Bayesian hierarchical approach.
-- Linear median regression modeling with a mixture of Pólya Trees prior for the error distribution.

Hanson & Johnson
Müller; Walker et.al
Nov5 Semiparametric Regression, Bayesian approaches.
-- Semiparametric linear mixed-effects modeling using the Dirichlet Process prior and the Mixture of Dirichlet Process prior.
-- Applications of the model to longitudinal data analysis, and for meta-analysis.
Kleinman&Ibrahim98a
Nov12 Semiparametric Regression, the Bayesian hierarchical approach.
-- Semiparametric generalized linear mixed-effects modeling using the Dirichlet Process prior and the Mixture of Dirichlet Process prior.
-- Applications of the model to longitudinal data analysis.
Kleinman&Ibrahim98b
Nov19 Semiparametric Regression, the Bayesian hierarchical approach.
-- Semiparametric binary regression using the Dirichlet Process prior and the mixture of Dirichlet Process prior.
-- Semiparametric binary regression under the Centrally-Standardized Dirichlet Process prior
(i.e., for the inverse link-function, putting a prior on the entire space of cdfs.
Thus, the inverse link function is treated as an unknown parameter, instead of a known fixed parameter, such as the logistic cdf)
Newton et al
Nov26

Semiparametric Regression, the Bayesian hierarchical approach.
-- Semiparametric ordinal linear mixed-effects modeling using the Dirichlet Process prior and Mixture of Dirichlet Process Prior.
-- Applications of the models.
Kleinman&Ibrahim98b
Dec3 Semiparametric Regression, the Bayesian hierarchical approach.
-- Psychometric applications: Semiparametric Rasch modeling using the Dirichlet Process prior and Mixture of Dirichlet Process Prior.
-- Receiver Operating Curve (ROC) Analysis with nonparametric priors.
-- Survival Analysis: Accelerated Failure Time (AFT) Regression Modeling (of interval-censored data) using the Dirichlet Process prior.
Dec12 TAKE-HOME FINAL EXAM DUE (Exam week)
Please leave paper in my mailbox in Room 3233, or under my office door at Room 1034.

 

Grading Policy:
The Midterm Exam and the Final Exam are each worth 50% of the final grade (I can only accept hard copies of the completed exams).
Final grades will be given out according to the following scale:

A
90% - 100%
B
79% - 89%
C
68% - 78%
D
57% - 67%
F
56% - Lower

Students will spend substantial amounts of time reading, and using the computer.

An "A" grade for the course is guaranteed by completing both exams accurately, and on time.
While both exams are take-home, attendance in the course is important to their successful completion.
I am fine with group-based completion of the exam, when necessary. (Of course, I cannot accept two (or more) people handing in exactly the same completed exam).
There are no exceptions to the above grading scale, and no extra credit work will be accepted.
I can only accept hard-copies of the exams (not electronic copies).
An exam that is completed after the due date receives a grade penalty of 10%*(# weeks after due date).
Incomplete grades will be considered for students with extenuating circumstances.
Poor performance on assignments will not be considered in a request for an incomplete.


Disability Services:
UIC strives to ensure the accessibility of programs, classes, and services to students with disabilities. Reasonable accommodations can be arranged for students with various types of disabilities, such as documented learning disabilities, vision, or hearing impairments, and emotional or physical disabilities. If you need accommodations for this class, please let your instructor know your needs and he/she will help you obtain the assistance you need in conjunction with the Office of Disability Services (1190 SSB, 413-2183).