Saturday, January 26, 2019

Biographical information on Carl Friedrich Gauss (in English)

There aren't that many biographies of Gauss in English (there are more in German); Gauss was less colorful and less public than Euler, so there is less for a non-mathematical historian to work with. By contrast, the MAA edition of Dunnington includes many references to the literature (up to 2004) in the history of mathematics, statistics and physics on studies of Gauss' research corpus. (See the MacTutor archive page on Gauss for an online list of references.)
  • monographs (from primary sources)
    • Dunnington, Gauss: Titan of Science. MAA, 2004, 0883855380. 
      • (Reprint edition of 1955 book, with critical commentary and an annotated bibliography.)
  • biographies (from secondary sources)
    • academic readers
      • Buhler, Gauss: A Biographical Study. Springer-Verlag, 1981, 3540106626.
      • Hall, Carl Friedrich Gauss: A Biography. MIT Press, 1970, 0262080400.
    • general readers
      • Schaaf, Carl Friedrich Gauss: Prince of Mathematicians. Franklin Watts, 1964,
        0531008770.
    • youth readers (semi-fictionalized)
      • Tent, The Prince of Mathematics: Carl Friedrich Gauss. AK Peters, 2006, 1568812612.
  • memoires
    • von Waltershausen, Carl Friedrich Gauss: A Memorial. [archive.org]

Wednesday, February 21, 2018

Video series relevant to ML

You can go top-down from code to theory (the fast.ai way) or bottom-up from theory to code (the academic way). I'm going to assume you're in it for the long haul and want to understand the principles (to the degree that there are principles) first; as such, you'd take a course like Stanford's CS 229 before courses on deep learning. (You can actually take EE 263, CS 229, CS 224N, CS 231N, CS 234 remotely as a non-degree student, trading ~$1000/unit for graded assessment.)

Videos give intuition, but won't teach you to do anything useful. You need to do exercises. If the book/psets used in the courses don't have solutions, find another book that does and work some of those exercises as warm-up.  (Reading two books at once is a common math study tactic.)

Background for intro ML classes
  • Harvard Stats 110, Probability Theory: YouTube
    • A first course in calculus-based probability theory is pretty much assumed in any serious discussion of ML. This one seems typical, based on Blitzstein's book.
    • For CS 229, you might be able to get away with 1-2 weeks of probability theory covered in a typical discrete math course, but you'd have to do some additional work since facility with continuous distributions (such as the univariate and multivariate normal distributions) is assumed all over the place in ML.
  • Mathematical Statistics: ?
    • I don't know of a good video course on mathematical statistics. 
    • For CS 229, it's helpful to have this background at the Casella & Berger level (sufficient statistics, the significance of exponential families, etc.) for your own understanding, but you can survive without it since this level of rigor is never actually required in psets.
  • MIT 18.06, Linear Algebra: YouTube (1999 lectures), YouTube (2011 sections)
    • This is a first course in applied linear algebra, at a lower-division level of rigor but working up through valuable applied topics like SVD, basic linear dynamical systems, least squares and pseudoinverses, etc.
    • For CS 229, you might be able to get away with the linear algebra covered in the usual ABET combined linear algebra / differential equations course as taught to engineers in sophomore year (i.e., maybe 5-7 weeks of instruction in linear algebra). But for your own sanity, you should probably buy Strang's Introduction and refresh/extend your understanding using this course.
  • Stanford EE 263, Linear Dynamical Systems (2008): YouTube
    • The first 1/3 of this course reviews the material of a course like MIT 18.06 or Stanford EE 103 (based on Boyd's undergrad book). (You can also use EE 263 as a bootcamp version of the prerequisites, but since most students will have actually taken the linear algebra prerequisite, it will be tough to compete in the course itself.) The middle 1/3 or so is stuff like least norm, SVD, etc. This is a quick introduction to a lot of relevant methods for fitting, optimization, etc. The final 1/3 or so covers linear dynamical systems (which are relevant to reinforcement learning) and a few additional relevant linear algebra topics such as Cayley-Hamilton. You should be ok with sophomore-level differential equations; for the latter third of the course, it is assumed that you can (or can learn to) apply the Laplace Transform, so an upper-division diffeq course is helpful (but not necessary if you're ok with applying it as a black box).
    • For CS 229, certain bits relating to control in reinforcement learning will make a lot more sense with this background. Also, most of the linear algebra proofs in CS 229 will be at exactly this level, so it's very good practice.
 Survey / Introduction
  • Stanford CS 229, Machine Learning (2009): YouTube 
    • CS 229 is not an applied course on ML. And the emphasis is very different from the "Andrew Ng" Coursera ML course, which focuses on rudimentary understanding of a few ML algorithms. CS 229 is a quick overview of the mathematics (mostly at a lower-division level of rigor) behind regression, SVMs, EM, basic RL, and so on. If your linear algebra background is weaker than 18.06 and (in a few places) EE 263, this will severely limit your understanding; you have to be able to absorb the material from the lecture notes and not expect much more than intuition from the lectures.
    • While the level of rigor is lower-division (meaning, you use calculus rather than analysis), you had better be very facile with reading and writing proofs. Most pset and exam proofs are calculations (i.e., "show that" by rewriting one expression into another expression), but if you don't feel pretty comfortable with proof by counterexample, proof by contradiction, etc. you are going to have issues with 1-3 problems per pset.
    • If you take this course for credit, be aware that an awful lot of people will already be familiar with all of the material (e.g., from watching all of the 2009 videos and doing the psets). Moreover, many seem to have access to previous pset solutions. So it's a real grind to do this honestly. Also, many students will already have teams set up and semi-started on their projects before the quarter starts (e.g., collecting data) so do not measure progress by the stated milestone dates.
  •  UCL, Introduction to RL (2015): YouTube 
Deep Learning
  • UC Berkeley CS 294-129 (2016), Deep Learning: YouTube
  • Stanford CS 231N, Convolutional Neural Networks for Computer Vision  (2017): YouTube
  • Stanford CS 224N, Deep Learning for NLP (2017): YouTube
  • UC Berkeley CS 294-112 (2017), Deep RL: YouTube (Spring), YouTube (Fall) 
Less directly relevant stuff:
  • Stanford EE 364AB, Convex Optimization  (2008):  YouTube 
    • All of ML involves optimization in some form or another, so many people recommend this course (based on Boyd's book) if you're interested in statistical learning theory.
    • You can definitely get by in CS 229 without this, and convex optimization has little direct relevance to deep learning (for example).  But I think the serious ML students take it for culture (e.g., as prep for Stats 231/CS 229T).
    • If you take this course for credit, you should take EE 263, if only to gird yourself for the 24-hour take-home exams.
  • UCCS Math 535, Applied Functional Analysis: YouTube
    • A course based on Kreyszig. Caveat: I can't vouch for this as I didn't go through this series; I took a course on Fourier analysis that used the Hilbert space method.
    • You don't need this for CS 229 (for example, uniform convergence is mentioned exactly once, and only in a hand-wavy way; Hilbert spaces are never mentioned). But deeper discussions in topics like kernel methods and statistical learning theory are often framed in the language of functional analysis.

Saturday, December 24, 2016

Studies in the History of Statistics and Probability, Volume 2


Quoted from https://www.york.ac.uk/depts/maths/histstat/bib/studies.htm, and annotated with links. As with Volume 1, most are now online; again, I've noted those outside of the Studies in italic, and those I have been unable to find in bold.

[...]

STUDIES IN THE HISTORY OF STATISTICS AND PROBABILITY II

Edited by M G Kendall and R L Plackett (QUARTO S 0.9 STU)

CONTENTS

  1. On the possible and probable in Ancient Greece by S Sambursky
  2. Probability in the Talmud by N L Rabinovitch
  3. Combinations and probabilities in rabbinic literature by N L Rabinovitch
  4. A Budget of paradoxes by H L Seal
  5. An argument for Divine Providence, taken from the constant regularity observ'd in the births of both sexes by J Arbuthnot
  6. Measurement in the study of society by M G Kendall [book]
  7. The early history of index numbers by M G Kendall
  8. Abraham De Moivre's 1733 derivation of the normal curve: a bibliographical note by R S Daw and E S Pearson
  9. The historical development of the use of generating functions in probability theory by H L Seal
  10. Boscovich and the combination of observations by C Eisenhart [book]
  11. Daniel Bernoulli on the normal law by O B Sheynin
  12. D Bernoulli's work on probability by O B Sheynin
  13. Progress in the middle of the eighteenth century. Süssmilch and his contemporaries. Estimates and enumerations of population. Progress of theory at the close of the eighteenth century by H Westergaard. [book]
  14. Leading British statisticians of the nineteenth century by P J FitzPatrick
  15. Notes on the history of quantification in sociology trends, sources and problems by P F Lazarsfeld
  16. Laplace, Fisher, and the discovery of the concept of sufficiency by S M Stigler
  17. The discovery of the method of least squares by R L Plackett
  18. Development of the notion of statistical dependence by H O Lancaster
  19. Florence Nightingale as statistician by E W Kopf
  20. On the history of some statistical laws of distribution by O B Sheynin
  21. The work of Ernst Abbe by M G Kendall
  22. Entropy, probability and information by M G Kendall
  23. A history of random processes. I. Brownian motion from Brown to Perrin by S G Brush
  24. Branching processes since 1873 by D G Kendall
  25. The simple branching process, a turing point test and a fundamental inequality: a historical note on I.J. Bienaymé by C C Heyde and E Seneta
  26. Simon Newcomb, Percy Daniell, and the history of robust estimation 1885-1920 by S M Stigler
  27. The hypothesis of elementary errors and the Scandinavian school in statistical theory by C-E Särndal
  28. On the history of certain expansions used in mathematical statistics by H Cramér
  29. Historical survey of the development of sampling theories and practice by You Poh Seng
  30. Sir Arthur Lyon Bowley (1869-1957) by W F Maunder
  31. Note on the history of sampling methods in Russia by S S Zarkovic
  32. A supplement to ``Note on the history of sampling methods in Russia'' by S S Zarkovic.


STUDIES IN THE HISTORY OF STATISTICS AND PROBABILITY

The two collections Studies in the History of Statistics and Probability, edited by E S Pearson and M G Kendall, Griffin 1970 and Studies in the History of Statistics and Probability, Volume II, edited by M G Kendall and R L Plackett, Griffin 1977, contain all of numbers I to XXXII of the occasional series Studies in the History of Statistics and Probability. There are in fact two diferent papers numbered XXI, one by O B Sheynin and the other by M G Kendall; however, the latter, hereafter denoted XXI bis is unique in that it first appeared in Rev. Int. Stat. Inst. rather than in Biometrika, which is where all the other papers first appeared.

[...]

Numbers XXI bis and XXII to XXXII appear in Volume II as papers with the following Arabic numbers:
XXI bis - 7; XXII - 2; XXIII - 11; XXIV - 3; XXV - 20; XXVI - 21; XXVII - 27; XXVIII - 28; XXIX - 17; XXX - 8; XXXI - 25; XXXII - 16.

Studies in the History of Statistics and Probability, Volume 1

Quoted from https://www.york.ac.uk/depts/maths/histstat/bib/studies.htm, and annotated with links to JSTOR (since 25 out of 29 were published in Biometrika). I've noted those outside of the Studies in italic, and those I have been unable to find in bold.


STUDIES IN THE HISTORY OF STATISTICS AND PROBABILITY

Edited by E S Pearson and M G Kendall (QUARTO S 0.9 STU)

CONTENTS

  1. Dicing and gaming by F N David
  2. The beginnings of probability calculus by M G Kendall
  3. A Note on playing cards by M G Kendall
  4. The Book of Fate by M G Kendall
  5. Random mechanisms in Talmudic literature by A M Hasofer
  6. Where shall the history of statistics begin? by M G Kendall
  7. Medical statistics from Graunt to Farr by M Greenwood
  8. The principle of the arithmetic mean by R L Plackett
  9. A note on the early solutions of the problem of the duration of play by A R Thatcher
  10. An essay towards solving a problem in the doctrine of chances by T Bayes (with a biographical note by G A Barnard)
  11. The most probable choice between several discrepant observations and the formulation therefrom of the most likely induction by D Bernoulli (with observations by L Euler and an introductory note by M G Kendall)
  12. A note on the history of the graphical presentation of data by E Royston
  13. Thomas Young on coincidences by M G Kendall
  14. Notes on the history of correlation by Karl Pearson
  15. The historical development of the Gauss linear model by H L Seal
  16. On the early history of the law of large numbers by O B Sheynin
  17. A note on the early statistical study of literary style by C B Williams
  18. De Morgan and the statistical study of literary style by R D Lord
  19. Isaac Todhunter's History of the Mathematical Theory of Probability by M G Kendall
  20. Francis Ysidro Edgeworth, 1845-1926 by M G Kendall
  21. Walter Frank Raphael Weldon, 1860-1926 by Karl Pearson
  22. Some incidents in the early history of biometry and statistics 1890-94 by E S Pearson
  23. Some reflexions on continuity in the development of mathematical statistics, 1885-1920 by E S Pearson
  24. William Sealy Gosset, 1876-1937
    (1) "Student" as a man by L McMullen
    (2) "Student" as a statistician by E S Pearson
  25. Some early correspondence between W S Gosset, R A Fisher and Karl Pearson with notes and comments by E S Pearson
  26. George Udny Yule, 1871-1951 by M G Kendall
  27. Karl Pearson 1857 (1957). A centenary lecture delivered at University College London by J B S Haldane
  28. Ronald Aylmer Fisher, 1890-1962 by M G Kendall
  29. The Neyman-Pearson story: 1926-1934. Historical sidelights on an episode in Anglo-Polish collaboration
[...]

STUDIES IN THE HISTORY OF STATISTICS AND PROBABILITY

The two collections Studies in the History of Statistics and Probability, edited by E S Pearson and M G Kendall, Griffin 1970 and Studies in the History of Statistics and Probability, Volume II, edited by M G Kendall and R L Plackett, Griffin 1977, contain all of numbers I to XXXII of the occasional series Studies in the History of Statistics and Probability. There are in fact two diferent papers numbered XXI, one by O B Sheynin and the other by M G Kendall; however, the latter, hereafter denoted XXI bis is unique in that it first appeared in Rev. Int. Stat. Inst. rather than in Biometrika, which is where all the other papers first appeared.

Numbers I to XXI appear in the first volume as papers with the following Arabic numbers:
I - 1; II - 2; III - 12; IV - 17; V - 3; VI - 9; VII - 8; VIII - 18; IX - 10; X - 6; XI - 11; XII - 4; XIII - 19; XIV - 22; XV - 15; XVI - 5; XVII - 23; XVIII - 13; XIX - 20; XX - 25; XXI - 16.

Tuesday, October 11, 2016

Online analysis courses

Does it make sense to teach mathematical ({real,complex,functional}) analysis in a distance learning format? In the transition from "math as a bunch of mechanical procedures" (where problem sets can be graded fairly mechanically) to "math as idiomatic communication between mathematicians" (aka reading and writing proofs), somebody who is already a competent mathematician has to actually critique the proofs being written by the novice students.  From the outside, I somehow imagine it would be like trying to teach a writing or design discipline by distance learning.  

And yet, there are a few such distance learning courses. (I'm talking about for-credit courses, from non-profit universities.)  Aside from the Harvard Extension courses (which have pretty specific syllabi posted online), it's very unclear how "proof-y" some of these courses are, or whether they are "advanced calculus"/"advanced engineering math"-type courses.  
  • Use of texts like Ross and Abbott indicate that instructors are trying to make the course "friendly" for students who aren't trying for math Ph.D.s (e.g., prospective HS/CC math teachers). For example, at Stanford, non-honors (Math 115) uses Ross while honors (Math 171) uses Johnsonbaugh & Pfaffenberger; at Berkeley, non-honors (Math 104) uses Ross while honors (Math H104) uses Pugh or baby Rudin. You see frank comments from instructors about the perception of books/courses for the "average" vs. the "elite"... (See also this thread about H104.)
  • Use of full-on "transition" texts like Lay are even more likely aimed at prospective math teachers.
  • Some of the texts, like Brown/Churchill, are aimed at engineering students and not very rigorous (proof-y) compared to texts aimed at math students. 
(From the outside, it's interesting to me that distance learning courses don't try to teach using proof systems like Mizar and then auto-grade. Since half of the exercise of "fill in the gaps in Rudin's proofs" is to spot the gaps in an idiomatically-written proof and learn to fill them in for oneself, then assigning the task of mechanically filling gaps in proof templates misses the mark.)

Anyway, digressions aside, here are some of the courses I've found (in no meaningful order).Textbooks are in [brackets].
  1. Harvard Extension School (math courses)
    1. MATH E-23A Linear Algebra and Real Analysis I [Ross]
    2. MATH E-216 Real Analysis, Convexity, and Optimization [Luenberger]
  2. SUNY Empire State
    1. REAL ANALYSIS: THE THEORY OF CALCULUS (SMT-274344) [Abbott]
    2. COMPLEX VARIABLES (SMT-273314) [Brown/Churchill] 
  3. University of West Florida (M.S. program; non-degree option) 
    1. MAA 6306 Real Analysis
    2. MAA 6426 Complex Analysis
  4. Emporia State, Kansas (M.S. program; non-degree option)
    1. MA 734 Complex Variables
    2. MA 735 Advanced Calculus I [Bressoud]
    3. MA 736 Advanced Calculus II [Wade]
  5. UMUC 
    1. Concepts of Real Analysis I (MATH 301) [Lebl]
  6. University of Idaho (historical course listing)
    1. Math 420 Complex Variables [Brown/Churchill] 
    2. Math 471 Introduction to Analysis I [Fitzpatrick]
    3. Math 472 Introduction to Analysis II [Fitzpatrick]
  7. Texas Tech (certificate program; non-degree option)
    1. MATH 5366 - Introduction to Analysis I 
    2. MATH 5367 - Introduction to Analysis II
  8. Chadron State College, Nebraska
    1. MATH 434 INTRODUCTORY ANALYSIS [Lay]
  9. Indiana University East (certificate program; non-degree option)
    1. MATH-M 413 Intro to Real Analysis I [Lay]
    2. MATH-M 414 Intro to Real Analysis II [Lay]
    3. MATH-M 511 Real Variables I [Lay]
    4. MATH-M 512 Real Variables II  
There are a number of other courses that probably have more restricted enrollment. Stanford's Online High School program lists courses in real analysis [Ross] and complex analysis [Brown/Churchill]. Wow!  But I doubt many people reading this are high-schoolers trying to take analysis. Other programs may or may not allow enrollment a la carte, as the courses are part of undergraduate math programs or graduate programs for math teachers - presumably, any mathematics department offering an online math degree must offer an analysis course. But if analysis isn't a core requirement, it may not be offered regularly.
  1. University of Houston (M.A. program)
    1. MATH 5333: ANALYSIS [Lay]
    2. MATH 5334: COMPLEX ANALYSIS
  2.  Texas A&M (teaching M.S.) 
    1. Math 615 - Intro to Classical Analysis
There are a few other courses at private schools. I only list these separately because, unlike Harvard/Stanford and state schools, I have no idea about what these schools are about. They do at least appear to be not-for-profit schools.
  1. Ottawa University, Kansas
    1. MAT 45143 Introduction to Real Analysis 
  2. Southern New Hampshire University
    1. MAT 470 Real Analysis
Bookmarks
  1. Quora
  2. Reddit 
  3. PhysicsForums 
  4. Stack Exchange

Monday, September 26, 2016

the maximum page limit in OmniPage

I use OmniPage Ultimate (v19) all the time. I am slowly scanning in my academic book collection and converting the books into searchable PDF. If I was a hard-core FOSS guy I would use some kind of Tesseract-based workflow, but OmniPage just sort of works (once you get the settings tweaked right), it's super-fast, and I'm already time-starved rather than cash-starved. (I do wish it did a better job with equations, though.)

The thing about academic books is that they are often very long, and the OmniPage Batch Manager (aka DocuDirect) tries to limit you to 500 pages per output file. It's actually ok if your input file is longer, but the output files are chunked into units of (at most) 500 pages, which you have to combine by hand. I imagine this may have to do with internal implementation details; or it may just be something to upsell you to their SDK product. I have no idea. They implement this by limiting what you can enter as the maximum page limit in the Batch Manager Options GUI (i.e., as a text entry validation).

However, now that PCs routinely have large memories, and 500 pages is not an exact limit anyway, you may want to risk the occasional OmniPage crash and use a larger page limit.

It turns out that the Batch Manager options are stored in:
C:\Users\<user>\AppData\Roaming\ScanSoft\OmniPage 19\Job\BMSettings.dat
If you can use a binary editor (like hexl-mode in emacs) you can identify which bytes correspond to the page limit and set them.
  1. MAKE A COPY of your current BMSettings.dat in case you trash your working copy and want your current settings back.
  2. Open the DocuDirect GUI and use the Options pane to set the page limit to something between 256 and 500 that is easily detectable as two hex bytes. For example, 319 = 0x1FF.  Fully close DocuDirect (this means exiting the OmniPage Agent in your toolbar, not just closing the GUI).
  3. Open BMSettings.dat in your binary editor and look for the string "SetPageLimit" (hexl-mode helpfully shows you the ASCII in a display next to the hex values). Then look for your two chosen hex bytes (e.g., 0x01 and 0xFF) in the dozen-odd bytes following it. Change those two bytes to something that represent a larger page limit, like 0x13 and 0x88 (= 5000), respectively.
  4. Now open the DocuDirect GUI again. You should now see your chosen page limit in the Options pane. (If you touch this value in the GUI, you will be subject to the "500 page" validation again, so don't do it.)



Wednesday, May 7, 2014

1dollarscan tips

This isn't about how to optimize your 1dollarscan bill.  This about the things in 1dollarscan that I've used that worked and others that take some care.  I haven't used all of their options so YMMV.
  1. As advertised, 1dollarscan does actually scan your book and does not give you a scan of somebody else's book.  So your own annotations and highlighting are preserved, for better or worse. 
  2. Very occasionally, they do skip pages and it doesn't get caught in QA. I haven't ever had this happen with regular pulp paper books and trade paperbacks, but it has happened with academic books that are thicker and often have either thick binding glue (for durability) or very thin paper (because of book length).
  3. Sometimes they do mis-cut books (i.e., cut off part of the content) and don't say anything about it.  Again, this is more likely with books with thick binding glue.  If you think there might be an issue, you might attach a note for the operator.
  4. If you are scanning an expensive/hard-to-find book, buy the rescan option ($0.20/set). They'll happily work with you to address scan quality issues (e.g., light scans).  And if they do mess up and miss some pages, there is no other way to fix that problem. The fact that they only keep the scanned originals for two weeks is kind of a drag if you dump big boxes of books on them, though, since it means you don't have much time to check all of those scans for completeness and quality.
  5. 600dpi scans are expensive ($3/set instead of $1/set) but useful for books with tiny print (like academic books).  However, it looks like 600dpi bilevel (b&w) scans are converted from 600dpi color scans, so if you have lots of free disk space and have the ability to reprocess the scans, it may be worth converting from color to bilevel yourself because you can probably do a better job than they can of filtering, thresholding (thresholding works better than dithering with scans that have more text/lines than patterns/halftones), and denoising the color images into bilevel images with clean text.  (I wish they would just do native bilevel scans, since in my experience a desktop scanner does a better job of producing a clean bilevel scan than an untuned software conversion from color.)  I say this from experience: the 600dpi bilevel scans I've received from them are noticeably worse than the bilevel images I've been able to produce from their 600dpi color scans.
  6. Also, if you get color scans you can remove highlighting and correct other color problems in the process of converting to a bilevel image, whereas if you select bilevel or grayscale scans you're stuck with whatever artifacts were introduced by their color scan conversion process (which was not customized for your book).  (Note that for scans of black & white documents, ordering grayscale doesn't save you any disk space over ordering color - they're using JPEG compression on both anyway.)
(None of this should be construed as a complaint about the service - I use their service all the time and have given them more money than I want to admit.  The fact that they exist means that I don't have to keep a kid-unfriendly guillotine in my house, do continuous cleaning/maintenance on my desktop scanner, etc.)