Mercurial > emacs
annotate doc/misc/ert.texi @ 112379:b14f98859016
Merge: build from gnulib a bit better; document autoreconf.
author | Paul Eggert <eggert@cs.ucla.edu> |
---|---|
date | Wed, 19 Jan 2011 15:59:52 -0800 |
parents | 98399365dc54 |
children |
rev | line source |
---|---|
112201
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
1 \input texinfo |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
2 @c %**start of header |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
3 @setfilename ../../info/ert |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
4 @settitle Emacs Lisp Regression Testing |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
5 @c %**end of header |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
6 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
7 @dircategory Emacs |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
8 @direntry |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
9 * ERT: (ert). Emacs Lisp Regression Testing. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
10 @end direntry |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
11 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
12 @copying |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
13 Copyright @copyright{} 2008, 2010, 2011 Free Software Foundation, Inc. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
14 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
15 @quotation |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
16 Permission is granted to copy, distribute and/or modify this document |
112355
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
17 under the terms of the GNU Free Documentation License, Version 1.3 or |
112201
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
18 any later version published by the Free Software Foundation; with no |
112355
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
19 Invariant Sections, with the Front-Cover texts being ``A GNU Manual,'' |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
20 and with the Back-Cover Texts as in (a) below. A copy of the license |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
21 is included in the section entitled ``GNU Free Documentation License'' |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
22 in the Emacs manual. |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
23 |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
24 (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
25 modify this GNU manual. Buying copies from the FSF supports it in |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
26 developing GNU and promoting software freedom.'' |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
27 |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
28 This document is part of a collection distributed under the GNU Free |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
29 Documentation License. If you want to distribute this document |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
30 separately from the collection, you can do so by adding a copy of the |
98399365dc54
* doc/misc/ert.texi: Relicense under GFDL 1.3+, and standardize license notice.
Glenn Morris <rgm@gnu.org>
parents:
112201
diff
changeset
|
31 license to the document, as described in section 6 of the license. |
112201
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
32 @end quotation |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
33 @end copying |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
34 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
35 @node Top, Introduction, (dir), (dir) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
36 @top ERT: Emacs Lisp Regression Testing |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
37 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
38 ERT is a tool for automated testing in Emacs Lisp. Its main features |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
39 are facilities for defining tests, running them and reporting the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
40 results, and for debugging test failures interactively. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
41 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
42 ERT is similar to tools for other environments such as JUnit, but has |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
43 unique features that take advantage of the dynamic and interactive |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
44 nature of Emacs. Despite its name, it works well both for test-driven |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
45 development (see |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
46 @url{http://en.wikipedia.org/wiki/Test-driven_development}) and for |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
47 traditional software development methods. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
48 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
49 @menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
50 * Introduction:: A simple example of an ERT test. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
51 * How to Run Tests:: Run tests in your Emacs or from the command line. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
52 * How to Write Tests:: How to add tests to your Emacs Lisp code. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
53 * How to Debug Tests:: What to do if a test fails. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
54 * Extending ERT:: ERT is extensible in several ways. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
55 * Other Testing Concepts:: Features not in ERT. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
56 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
57 @detailmenu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
58 --- The Detailed Node Listing --- |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
59 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
60 How to Run Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
61 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
62 * Running Tests Interactively:: Run tests in your current Emacs. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
63 * Running Tests in Batch Mode:: Run tests in emacs -Q. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
64 * Test Selectors:: Choose which tests to run. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
65 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
66 How to Write Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
67 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
68 * The @code{should} Macro:: A powerful way to express assertions. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
69 * Expected Failures:: Tests for known bugs. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
70 * Tests and Their Environment:: Don't depend on customizations; no side effects. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
71 * Useful Techniques:: Some examples. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
72 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
73 How to Debug Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
74 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
75 * Understanding Explanations:: How ERT gives details on why an assertion failed. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
76 * Interactive Debugging:: Tools available in the ERT results buffer. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
77 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
78 Extending ERT |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
79 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
80 * Defining Explanation Functions:: Teach ERT about more predicates. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
81 * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
82 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
83 Other Testing Concepts |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
84 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
85 * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
86 * Fixtures and Test Suites:: How ERT differs from tools for other languages. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
87 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
88 @end detailmenu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
89 @end menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
90 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
91 @node Introduction, How to Run Tests, Top, Top |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
92 @chapter Introduction |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
93 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
94 ERT allows you to define @emph{tests} in addition to functions, |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
95 macros, variables, and the other usual Lisp constructs. Tests are |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
96 simply Lisp code --- code that invokes other code and checks whether |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
97 it behaves as expected. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
98 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
99 ERT keeps track of the tests that are defined and provides convenient |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
100 commands to run them to verify whether the definitions that are |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
101 currently loaded in Emacs pass the tests. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
102 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
103 Some Lisp files have comments like the following (adapted from the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
104 package @code{pp.el}): |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
105 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
106 @lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
107 ;; (pp-to-string '(quote quote)) ; expected: "'quote" |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
108 ;; (pp-to-string '((quote a) (quote b))) ; expected: "('a 'b)\n" |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
109 ;; (pp-to-string '('a 'b)) ; same as above |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
110 @end lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
111 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
112 The code contained in these comments can be evaluated from time to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
113 time to compare the output with the expected output. ERT formalizes |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
114 this and introduces a common convention, which simplifies Emacs |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
115 development, since programmers no longer have to manually find and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
116 evaluate such comments. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
117 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
118 An ERT test definition equivalent to the above comments is this: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
119 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
120 @lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
121 (ert-deftest pp-test-quote () |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
122 "Tests the rendering of `quote' symbols in `pp-to-string'." |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
123 (should (equal (pp-to-string '(quote quote)) "'quote")) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
124 (should (equal (pp-to-string '((quote a) (quote b))) "('a 'b)\n")) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
125 (should (equal (pp-to-string '('a 'b)) "('a 'b)\n"))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
126 @end lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
127 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
128 If you know @code{defun}, the syntax of @code{ert-deftest} should look |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
129 familiar: This example defines a test named @code{pp-test-quote} that |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
130 will pass if the three calls to @code{equal} all return true |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
131 (non-nil). |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
132 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
133 @code{should} is a macro with the same meaning as @code{assert} but |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
134 better error reporting. @xref{The @code{should} Macro}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
135 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
136 Each test should have a name that describes what functionality the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
137 test tests. Test names can be chosen arbitrarily --- they are in a |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
138 namespace separate from functions and variables --- but should follow |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
139 the usual Emacs Lisp convention of having a prefix that indicates |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
140 which package they belong to. Test names are displayed by ERT when |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
141 reporting failures and can be used when selecting which tests to run. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
142 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
143 The empty parentheses @code{()} in the first line don't currently have |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
144 any meaning and are reserved for future extension. They also make |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
145 @code{ert-deftest}'s syntax more similar to @code{defun}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
146 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
147 The docstring describes what feature this test tests. When running |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
148 tests interactively, the first line of the docstring is displayed for |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
149 tests that fail, so it is good if the first line makes sense on its |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
150 own. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
151 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
152 The body of a test can be arbitrary Lisp code. It should have as few |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
153 side effects as possible; each test should be written to clean up |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
154 after itself, leaving Emacs in the same state as it was before the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
155 test. Tests should clean up even if they fail. @xref{Tests and Their |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
156 Environment}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
157 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
158 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
159 @node How to Run Tests, How to Write Tests, Introduction, Top |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
160 @chapter How to Run Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
161 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
162 You can run tests either in the Emacs you are working in, or on the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
163 command line in a separate Emacs process in batch mode (i.e., with no |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
164 user interface). The former mode is convenient during interactive |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
165 development, the latter is useful to make sure that tests pass |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
166 independently of your customizations, allows tests to be invoked from |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
167 makefiles and scripts to be written that run tests in several |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
168 different Emacs versions. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
169 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
170 @menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
171 * Running Tests Interactively:: Run tests in your current Emacs. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
172 * Running Tests in Batch Mode:: Run tests in emacs -Q. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
173 * Test Selectors:: Choose which tests to run. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
174 @end menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
175 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
176 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
177 @node Running Tests Interactively, Running Tests in Batch Mode, How to Run Tests, How to Run Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
178 @section Running Tests Interactively |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
179 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
180 You can run the tests that are currently defined in your Emacs with |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
181 the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. ERT will pop |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
182 up a new buffer, the ERT results buffer, showing the results of the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
183 tests run. It looks like this: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
184 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
185 @example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
186 Selector: t |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
187 Passed: 31 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
188 Failed: 2 (2 unexpected) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
189 Total: 33/33 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
190 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
191 Started at: 2008-09-11 08:39:25-0700 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
192 Finished. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
193 Finished at: 2008-09-11 08:39:27-0700 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
194 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
195 FF............................... |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
196 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
197 F addition-test |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
198 (ert-test-failed |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
199 ((should |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
200 (= |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
201 (+ 1 2) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
202 4)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
203 :form |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
204 (= 3 4) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
205 :value nil)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
206 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
207 F list-test |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
208 (ert-test-failed |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
209 ((should |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
210 (equal |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
211 (list 'a 'b 'c) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
212 '(a b d))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
213 :form |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
214 (equal |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
215 (a b c) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
216 (a b d)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
217 :value nil :explanation |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
218 (list-elt 2 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
219 (different-atoms c d)))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
220 @end example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
221 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
222 At the top, there is a summary of the results: We ran all tests in the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
223 current Emacs (@code{Selector: t}), 31 of them passed, and 2 failed |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
224 unexpectedly. @xref{Expected Failures}, for an explanation of the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
225 term @emph{unexpected} in this context. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
226 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
227 The line of dots and @code{F}s is a progress bar where each character |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
228 represents one test; it fills while the tests are running. A dot |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
229 means that the test passed, an @code{F} means that it failed. Below |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
230 the progress bar, ERT shows details about each test that had an |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
231 unexpected result. In the example above, there are two failures, both |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
232 due to failed @code{should} forms. @xref{Understanding Explanations}, |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
233 for more details. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
234 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
235 In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
236 buttons. Each name of a function or macro in this buffer is a button; |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
237 moving point to it and typing @kbd{RET} jumps to its definition. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
238 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
239 Pressing @kbd{r} re-runs the test near point on its own. Pressing |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
240 @kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
241 definition of the test near point (@kbd{RET} has the same effect if |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
242 point is on the name of the test). On a failed test, @kbd{b} shows |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
243 the backtrace of the failure. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
244 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
245 @kbd{l} shows the list of @code{should} forms executed in the test. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
246 If any messages were generated (with the Lisp function @code{message}) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
247 in a test or any of the code that it invoked, @kbd{m} will show them. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
248 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
249 By default, long expressions in the failure details are abbreviated |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
250 using @code{print-length} and @code{print-level}. Pressing @kbd{L} |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
251 while point is on a test failure will increase the limits to show more |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
252 of the expression. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
253 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
254 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
255 @node Running Tests in Batch Mode, Test Selectors, Running Tests Interactively, How to Run Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
256 @section Running Tests in Batch Mode |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
257 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
258 ERT supports automated invocations from the command line or from |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
259 scripts or makefiles. There are two functions for this purpose, |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
260 @code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
261 They can be used like this: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
262 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
263 @example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
264 emacs -batch -L /path/to/ert -l ert.el -l my-tests.el -f ert-run-tests-batch-and-exit |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
265 @end example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
266 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
267 This command will start up Emacs in batch mode, load ERT, load |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
268 @code{my-tests.el}, and run all tests defined in it. It will exit |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
269 with a zero exit status if all tests passed, or nonzero if any tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
270 failed or if anything else went wrong. It will also print progress |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
271 messages and error diagnostics to standard output. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
272 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
273 You may need additional @code{-L} flags to ensure that |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
274 @code{my-tests.el} and all the files that it requires are on your |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
275 @code{load-path}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
276 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
277 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
278 @node Test Selectors, , Running Tests in Batch Mode, How to Run Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
279 @section Test Selectors |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
280 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
281 Functions like @code{ert} accept a @emph{test selector}, a Lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
282 expression specifying a set of tests. Test selector syntax is similar |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
283 to Common Lisp's type specifier syntax: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
284 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
285 @itemize |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
286 @item @code{nil} selects no tests. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
287 @item @code{t} selects all tests. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
288 @item @code{:new} selects all tests that have not been run yet. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
289 @item @code{:failed} and @code{:passed} select tests according to their most recent result. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
290 @item @code{:expected}, @code{:unexpected} select tests according to their most recent result. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
291 @item A string selects all tests that have a name that matches the string, a regexp. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
292 @item A test selects that test. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
293 @item A symbol selects the test that the symbol names. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
294 @item @code{(member TESTS...)} selects TESTS, a list of tests or symbols naming tests. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
295 @item @code{(eql TEST)} selects TEST, a test or a symbol naming a test. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
296 @item @code{(and SELECTORS...)} selects the tests that match all SELECTORS. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
297 @item @code{(or SELECTORS...)} selects the tests that match any SELECTOR. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
298 @item @code{(not SELECTOR)} selects all tests that do not match SELECTOR. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
299 @item @code{(tag TAG)} selects all tests that have TAG on their tags list. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
300 @item @code{(satisfies PREDICATE)} Selects all tests that satisfy PREDICATE. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
301 @end itemize |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
302 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
303 Selectors that are frequently useful when selecting tests to run |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
304 include @code{t} to run all tests that are currently defined in Emacs, |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
305 @code{"^foo-"} to run all tests in package @code{foo} --- this assumes |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
306 that package @code{foo} uses the prefix @code{foo-} for its test names |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
307 ---, result-based selectors such as @code{(or :new :unexpected)} to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
308 run all tests that have either not run yet or that had an unexpected |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
309 result in the last run, and tag-based selectors such as @code{(not |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
310 (tag :causes-redisplay))} to run all tests that are not tagged |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
311 @code{:causes-redisplay}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
312 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
313 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
314 @node How to Write Tests, How to Debug Tests, How to Run Tests, Top |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
315 @chapter How to Write Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
316 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
317 ERT lets you define tests in the same way you define functions. You |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
318 can type @code{ert-deftest} forms in a buffer and evaluate them there |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
319 with @code{eval-defun} or @code{compile-defun}, or you can save the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
320 file and load it, optionally byte-compiling it first. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
321 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
322 Just like @code{find-function} is only able to find where a function |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
323 was defined if the function was loaded from a file, ERT is only able |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
324 to find where a test was defined if the test was loaded from a file. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
325 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
326 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
327 @menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
328 * The @code{should} Macro:: A powerful way to express assertions. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
329 * Expected Failures:: Tests for known bugs. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
330 * Tests and Their Environment:: Don't depend on customizations; no side effects. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
331 * Useful Techniques:: Some examples. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
332 @end menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
333 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
334 @node The @code{should} Macro, Expected Failures, How to Write Tests, How to Write Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
335 @section The @code{should} Macro |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
336 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
337 Test bodies can include arbitrary code; but to be useful, they need to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
338 have checks whether the code being tested (or @emph{code under test}) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
339 does what it is supposed to do. The macro @code{should} is similar to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
340 @code{assert} from the cl package, but analyzes its argument form and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
341 records information that ERT can display to help debugging. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
342 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
343 This test definition |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
344 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
345 @lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
346 (ert-deftest addition-test () |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
347 (should (= (+ 1 2) 4))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
348 @end lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
349 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
350 will produce this output when run via @kbd{M-x ert}: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
351 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
352 @example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
353 F addition-test |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
354 (ert-test-failed |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
355 ((should |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
356 (= |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
357 (+ 1 2) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
358 4)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
359 :form |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
360 (= 3 4) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
361 :value nil)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
362 @end example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
363 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
364 In this example, @code{should} recorded the fact that (= (+ 1 2) 4) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
365 reduced to (= 3 4) before it reduced to nil. When debugging why the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
366 test failed, it helps to know that the function @code{+} returned 3 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
367 here. ERT records the return value for any predicate called directly |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
368 within @code{should}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
369 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
370 In addition to @code{should}, ERT provides @code{should-not}, which |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
371 checks that the predicate returns nil, and @code{should-error}, which |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
372 checks that the form called within it signals an error. An example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
373 use of @code{should-error}: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
374 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
375 @lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
376 (ert-deftest test-divide-by-zero () |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
377 (should-error (/ 1 0) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
378 :type 'arith-error)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
379 @end lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
380 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
381 This checks that dividing one by zero signals an error of type |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
382 @code{arith-error}. The @code{:type} argument to @code{should-error} |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
383 is optional; if absent, any type of error is accepted. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
384 @code{should-error} returns an error description of the error that was |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
385 signalled, to allow additional checks to be made. The error |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
386 description has the format @code{(ERROR-SYMBOL . DATA)}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
387 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
388 There is no @code{should-not-error} macro since tests that signal an |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
389 error fail anyway, so @code{should-not-error} is effectively the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
390 default. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
391 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
392 @xref{Understanding Explanations}, for more details on what |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
393 @code{should} reports. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
394 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
395 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
396 @node Expected Failures, Tests and Their Environment, The @code{should} Macro, How to Write Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
397 @section Expected Failures |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
398 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
399 Some bugs are complicated to fix or not very important and are left as |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
400 @emph{known bugs}. If there is a test case that triggers the bug and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
401 fails, ERT will alert you of this failure every time you run all |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
402 tests. For known bugs, this alert is a distraction. The way to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
403 suppress it is to add @code{:expected-result :failed} to the test |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
404 definition: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
405 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
406 @lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
407 (ert-deftest future-bug () |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
408 "Test `time-forward' with negative arguments. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
409 Since this functionality isn't implemented yet, the test is known to fail." |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
410 :expected-result :failed |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
411 (time-forward -1)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
412 @end lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
413 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
414 ERT will still display a small @code{f} in the progress bar as a |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
415 reminder that there is a known bug, and will count the test as failed, |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
416 but it will be quiet about it otherwise. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
417 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
418 An alternative to marking the test as a known failure this way is to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
419 delete the test. This is a good idea if there is no intent to fix it, |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
420 i.e., if the behavior that was formerly considered a bug has become an |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
421 accepted feature. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
422 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
423 In general, however, it can be useful to keep tests that are known to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
424 fail. If someone wants to fix the bug, they will have a very good |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
425 starting point: an automated test case that reproduces the bug. This |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
426 makes it much easier to fix the bug, demonstrate that it is fixed, and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
427 prevent future regressions. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
428 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
429 ERT displays the same kind of alerts for tests that pass unexpectedly |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
430 that it displays for unexpected failures. This way, if you make code |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
431 changes that happen to fix a bug that you weren't aware of, you will |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
432 know to remove the @code{:expected-result} clause of that test and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
433 close the corresponding bug report, if any. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
434 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
435 Since @code{:expected-result} evaluates its argument when the test is |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
436 loaded, tests can be marked as known failures only on certain Emacs |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
437 versions, specific architectures, etc.: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
438 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
439 @lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
440 (ert-deftest foo () |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
441 "A test that is expected to fail on Emacs 23 but succeed elsewhere." |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
442 :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
443 :failed |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
444 :passed) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
445 ...) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
446 @end lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
447 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
448 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
449 @node Tests and Their Environment, Useful Techniques, Expected Failures, How to Write Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
450 @section Tests and Their Environment |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
451 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
452 The outcome of running a test should not depend on the current state |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
453 of the environment, and each test should leave its environment in the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
454 same state it found it in. In particular, a test should not depend on |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
455 any Emacs customization variables or hooks, and if it has to make any |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
456 changes to Emacs' state or state external to Emacs such as the file |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
457 system, it should undo these changes before it returns, regardless of |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
458 whether it passed or failed. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
459 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
460 Tests should not depend on the environment because any such |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
461 dependencies can make the test brittle or lead to failures that occur |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
462 only under certain circumstances and are hard to reproduce. Of |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
463 course, the code under test may have settings that affect its |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
464 behavior. In that case, it is best to make the test @code{let}-bind |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
465 all such settings variables to set up a specific configuration for the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
466 duration of the test. The test can also set up a number of different |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
467 configurations and run the code under test with each. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
468 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
469 Tests that have side effects on their environment should restore it to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
470 its original state because any side effects that persist after the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
471 test can disrupt the workflow of the programmer running the tests. If |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
472 the code under test has side effects on Emacs' current state, such as |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
473 on the current buffer or window configuration, the test should create |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
474 a temporary buffer for the code to manipulate (using |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
475 @code{with-temp-buffer}), or save and restore the window configuration |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
476 (using @code{save-window-excursion}), respectively. For aspects of |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
477 the state that can not be preserved with such macros, cleanup should |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
478 be performed with @code{unwind-protect}, to ensure that the cleanup |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
479 occurs even if the test fails. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
480 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
481 An exception to this are messages that the code under test prints with |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
482 @code{message} and similar logging; tests should not bother restoring |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
483 the @code{*Message*} buffer to its original state. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
484 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
485 The above guidelines imply that tests should avoid calling highly |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
486 customizable commands such as @code{find-file}, except, of course, if |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
487 such commands are what they want to test. The exact behavior of |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
488 @code{find-file} depends on many settings such as |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
489 @code{find-file-wildcards}, @code{enable-local-variables}, and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
490 @code{auto-mode-alist}. It is difficult to write a meaningful test if |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
491 its behavior can be affected by so many external factors. Also, |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
492 @code{find-file} has side effects that are hard to predict and thus |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
493 hard to undo: It may create a new buffer or may reuse an existing |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
494 buffer if one is already visiting the requested file; and it runs |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
495 @code{find-file-hook}, which can have arbitrary side effects. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
496 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
497 Instead, it is better to use lower-level mechanisms with simple and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
498 predictable semantics like @code{with-temp-buffer}, @code{insert} or |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
499 @code{insert-file-contents-literally}, and activating the desired mode |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
500 by calling the corresponding function directly --- after binding the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
501 hook variables to nil. This avoids the above problems. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
502 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
503 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
504 @node Useful Techniques, , Tests and Their Environment, How to Write Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
505 @section Useful Techniques when Writing Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
506 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
507 Testing simple functions that have no side effects and no dependencies |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
508 on their environment is easy. Such tests often look like this: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
509 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
510 @lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
511 (ert-deftest ert-test-mismatch () |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
512 (should (eql (ert--mismatch "" "") nil)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
513 (should (eql (ert--mismatch "" "a") 0)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
514 (should (eql (ert--mismatch "a" "a") nil)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
515 (should (eql (ert--mismatch "ab" "a") 1)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
516 (should (eql (ert--mismatch "Aa" "aA") 0)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
517 (should (eql (ert--mismatch '(a b c) '(a b d)) 2))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
518 @end lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
519 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
520 This test calls the function @code{ert--mismatch} several times with |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
521 various combinations of arguments and compares the return value to the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
522 expected return value. (Some programmers prefer @code{(should (eql |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
523 EXPECTED ACTUAL))} over the @code{(should (eql ACTUAL EXPECTED))} |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
524 shown here. ERT works either way.) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
525 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
526 Here's a more complicated test: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
527 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
528 @lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
529 (ert-deftest ert-test-record-backtrace () |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
530 (let ((test (make-ert-test :body (lambda () (ert-fail "foo"))))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
531 (let ((result (ert-run-test test))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
532 (should (ert-test-failed-p result)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
533 (with-temp-buffer |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
534 (ert--print-backtrace (ert-test-failed-backtrace result)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
535 (goto-char (point-min)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
536 (end-of-line) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
537 (let ((first-line (buffer-substring-no-properties (point-min) (point)))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
538 (should (equal first-line " signal(ert-test-failed (\"foo\"))"))))))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
539 @end lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
540 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
541 This test creates a test object using @code{make-ert-test} whose body |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
542 will immediately signal failure. It then runs that test and asserts |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
543 that it fails. Then, it creates a temporary buffer and invokes |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
544 @code{ert--print-backtrace} to print the backtrace of the failed test |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
545 to the current buffer. Finally, it extracts the first line from the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
546 buffer and asserts that it matches what we expect. It uses |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
547 @code{buffer-substring-no-properties} and @code{equal} to ignore text |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
548 properties; for a test that takes properties into account, |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
549 @code{buffer-substring} and @code{ert-equal-including-properties} |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
550 could be used instead. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
551 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
552 The reason why this test only checks the first line of the backtrace |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
553 is that the remainder of the backtrace is dependent on ERT's internals |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
554 as well as whether the code is running interpreted or compiled. By |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
555 looking only at the first line, the test checks a useful property |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
556 --- that the backtrace correctly captures the call to @code{signal} that |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
557 results from the call to @code{ert-fail} --- without being brittle. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
558 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
559 This example also shows that writing tests is much easier if the code |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
560 under test was structured with testing in mind. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
561 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
562 For example, if @code{ert-run-test} accepted only symbols that name |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
563 tests rather than test objects, the test would need a name for the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
564 failing test, which would have to be a temporary symbol generated with |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
565 @code{make-symbol}, to avoid side effects on Emacs' state. Choosing |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
566 the right interface for @code{ert-run-tests} allows the test to be |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
567 simpler. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
568 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
569 Similarly, if @code{ert--print-backtrace} printed the backtrace to a |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
570 buffer with a fixed name rather than the current buffer, it would be |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
571 much harder for the test to undo the side effect. Of course, some |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
572 code somewhere needs to pick the buffer name. But that logic is |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
573 independent of the logic that prints backtraces, and keeping them in |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
574 separate functions allows us to test them independently. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
575 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
576 A lot of code that you will encounter in Emacs was not written with |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
577 testing in mind. Sometimes, the easiest way to write tests for such |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
578 code is to restructure the code slightly to provide better interfaces |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
579 for testing. Usually, this makes the interfaces easier to use as |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
580 well. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
581 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
582 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
583 @node How to Debug Tests, Extending ERT, How to Write Tests, Top |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
584 @chapter How to Debug Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
585 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
586 This section describes how to use ERT's features to understand why |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
587 a test failed. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
588 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
589 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
590 @menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
591 * Understanding Explanations:: How ERT gives details on why an assertion failed. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
592 * Interactive Debugging:: Tools available in the ERT results buffer. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
593 @end menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
594 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
595 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
596 @node Understanding Explanations, Interactive Debugging, How to Debug Tests, How to Debug Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
597 @section Understanding Explanations |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
598 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
599 Failed @code{should} forms are reported like this: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
600 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
601 @example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
602 F addition-test |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
603 (ert-test-failed |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
604 ((should |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
605 (= |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
606 (+ 1 2) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
607 4)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
608 :form |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
609 (= 3 4) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
610 :value nil)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
611 @end example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
612 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
613 ERT shows what the @code{should} expression looked like and what |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
614 values its subexpressions had: The source code of the assertion was |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
615 @code{(should (= (+ 1 2) 4))}, which applied the function @code{=} to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
616 the arguments @code{3} and @code{4}, resulting in the value |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
617 @code{nil}. In this case, the test is wrong; it should expect 3 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
618 rather than 4. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
619 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
620 If a predicate like @code{equal} is used with @code{should}, ERT |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
621 provides a so-called @emph{explanation}: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
622 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
623 @example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
624 F list-test |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
625 (ert-test-failed |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
626 ((should |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
627 (equal |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
628 (list 'a 'b 'c) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
629 '(a b d))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
630 :form |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
631 (equal |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
632 (a b c) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
633 (a b d)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
634 :value nil :explanation |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
635 (list-elt 2 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
636 (different-atoms c d)))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
637 @end example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
638 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
639 In this case, the function @code{equal} was applied to the arguments |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
640 @code{(a b c)} and @code{(a b d)}. ERT's explanation shows that |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
641 the item at index 2 differs between the two lists; in one list, it is |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
642 the atom c, in the other, it is the atom d. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
643 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
644 In simple examples like the above, the explanation is unnecessary. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
645 But in cases where the difference is not immediately apparent, it can |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
646 save time: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
647 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
648 @example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
649 F test1 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
650 (ert-test-failed |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
651 ((should |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
652 (equal x y)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
653 :form |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
654 (equal a a) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
655 :value nil :explanation |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
656 (different-symbols-with-the-same-name a a))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
657 @end example |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
658 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
659 ERT only provides explanations for predicates that have an explanation |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
660 function registered. @xref{Defining Explanation Functions}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
661 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
662 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
663 @node Interactive Debugging, , Understanding Explanations, How to Debug Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
664 @section Interactive Debugging |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
665 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
666 Debugging failed tests works essentially the same way as debugging any |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
667 other problems with Lisp code. Here are a few tricks specific to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
668 tests: |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
669 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
670 @itemize |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
671 @item Re-run the failed test a few times to see if it fails in the same way |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
672 each time. It's good to find out whether the behavior is |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
673 deterministic before spending any time looking for a cause. In the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
674 ERT results buffer, @kbd{r} re-runs the selected test. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
675 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
676 @item Use @kbd{.} to jump to the source code of the test to find out what |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
677 exactly it does. Perhaps the test is broken rather than the code |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
678 under test. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
679 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
680 @item If the test contains a series of @code{should} forms and you can't |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
681 tell which one failed, use @kbd{l}, which shows you the list of all |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
682 @code{should} forms executed during the test before it failed. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
683 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
684 @item Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
685 the test with debugging enabled, this will enter the debugger and show |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
686 the backtrace as well; but the top few frames shown there will not be |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
687 relevant to you since they are ERT's own debugger hook. @kbd{b} |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
688 strips them out, so it is more convenient. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
689 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
690 @item If the test or the code under testing prints messages using |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
691 @code{message}, use @kbd{m} to see what messages it printed before it |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
692 failed. This can be useful to figure out how far it got. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
693 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
694 @item You can instrument tests for debugging the same way you instrument |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
695 @code{defun}s for debugging --- go to the source code of the test and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
696 type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
697 re-run the test with @kbd{r} or @kbd{d}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
698 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
699 @item If you have been editing and rearranging tests, it is possible that |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
700 ERT remembers an old test that you have since renamed or removed --- |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
701 renamings or removals of definitions in the source code leave around a |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
702 stray definition under the old name in the running process, this is a |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
703 common problem in Lisp. In such a situation, hit @kbd{D} to let ERT |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
704 forget about the obsolete test. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
705 @end itemize |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
706 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
707 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
708 @node Extending ERT, Other Testing Concepts, How to Debug Tests, Top |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
709 @chapter Extending ERT |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
710 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
711 There are several ways to add functionality to ERT. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
712 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
713 @menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
714 * Defining Explanation Functions:: Teach ERT about more predicates. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
715 * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
716 @end menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
717 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
718 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
719 @node Defining Explanation Functions, Low-Level Functions for Working with Tests, Extending ERT, Extending ERT |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
720 @section Defining Explanation Functions |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
721 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
722 The explanation function for a predicate is a function that takes the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
723 same arguments as the predicate and returns an @emph{explanation}. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
724 The explanation should explain why the predicate, when invoked with |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
725 the arguments given to the explanation function, returns the value |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
726 that it returns. The explanation can be any object but should have a |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
727 comprehensible printed representation. If the return value of the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
728 predicate needs no explanation for a given list of arguments, the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
729 explanation function should return nil. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
730 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
731 To associate an explanation function with a predicate, add the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
732 property @code{ert-explainer} to the symbol that names the predicate. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
733 The value of the property should be the symbol that names the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
734 explanation function. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
735 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
736 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
737 @node Low-Level Functions for Working with Tests, , Defining Explanation Functions, Extending ERT |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
738 @section Low-Level Functions for Working with Tests |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
739 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
740 Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch} |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
741 are implemented on top of the lower-level test handling code in the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
742 sections named ``Facilities for running a single test'', ``Test |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
743 selectors'', and ``Facilities for running a whole set of tests''. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
744 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
745 If you want to write code that works with ERT tests, you should take a |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
746 look at this lower-level code. Symbols that start with @code{ert--} |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
747 are internal to ERT, those that start with @code{ert-} but not |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
748 @code{ert--} are meant to be usable by other code. But there is no |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
749 mature API yet. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
750 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
751 Contributions to ERT are welcome. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
752 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
753 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
754 @node Other Testing Concepts, , Extending ERT, Top |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
755 @chapter Other Testing Concepts |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
756 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
757 For information on mocks, stubs, fixtures, or test suites, see below. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
758 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
759 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
760 @menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
761 * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
762 * Fixtures and Test Suites:: How ERT differs from tools for other languages. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
763 @end menu |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
764 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
765 @node Mocks and Stubs, Fixtures and Test Suites, Other Testing Concepts, Other Testing Concepts |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
766 @section Other Tools for Emacs Lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
767 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
768 Stubbing out functions or using so-called @emph{mocks} can make it |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
769 easier to write tests. See |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
770 @url{http://en.wikipedia.org/wiki/Mock_object} for an explanation of |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
771 the corresponding concepts in object-oriented languages. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
772 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
773 ERT does not have built-in support for mocks or stubs. The package |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
774 @code{el-mock} (see @url{http://www.emacswiki.org/emacs/el-mock.el}) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
775 offers mocks for Emacs Lisp and can be used in conjunction with ERT. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
776 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
777 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
778 @node Fixtures and Test Suites, , Mocks and Stubs, Other Testing Concepts |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
779 @section Fixtures and Test Suites |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
780 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
781 In many ways, ERT is similar to frameworks for other languages like |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
782 SUnit or JUnit. However, two features commonly found in such |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
783 frameworks are notably absent from ERT: fixtures and test suites. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
784 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
785 Fixtures, as used e.g. in SUnit or JUnit, are mainly used to provide |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
786 an environment for a set of tests, and consist of set-up and tear-down |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
787 functions. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
788 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
789 While fixtures are a useful syntactic simplification in other |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
790 languages, this does not apply to Lisp, where higher-order functions |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
791 and `unwind-protect' are available. One way to implement and use a |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
792 fixture in ERT is |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
793 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
794 @lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
795 (defun my-fixture (body) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
796 (unwind-protect |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
797 (progn [set up] |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
798 (funcall body)) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
799 [tear down])) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
800 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
801 (ert-deftest my-test () |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
802 (my-fixture |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
803 (lambda () |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
804 [test code]))) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
805 @end lisp |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
806 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
807 (Another way would be a @code{with-my-fixture} macro.) This solves |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
808 the set-up and tear-down part, and additionally allows any test |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
809 to use any combination of fixtures, so it is more flexible than what |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
810 other tools typically allow. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
811 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
812 If the test needs access to the environment the fixture sets up, the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
813 fixture can be modified to pass arguments to the body. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
814 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
815 These are well-known Lisp techniques. Special syntax for them could |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
816 be added but would provide only a minor simplification. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
817 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
818 (If you are interested in such syntax, note that splitting set-up and |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
819 tear-down into separate functions, like *Unit tools usually do, makes |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
820 it impossible to establish dynamic `let' bindings as part of the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
821 fixture. So, blindly imitating the way fixtures are implemented in |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
822 other languages would be counter-productive in Lisp.) |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
823 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
824 The purpose of test suites is to group related tests together. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
825 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
826 The most common use of this is to run just the tests for one |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
827 particular module. Since symbol prefixes are the usual way of |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
828 separating module namespaces in Emacs Lisp, test selectors already |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
829 solve this by allowing regexp matching on test names; e.g., the |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
830 selector "^ert-" selects ERT's self-tests. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
831 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
832 Other uses include grouping tests by their expected execution time to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
833 run quick tests during interactive development and slow tests less |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
834 frequently. This can be achieved with the @code{:tag} argument to |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
835 @code{ert-deftest} and @code{tag} test selectors. |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
836 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
837 @bye |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
838 |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
839 @c LocalWords: ERT Hagelberg Ohler JUnit namespace docstring ERT's |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
840 @c LocalWords: backtrace makefiles workflow backtraces API SUnit |
c5917804ecad
Add ERT, a tool for automated testing in Emacs Lisp.
Christian Ohler <ohler@gnu.org>
parents:
diff
changeset
|
841 @c LocalWords: subexpressions |