From 453a55e9c8da9e03fbb735946bacd51fefd36380 Mon Sep 17 00:00:00 2001 From: Freddy Vulto Date: Fri, 2 Oct 2009 10:53:00 +0200 Subject: [PATCH] (testsuite) Fix assert_bash_exec to match prompt Added documentation to "Testing" chapter: - Running tests via cron - Specifying bash binary --- doc/html~/main.html | 49 ++++++++++++++++++++++++++++---------------- doc/testing.txt | 38 ++++++++++++++++++++++++++++++++-- test/lib/library.exp | 2 +- 3 files changed, 68 insertions(+), 21 deletions(-) diff --git a/doc/html~/main.html b/doc/html~/main.html index 86ce6d81..a4e5d6ef 100644 --- a/doc/html~/main.html +++ b/doc/html~/main.html @@ -1,5 +1,5 @@ -Bash-completion

Bash-completion

Freddy Vulto (FVu)

Revision History
Revision 1.0Mar 2009FV(

Table of Contents

Preface
I. Bash completion
1. Environment variables
II. Coding Style Guide
2. Introduction
III. Automated testing
3. Introduction
4. Installing DejaGnu
4.1. Debian/Ubuntu
5. Structure
5.1. Main areas (DejaGnu tools)
5.2. Running the tests
6. Maintenance
6.1. Adding a completion test
6.2. Fixing a completion test
6.3. Fixing a unit test
7. Rationale
7.1. Naming conventions
7.1.1. Test suite or testsuite
7.1.2. script/generate
8. Reference
9. Test context
9.1. What happens when tests are run?
9.1.1. completion
9.1.2. install
9.1.3. unit
9.2. bashrc
9.3. inputrc
Index

List of Examples

9.1. What happens when completion tests are run?
9.2. What happens when install tests are run?
9.3. What happens when unit tests are run?

Preface

Bash completion extends bashs standard completion behavior to achieve +Bash-completion

Bash-completion

Freddy Vulto (FVu)

Revision History
Revision 1.0Mar 2009FV(

Table of Contents

Preface
I. Bash completion
1. Environment variables
II. Coding Style Guide
2. Introduction
III. Automated testing
3. Introduction
4. Installing dependencies
4.1. Debian/Ubuntu
4.2. Fedora/RHEL/CentOS
5. Structure
5.1. Main areas (DejaGnu tools)
6. Running the tests
6.1. Running tests via cron
6.2. Specifying bash binary
7. Maintenance
7.1. Adding a completion test
7.2. Fixing a completion test
7.3. Fixing a unit test
8. Rationale
8.1. Naming conventions
8.1.1. Test suite or testsuite
8.1.2. script/generate
9. Reference
10. Test context
10.1. What happens when tests are run?
10.1.1. completion
10.1.2. install
10.1.3. unit
10.2. bashrc
10.3. inputrc
Index

List of Examples

10.1. What happens when completion tests are run?
10.2. What happens when install tests are run?
10.3. What happens when unit tests are run?

Preface

Bash completion extends bashs standard completion behavior to achieve complex command lines with just a few keystrokes. This project was conceived to produce programmable completion routines for the most common Linux/UNIX commands, reducing the amount of typing sysadmins @@ -33,7 +33,7 @@ and programmers need to do on a daily basis.

Part II. Coding Style Guide

Table of Contents

2. Introduction

Chapter 2. Introduction

This document attempts to explain the basic styles and patterns that are used in the bash completion. New code should try to conform to these standards so that it is as easy to maintain as existing code. Of course every rule has an exception, but it’s important to know the rules nonetheless!

This is particularly directed at people new to the bash completion codebase, who are in the process of getting their code reviewed. Before getting a review, please read over this document and make sure your code conforms to the recommendations here.

Part III. Automated testing

Table of Contents

3. Introduction
4. Installing DejaGnu
4.1. Debian/Ubuntu
5. Structure
5.1. Main areas (DejaGnu tools)
5.2. Running the tests
6. Maintenance
6.1. Adding a completion test
6.2. Fixing a completion test
6.3. Fixing a unit test
7. Rationale
7.1. Naming conventions
7.1.1. Test suite or testsuite
7.1.2. script/generate
8. Reference
9. Test context
9.1. What happens when tests are run?
9.1.1. completion
9.1.2. install
9.1.3. unit
9.2. bashrc
9.3. inputrc

Chapter 3. Introduction

The bash-completion package contains an automated test suite. Running the tests should help verifying that bash-completion works as expected. The tests are also very helpful in uncovering software regressions at an early stage.

The bash-completion test suite is written on top of the DejaGnu testing framework. DejaGnu is written in Expect, which in turn uses Tcl — Tool command language.

Chapter 4. Installing DejaGnu

Table of Contents

4.1. Debian/Ubuntu

Installing DejaGnu should be easy using your local package manager.

4.1. Debian/Ubuntu

On Debian/Ubuntu you can use apt-get:

sudo apt-get install dejagnu

This should also install the necessary expect and tcl packages.

Chapter 5. Structure

Table of Contents

5.1. Main areas (DejaGnu tools)
5.2. Running the tests

5.1. Main areas (DejaGnu tools)

The tests are grouped into different areas, called tool in DejaGnu:

+

Part II. Coding Style Guide

Table of Contents

2. Introduction

Chapter 2. Introduction

This document attempts to explain the basic styles and patterns that are used in the bash completion. New code should try to conform to these standards so that it is as easy to maintain as existing code. Of course every rule has an exception, but it’s important to know the rules nonetheless!

This is particularly directed at people new to the bash completion codebase, who are in the process of getting their code reviewed. Before getting a review, please read over this document and make sure your code conforms to the recommendations here.

Part III. Automated testing

Table of Contents

3. Introduction
4. Installing dependencies
4.1. Debian/Ubuntu
4.2. Fedora/RHEL/CentOS
5. Structure
5.1. Main areas (DejaGnu tools)
6. Running the tests
6.1. Running tests via cron
6.2. Specifying bash binary
7. Maintenance
7.1. Adding a completion test
7.2. Fixing a completion test
7.3. Fixing a unit test
8. Rationale
8.1. Naming conventions
8.1.1. Test suite or testsuite
8.1.2. script/generate
9. Reference
10. Test context
10.1. What happens when tests are run?
10.1.1. completion
10.1.2. install
10.1.3. unit
10.2. bashrc
10.3. inputrc

Chapter 3. Introduction

The bash-completion package contains an automated test suite. Running the tests should help verifying that bash-completion works as expected. The tests are also very helpful in uncovering software regressions at an early stage.

The bash-completion test suite is written on top of the DejaGnu testing framework. DejaGnu is written in Expect, which in turn uses Tcl — Tool command language.

Chapter 4. Installing dependencies

Table of Contents

4.1. Debian/Ubuntu
4.2. Fedora/RHEL/CentOS

Installing dependencies should be easy using your local package manager.

4.1. Debian/Ubuntu

On Debian/Ubuntu you can use apt-get:

sudo apt-get install dejagnu tcllib

This should also install the necessary expect and tcl packages.

4.2. Fedora/RHEL/CentOS

On Fedora and RHEL/CentOS (with EPEL) you can use yum:

sudo yum install dejagnu tcllib

This should also install the necessary expect and tcl packages.

Chapter 5. Structure

Table of Contents

5.1. Main areas (DejaGnu tools)

5.1. Main areas (DejaGnu tools)

The tests are grouped into different areas, called tool in DejaGnu:

completion
Functional tests per completion. @@ -45,11 +45,14 @@ and programmers need to do on a daily basis.

unit
Unit tests for bash-completion helper functions. -

Each tool has a slightly different way of loading the test fixtures, see Test context below.

5.2. Running the tests

The tests are run by calling runtest in the test directory:

runtest --outdir log --tool completion
+

Each tool has a slightly different way of loading the test fixtures, see Test context below.

Chapter 6. Running the tests

Table of Contents

6.1. Running tests via cron
6.2. Specifying bash binary

The tests are run by calling runtest in the test directory:

runtest --outdir log --tool completion
 runtest --outdir log --tool install
 runtest --outdir log --tool unit

The commands above are already wrapped up in shell scripts within the test directory:

./runCompletion
 ./runInstall
-./runUnit

Chapter 6. Maintenance

Table of Contents

6.1. Adding a completion test
6.2. Fixing a completion test
6.3. Fixing a unit test

6.1. Adding a completion test

You can run cd test && ./generate to add a test.

6.2. Fixing a completion test

Let’s consider this real-life example where an ssh completion bug is fixed. First you’re triggered by unsuccessful tests:

$ ./runCompletion
+./runUnit

6.1. Running tests via cron

The test suite requires a connected terminal (tty). When invoked via cron, no tty is connected and the test suite may respond with this error:

can't read "multipass_name": no such variable

To run the tests successfully via cron, connect a terminal by redirecting stdin from a tty, e.g. /dev/tty50:

./runUnit < /dev/tty50

If the process doesn’t run as root, root will have to change the owner and +permissions of the tty.

In Linux, you can press alt-Fx or ctrl-alt-Fx to switch the console from +/dev/tty1 to tty7. There are many more /dev/tty* which are not accessed via +function keys. To be safe, use a tty greater than tty7?

6.2. Specifying bash binary

The test suite standard uses bash as found in the tcl path (/bin/bash). Using --tool_exec you can specify which bash binary you want to run the test suite against, e.g.:

./runUnit --tool_exec /opt/bash-4.0/bin/bash

Chapter 7. Maintenance

Table of Contents

7.1. Adding a completion test
7.2. Fixing a completion test
7.3. Fixing a unit test

7.1. Adding a completion test

You can run cd test && ./generate to add a test.

7.2. Fixing a completion test

Let’s consider this real-life example where an ssh completion bug is fixed. First you’re triggered by unsuccessful tests:

$ ./runCompletion
 ...
                 === completion Summary ===
 
@@ -78,7 +81,7 @@ expect: does "ssh -F fixtures/ssh/config ls\r\nls           lsattr       lsb_rel
 ...
                 === completion Summary ===
 
-# of expected passes            2

6.3. Fixing a unit test

Now let’s consider a unit test failure. First you’re triggered by unsuccessful tests:

$ ./runUnit
+# of expected passes            2

7.3. Fixing a unit test

Now let’s consider a unit test failure. First you’re triggered by unsuccessful tests:

$ ./runUnit
 ...
                 === unit Summary ===
 
@@ -90,9 +93,9 @@ FAIL: Environment should stay clean

In this case it appears

Now, if we haven’t already figured out the problem, we can have a detailed look in dbg.log to find out what’s going wrong. Open dbg.log and search for UNRESOLVED (or FAIL if that’s what you’re looking for):

FAIL: Environment should stay clean

From there, search up for the first line saying:

expect: does "..." match regular expression "..."

This tells you where the actual output differs from the expected output. In this case it looks like the the function _known_hosts_real is unexpectedly modifying global variables cur and flag. In case you need to modify the test:

$ vi lib/unit/_known_hosts_real.exp

Chapter 7. Rationale

Table of Contents

7.1. Naming conventions
7.1.1. Test suite or testsuite
7.1.2. script/generate

7.1. Naming conventions

7.1.1. Test suite or testsuite

The primary Wikipedia page is called +# of unexpected failures 1

Now, if we haven’t already figured out the problem, we can have a detailed look in dbg.log to find out what’s going wrong. Open dbg.log and search for UNRESOLVED (or FAIL if that’s what you’re looking for):

FAIL: Environment should stay clean

From there, search up for the first line saying:

expect: does "..." match regular expression "..."

This tells you where the actual output differs from the expected output. In this case it looks like the the function _known_hosts_real is unexpectedly modifying global variables cur and flag. In case you need to modify the test:

$ vi lib/unit/_known_hosts_real.exp

Chapter 8. Rationale

Table of Contents

8.1. Naming conventions
8.1.1. Test suite or testsuite
8.1.2. script/generate

8.1. Naming conventions

8.1.1. Test suite or testsuite

The primary Wikipedia page is called test suite and not testsuite, so that’s what this -document sticks to.

7.1.2. script/generate

The name and location of this code generation script come from Ruby on Rails' script/generate.

Chapter 8. Reference

Within test scripts the following library functions can be used:

Chapter 9. Test context

Table of Contents

9.1. What happens when tests are run?
9.1.1. completion
9.1.2. install
9.1.3. unit
9.2. bashrc
9.3. inputrc

The test environment needs to be put to fixed states when testing. For instance the bash prompt (PS1) is set to the current test directory, followed by an ampersand (@). The default settings for bash reside in config/bashrc and config/inputrc.

For each tool (completion, install, unit) a slightly different context is in effect.

9.1. What happens when tests are run?

9.1.1. completion

When the completions are tested, invoking DejaGnu will result in a call to completion_start() which in turn will start bash --rcfile config/bashrc.

Example 9.1. What happens when completion tests are run?

               | runtest --tool completion
+document sticks to.

8.1.2. script/generate

The name and location of this code generation script come from Ruby on Rails' script/generate.

Chapter 9. Reference

Within test scripts the following library functions can be used:

Chapter 10. Test context

Table of Contents

10.1. What happens when tests are run?
10.1.1. completion
10.1.2. install
10.1.3. unit
10.2. bashrc
10.3. inputrc

The test environment needs to be put to fixed states when testing. For instance the bash prompt (PS1) is set to the current test directory, followed by an ampersand (@). The default settings for bash reside in config/bashrc and config/inputrc.

For each tool (completion, install, unit) a slightly different context is in effect.

10.1. What happens when tests are run?

10.1.1. completion

When the completions are tested, invoking DejaGnu will result in a call to completion_start() which in turn will start bash --rcfile config/bashrc.

Example 10.1. What happens when completion tests are run?

               | runtest --tool completion
                V
     +----------+-----------+
     |  lib/completion.exp  |
@@ -117,7 +120,7 @@ document sticks to.

Setting up bash once within completion_start() has the speed advantage that bash - and bash-completion - need only initialize once when testing multiple completions, e.g.:

    runtest --tool completion alias.exp cd.exp

9.1.2. install

Example 9.2. What happens when install tests are run?

                 | runtest --tool install
+    +----------------------+

Setting up bash once within completion_start() has the speed advantage that bash - and bash-completion - need only initialize once when testing multiple completions, e.g.:

    runtest --tool completion alias.exp cd.exp

10.1.2. install

Example 10.2. What happens when install tests are run?

                 | runtest --tool install
                  V
             +----+----+
             | DejaGnu |
@@ -131,7 +134,7 @@ document sticks to.

9.1.3. unit

Example 9.3. What happens when unit tests are run?

               | runtest --tool unit
+    +-------------------------+

10.1.3. unit

Example 10.3. What happens when unit tests are run?

               | runtest --tool unit
                V
           +----+----+
           | DejaGnu |
@@ -141,10 +144,7 @@ document sticks to.

9.2. bashrc

This is the bash configuration file (bashrc) used for testing:

# -*- mode: shell-script; sh-basic-offset: 8; indent-tabs-mode: t -*-
-# ex: ts=8 sw=8 noet filetype=sh
-#
-# bashrc file for DejaGnu testsuite
+    +----------------------+

10.2. bashrc

This is the bash configuration file (bashrc) used for testing:

# bashrc file for DejaGnu testsuite
 
         # Use emacs key bindings
 set -o emacs
@@ -171,10 +171,15 @@ unset -v \
     COMP_TAR_INTERNAL_PATHS
 
         # Load bash testsuite helper functions
-. lib/library.sh

9.3. inputrc

This is the readline configuration file (inputrc) used for testing:

# -*- mode: shell-script; sh-basic-offset: 8; indent-tabs-mode: t -*-
-# ex: ts=8 sw=8 noet filetype=sh
-#
-# Readline init file for DejaGnu testsuite
+. lib/library.sh
+
+# Local variables:
+# mode: shell-script
+# sh-basic-offset: 8
+# sh-indent-comment: t
+# indent-tabs-mode: t
+# End:
+# ex: ts=8 sw=8 noet filetype=sh

10.3. inputrc

This is the readline configuration file (inputrc) used for testing:

# Readline init file for DejaGnu testsuite
 # See: info readline
 
         # Press TAB once (instead of twice) to auto-complete
@@ -186,4 +191,12 @@ set completion-query-items -1
         # Display completions sorted horizontally, not vertically
 set print-completions-horizontally on
         # Don't use pager when showing completions
-set page-completions off

Index

+set page-completions off + +# Local variables: +# mode: shell-script +# sh-basic-offset: 8 +# sh-indent-comment: t +# indent-tabs-mode: t +# End: +# ex: ts=8 sw=8 noet filetype=sh

Index

diff --git a/doc/testing.txt b/doc/testing.txt index 14cf6ff4..b54779a1 100644 --- a/doc/testing.txt +++ b/doc/testing.txt @@ -21,7 +21,7 @@ Debian/Ubuntu On Debian/Ubuntu you can use `apt-get`: ------------- -sudo apt-get install dejagnu +sudo apt-get install dejagnu tcllib ------------- This should also install the necessary `expect` and `tcl` packages. @@ -54,8 +54,11 @@ The tests are grouped into different areas, called _tool_ in DejaGnu: Each tool has a slightly different way of loading the test fixtures, see <> below. + + + Running the tests -~~~~~~~~~~~~~~~~~ +----------------- The tests are run by calling `runtest` in the test directory: ----------------------- @@ -71,6 +74,37 @@ The commands above are already wrapped up in shell scripts within the `test` dir ----------------------- +Running tests via cron +~~~~~~~~~~~~~~~~~~~~~~ + +The test suite requires a connected terminal (tty). When invoked via cron, no tty is connected and the test suite may respond with this error: +--------------------------------------------- +can't read "multipass_name": no such variable +--------------------------------------------- + +To run the tests successfully via cron, connect a terminal by redirecting stdin from a tty, e.g. /dev/tty50: +---------------------- +./runUnit < /dev/tty50 +---------------------- + +If the process doesn't run as root, root will have to change the owner and +permissions of the tty. + +In Linux, you can press alt-Fx or ctrl-alt-Fx to switch the console from +/dev/tty1 to tty7. There are many more /dev/tty* which are not accessed via +function keys. To be safe, use a tty greater than tty7? + + +Specifying bash binary +~~~~~~~~~~~~~~~~~~~~~~ + +The test suite standard uses `bash` as found in the tcl path (/bin/bash). Using `--tool_exec` you can specify which bash binary you want to run the test suite against, e.g.: + +---------------- +./runUnit --tool_exec /opt/bash-4.0/bin/bash +---------------- + + Maintenance diff --git a/test/lib/library.exp b/test/lib/library.exp index 10d4ddac..4ca6b96c 100644 --- a/test/lib/library.exp +++ b/test/lib/library.exp @@ -31,7 +31,7 @@ proc assert_bash_exec {{aCmd ""} {title ""} {prompt /@} {out -1}} { # Remove $prompt suffix from output set results [ string range $results 0 [ - expr [string length $results] - [string length "/@"] - 1 + expr [string length $results] - [string length $prompt] - 1 ] ] if {$out == -1 && [string length $results] > 0} {