#!/bin/sh # ---------------------------------------------------------------------------- # Run the VisIt test suite # # Each .py file in the tests directory contains tags indicating the class(es) # of tests the file belongs to. Additionally, each test can be run in one # of several modes. Classes are used to select WHICH tests to run. Modes are # used to decide HOW to run those tests. # # CLASSES are identified in two places; in the test .py files # to identify which tests support belong to which classes and on the # command line to run-test to select which classes of tests to run. # # MODES is used to control global modes in which tests are performed. The # current list of supported modes is "serial","parallel" and # "scalable". The runtest script accepts a # "-m(odes)" argument which identifies the mode(s) in which tests should be # performed. It is a comma separated list of modes. Note that it does # not make sense to pass a list of modes to runtest with mutually exclusive # modes. For example, it would not make sense to pass "serial,parallel". # To run both modes, runtest would have to be executed twice. # # Finally, note that regardless # of the mode of a test, the resultant image SHOULD be identical to its # associated baseline. If we ever add a mode to do testing using hardware # rendering, this may not be the case due to minor differences in how the # hardware and software may render. # # CLASSES is simple and arbitrary selection mechanism for selecting classes # of tests to run. Each test identifies in a comment block in the .py file # for the test the classes that it is. Then, the -classes argument to runtest # will find and run only those tests in the specified list of classes. The # initial list of classes is "nightly" # # Finally, if you runtest as you always have, it will behave as it always # has. That is, it will execute all tests sequentially with no attempt at # scalable rendering or parallel. # # Programmer: Jeremy Meredith # Date: April 17, 2002 # # Modifications: # Jeremy Meredith, Thu Jul 18 23:50:04 PDT 2002 # Moved tests into categorized subdirectories. Made successful test # cases output as such without needing the -verbose flag. # Set the display to elysium for now, until full -nowin mode works. # # Jeremy Meredith, Fri Jul 19 16:16:15 PDT 2002 # Allowed specific files to be passed on the commandline. # Made the main page be links to subpages, one per test file. # Added a return code for small differences. # # Jeremy Meredith, Thu Aug 29 15:11:01 PDT 2002 # Added copying of log file to html directory. Reversed order of # past results. # # Jeremy Meredith, Thu Nov 14 13:34:51 PST 2002 # Added code to make sure permissions were set up correctly. # # Jeremy Meredith, Fri Nov 15 12:54:51 PST 2002 # Added a "summary" file. Allowed running without the PIL library. # # Jeremy Meredith, Mon Apr 14 09:57:21 PDT 2003 # Propagate -verbose to the CLI. # # Jeremy Meredith, Mon Jun 9 17:52:25 PDT 2003 # Added "." to the python path. This lets us pick up modules # from the test directory. # # Mark C. Miller, 02Jul03 # Added modes and classes functionality. Classes allows you to select # tests by class and modes allows you to select global operating modes. # Initial implementation supports "Scalable_Rendering" and "Parallel" # # Jeremy Meredith, Mon Aug 18 15:19:23 PDT 2003 # Added timings. # # Mark C. Miller, 18Dec03 # Removed use of modes as a selection criteria for tests. # Updated documentation in pre-amble to this shell script. # # Mark C. Miller, 15Jan04 # Modified html output to support multi-mode testing # Added a -skip command line option to specify a list of modes/tests to skip # Added a -limit command line option to specify per-test elapsed time limit # # Mark C. Miller, Sat Jan 31 22:58:37 PST 2004 # Added -dbg flag for debugging. Modified 'post' functionality to # do highlighting for previous test history. # # Jeremy Meredith, Mon Feb 2 12:40:57 PST 2004 # Removed my display from the script. Nowin mode is fully functional. # # Mark C. Miller, Tue Mar 2 17:06:46 PST 2004 # Added -i and -e command-line switches # # Brad Whitlock, Tue Mar 30 09:37:24 PDT 2004 # I added code to finish off the HTML page for a crashed test so we can # see how far it got before crashing. I also changed the HTML generation # here so tests that are skipped don't have links to HTML pages that did # not get generated due to a skipped test. # # Mark C. Miller, Tue Mar 30 08:37:26 PST 2004 # Added -dump option. Made most variables that represent CL arguments # be string valued and either empty string or specific value # # Brad Whitlock, Tue Mar 30 17:13:26 PST 2004 # Added -notrackmem option to turn off the nre memory usage tracking # features. # # Brad Whitlock, Thu Apr 8 15:56:19 PST 2004 # Added better support for deleting a lot of files. # # Jeremy Meredith, Wed Apr 28 10:51:54 PDT 2004 # Added check to make sure timings existed before trying to use them. # # Jeremy Meredith, Thu Aug 12 10:17:57 PDT 2004 # Added runtime in seconds to test case output. # # Mark C. Miller, Mon Nov 22 17:42:22 PST 2004 # Added -nonowin option # # Mark C. Miller, Mon Nov 29 18:52:41 PST 2004 # Changed how interactive and noPIL options are handled # # Hank Childs, Thu Mar 3 09:22:05 PST 2005 # Added dynamic load balancing (dlb) mode. # # Mark C. Miller, Mon Mar 21 19:01:04 PST 2005 # Made message about rebuilding test data conditional on verbose setting # # Jeremy Meredith, Fri Apr 1 10:16:28 PST 2005 # Made the non-purify post-to-host be sunburn instead of localhost. # # Jeremy Meredith, Fri Apr 8 10:49:19 PDT 2005 # Added -e to some echo commands. Apparently this needs to be explicitly # set on linux (i.e. -E is the default) instead of the old behavior we # are used to from the suns. # # Mark C. Miller, Thu Apr 21 09:37:41 PDT 2005 # Changed much of how purify runs work. Now, purify is dealt with like # other modes. There is no longer a -purify option. Also, eliminated # purify baseline errorcounts and much of the html output for it # Added code to setup for a purify run. # # Brad Whitlock, Tue Apr 26 10:54:51 PDT 2005 # I changed the test for dlb so it does not get executed when it should # not be executed. # # Mark C. Miller, Wed May 11 08:11:27 PDT 2005 # Added args for minor/major error threshold. Added "Passed" message # for case when pass has non-zero error but is below threshold # # Jeremy Meredith, Tue Jun 7 11:01:24 PDT 2005 # Added a "-lessverbose", or "-lv", flag that prints normal status # without all the progress messages. # # Hank Childs, Wed Aug 24 16:49:49 PDT 2005 # Add support for test specific config files. # # Jeremy Meredith, Wed Sep 7 12:05:23 PDT 2005 # Enhanced grep for finding runtimes for mode-specific tests. # # Mark C. Miller, Mon Jan 23 16:11:59 PST 2006 # Made it so gzip'd purify results can be viewed from netscape even # if you don't "post" the results to the website # # Mark C. Miller, Wed Aug 16 15:48:21 PDT 2006 # Added logic to save off core files to cores dir # # Mark C. Miller, Tue Sep 5 12:06:58 PDT 2006 # Added code to quiet output from re-makes of data and purify targets # Added copy of runtimes.txt to visit's public_html dir # # Mark C. Miller, Mon Nov 20 22:47:10 PST 2006 # In regenerating the top-level index.html file, made find commands faster # by maxing the depth at 1 and grep commands faster by stopping after # first match. # # Mark C. Miller, Tue Nov 28 23:13:08 PST 2006 # Changed CL args for controlling error threshold. Changed text of # some HTML from "Minor" and "Major" to "Acceptable" and "Unacceptable" # Removed DiffUsingCkSum stuff. Improved logic to use 'diffState' for # indicating state of difference result. # # Mark C. Miller, Tue Nov 28 23:50:15 PST 2006 # Changed maxdiff to meddiff # # Mark C. Miller, Wed Nov 29 08:19:52 PST 2006 # Changed meddiff to avgdiff # # Mark C. Miller, Sun Dec 3 12:20:11 PST 2006 # Added -vargs option to support all CL args that are merely passed onto # visit as opposed to being interpreted here (or in Testing.py). This # resulted in removing -dump, -dbg options. Also added ability for # individual tests to set env. variables. # # Mark C. Miller, Mon Mar 19 19:24:02 PDT 2007 # Backgrounded the actual VisIt test jobs that get run. Added limitTest # shell script to limit the amount of time any test runs. Needed to # re-direct output in the main loop in a bit of a funky way. # # Mark C. Miller, Tue Mar 20 09:48:03 PDT 2007 # Removed re-direction at end of 'for t in $testcases' loop. That was # causing the whole loop to get executed in a subshell. Removed use # of 'kill -9' on the limitTest script (see comment there). Set default # timeout to be 5 minutes (300 seconds). Removed artificially low # 120 second time out used to confirm limitTest was working. # # Mark C. Miller, Tue Mar 27 08:39:55 PDT 2007 # Removed backgrounding of 'interactive' runs as well as limiting the time # for them to run. Removed setting of PYTHONPATH var to include # /usr/gapps/visit/python+/sunos_57/python2.1 # # Hank Childs, Thu May 17 10:23:50 PDT 2007 # Make executable path be "../src/bin/visit", instead of "../bin/visit", # since that is the right path for the new Subversion repo. # # Jeremy Meredith, Thu May 17 14:44:15 EDT 2007 # I made the html, diff, and current directories be forcibly removed # and re-created. (This was previously unsafe because they were # clearcase elements, but I have removed them from SVN control.) # # Mark C. Miller, Mon Aug 27 18:00:06 PDT 2007 # Change handling to test-specific enviornment to instead of creating # a file for each test, put the relevant strings into a shell var that # is expanded on the test line. # # Mark C. Miller, Tue Aug 28 16:23:20 PDT 2007 # Added ability for test to specify how to invoke VisIt to run itself. # Added setting of $nowin="" for interactive mode # # Mark C. Miller, Mon Dec 17 16:06:48 PST 2007 # Added ability to skip specific test cases in a .py file # # Mark C. Miller, Tue Jan 29 14:59:48 PST 2008 # Modified setting of PYTHONPATH to include # # Mark C. Miller, Wed Feb 20 21:18:29 PST 2008 # Added logic to limit engine process ids # # Mark C. Miller, Tue Mar 4 18:36:42 PST 2008 # Reduced time to wait to get engine pids from 25 to 10 seconds; check # for existence of engine_pids.txt file before trying to read it. # # Mark C. Miller, Fri May 16 08:39:57 PDT 2008 # Fixed problems with detecting and saving cores. # # Tom Fogal, Fri Jul 11 14:05:08 EDT 2008 # I added a mode for IceT rendering. # # Mark C. Miller, Tue Aug 5 16:31:43 PDT 2008 # Increased default time limit for a given .py file test from 300 to # 600 seconds. # # Mark C. Miller, Mon Dec 8 23:36:31 PST 2008 # I added logic to provide useful error messages when it looks like the # test ran 'ok' but developer negected to call Exit() from .py file. # # Cyrus Harrison, Tue Dec 8 09:59:42 PST 2009 # Updated to reflect reorg of data directory. # # Tom Fogal, Thu Mar 25 14:43:08 MDT 2010 # Acceptable differences ("-pixdiff X") shouldn't count as a failure. # # Mark C. Miller, Sun Mar 28 22:59:47 PDT 2010 # Added a 'cifrendly' mode where if a test fails, it is run a second time # with verbose and debugging ON. Verbose output gets capture in ci logs # automatically and debug logs are cat'd so they appear there too. # # Mark C. Miller, Mon Mar 29 08:40:44 PDT 2010 # Made output of debug logs in cifriendly mode a little more efficient # and human friendly. Only those logs that contain indications of error # (e.g. 'SIG' or 'EXCEPT' or 'Except) are cat'd. When a log is cat'd, each # line is pre-pended with the test name and log name. # # Mark C. Miller, Thu Apr 1 14:26:36 PDT 2010 # Added ulimit command. # # Eric Brugger, Thu Apr 22 12:46:21 PDT 2010 # I modified the script so that the summary for each test will read # "Succeeded With Skips" or "Acceptable With Skips" when a partially # skipped test is successful and "Unacceptable" when a partially skipped # test has a non-skipped test fail. Previously all these cases simply # had "Partially Skipped". # # Eric Brugger, Fri Apr 23 14:38:05 PDT 2010 # I modified the parsing of the modes command so that it only considers # the key words serial, parallel, and scalable in an arbitrary order. # Specifically, if it sees the keyword serial, it runs the serial test, # if it sees the keyword parallel in the absence of the keyword scalable, # it runs the parallel test, and if it sees the keywords parallel and # scalable, it runs the test scalable,parallel test. This change allows # it to ignore any arbitrary host name. # # Mark C. Miller, Wed May 5 08:28:46 PDT 2010 # Removed some bash-specific coding I introduced in Mar 29, 2010 update. # Removed use of an array shell variable and 'let' statements. # # Mark C. Miller, Fri Jul 16 10:18:12 PDT 2010 # Allow just scalable or parallel to be parsed from VISIT_TEST_MODES # # Mark C. Miller, Mon Nov 1 12:15:04 PDT 2010 # Pass '-exec-timeout' and '-idle-timeout' instead of just '-timeout' # # Cyrus Harrison, Fri Jan 13 13:31:07 PST 2012 # Added hooks for beta report generation. # # Cyrus Harrison, Fri Apr 6 17:49:29 PDT 2012 # Use explicit call to python for beta report gen. # # Eric Brugger, Wed Aug 1 11:59:16 PDT 2012 # I corrected a bug where all the tests that were executed after a test # generated a core file were flagged as having failed. This was caused # by the "file" command now returning the full path of the executable # instead of just the executable name, causing the "mv" command to fail. # Now the script just takes the portion of the name after the last "\". # # Cyrus Harrison, Mon Aug 13 16:48:14 PDT 2012 # Guts are now python, use visit_test_suite.py as main entry point. # # Mark C. Miller, Tue Oct 9 10:42:24 PDT 2012 # Surround $@ passed to python with double quotes ("). Otherwise, # something like '--vargs "-debug 5"' winds up getting passed as 3 sep. # words: --vargs, "-debug, 5" # ---------------------------------------------------------------------------- # Call python test suite: python visit_test_suite.py "$@"