Subject: ANNOUNCE: Comm.pl, a new expect/chat2.pl pkg, Beta1
From: [email-masked] (Eric Arnold)
Date: 8 Oct 1995 20:24:39 GMT
Newsgroups: comp.lang.perl.announce,comp.lang.perl.misc

Hello, all.  Recently, Tom made the deeply disturbing heretical statement:

  > You should probably just give up on perl for expect stuff.  
  > 
  > Anyone caring to dispute this suggestion is welcome to provide an
  > expect-like module in perl5.  I mean a real one, fully portable, 
  > delightfully elegant and easy to use.
  > 
  > Failing that, use tcl -- and suffer.
  > 
  > --tom
  > -- 
  > Tom Christiansen      Perl Consultant, Gamer, Hiker   [email-masked]

So, "caring to dispute this", since I just know Perl can be used simply and
easily to do expect-like stuff, with the much better Perl language support,
I'm providing the package I've been keeping up to date over the last
couple years, with modifications to address:

  "a real one"
  "fully portable"
  "delightfully elegant and easy to use"

The, "module in perl5", bit is open to discussion, as is all of it.  Comments
and suggestions are welcome.

-Eric

What:

  "Comm.pl", a successor to "chat2.pl", providing a high level interface to:

    - STREAM/UDP sockets
    - pseudo-tty control
    - Revamped "expect"-like functionality (plus "interact").
    - ioctl/stty terminal mode control

  other things:

    - Support for BSD & SVR4 flavors (so far, tested with SunOS4.x, Solaris2.x)
    - sample client/server and expect scripts

Why:

  - "chat2.pl" doesn't have SVR4 support.  People have been posted a
    lot with questions about getting SOCK_STREAM right, or how to get
    a pty.

  - The Expect pattern/action pair only confuses people the way that chat2
    emulated it.  A Perl "expect" should be simple; it should not be
    trying to execute code given as parameters; that's what the Perl
    interpreter is for.  Also, the TCL Expect program has a whole lot
    of stuff that we don't really need because it's available via other
    methods in Perl (i.e. "send_tty").

Issues:

  ----------------------------------------
  What should "expect" look like?
  ----------------------------------------

    - All a Perl expect function really needs to do is to look through
      input (non-blocking, without need for newlines) and return
      whether a pattern was found or not.

      I've changed the interface to:

	( $match, $err, $before, $after ) = 
		  &expect( $fh, $timeout, 'regexp1', 'regexp2' );

	die "expect failed, err($err), last seen($before)" if $err;

	$_ = $match ; SWITCH : 
	{
	  /regexp1/ && do { print $fh "something\n" ; next SWITCH };
	  /regexp2/ && do { print $fh "other\n" ; next SWITCH };
	  # default
	}
	# or if .. else; whatever you want.

      or for more simple situations:

	&expect( $fh, $timeout, 'regexp1' ) || die "regexp1 failed, err=$!";
	print $fh "something\n";
	&expect( $fh, $timeout, 'regexp2' ) || die "regexp2 failed, err=$!";
	print $fh "something else\n";

    - When something doesn't work right, people are often confused as to
      what is going wrong.  That's why I added the "$before" and
      "$after" variables, so people can [be encouraged to] see what's
      being read or discarded, and why their pattern isn't matching.

    - The "interact()" interface is:

	( $match, $err ) = &interact( "optional string patterns for STDIN",..., 
			$Proc_pty_handle, "optional regex patterns", ... );

      I don't know how offensive it is to have $Proc_pty_handle be the
      delimeter between the string patterns and the regex patterns.  I
      wanted to provide triggers for both the user (STDIN) and the process
      ($Proc_pty_handle), and that seemed expedient.
    
    - Nit:  I wanted to be able to set $! from within "expect()" so
      that when the user used the short (!wantarray) form, I could
      still give error feedback.  I found that $! can only be set to
      valid values for "errno".  Then I found that I couldn't set and
      make it stick until the results had been evaluated by the
      caller.  Rats.

  ----------------------------------------
  Should it be a Perl5 module?  
  Does it need to be object oriented?  
  Abandon Perl4 support?
  ----------------------------------------

    I can't decide whether to abandon Perl4, yet.  Nothing about
    Comm.pl, as it is, really screams for Perl5 functionality, though
    it would be convenient to use "Exporter.pm", "Socket.pm", and array
    refs for "interact()".  I have a real bias toward Perl5, but I don't
    know whether the world shares it.  Speak up if you're using Perl4,
    and you can't upgrade for some reason.

  ----------------------------------------
  Portability?
  ----------------------------------------

    Pseudo-tty portability is a problem:  one version uses [some] BSD
    backward compatibility in Solaris, and the other version uses some
    scarey bit hacking, which might not be any more portable:

      # ptsname - not portable probably: assumes 14 bit minor numbers.
      # only a problem if it's less than 14bits, I think.
      print "rdev=$rdev\n";
      $rdev &= (1<<14) - 1;
      $slave = "/dev/pts/$rdev";

    The main problem being that Perl has no interface to ptm functions
    like "grantpt" and "ptsname".  Making a C linkable module seems
    like overkill.  Is there a POSIX standard for this, which could place
    it in the POSIX module?

    Perl5 does have a good socket module, which should be used, I guess,
    but it doesn't have Streams stuff like "I_PUSH", etc.  So, system
    dependencies will still have to live in the Comm.pl package.  

    This then gets back to the question of whether to write it as Perl5-only,
    or retain the current Perl4/Perl5 compatibility.

------------------------------------cut here------------------------------------

$HEADER = <<'EOF';

                        "Comm.pl"

This is a free library of IPC goodies.  There is no warrenty, but I'd
be happy to get ideas for improvements.

		- [email-masked]

It's been tested with Perl4/Perl5 and SunOS4.x and Solaris2.3 - 2.5.
It's normally put into a file and "require"'d, but can also be simply
concatinated to the end of some other perl script.  If you do that, use:
	require "Comm.pl" unless defined &Comm'init;

A lot was borrowed from "chat2.pl"(Randal L. Schwartz), and then
diverged as its goals became generalized client/server IPC, support for
SVR4/Solaris, and to facilitate my "shelltalk" program.  Since then, I/we've
been using it for all sorts of stuff.

See the end of this file for example programs demonstrating usage.

Function summary:

  (Remember to use prefixes (i.e. "&Comm'init") for anything not exported.)
  (All file handles passed up from these functions are exported into the
  caller's package.)

  &Comm'init();		# Under normal circumstances, "require" will call init
			# for you.

  &Comm'init(1.2);	# If first arg is numberic, it specifies a desired 
			# version for compatibility

  &Comm'init("funcname",...);	# Tell it to export specified function(s)

    # By default, init() will export these functions:
      open_port, open_listen, open_udp_port, open_proc ,
      send_to, recv_from, accept_it, select_it ,
      expect , interact , 
      close_noshutdown, close_it, stty_sane, stty_raw, stty_ioctl
    
    # If you don't want this to happen, you'll have to comment out the call
    # to "&init;" in this package.  Sorry.

  open_port :
  ---------

    # Open a STREAM socket connection to a host:
    $handle = &open_port($host, $port, $timeout);

  open_listen :
  -----------

    # Open a STREAM listen socket on your host:

    $handle = &open_listen( $port );

    # Or you can specify the $host if you need to listen on an address 
    # other than:  `uname -n` (E.g. if you have a second ethernet)

    $handle = &open_listen( $host, $port );

  select_it :
  ---------

    # Give it a timeout and a list of handles, and it tells you which ones
    # have data ready (or some condition, like EOF).  It's called "select_it"
    # so it won't clash with "select".

    @ready_handles = &select_it( $timeout, $handle1, $handle2, ..... );

  accept_it :
  ---------

    # Complement to "open_listen":

    ( $new_handle, $rem_host ) = &accept_it( $handle );

  open_proc :
  ---------

    # Set up a pseudo-tty, and start "$Command" running in it.

    ( $Proc_pty_handle, $Proc_tty_handle, $Proc_pid ) = 
	&open_proc($Command);

  wait_nohang :
  -----------

    # Does a portable wait4/waitpid.  Used mostly internally.  Not exported.
    &Comm'wait_nohang;

  expect :
  ------

    # This function scans an input stream for a pattern, without blocking,
    # a la "sysread()".
    #
    # Patterns are scanned in the order given, so later patterns can contain
    # general defaults that won't be examined unless the earlier patterns
    # have failed.  Be careful of timing problems, however.  If you specify
    # a very general pattern later in the list, it might match undesireably
    # if a partial packet of data is received.  

    # $err can contain "TIMEOUT" or "EOF".  $before will contain anything
    # before $match, or everything seen if $err is set.  $after contains
    # everything discarded after $match.

    ( $match, $err, $before, $after ) = 
		&expect( $fh, $timeout, 'regexp1', 'regexp2' );
    # or
    $match = &expect( $fh, $timeout, 'regexp1', 'regexp2', ... );

    # You can give it any file handle, but remember to pass the type glob,
    # so it can be used in a different package namespace:

    open(RDR, "somecommand|");
    # or
    &open3(WRT, RDR, ERR, 'somecommand' );
    ( $match, $err, $before, $after ) = expect( *RDR, 1, $pattern );

  interact :
  --------

    # This connects a process opened with "open_proc()" to the user via
    # STDIN, and allows them to "interact".
    #
    # You specify patterns to trigger return of control to your script, which
    # can be matched either in STDIN or the process file handle.  The
    # $Proc_pty_handle serves as a delimeter between string patterns for STDIN,
    # and regex patterns for $Proc_pty_handle.
    #
    # Any pattern matched for STDIN isn't sent to the process.  Therefore,
    # patterns for STDIN are treated only as strings (it's too hard to
    # figure out partial matches on a regex).

    &stty_raw(STDIN);
    ( $match, $err ) = &interact( "optional string patterns for STDIN", ..., 
			$Proc_pty_handle, "optional regex patterns", ... );
    &stty_sane(STDIN);

  open_udp_port :
  -------------

    # Open a UDP port.  There are more variations possible for UDP ports,
    # so the arguments you can give are more variable:

    # Just open a UDP socket on your host.  You'll have to use "send_to"
    # if you want to do more than read from it.  If you don't specify
    # a host (i.e. ""), it uses `uname -n`.  If you don't specify a port
    # (i.e. 0 ) it will assign a port for you.

    $handle = &open_udp_port( "", 0 )

    # Set up a connected UPD port.  You can "print" to this handle:

    $handle = &open_udp_port( "local_addr", 5050, "remotehost", 5050 )
      etc.

  send_to :
  -------

    # This is a convenience interface function to "send()".  It packs up
    # the appropriate binary structure from the remote address and port.

    &send_to( $handle, $buf, $flags, $remote_addr, $remote_port );

  sockaddr_struct :
  ---------------

    # If you're really pressed for performance, you can save a packed struct,
    # and use "send()", which saves some overhead with each call:

    $remote_sockaddr = &sockaddr_struct( $remote_addr, $remote_port );
    send( $handle, $buf, 0, $remote_sockaddr ) || die "send $!";

  recv_from :
  ---------

    # This is another convenience function, which unpacks the returned struct
    # from "recv()", and tells you what address and port the data came from.
    # You have to pass it a glob (i.e. *buf) so it can fill that variable with
    # the data.

    ( $addr, $port ) = &recv_from($handle, *buf, 10000, 0);

  close_it :
  --------

    # This will either call "shutdown()" if the handle is a socket type,
    # or kill the child process if the handle is a pty type.

    &close_it( $handle )

  close_noshutdown :
  ----------------

    # Use this when a parent forks a child to handle a request on a socket
    # file handle.  The parent would like to close the file handle, but
    # leave the socket alive so the child can continue to read/write it
    # (the child inherited the file handle and therefore the socket) 

    &close_noshutdown( $handle );

  stty_sane , stty_raw, stty_ioctl :
  --------------------------------
    
    # These use "stty" to set the terminal modes the first time through,
    # because "stty" is easy and portable.  The binary ioctl struct
    # containing the modes is then cached for subsequent calls to
    # "ioctl()", which is much faster for switching between modes, but is
    # a pain to make portable.  "$Proc_tty_handle" can be "STDIN".

    &stty_sane( $Proc_tty_handle );
    &stty_raw( $Proc_tty_handle );

    # "stty_raw/sane" use "stty_ioctl".  See the header for
    # "get_ioctl_from_stty" for more information about getting and saving
    # binary ioctl structs.

    &stty_ioctl( $Proc_tty_handle, "stty intr '^c'" );

  open_dupsockethandle, open_dupprochandle :
  ----------------------------------------

    # I don't know if anybody will ever use these.  They dup file
    # handles, which will fool the utilities here into thinking that
    # your file handle (created from some other package) was actually
    # created by a routine in here.

    &open_dupsockethandle($handle);
    &open_dupprochandle($handle);

Misc:

  $Debug is "inherited" from $main'Debug

Portability bug-a-boos:

  - There are two versions of getpty().  getpty_svr4() tries to do the
    right SVR4 thing, although without direct access to the right function
    calls :-(.  getpty() also works for SVR4/Solaris, using some partial BSD
    backward compatibility.  Neither is all too clean.

  - Once I decide to bite the bullet, and give up support for perl4, it
    should use "use Socket" for all the socket defines like SOCK_STREAM.
    There's no getting around putting some of the other defines directly
    in here, I think (i.e. I_STR).

Bugs:

  - There used to be some odd problems with the value for SOCK_STREAM,
    depending on whether it was perl4 or perl5 and whether it was compiled
    under SunOS or Solaris, but it seems to be better now.

[....code deleted for post to comp.lang.perl.announce. See comp.lang.perl.misc
or soon, http://franz.ww.TU-Berlin.DE:80/modulelist/ .]