                      =============================
                        Likewise Storage Services
                      =============================

Build and Installation
======================

The build system is based on MakeKit (http://mkbuild.sourceforge.net)
and is fairly self-contained, so you shouldn't need anything in
particular other than gcc, GNU make, and the below prerequisites.
Note that some of the bundled libraries (krb5, openldap, etc.) require
recent versions of GNU autoconf, automake, and libtool.

While the source code is theoretically portable to a variety of
platforms, Linux and FreeBSD are the primary targets.

Build prerequisites:

  * RHEL/CentOS/Fedora:

    $ yum install gcc glibc-devel pam-devel ncurses-devel flex bison \
          rpm-build rpm-devel autoconf automake libtool libattr-devel

    If you are on a 64-bit system, also install the following
    (necessary for 32-bit pam/nsswitch modules):

    $ yum install glibc-devel.i686 libgcc.i686 pam-devel.i686

  * Ubuntu/Debian:

    $ sudo apt-get install build-essential fakeroot devscripts \
          debhelper autoconf automake libtool libncurses5-dev flex \
          bison libpam0g-dev libattr1-dev

    If you are on a 64-bit system, also install the following
    (necessary for 32-bit pam/nsswitch modules):

    $ sudo apt-get install ia32-libs libc6-dev-i386 gcc-multilib

    The default version of awk on some Ubuntu systems has a known bug
    which causes it to segfault when attempting to build.  You can
    work around this by installing GNU awk:

    $ sudo apt-get install gawk

    See https://launchpad.net/ubuntu/+source/mawk/+bug/23494 for
    details.

Build steps:

  * Create a build directory:

    $ mkdir debug && cd debug

  * Configure the build:

    $ ../configure --debug

  * Build packages:

    $ make -jX package

    Replace X with 2 times the number of logical CPUs present.

You will find installable packages in the
package/{deb,rpm}/likewise-storage directory.

If you do not want to install using packages, you can do the following
instead, but be sure to read the note below:

  $ make -jX
  $ sudo make install

You may view the complete set of configure options by running

  $ ../configure --help

Note: When not installing using a package, you will need to start lwsmd
and import registry files manually:

  $ sudo /etc/init.d/lwsmd start
  $ sudo sh -c 'for file in /opt/likewise/share/config/*.reg; do \
       /opt/likewise/bin/lwregshell import $file; \
       done'
  $ sudo /etc/init.d/lwsmd reload


Likewise Registry Service
=========================

The Likewise Registry Service (lwreg) is the configuration data store
used by all Likewise services.  You can view and modify the registry
settings by running /opt/likewise/bin/lwregshell as the root user.
For example:

  $ sudo /opt/likewise/bin/lwregshell
  \> cd HKEY_THIS_MACHINE\

  HKEY_THIS_MACHINE\> cd Services

  HKEY_THIS_MACHINE\Services> cd lsass

  HKEY_THIS_MACHINE\Services\lsass> ls

  [HKEY_THIS_MACHINE\Services\lsass]
     "Arguments"    REG_SZ          ""
     "Dependencies" REG_SZ          "netlogon lwio lwreg rdr  npfs"
     "Description"  REG_SZ          "Likewise Security and Authentication Subsystem"
     "FdLimit"      REG_DWORD       0x00000400 (1024)
     "Path"         REG_SZ          "/opt/likewise/lib/lw-svcm/lsass.so"

  [HKEY_THIS_MACHINE\Services\lsass\Accounts]
  [HKEY_THIS_MACHINE\Services\lsass\Parameters]
  [HKEY_THIS_MACHINE\Services\lsass\Privileges]

You can also use /opt/likewise/bin/lw-edit-reg to load up all or a
portion of the registry in your favorite text editor.  For example, to
edit all lsass settings:

  $ sudo /opt/likewise/bin/lw-edit-reg -s lsass

You can specify an editor with the EDITOR environment variable or the
-e option.


Likewise Service Manager
========================

The Likewise Service Manager (lwsmd) loads and manages all Likewise
services.

The lwsmd daemon itself is managed using a standard init script:

  $ sudo /etc/init.d/lwsmd start
  * Starting Likewise Service Manager: lwsmd              [ OK ]

By default, only lwsmd itself is started.  The "likewise" init script
instructs lwsmd to load all services configured to be automatically
started in the registry:

  $ sudo /etc/init.d/likewise start
  * Starting Likewise Storage Services                    [ OK ]

You can use the "lwsm" command to inspect and control services:

  $ /opt/likewise/bin/lwsm list
  lwreg       [service]    running (27463)
  dcerpc      [service]    stopped
  lsass       [service]    running (27491)
  lwio        [service]    running (27472)
  netlogon    [service]    running (27482)
  npfs        [driver]     running (27472)
  pvfs        [driver]     running (27472)
  rdr         [driver]     running (27472)
  srv         [driver]     running (27472)
  srvsvc      [service]    running (27527)

  $ sudo /opt/likewise/bin/lwsm stop lsass
  Stopping service reverse dependency: srvsvc
  Stopping service reverse dependency: srv
  Stopping service: lsass

  $ sudo /opt/likewise/bin/lwsm start srvsvc
  Starting service dependency: lsass
  Starting service dependency: srv
  Starting service: srvsvc

Likewise services are actually shared object (.so) files that are
dynamically loaded into container processes.  Services can be
configured to run in dedicated containers for fault tolerance
(the default) or to share containers to reduce memory usage
and IPC overhead.


Authentication
==============

Likewise Storage Services can run in a standalone mode or as
part of an Active Directory domain.

Use 'lw-lsa join <domain name> <user>' to join an AD domain.  The
domain name must be specified as a fully-qualified DNS name, and the
specified user must have sufficient rights to create machine accounts
in AD (e.g. an administrator).  You will be prompted for a password.
For example:

  $ sudo /opt/likewise/bin/lw-lsa join my.domain.com administrator
  administrator@MY.DOMAIN.COM's password:
  Joining to AD Domain: MY.DOMAIN.COM
  With Computer DNS Name: computer.my.domain.com

  SUCCESS!

After joining a domain, domain administrators will have full access to
set up shares as detailed in the next section.  If you want to
configure shares as a user who is not a domain admin, you will need
to add the user to the local administrators group as follows:

 $ sudo /opt/likewise/bin/lw-lsa mod-group --add-members 'MYDOMAIN\username' administrators
 Successfully modified group administrators

If not using Active Directory authentication, you may enable the local
administrator account (disabled by default) and give it a password:

 $ sudo /opt/likewise/bin/lw-lsa mod-user --set-password <password> --enable-user administrator

You can also add your own users and groups using the "add-user" and
"add-group" subcommands.  Pass "-h" for help information.


Storage
=======

Likewise provides an SMB and SMB2 protocol stack that supports
the following clients:

  * Windows: XP, Server 2003, Vista, Server 2008, 7, Server 2008 R2
  * Mac OS X: 10.5 and later
  * Linux: mount.cifs, smbclient (including gvfs and kio smb:// handlers)

No support is planned for Windows 9x or Mac OS X releases prior to 10.5.

The file server is composed of several drivers which are loaded by
lwio, the Likewise I/O Manager:

  * srv: the SMB/SMB2/CIFS protocol frontend
  * pvfs: the file system backend
  * npfs: ephemeral named pipe filesystem (used for RPC calls)

By default, the file server maps the C$ share to /lwcifs.  In order to
provide full-fidelity Windows semantics (security descriptors,
oplocks, etc.), many operations such as permissions checks are
performed in user-space and are not delegated to the underlying
filesystem.  As such, you should avoid accessing or manipulating files
directly through /lwcifs, and use an appropriate SMB/CIFS client
instead.

On Linux, the pvfs driver makes extensive use of extended attributes
to store Windows-like file metadata such as security descriptors and
DOS file attributes.  You will need to enable user extended attributes
on your root filesystem by adding the "user_xattr" option to the
appropriate entry in /etc/fstab.  You can then remount your filesystem
on the fly with:

  $ mount -o remount /

The "srvsvc" service implements the share enumeration and management
RPC interface, allowing you to create shares with standard tools such
as Windows MMC.  You can also use the "lwnet" command to manage them locally,
e.g:

  $ /opt/likewise/bin/lwnet share add newshare='C:\lwcifs\newshare' \
        --read-write --allow 'MYDOMAIN\mygroup'
  $ /opt/likewise/bin/lwnet share
    Share name  Resource            Remark
  -------------------------------------------------
    newshare    C:\lwcifs\newshare
    C$          C:\lwcifs           Default Share
    IPC$        C:\                 Remote IPC


Troubleshooting
===============

Likewise services have a flexible logging system that is centrally
controlled through lwsmd.  A log message has the following parts:

  * Timestamp: when the message was logged
  * Level: describes the type and importance of a message
      - always:    always logged
      - error:     error condition
      - warning:   non-critical but abnormal condition
      - info:      informational message
      - verbose:   developer-oriented informational message
      - debug:     message to assist in debugging
      - trace:     message tracing call execution
  * Facility: describes which subsystem generated the message.
    Examples:
      - lsass (messages from lsass service)
      - lwio-ipc (message from lwio IPC server)
  * Metadata: thread id, function, source file, and line number (may be omitted)
  * Message: the message itself

By default, all services log messages up to warning level to syslog
(DAEMON facility).  Logging settings are controlled on a per-container
basis.  Within a container, each facility may be configured with a
separate log target (syslog or plain file) and maximum log level.
There is also a default target and level for facilities that have not
been explicitly configured.  The configuration of running services may
be changed on the fly using 'lwsm set-log-level' and 'lwsm
set-log-target'.  These commands each begin with a service name (to
identify which container to configure) and a facility name.
Specifying '-' for the service name will configure lwsmd itself, and
specifying '-' for a facility name will configure the default target
or log level instead of a particular facility.  For example, the
following command will set the default log level to verbose in the
container which hosts the lsass service:

  $ sudo /opt/likewise/bin/lwsm set-log-level lsass - verbose

The next two commands redirect logging for the lsass-ipc facility in
the same container to a file and raise the log level to trace:

  $ sudo /opt/likewise/bin/lwsm set-log-target lsass lsass-ipc file /tmp/lsass-ipc.log
  $ sudo /opt/likewise/bin/lwsm set-log-level lsass lsass-ipc trace

You can use 'lwsm get-log' to see all logging settings for a container:

  $ /opt/likewise/bin/lwsm get-log lsass
  <default>: syslog LOG_DAEMON at VERBOSE
  lsass-ipc: file /tmp/lsass-ipc.log at TRACE

Setting the log level for a particular facility to '-' will make it
use the default log level instead:

  $ sudo /opt/likewise/bin/lwsm set-log-level lsass lsass-ipc -
  $ /opt/likewise/bin/lwsm get-log lsass
  <default>: syslog LOG_DAEMON at VERBOSE
  lsass-ipc: file /tmp/lsass-ipc.log at default

Setting the log target to '-' will make it use the default log target.
If both the target and log level for a facility are set to the
default, the facility is removed from the configuration list:

  $ sudo /opt/likewise/bin/lwsm set-log-target lsass lsass-ipc -
  $ /opt/likewise/bin/lwsm get-log lsass
  <default>: syslog LOG_DAEMON at VERBOSE

A common task when troubleshooting is to temporarily redirect a
facility of interest to a file, raise the log level, and then reset
everything back to defaults when done.  The 'lwsm tap-log' command
will redirect a facility (including '-') to stdout of your terminal
and simultaneously change the log level.  Hitting ctrl-c will exit the
command and revert to the previous logging settings.  For example:

  $ sudo /opt/likewise/bin/lwsm tap-log lsass lsass-ipc trace | tee /tmp/lsass-ipc.log
  20110927121313:ALWAYS:lsass-ipc: Logging started
  20110927121313:ALWAYS:lsass-ipc:0x7f7d9472a700:LwSmSetMaxLogLevel():lwsm/server/logger.c:619: Log level changed to TRACE
  ^C
  $ /opt/likewise/bin/lwsm get-log lsass
  <default>: syslog LOG_DAEMON at VERBOSE

The Likewise srv service also provides verbose protocol level debugging that 
must be enabled independently of the logging methods detailed above. 
The protocol enhanced logging must be enabled and disabled via the following 
registry  key:

HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\srv\logging\EnableLogging

Once the enhanced protocol logging is enabled and lwio has been restarted then
the aforementioned tap-log facility may be used extract the protocol log events.

The following is an example of enabling srv enhanced protocol logging via an
interactive lwregshell session. Note that after manipulating the registry
values, lwio must be restarted.

    # Start lwregshell session
    #
    lwregshell

    # cd to srv logging registery entry:
    #
    \> cd HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\srv\logging

    # List keys/values
    #
    HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\srv\logging> ls
    [HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\srv\logging]
    +  "EnableLogging"       REG_DWORD       0x00000000 (0)
    +  "Filters"             REG_MULTI_SZ[0] "*,smb2,*,debug"
       "MaxRequestLogLength" REG_DWORD       0x00000100 (256)

    # Enable logging
    #
    HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\srv\logging> set_value "EnableLogging" 1
    

    # List keys/values
    #
    HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\srv\logging> ls
    [HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\srv\logging]
    +  "EnableLogging"       REG_DWORD       0x00000001 (1)
    +  "Filters"             REG_MULTI_SZ[0] "*,smb2,*,debug"
   "MaxRequestLogLength" REG_DWORD       0x00000100 (256)

    # exit
    #
    HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\srv\logging> quit

Restart lwio:
lwsm restart lwio

More details on srv protocol enhanced logging are available:
    1.) in the source code at:
    lwio/server/srv/utils/logfilter.c

    2.) the lwio srv design at:
    lwio/server/srv/docs/lwio_srv_driver.doc


If logging is not enough, the 'lwsm gdb' command will conveniently
attach gdb to a service, first starting it if it is not running:

  $ sudo /opt/likewise/bin/lwsm gdb lsass

If a service is failing or crashing on startup, it may not be possible
to attach gdb to it after the fact.  In this case, it is possible to
run a container process manually before starting the service.  First,
make sure all services in the container are stopped.  You may find the
container name for a service by using 'lwsm info <service>' and
looking at the "Service Group" line.

  $ /opt/likewise/bin/lwsm info lsass
  Service: lsass
  Description: Likewise Security and Authentication Subsystem
  Categories: service
  Path: /opt/likewise/lib/lw-svcm/lsass.so
  Arguments:
  Dependencies: netlogon lwio lwreg rdr npfs
  Service Group: lsass
  File descriptor limit: 1024
  $ sudo /opt/likewise/bin/lwsm stop lsass
  Stopping service reverse dependency: srvsvc
  Stopping service reverse dependency: srv
  Stopping service: lsass

Next, manually run 'lwsmd --container <name>' to start the container.
You may run the container within gdb, valgrind, or any other debugging
or code instrumentation utility.

  $ sudo /opt/likewise/sbin/lwsmd --container lsass
  20110927122024:ALWAYS: Logging started
  20110927122024:INFO: Likewise Service Manager starting up
  20110927122024:INFO:lwsm-ipc: Listener started
  20110927122024:VERBOSE: Starting IPC server
  20110927122024:INFO: Registering as service container for group: lsass
  20110927122024:INFO: Likewise Service Manager startup complete
  20110927122024:VERBOSE:lwsm-ipc: (session:df6e2580f626c918-f9e563f72602361a) Connected <local euid:0 egid:0 pid:27435>

In a separate terminal, you can now start the service as usual:

  $ sudo /opt/likewise/bin/lwsm start lsass
  Starting service: lsass

You should then see the service loaded and run within your container
session.

By default, lwsmd will automatically restart any service that dies
unexpectedly or fails to initialize within a 15 second "stuck timeout"
period.  These behaviors may be inconvenient when debugging, and can be
temporarily modified as follows:

  $ sudo /opt/likewise/bin/lwsm set watchdog off
  $ sudo /opt/likewise/bin/lwsm set stuck-timeout 3600000 # 1hr

You can view the state of the watchdog and other settings with:

  $ /opt/likewise/bin/lwsm settings
  watchdog: off
  stuck-timeout: 3600000
  restart-limit: 2
  restart-period: 30
