root/freepbx/branches/experimental/vgps/README

---------------------------------------------------------------------------
VoicePulse Global Provisioning System (VGPS)
by Ketan Patel
Copyright (C) 2006 VoicePulse Inc.

The VoicePulse Global Provisioning System (VGPS) is a PHP/MySQL-based 
provisioning system intended to generate vendor-specific configuration files 
for via a generic HTTP API.
---------------------------------------------------------------------------

TABLE OF CONTENTS

  1.0 Background
  2.0 Concepts
    2.1 Templates
    2.2 Generic Features
  3.0 Architecture
    3.1 Models
    3.2 Devices
    3.3 Generic Settings
    3.4 Generic Mappings
    3.5 Generic Translations
    3.6 Device Settings
  4.0 Installation
    4.1 Download
    4.2 Extract
    4.3 Database
    4.4 Test
    4.5 Requirements
  5.0 Integration
  6.0 Contributing
    6.1 Templates
    6.2 SQL Script
    6.3 Test.php

1.0 BACKGROUND

  The VoicePulse Global Provisioning System is the latest version of 
  VoicePulse's in-house provisioning system for Voice-over-IP phone
  configuration files.  As most others who have tried to tackle the issue
  of mass configuring phones, we have gone through the usual revisions:
  
    1. Hard-coded script: This is usually a perl script with the majority
    of the configuration file text hard-coded and a couple variables looked
    up from a database.  This is a very quick and simple solution until you
    need to support multiple phones.  Scripts are usually setup via cron to 
    batch generate configuration files at some interval.  This soon 
    becomes a problem because the batch job takes longer and longer as your 
    customer base grows.  Also, batch jobs prevent settings from taking 
    effect immediately as expected by customers.
    
    2. Fully database driven: The next step is usually an over-compensation
    for the hard-coded script.  "Hey, if I just create a database with one
    table for every manufacturer and one column for every setting, it'll be
    infinitely flexible."  That's true, but it's also very difficult to add
    a new device because you have to create a SQL script with every possible 
    setting (or leave out settings and be required to change the schema when
    you want to support a new setting).

  On our third attempt at coming up with a provisioning system that would
  handle multiple manufacturers, we finally came up with a solution that
  would:
  
    1. Support any text or XML based configuration file
    2. Allow you to make global changes just by editing a text/XML file
       (e.g. Set a new proxy, DNS server, etc)
    3. Allow your customer website or GUI to make setting changes without
       having to know what type of device is being configured
       (e.g. Turn on call waiting for an Aastra, Cisco, Linksys, Sipura, etc
       phone using the same function call)
    4. Provide a simple HTTP API for integration with GUIs written in any
       language
  
2.0 CONCEPTS

  There are two key concepts that drive the VoicePulse Global Provisioning
  System -- templates and generic features.

  2.1 TEMPLATES
  
    Using templates in their original form allows for much easier implementation
    and troubleshooting of configuration files.  We generally obtain templates 
    from documentation or utilities (such as Sipura's spc.exe).  It is not 
    mandatory that the template have every possible setting supported by
    the device, but it's easier to start with a known working template than 
    creating one by hand.
    
    When a request to regenerate a configuration file is submitted via the API,
    VGPS will determine the model of device, open it's template file and modify
    just the unique settings for that device before saving the unique output 
    configuration file.
    
    Template files are found in the "input" directory.  The name of the file
    is entered into the `models` table when support for that model is added to
    the database.

  2.2 GENERIC FEATURES
  
    Previous systems have required the calling application (your GUI or 
    customer service management interface) to know the exact name and value
    that needs to be set in the configuration file.  This means if you want to
    support a different model device, all of your applications have to know
    how to translate that request for different devices (e.g. to set the SIP
    username on a Linksys, output "<User_ID_1_>VoicePulseUser</User_ID_1_>", but
    to set it on a Grandstream, output "P35 = VoicePulseUser").  This can make
    it almost impossible to support a new device if there are numerous 
    applications which must be updated to do so.
    
    A key concept of the VGPS is the generic feature.  Using generic features, 
    you can tell VGPS:
    
      Set "line_01_username" to "VoicePulseUser" for device "000E01234567"

    Since you have already entered the device MAC address and model into VGPS,
    it will automatically look up the model, and translate your request into:

      "<User_ID_1_>VoicePulseUser</User_ID_1_>" for a Linksys phone
      
      or
      
      "sip line1 user name: VoicePulseUser" for an Aastra phone 
      
      or
      
      "P35 = VoicePulseUser" for a Grandstream phone
    
    This means your GUI no longer has to have any knowledge of device-specific
    settings!  If you add a new device to the system at a later time, your GUI
    does not have to be updated!

3.0 ARCHITECTURE

  This section will describe the major entities in the VGPS.  Understanding 
  how these entities are related will make it easier for you to use and improve 
  VGPS.

  3.1 MODELS
  
    A model is defined for each manufacturer you are provisioning devices for.  
    You can also define multiple models for different firmware versions of the 
    same device if the configuration files are different.  You may also create 
    different models of the same device if you want to use  different templates 
    for different deployments of the same device (e.g. Linksys PAP2T's in the 
    main office and a remote office).  A model is defined as:
    
    MODEL_ID: A unique ID for this model (primary key).
    
    NAME: The descriptive name of this model (e.g. Linksys PAP2T) 

    TAG: An internal name for this model (e.g. linksys_pap2t).  Use all 
      lowercase letters and underscores for spaces.

    FORMAT: Either XML or TEXT.  If you use XML, all generic mappings must be
      specified using XPath to find the right node.  If you use TEXT, the REGEX
      field must specify how to parse each line into a key/value pair.

    REGEX: Must be specified if FORMAT is TEXT.  This is a regular expression
      that describes how to parse the configuration file on a line-by-line 
      basis.  The sample models already include the common formats:
      
        key with spaces: value
        key_without_spaces: value
        key = value (<-- alphanumeric key)
        123 = value (<-- numeric key)

    INPUT_FILENAME: The template input path + filename.  This is the file that
      is usually supplied or generated from the manufacturer and contains all
      possible configuration settings.

    OUTPUT_FILENAME: The device-specific configuration file's output filename.
      You can use tokens [mac] or [MAC] in the output filename.  These will be
      replaced by the lowercase MAC address or uppercase MAC address when the
      final file is saved.

    POST_PROCESSING: This is a command that is run after the device-specific
      configuration file has been generated.  This can be used for compiling the
      output file into a binary (e.g. for Cisco ATAs).  This can also be used
      for encrypting the output file or copying it to a different location.

  3.2 DEVICES
  
    A device is defined for every unique device that configuration files will
    be generated for.  The MAC address of the device is the unique identifier
    used in all API calls.  A device is defined as:
    
    DEVICE_ID: A unique ID for this device (primary key).
    
    MODEL_ID: The ID from the MODELS table that specifies the model of this
      device.
    
    MAC_ADDRESS: The MAC address of this device.  The MAC address of the device 
      is the unique identifier used in all API calls.  
    
    LOCATION: An optional text field to be used to specify the location of this
      device.
    
    NOTE: An optional text field to be used to keep notes on this device.
    
  3.3 GENERIC SETTINGS
  
    A generic setting is defined for every configuration file setting you may
    want to set, regardless of the model of the device.  The database is pre-
    populated with generic settings for the most common settings -- proxy, 
    username, password, display name, and call waiting.

    GENERIC_SETTING_ID: A unique ID for this generic setting (primary key).
    
    NAME: The descriptive name of this generic setting (e.g. Line 01 Username) 
    
    TAG: A programmatic name for this generic setting (e.g. line_01_username).  
      Use all lowercase letters and underscores for spaces.  This value will 
      be used when you try to set the value of a setting through the API.
      
  3.4 GENERIC MAPPINGS
  
    A generic mapping is defined for every combination of model and generic
    setting.  This is because you must tell the system what to translate the 
    generic settings into for a specific model (e.g. for a Linksys PAP2T, 
    generic setting line_01_username translates into User_ID_1_ in the final
    device-specific configuration file).
  
    GENERIC_MAPPING_ID: A unique ID for this generic mapping (primary key).
    
    MODEL_ID: The ID from the MODELS table that specifies the model of this
      mapping (e.g. Linksys PAP2T).
      
    GENERIC_SETTING_ID: The ID from the GENERIC_SETTINGS table that specifies
      the generic setting of this mapping (e.g. line_01_username).
    
    PROFILE_KEY: If the format of the configuration file is TEXT, this should
      be the key of the key/value pair.  If the format of the configuraion file
      is XML, this should be the XPath of the xml node where the setting is
      (e.g. /flat-profile/User_ID_1_ for Linksys PAP2T).
      
  3.5 GENERIC TRANSLATIONS
  
    A generic translation is defined for every combination of model and generic
    setting.  This is because you must tell the system what to translate the
    generic settings into for a specific model (e.g. for a Linksys PAP2T,
    generic value 0 translates into No in the final device-specific 
    configuration file.  If there is no translation, the value supplied by the
    API will be set directly, with no translation.
    
    GENERIC_TRANSLATION_ID: A unique ID for this generic translation 
      (primary key).  
    
    MODEL_ID: The ID from the MODELS table that specifies the model of this
      mapping (e.g. Linksys PAP2T).

    GENERIC_SETTING_ID: The ID from the GENERIC_SETTINGS table that specifies
      the generic setting of this mapping (e.g. line_01_username).
    
    GENERIC_VALUE: The generic value you will call the API with when setting 
      this generic setting (e.g. 0).
      
    PROFILE_VALUE: The value that should be used in the configuration file 
      (e.g. No).

  3.6 DEVICE SETTINGS
  
    This table holds the device settings that have been defined for each 
    device.  The values stored in this table are translated into their final 
    key/pair values.
    
    * This values in this table should not be set/modified directly. *

4.0 INSTALLATION

  4.1 DOWNLOAD
  
    Download the necessary files from http://www.sourceforge.net/vgps/.
    
  4.2 EXTRACT
  
    To uncompress and extract the VGPS files, run:
    
      tar -zxvf vgps-0.1.tar.gz
    
    Since VGPS is a web based system, you probably want to extract the files
    into:
    
      /var/www/html/vgps/ (or your equivalent web root directory)
      
  4.3 DATABASE
  
    A SQL script is provided to initialize the VGPS database.  To run the 
    script, type the following from the command line (assumes Mysql is running
    on localhost):
  
      mysql -h localhost -u root -p < vgps.sql

    WARNING! This will drop any existing VGPS database!
    
    After running the script, you should see the following in your Mysql 
    database:
    
      mysql> use voicepulse_global_provisioning_system;
      Database changed
      mysql> show tables;
      +-------------------------------------------------+
      | Tables_in_voicepulse_global_provisioning_system |
      +-------------------------------------------------+
      | device_settings                                 |
      | devices                                         |
      | generic_mappings                                |
      | generic_settings                                |
      | generic_translations                            |
      | models                                          |
      +-------------------------------------------------+
      6 rows in set (0.00 sec)

  4.4 TEST
  
    To test your installation, change to the directory you extracted the files
    into.  Assuming you extracted them into /var/www/html/vgps/ and the URL to
    reach those files is http://localhost/vgps/:
    
      cd /var/www/html/vgps/
      php test.php
      
    If the test is successful, you should see:
    
      [root@host vgps]# php test.php
      Content-type: text/html
      X-Powered-By: PHP/4.3.11
      
      SUCCESS
      SUCCESS
      SUCCESS
      ...
      
    You should also see the generated configuration files for the sample devices
    in the output directory.
    
      [root@host vgps]# cd output
      [root@host output]# ls
      000413000000.htm  00085D000000.cfg  000B82000000.cfg  000E08000000.xml  
      phone0004F2000000.cfg
      
    If the URL of your VGPS installation is not http://localhost/vgps/, you must
    edit $root_url in test.php.

  4.5 REQUIREMENTS
  
    PHP 4.3.11 or greater
    Mysql 4 or greater
    Apache
    
5.0 INTEGRATION

  To integrate your systems with the VGPS, you have to submit HTTP GET requests
  to the execute.php file.  A working sample can be found in index.php and 
  client_lib.php.
  
  ADD/MODIFY A DEVICE
  
    http://localhost/vgps/execute.php?op=set_device&model=$model&mac=$mac&location=$location&note=$note
    
    Returns a string containing:
    
      ERROR: Error message with details
      or
      SUCCESS
  
  DELETE A DEVICE
  
    http://localhost/vgps/execute.php?op=delete_device&mac=$mac
    
    Returns a string containing:
    
      ERROR: Error message with details
      or
      SUCCESS
        
  SET A DEVICE SETTING
  
    http://localhost/vgps/execute.php?op=set_setting&mac=$mac&key=$key&value=$value
    
    Returns a string containing:
    
      ERROR: Error message with details
      or
      SUCCESS
  
  DELETE A DEVICE SETTING
  
    http://localhost/vgps/execute.php?op=delete_setting&mac=$mac&key=$key
    
    Returns a string containing:
    
      ERROR: Error message with details
      or
      SUCCESS
  
  REGENERATE
  
    http://localhost/vgps/execute.php?op=regen&mac=$mac
    
    Returns a string containing:
    
      ERROR: Error message with details
      or
      SUCCESS
  
  LIST DEVICES
  
    http://localhost/vgps/execute.php?op=list_devices
    
    Returns a string with a header row, data fields delimited with ||, and 
    rows delimited with \n:
    
      mac_address||location||note||name||tag
      000413000000||||Sample Snom 300||Snom 300||snom_300
      0004F2000000||||Sample Polycom Soundpoint IP 501||Polycom Soundpoint IP 501||polycom_soundpoint_ip_501
      00085D000000||||Sample Aastra 480i||Aastra 480i||aastra_480i
      000B82000000||||Sample Grandstream GXP2000 1.1.0.16||Grandstream GXP2000 1.1.0.16||grandstream_gxp2000_1_1_0_16
      000E08000000||||Sample Linksys PAP2||Linksys PAP2T||linksys_pap2t

  LIST DEVICE SETTINGS
  
    http://localhost/vgps/execute.php?op=list_device_settings&mac=$mac
    
    Returns a string with data fields delimited with ||, and rows delimited 
    with \n:
        
      line_01_password||vppassword
      line_01_proxy||sip.voicepulse.com
      line_01_username||vpusername

  
  LIST MODELS
  
    http://localhost/vgps/execute.php?op=list_models
    
    Returns a string with a header row, data fields delimited with ||, and 
    rows delimited with \n:
    
      name||tag
      Aastra 480i||aastra_480i
      Grandstream GXP2000 1.1.0.16||grandstream_gxp2000_1_1_0_16
      Linksys PAP2T||linksys_pap2t
      Polycom Soundpoint IP 501||polycom_soundpoint_ip_501
      Snom 300||snom_300

6.0 CONTRIBUTING

  If you would like VGPS to support a device that is currently not included in
  this version, it's very easy to do so!
  
  6.1 TEMPLATES
  
    The first step in supporting a new device is providing the template file
    that will go in the input directory.  We would prefer templates that were
    provided directly from the manufacturer or created by a utility provided by
    the manufacturer.
    
  6.2 SQL SCRIPT
  
    The next step is to provide the appropriate data to be entered into Mysql.
    The SQL script that comes with VGPS, vgps.sql, shows how to populate the
    data for a new device.  We will walk through the required items below:
    
    # First, we need to insert a new model:
    INSERT INTO `voicepulse_global_provisioning_system`.`models` (`name`, `tag`, `format`, `regex`, `input_filename`, `output_filename`, `post_processing`) VALUES ('Linksys PAP2T', 'linksys_pap2t', 'XML', '', 'input/linksys_pap2t.xml', 'output/[MAC].xml', '');
    
    # We select the @MODEL_ID for use in the statements to follow:
    SET @MODEL_ID = (SELECT LAST_INSERT_ID());
    
    # Since we have already defined the most common generic mappings across all
    # devices, we'll use those to pre-populate the GENERIC_MAPPINGS table:
    INSERT INTO `voicepulse_global_provisioning_system`.`generic_mappings` (`model_id`, `generic_setting_id`, `profile_key`)
    SELECT @MODEL_ID, generic_setting_id, tag FROM `voicepulse_global_provisioning_system`.`generic_settings`;
    
    # Now that the GENERIC_MAPPINGS table has a copy of the most common 
    # mappings, we will update those rows with the actual values.  In the
    # following rows, you should set the first `profile_key` to the actual key 
    # used in the device-specific configuration file you are trying to support.
    # Follow this pattern:
    # UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '[SET_THE_PROFILE_KEY_HERE]' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'admin_password';
    #
    # The most common mappings (admin password, line_01_proxy, line_01_username,
    # etc) are all provided below.  The lines below are for a configuration file
    # that is in XML format, so the `profile_key` is in XPath format.
    #

    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/Admin_Passwd' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'admin_password';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/Proxy_1_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_01_proxy';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/User_ID_1_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_01_username';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/Password_1_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_01_password';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/Display_Name_1_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_01_display_name';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/CW_Setting_1_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_01_call_waiting';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/Proxy_2_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_02_proxy';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/User_ID_2_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_02_username';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/Password_2_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_02_password';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/Display_Name_2_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_02_display_name';
    UPDATE `voicepulse_global_provisioning_system`.`generic_mappings` SET `profile_key` = '/flat-profile/CW_Setting_2_' WHERE `model_id` = @MODEL_ID AND `profile_key` = 'line_02_call_waiting';
    
    # Next, if you have settings with an ON/OFF value, such as Call Waiting,
    # you should set a mapping.  In the example below, we set a mapping for
    # 0 = No and 1 = Yes.  If the device you are trying to add uses the words
    # "Off" and "On" instead of "No" or "Yes", this is where you would specify
    # that.
    
    # First, select the generic setting you need a mapping for:
    SET @GENERIC_SETTING_ID = (SELECT generic_setting_id FROM `voicepulse_global_provisioning_system`.`generic_settings` WHERE `tag` = 'line_01_call_waiting');
    
    # Next, insert the two mappings:
    INSERT INTO `voicepulse_global_provisioning_system`.`generic_translations` (`model_id`, `generic_setting_id`, `generic_value`, `profile_value`)
    SELECT @MODEL_ID, @GENERIC_SETTING_ID, '0', 'No';
    INSERT INTO `voicepulse_global_provisioning_system`.`generic_translations` (`model_id`, `generic_setting_id`, `generic_value`, `profile_value`)
    SELECT @MODEL_ID, @GENERIC_SETTING_ID, '1', 'Yes';
    
    # Again, select the generic setting you need a mapping for:
    SET @GENERIC_SETTING_ID = (SELECT generic_setting_id FROM `voicepulse_global_provisioning_system`.`generic_settings` WHERE `tag` = 'line_02_call_waiting');
    
    # Again, insert the two mappings:
    INSERT INTO `voicepulse_global_provisioning_system`.`generic_translations` (`model_id`, `generic_setting_id`, `generic_value`, `profile_value`)
    SELECT @MODEL_ID, @GENERIC_SETTING_ID, '0', 'No';
    INSERT INTO `voicepulse_global_provisioning_system`.`generic_translations` (`model_id`, `generic_setting_id`, `generic_value`, `profile_value`)
    SELECT @MODEL_ID, @GENERIC_SETTING_ID, '1', 'Yes';
    
    # Finally, you should insert a sample device for testing purposes.
    INSERT INTO `voicepulse_global_provisioning_system`.`devices` (model_id, mac_address, location, note) VALUES (@MODEL_ID, '000E08000000', '', 'Sample Linksys PAP2');
    
  6.3 TEST.PHP
  
    Finally, you should modify test.php to call the three set_setting API calls
    using the MAC address of the sample device you have added.  If test.php runs
    successfully, you should see the generated output file in the output 
    directory.  This file should have the values of the line 1 username, 
    password, and proxy set to the values used in test.php.